STFT-VAE is a continuous VAE neural audio codec that uses STFT, transformer encoder-decoder, and ISTFT to achieve a very low latent frame rate (3.125 Hz) while preserving phase and outperforming SNAC on objective metrics.

Stars

43

7-day growth

No data

Forks

2

Open issues

0

License

MIT

Last updated

2026-07-12

AI repository intelligence
FR-AI / ANALYSIS

Why it is worth attention

It achieves 4–15× lower frame rate than comparable codecs (Mimi, SNAC, MioCodec) yet outscores SNAC on all tested metrics (SI-SDR, PESQ, STOI), and it preserves phase directly through STFT/ISTFT without a learned vocoder.

Who it is for

  • Audio machine learning researchers
  • Sound designers and creative coders
  • Engineers building low-dimensional audio representations
  • Developers exploring continuous latent audio models

Use cases

  • Low-rate audio latent extraction for downstream tasks (e.g., classification, generation)
  • Round-trip audio reconstruction with extremely compact latent space
  • Listening to side-by-side comparisons of codec quality via the demo
  • Benchmarking novel audio codecs against a strong continuous baseline

Strengths

  • Ultra-low latent rate (3.125 Hz) with strong reconstruction quality
  • Phase is preserved through STFT/ISTFT, removing the need for a separate vocoder
  • Outperforms SNAC on all objective metrics (SI-SDR, PESQ, STOI)
  • Pretrained model available via HuggingFace with simple pip install and CLI

Considerations

  • Only one pretrained model (24 kHz, mono) is provided; no support for other sample rates or stereo
  • No quantization is applied, so the continuous latent cannot be used directly for compression or storage
  • Limited to 30-second clips in the CPU demo; inference dependencies (torch, torchcodec) may be heavy

README quick start

Usage

Description

Inference for the STFT-VAE continuous audio codec (24kHz, 3.125Hz latent)

Related repositories

Similar projects matched by category, topics, and programming language.

lopopolo
Featured
lopopolo GitHub avatar

harness-engineering

Harness Engineering is a methodology for improving coding agent outputs by carefully crafting the environment around them—providing curated context, tools, and executable constraints that encode an organization’s nonfunctional requirements and cumulative lessons.

AI & Machine LearningAI Agents
2,390
slvDev
Featured
slvDev GitHub avatar

esp32-ai

A 28.9 million parameter language model runs on an $8 ESP32-S3 microcontroller entirely on-device, generating simple stories at about 9.5 tokens per second.

AI & Machine LearningLarge Language Models
1,960
littledivy
Featured
littledivy GitHub avatar

mimic

mimic captures traffic from any iOS or web app and automatically generates a Python client library that lets you call the app's API like a regular library.

AI & Machine Learning
1,482