Alexandre Défossez "Discrete and continuous tokenization for audio-text modeling"

Audio signals are inherently stochastic. Naive representations would require tens of thousands of auto-regressive steps per seconds, make them incompatible with LM based methods. We will see how using adversarial auto-encoder with a discrete information bottleneck, one can reduce that drastically to a few hundred, making audio language modeling tractable with custom architectures. Yet such discrete tokenizations have limits: higher quality requires more tokens, which accounting for more computation time than the content itself. We will thus open up on recent advances in continuous tokenization and generation for audio.