Benchmarking Language Modeling for Lossless Compression of Full-Fidelity Audio

AI summary

Overview

• Research area: Lossless audio compression using autoregressive (AR) "language" models trained directly on raw audio waveforms, benchmarked on full-fidelity audio (music, speech, bioacoustics, sound effects).
• Technical level: Intermediate. Readers should be comfortable with bit depth, sampling rates, and the basic idea that a model's predictive likelihood can be turned into compressed bits via arithmetic coding.
• Scope: The paper proposes a byte-level tokenization scheme called Trilobyte, and benchmarks LM-based lossless compression against FLAC at 8-, 16-, and 24-bit across 16 datasets spanning 16kHz–48kHz. (arXiv:2603.08683v2 [cs.SD], by Phillip Long, Zachary Novack, and Chris Donahue, of UC San Diego and Carnegie Mellon University.)

What This Paper Is About

Prior work showed that AR models trained on raw audio can act as lossless compressors via arithmetic coding, but that work was limited to 8-bit audio at 16kHz — a fidelity almost never used in real distribution. The problem is that treating each audio sample as a token makes the vocabulary explode exponentially: 2^16 = 65,536 tokens for 16-bit audio and 2^24 = 16,777,216 tokens for 24-bit audio. The paper's goal is to test whether LM-based compression actually scales to the CD-quality (44.1kHz, 16-bit) and professional (24-bit) regimes, and to introduce a tokenization scheme that makes 24-bit modeling tractable in the first place.

Key Contributions

1. Trilobyte, a byte-level, hierarchical tokenization schema that decomposes each b-bit sample into B = ⌈b/8⌉ bytes, changing vocabulary scaling from exponential O(2^b) to constant O(1), with a fixed vocabulary of |𝒱| = 256 regardless of bit depth (at the cost of increasing sequence length by ⌈b/8⌉). This enables the first tractable 24-bit LM-based lossless compression.
2. The first comprehensive benchmark of LM-based lossless compression on full-fidelity audio (16- and 24-bit), across diverse domains (music, speech, bioacoustics, sound effects), sampling rates (16–48kHz), and bit depths (8, 16, 24).
3. Evidence characterizing the gap between learned and traditional compressors across bit depths, showing that bit depth — not sampling rate or audio domain — is the limiting factor.
4. An open-source generalist codec: a single 24-bit Trilobyte model trained on all datasets with lower-byte masking that performs comparably to dataset-specific models.
5. Additional negative-result experiments on neural audio codecs as replacements for linear predictive coding (Appendix B) and on in-context compression with pretrained Llama-2-7B and Llama-2-13B (Appendix C).

Main Findings

• 8-bit: large gains over FLAC. Standard sample-level tokenization and Trilobyte are identical at 8-bit (1 byte per sample, vocabulary of 256). Both substantially outperform FLAC, achieving 370%, 163%, and 119% improvements on Beethoven, YouTube Mix, and SC09 respectively. The overall average improvement over FLAC at 8-bit is 217%, consistent with prior work. Trilobyte's 8-bit compression rates span a wide 2.08–7.94x, indicating that gains depend heavily on domain structure (solo piano music compresses far better than multi-speaker SC09).

• 16-bit: consistent but modest gains. AR modeling consistently beats FLAC at 16-bit, with 15%, 31%, and 21% improvements on VCTK, MusDB18 Mono, and LibriSpeech respectively, and an 18% average improvement across domains. FLAC compression rate correlates strongly with Trilobyte compression rate at 16-bit (r = 0.92, p ≪ 0.01). Epidemic Sound reaches 3.40x with Trilobyte, a 29% improvement over FLAC (2.63x).

• Sampling rate matters less than bit depth. High sample rate datasets sometimes compress better than low rate ones — VCTK at 48kHz reaches 2.66x versus LibriSpeech at 16kHz at 2.11x.

• 24-bit: the barrier. The 16.7M-token vocabulary needed for sample-level tokenization is completely intractable at 24-bit, requiring approximately 12B parameters for the output projection matrix alone. Trilobyte's 256-token vocabulary makes 24-bit modeling feasible, but at 1.48x it falls 9% short of FLAC's 1.63x. The authors suggest that much of the least-significant-bit information at 24-bit is imperceptible noise, since 24-bit audio requires up to 144 dB of dynamic range to preserve the signal — and Rice coding in FLAC may be nearly optimal for that low-amplitude noise.

• Sample-level tokenization is not always worse. Sample-level modeling performs comparably to Trilobyte on some 16-bit datasets (2.66x on VCTK, 2.47x on Birdvox, 2.64x on MusDB18 Mono) but generally falls short, particularly on music.

• In-context LLMs underperform. Pretrained Llama-2-7B used for in-context compression underperforms both FLAC and Trilobyte across all datasets and bit depths except 8-bit SC09, where it reaches ~1.80x (and Appendix C reports ~1.8x versus FLAC's essentially 1x). Scaling from 7B to 13B gives minimal gains and often lower compression rates.

• A single transfer model works. Masking lower-significance bytes with a learned null token (dropout p = 0.1) lets one model handle multiple bit depths: a model trained on Commercial (24-bit) and MusDB18 Stereo (16-bit) achieves 1.49x (24-bit), 1.78x (16-bit), and 3.4x (8-bit) on 24-bit Commercial music, and 2.07x (16-bit) and 3.8x (8-bit) on 16-bit MusDB18 stereo. The generalist transfer model (final column of Table 1) performs similarly to per-dataset models.

• Expanded byte subvocabularies don't help. Explicit per-byte-position subvocabularies yielded negligible gains (< 0.003x), suggesting the constant vocabulary already learns separate byte distributions implicitly through autoregressive context.

• Stereo channel handling is a wash. Concatenating left and right channels in random order versus sample-level interleaving produced nearly identical compression rates.

• Neural codec residuals break Rice coding. Replacing FLAC's linear predictive coding with DAC, EnCodec, or a custom DAC variant on MusDB18 mixes (44.1kHz, 16-bit) underperforms FLAC (~1.8x): DAC and Custom DAC reach about 1.2x, and EnCodec actually increases file size (compression rate < 1.0x). Mean absolute residuals were 156.34 (FLAC), 1,603.54 (DAC), 18,376.66 (EnCodec), and 1,245.76 (Custom DAC) — an order of magnitude larger than FLAC. FLAC residuals follow a clear geometric distribution (what Rice coding assumes); NAC residuals are more uniform.

• Speed caveat. The authors acknowledge their ML approaches are orders of magnitude slower than FLAC, so modest compression wins are unlikely to justify computational costs in real deployment.

Methodology in Plain English

The core idea is that any autoregressive model that assigns a probability to each next sample can be paired with an arithmetic coder to produce a compressed bitstream, where better predictions (higher log-likelihood) mean smaller files. Because average per-token log likelihood corresponds directly to the expected bits per token, the authors can estimate compression rate from model loss without running a full arithmetic encoder. They define compression rate as R = A/C(A), where A is the original size and C(A) the compressed size, so R > 1 means the file shrank.

The key engineering move is Trilobyte: instead of treating each b-bit audio sample as one token (which gives a 2^b-sized vocabulary), it splits each sample into B = ⌈b/8⌉ bytes interleaved as MSB, middle byte(s), LSB, and predicts over 256 values at every position. The model is a standard decoder-only GPT-2-style Transformer with causal masking. At 8-bit this is exactly the same as sample-level tokenization.

The authors train models from scratch for a fixed 300K steps (90M parameters for Trilobyte and for 8-bit sample-level; 140M for 16-bit sample-level), and compare against FLAC at compression level 8 (the maximum). For a transfer experiment, they add a learned null token and randomly drop lower-significance bytes during training so one model can handle arbitrary bit depths. They also attempt an in-context baseline using pretrained Llama-2-7B (and 7B/13B in the appendix) over 1K or 1,000 randomly sampled 1,024-sample chunks per dataset, because that method is intractably slow.

Evaluation spans 16 datasets: 8-bit at 16kHz (Beethoven, YouTube Mix, SC09), 16-bit (LibriSpeech 16kHz, LJSpeech 22.05kHz, Birdvox 24kHz, MusDB18 44.1kHz, VCTK and Epidemic Sound 48kHz, commercial 16-bit at 44.1kHz), and 24-bit (commercial 24-bit at 44.1kHz). The commercial corpus contains 1,569 songs (120 hours) at 16-bit and 933 songs (70 hours) at 24-bit, the latter including high-resolution recordings up to 192kHz, all resampled to 44.1kHz for consistency.

Why This Matters

Impact on research. The paper reframes a key open question: LM-based lossless compression gains do not simply extrapolate from 8-bit to 16- and 24-bit. It supplies a reusable tokenization primitive (Trilobyte) compatible with any AR modeling framework, plus the first tractable 24-bit LM compressor and a benchmark suite covering domains and fidelities that prior work never touched. The negative results (neural codec substitution, in-context LLMs) map the boundaries of adjacent approaches. The authors note their empirical compression rates establish lower bounds across audio domains.

Real-world applications:

• Music archival and storage — professional recording and production workflows operate at CD-quality (44.1kHz, 16-bit) or better, and the paper's commercial datasets mirror the lossless files users actually want to compress: 1,569 songs at 16-bit and 933 at 24-bit.
• Professional 24-bit studio workflows — the paper reports the first tractable LM compression at 24-bit where sample-level approaches are completely infeasible.
• Speech and audiobook corpora — datasets like LibriSpeech, LJSpeech, and VCTK represent large-scale speech archives where modest per-file savings compound.
• Bioacoustics and environmental monitoring — Birdvox bird vocalizations and the Epidemic Sound effects library are the domains where compression is strongest (FLAC reaches ~6x on Birdvox at 8-bit, and Trilobyte hits 3.40x on Epidemic Sound at 16-bit).
• General-purpose lossless codecs — the released generalist transfer model is offered as a baseline for any-bit-depth learned lossless compression.

Industry relevance. FLAC remains the de facto standard, achieving typical compression rates of about 2x for CD-quality music. The paper's central practical message is that LM-based compression dominates at 8-bit but the advantage narrows to 18% on average at 16-bit and reverses to a 9% deficit at 24-bit — while being orders of magnitude slower. That makes near-term deployment unlikely, but the work charts where learned compression could eventually complement or replace traditional codecs.

Future Directions

• Closing the 24-bit gap. Trilobyte trails FLAC by 9% at 24-bit; the authors explicitly hope to close this gap in future work.
• Scaling and efficiency. The conclusion anticipates future research to scale the performance of these models and/or improve their efficiency, addressing the orders-of-magnitude speed disadvantage versus FLAC.
• Why learned gains shrink with bit depth. The paper hypothesizes that much of the least-significant-bit content at 24-bit is imperceptible noise that Rice coding nearly optimally captures, but leaves this as an explanation rather than a confirmed mechanism.
• Generalist any-bit-depth codecs. The transfer-masking result is described as an initial result; extending a single model across arbitrary bit depths, corpora, sample rates, and domains is left open.
• Better residual models for codec hybrids. Because neural audio codec residuals violate the geometric assumption Rice coding requires, alternative entropy coders matched to learned residual distributions are an untested direction.

Target Audience

Researchers and engineers working on neural audio compression, learned codecs, or autoregressive generative audio models; practitioners evaluating lossless codecs for music, speech, or bioacoustic archives; and anyone following the broader line of work on using language models as general-purpose compressors. It is also relevant to readers interested in tokenization design for high-cardinality discrete signals, since Trilobyte's byte-level decomposition is the paper's central technical device.

Authors’ abstract

Autoregressive "language" models (LMs) trained on raw waveforms can be repurposed for lossless audio compression, but prior work is limited to 8-bit audio, leaving open whether such approaches work for practical settings (16/24-bit) and can compete with existing codecs. We benchmark LM-based compression on full-fidelity audio across diverse domains (music, speech, bioacoustics), sampling rates (16kHz-48kHz), and bit depths (8, 16, 24-bit). Standard sample-level tokenization becomes intractable at higher bit depths due to vocabulary size (65K for 16-bit; 16.7M for 24-bit). We propose Trilobyte, a byte-level tokenization schema for full resolution audio, improving vocabulary scaling from $O(2^{b})$ to $O(1)$ and enabling the first tractable 24-bit LM-based lossless compression. While LMs consistently outperform FLAC and yield state-of-the-art compression at 8-bit and 16-bit, we observe that compression gains become more modest as bit depth increases beyond 8-bit.