Spectrograms, Mel Filterbanks, MFCCs, and Learned Front Ends

An audio representation is a contract between the sensor and the model. It fixes what gets kept and — silently — what gets thrown away. The easiest contract to read is one that somebody had to write down clause by clause.

ETSI ES 201 108 is that contract. The distributed speech recognition front-end standard, published in 2003, settles everything in numbers. The useful band runs from 64 Hz to half the sampling frequency. Clause 4.2.9 disposes of that band in one sentence: “This band is divided into 23 channels equidistant in mel frequency domain.” From the log outputs of those 23 channels, 13 cepstral coefficients are computed. The frame shift is 160/110/80 samples at 16/11/8 kHz — 10 ms in every case. Everything the model would ever see of the sound had to fit through 23 triangles and 13 numbers, once every 10 ms.

That was the price of the phone-to-server split. The standard was written so a handset could send features and a server could recognise them. The two ends had no way to negotiate afterwards. What would survive the trip had to be agreed in advance. RFC 3557 recorded the same settlement in 2003, and stated it plainly: “The coding algorithm, a standard mel-cepstral technique common to many speech recognition systems, supports three raw sampling rates: 8 kHz, 11 kHz, and 16 kHz.”

Three separate decisions sit inside a front end like that one. Mel filterbanks compress frequency according to a perceptual spacing. MFCCs go a step further and decorrelate a log spectrum. Learned front ends skip both recipes and optimize filters from data, under whatever objective is being trained.

ETSI took the middle option, and it took it because the middle option had won a measured comparison. Five candidates went into that 1980 test: mel-frequency cepstrum, linear-frequency cepstrum, linear prediction cepstrum, linear prediction spectrum and reflection coefficients. One downstream task ranked them. The abstract by Davis and Mermelstein names the winner: “A set of ten mel-frequency cepstrum coefficients computed every 6.4 ms resulted in the best performance, namely 96.5 percent and 95.0 percent recognition with each of two speakers.” Decorrelation was what made those features workable for the Gaussian mixtures of the day.

That reason has since expired, and the expiry is on the record too. Conformer, published in 2020, states its input in one line: “We extracted 80-channel filterbanks features computed from a 25ms window with a stride of 10ms.” Eighty channels rather than 23. No cepstral transform at all. It reaches 2.1%/4.3% WER on LibriSpeech test/test-other without a language model, and 1.9%/3.9% with one. A neural network does not need its inputs decorrelated on its behalf. So a pipeline that still carries the cepstral step under a neural encoder is not making a choice. It is inheriting one — from a model class it no longer uses, and from a 10 ms transport budget it no longer has to meet.

Inheriting a choice is easy, because choosing has an order to it and the order is easy to shortcut. Write down which cues the task actually depends on. Select the candidate front ends that could carry those cues. Fit any normalization statistics on training audio alone. Then ablate the representation and see which candidate the downstream numbers prefer.

The step that gets skipped is the second one. The 1980 comparison did not skip it. Five parametric representations went through the same word-recognition task — mel-frequency cepstrum, linear-frequency cepstrum, linear prediction cepstrum, linear prediction spectrum and reflection coefficients — before a winner was named at 96.5 percent and 95.0 percent for two speakers. That is what makes the result quotable forty years later. The field it ranked was a real field.

Put a single front end forward and nothing ever contradicts the cue list written at the start. It survives to the end as an untested assumption. The ablation has no rival representation with which to test it. It still returns a number. It simply ranks a field the cue list had already decided.

The third step is a smaller trap of the same shape. Normalization statistics fitted on validation or test audio move information across the split. The comparison at the end then grades a front end that will not exist at deployment.

What a domain-mismatched representation costs has been measured, on bird calls, with the numbers published. A 2023 study in Scientific Reports set embeddings from bird-trained classifiers against embeddings from models trained on general audio: VGGish, YAMNet, PSLA and AudioMAE, all AudioSet-trained. The mismatch showed up on every task they ran: “The embeddings extracted from the models trained on bird vocalization data consistently allowed higher quality classification than the embeddings trained on general audio datasets.”

The striking part is where the advantage held. It held on taxa that are not birds. Perch reached AUC 0.97 on bat species and 0.99 on Godwit calls. AudioMAE fell to 0.66 on Yellowhammer dialects. A representation fitted to one acoustic domain carried further into a neighbouring one than a representation fitted to audio in general did. And a gap from 0.99 to 0.66 is not one a deeper classifier recovers. What the front end did not encode is not in the input for any later stage to find.

Was the representation in your pipeline ever built around the frequencies this task lives in? A review that stops at downstream quality, on the tasks the representation already suits, has nowhere to ask that.

Set the three families side by side and the trade becomes easy to state. A linear or log spectrum retains explicit frequency bins. You can name what is in each one. Kept complex-valued, it also preserves phase — the quantity DCCRN turned into first place in the DNS challenge real-time track at 3.7M parameters.

Mel and MFCC features surrender those explicit bins in exchange for a compact, perceptually spaced description: 23 channels and 13 coefficients in ES 201 108, 80 channels in Conformer. In both cases the bin structure can be read off a clause.

A learned waveform front end surrenders the fixed definition altogether, in exchange for filters shaped by the objective. That is the trade LEAF was built to make: “In this work we show that we can train a single learnable frontend that outperforms mel-filterbanks on a wide range of audio signals, including speech, music, audio events and animal sounds, providing a general-purpose learned frontend for audio classification.”

The size of that trade is on the table. Under a common EfficientNetB0 encoder, with 40 filters for every front end, single-task test accuracy averaged 73.9% for mel-filterbanks, 72.9% for TD-fbanks, 73.1% for SincNet and 76.9% for LEAF. Three points of average accuracy, bought with the loss of a filter definition anyone can inspect. Each family buys something the other two do not. Which is why the argument that justifies one of them justifies neither of the others.

Choosing among the three families is not a question about audio in general. It is a question about the task, and four common tasks answer it four different ways. ASR, music information retrieval, bioacoustics and enhancement each ask the front end for different information. So each starts from a different list of task-relevant cues. A front end that clears one clears neither the other three nor the bioacoustics comparison above.

All four tasks need the same kind of evidence, and its shape is not in dispute, because it has been published. LEAF's comparison is legible only because one thing was held fixed and one thing varied. The same EfficientNetB0 encoder, 40 filters for every candidate, four front ends run through it — 73.9% for mel-filterbanks, 72.9% for TD-fbanks, 73.1% for SincNet, 76.9% for LEAF. Change the encoder between runs and those four numbers stop meaning anything about front ends at all.

The headline number belongs at matched budgets. The reason to collect anything beyond it is that it will sometimes disagree with reconstruction quality, or with probes for the cues the task claims to use. Agreement confirms what you already believed. Disagreement is the finding. It says the system is scoring well on something other than the cues written down at the start.

The per-task rows say the same thing from another direction. LEAF's 76.9% is an average. Birdsong detection inside it moved from 78.6% with mel-filterbanks to 81.4% with LEAF — a task-specific gap the average alone would have hidden.

The evaluation should also mark the edge of the range. That is the point where defaults from one domain, applied to a signal from another, leave the system nothing to do but abstain or fall back. It is the region where AudioMAE's embeddings sat at AUC 0.66 on Yellowhammer dialects.

All of this reduces to one piece of work you can do yourself, and there is a published template for it. LEAF, in 2021, ran precisely the four steps this lesson prescribes. It fixed the downstream model: one EfficientNetB0 encoder. It matched the budget across candidates: 40 filters for every front end. It trained several representations rather than one — mel-filterbanks, TD-fbanks, SincNet, LEAF. And it reported the results sliced by task rather than pooled, which is how birdsong detection at 78.6% against 81.4% is visible at all inside a single-task average of 73.9% against 76.9%. Davis and Mermelstein had done the same thing with five candidates in 1980. Neither result would exist if either team had put a single front end forward.

Write down three things. First, what your list of task-relevant cues takes for granted. Second, one counterexample to it: a signal or a band the list would wave through unexamined. Bat species and Godwit calls are the ready-made version, sitting outside a band that starts at 64 Hz and was drawn for speech. Third, what the ablation result would oblige you to change. The third is what makes the exercise binding. Without it, an ablation is a number you remain free to explain away.