Music Information Retrieval: Pitch, Beat, Chords, and Transcription

Music information retrieval converts audio into symbolic or semantic structure: pitch trajectories, note events, beats, tempo, chords, key, instruments, sections, or similarity. None of that is read off the signal. Each output takes something for granted about tuning, meter, notation, and the musical culture that produced all three. The taking-for-granted happens before a single model is trained.

So hold a low-level descriptor, a symbolic transcription and a semantic retrieval system against the music you actually have, before an architecture is fixed. The labels they need are not the same. The labels themselves are arguable.

How arguable has been measured. Two experts annotated the Rock Corpus. Scored against each other in 2015, they reached 0.932 weighted recall at root level. At tetrads they reached only 0.835. Humphrey and Bello ran that comparison. They write: “Additionally, it is curious to observe a non-negligible degree of disagreement between the two human perspectives, with more than a 15% discrepancy in the tetrads condition.” On The Beatles' “I Saw Her Standing There”, no two of the four reference annotations agreed above 65% under the tetrads rule.

What that does to a score is direct. Against the Isophonics reference, their DNN and kHMM systems scored 28% and 52% on that song. Against the four human annotations, the same two outputs matched at least one human for 89.1% and 92.3% of it. The audio did not change. The systems did not change. Only the sheet of paper they were held against did.

If the reference is one reading, where does that reading enter the system? At the first step, when the representation is chosen. Choosing a representation is how you define the musical unit: what counts as a note, a beat, a chord, a section. Everything downstream measures that unit and nothing else.

The last step cannot go back and check it. Evaluating musically inherits the unit and reports how faithfully the system hit it. A unit borrowed from the wrong tradition of meter is never caught. It is scored, and it can score well. Saying the assumption out loud is the cheapest step in the whole pipeline. The next section is what skipping it costs, in the numbers the standard itself prints.

A beat sequence can be well aligned with the music, wrong about which pulse is the pulse, and still take most of the marks. In 2012 Holzapfel and colleagues put a figure on it: “If the beat sequences are tapped at metrical levels related by a factor of two (but otherwise well aligned), this causes the score to drop from 100% to 66.7%.” Two-thirds of the headline number survives the mistake. For scale, the same authors note that completely unrelated beat sequences score around 25% on F-measure and closer to 18% on AMLt. A tracker locked to half the annotated tempo sits far nearer a perfect score than a random one. And AMLt counts double- or half-tempo and off-beat tapping as accurate outright.

The headline measure is narrow by design, and the design is written down. MIREX's Audio Beat Tracking task defines it as “F-measure - the standard calculation as used in onset evaluation but with a 70ms window.” That is a statement about timing precision and about nothing else. Alongside it the same task lists CMLc, CMLt, AMLc and AMLt, continuity-based measures “based on the longest continuously correctly tracked section”. MIREX scores them with the Beat Evaluation Toolbox. mir_eval, the widely used open-source implementation, encodes the same thresholds. Its default f_measure_threshold is 0.07 seconds. It documents CMLt as “Correct metric level, total accuracy (continuity not required)” and AMLt as “Any metric level, total accuracy (continuity not required)”.

That is the whole trap, in two lines of documentation. CMLt requires the correct metrical level. AMLt does not. A system on the wrong pulse therefore scores high on AMLt and low on CMLt. The distance between those two numbers is the failure — already computed, already printed, and invisible to anyone who quotes the F-measure alone.

The whole 66.7% problem lives inside one word. Beat sounds like a property of the recording. It is in fact a choice among metrical levels. Chroma, onset, tempo, and polyphonic transcription are words of the same kind: each assumes something about tuning, meter, or notation. Anyone writing about music information retrieval who blurs them hides that assumption. And a number is resting on it.

Those terms serve three different jobs, and the beat-tracking measures show why the difference is worth keeping straight. A low-level descriptor is a pitch, onset, loudness, or timbral feature over time. Symbolic transcription commits to notation. Semantic retrieval commits to meaning. The three are contested on different grounds: whether the signal was measured correctly, whether the right notation was chosen, whether people would agree about what the music is.

The 70 ms F-measure answers the first kind of question — were the detections in the right place in time. CMLt and AMLt answer the second — which pulse is the pulse. The third arrives as a gap between two numbers rather than as one score: 28% against Isophonics, 89.1% against four annotators. All of these come out of the same evaluation run. mir_eval's beat module computes the F-measure and the continuity measures alike. Usually only the first is quoted.

A release review therefore needs more than one kind of number, arranged so that a strong figure cannot cover for a useless system. Start by making “tolerance stated” mean something specific. MIREX's note-tracking evaluation, written up in 2009, does exactly that: “In the first scenario, a returned note event is assumed to be correct if its onset is within a +/-50 millisecond range of a ground-truth onset and its F0 is within +/- a quarter tone (3%) of the ground-truth pitch.” An optional second scenario adds an offset within 20% of the ground-truth note's duration or 50 ms, whichever is larger. mir_eval's transcription module carries the same thresholds as defaults: onset_tolerance 0.05 s, pitch_tolerance 50.0 cents, offset_ratio 0.2, offset_min_tolerance 0.05 s.

Then read what the evaluators say about their own reference. Onset-offset performance came out “relatively poor”, and they attribute it to the ground truth rather than to the systems. Offsets were labelled at very low loudness on isolated woodwind tracks and taken from MIDI for the piano set. So “in the woodwind dataset, the offset time may be overestimated, whereas in the MIDI-generated dataset, the offset may be underestimated.” The reference is arguable in both directions at once. A system is being marked against it.

Then show what happens when the tuning in the recording is not the tuning the model assumes. A quarter-tone tolerance is generous against small drift and useless against a repertoire whose degrees sit elsewhere. Show, too, whether the system abstains or falls back there rather than guessing confidently.

None of this survives in anyone's head between projects, which is why the last artifact is a sheet. An uncertainty sheet records where the labels are contestable. It is what another team needs in order to challenge a label this pipeline treated as settled. Record three things in each row: what defining the musical unit assumes, one counterexample, and what evaluating musically then triggers.

The metrical-level row is three short lines. The assumption is that the annotated metrical level is the musical one. The counterexample is any piece where a reading a factor of two away is defensible — the case that costs 100% to 66.7% on F-measure and nothing at all on AMLt. The trigger is to report CMLt and AMLt separately and to treat a wide gap between them as a finding.

Step 3 — use multiple annotations, record disagreement rather than forcing one reference — has a worked precedent. Dataset2, the SMC beat-tracking evaluation set, began with 678 extracted excerpts. A mutual-agreement threshold (BT-MMA ≤ 1 bit) selected 270 of them. The 19 highest-agreement excerpts were added as a cross-check: 289 pieces for annotation. Holzapfel and colleagues built it — five authors, four of them long-practising musicians. They first recorded spontaneous, uncorrected taps for all 289. Then they annotated them in Sonic Visualiser, with each annotation reviewed by a second author. And they left themselves the option of failing: “Each annotator was given the possibility to reject a file if the annotation process appeared intractable. This happened in 72 cases, resulting in 217 valid beat annotations for Dataset2.” Roughly a quarter of the excerpts they had deliberately selected went unannotated rather than being forced into one reading. A row that says no annotation was possible is a finding, not a gap.