SheetSage2: Coherent Lead-Sheet Transcription with Synthetic Supervision

AI summary

Overview

Research area: Music information retrieval (cs.SD), specifically automatic lead-sheet transcription — converting audio recordings into human-readable musical scores that encode rhythm, harmony, melody, and form.

Technical level: Advanced. The paper assumes familiarity with autoregressive sequence models, dynamic programming decoders, frame-level acoustic modeling, and music-theoretic concepts such as meter, key, chord spelling, and pitch intervals.

Scope: The paper describes SheetSage2, a unified transcription framework that pairs a MERT2-based audio encoder with synthetic MIDI-derived supervision, task-specific structured decoding, and autoregressive distillation into a single sequence-to-sequence model.

What This Paper Is About

Turning a recording into a lead sheet is hard because a system must jointly infer several musical dimensions at once — note positions and durations, bar lines and meter, tempo, chords, key, melody, and section markers — and organize them onto one shared, mutually consistent grid. Two obstacles stand in the way: complete audio–score pairs are scarce (existing datasets usually annotate only some attributes, sometimes with imprecise timing), and systems that make accurate local predictions can still produce globally incoherent scores, such as switching between half- and double-tempo readings or shifting octaves mid-melody. SheetSage2 addresses both by generating large volumes of synthetic audio with aligned labels from MIDI, decoding local predictions with musically structured constraint, and then distilling the result into an autoregressive model that needs no task-specific dynamic programming at inference.

Key Contributions

1. MIDI-derived supervision for audio music understanding. The authors extend content-based MIDI self-annotation into a scalable pipeline that renders MIDI collections into audio and derives aligned task labels (beat/tempo, meter/downbeat, melody, chord, key, structure) through symbolic analysis and pseudo-label bootstrapping, then show this synthetic supervision improves performance on real recordings.
2. Structured decoding for unified transcription. They build task-specific decoders with explicit tempo, meter, and pitch context — including a three-stage rhythm decoder with a shared tempo anchor and a Markovian melody decoder whose onset scores depend on the preceding pitch — to improve consistency within and across tasks.
3. Autoregressive distillation. They transfer the structured teacher's outputs into a single autoregressive student (SheetSage2-AR) that abandons task-specific dynamic programming at inference while largely retaining transcription performance.

The report notes that YuE2 (Yuan et al., 2026) first introduced SheetSage2 and the MERT2 encoder to supply symbolic and semantic supervision for music generation; this paper describes SheetSage2's method and evaluation in full.

Main Findings

• Broad benchmark leadership by a single model. Across eight benchmark collections (GTZAN, osu2017, GiantSteps, Chords1217, JAAH, HarmonixSet, RWC-Pop, and Rock Corpus), a single SheetSage2-AR model exceeds the strongest listed prior baseline on 12 of 15 benchmark–metric pairs, substantially improving over SheetSage1 and surpassing task-specific models on several benchmarks.
• Distillation preserves and sometimes improves accuracy. SheetSage2-AR exceeds SheetSage2-Prober on 10 of 15 benchmark–metric pairs, including both beat and downbeat benchmarks, showing that the autoregressive student retains Prober performance through a single event-generation interface.
• Specific headline scores. On osu2017 beat F1, AR reaches 93.01 versus SheetSage1's 91.80; on osu2017 downbeat F1, AR reaches 92.90 versus Prober's 92.79 and SheetSage1's 83.47; on JAAH major/minor chord, AR reaches 64.50 versus SheetSage1's 39.77; on Rock Corpus vocal-melody F1, AR reaches 67.08 versus SheetSage1's 49.19.
• Prior specialists still win some pairs. Beat This! leads GTZAN beat F1 at 89.01 versus AR's 86.27, and S-KEY leads GiantSteps key score at 72.09 versus AR's 77.73 (AR is higher) — the paper's own note is that AR does not exceed the prior baseline on every pair.
• Synthetic data helps most task averages. Adding synthetic data to real training data improves beat (+2.17), downbeat (+3.71), chord (+1.57), and structure (+1.58) task averages over real-only training, leaves the melody average nearly unchanged (+0.01), and slightly reduces the key average (−0.24).
• Synthetic-only training transfers to real audio. A synthetic-only model reaches 57.76% vocal-melody F1 on RWC-Pop, even though its synthetic training audio contains no singing, and remains competitive on chord and key estimation.
• Structured decoding improves tempo consistency. Measured by a "mixing ratio" (percentage of recordings whose predictions mix metrical levels), Prober and AR mix tempo levels far less than Beat This! on both osu2017 (2.96 and 1.48 versus 20.74) and Hooktheory (0.34 and 0.51 versus 17.30). The proposed decoder also beats P-DBN (madmom's DBN applied to Prober activations) on both sets.
• Pitch-interval dependencies improve octave consistency. On 333 common RWC full-melody passages, full decoding reduces the octave mixing ratio from 12.91% to 9.31% versus a pitch-marginal variant; SheetSage1 mixes octaves in 34.83% of passages, while AR reaches 11.71% without an explicit melody dynamic program at inference.

Methodology in Plain English

Step 1: Generate synthetic supervision from MIDI. Because complete audio–score pairs are rare, the authors mine large symbolic MIDI collections. The Los Angeles MIDI Dataset supplies 404,714 deduplicated files, but higher-level labels (key, chords, melody tracks) in MIDI are often missing or unreliable — for Beatles MIDI files, Raffel & Ellis (2016) reported a MIREX weighted key score of 0.400, rising to 0.842 after excluding C-major metadata. So the authors label MIDI with symbolic MIR models: they curate a small high-precision seed set using expert rules or metadata (for example, 17,134 MIDI files with identifiable melody-track names), fine-tune pretrained symbolic models on those seeds to make a MIDI auto-labeler, and bootstrap outward — the melody pool expands to 185,887 annotated songs (10.8×). Beat/tempo, meter/downbeat, and structure labels come from tick resolution, time-signature events, the metrical grid, and curated MIDI text markers (SLMS). The annotated Los Angeles collection and SLMS supply roughly 361,000 synthetic source records. They render MIDI with FluidSynth and the FluidR3 GM SoundFont and pair the labels with 24-kHz audio.

Step 2: Train two frame-wise "Prober" models on mixed data. Because annotations are partial, noisy, and inconsistent, the authors avoid training directly on event sequences. Instead, SheetSage2-Prober learns frame-level probabilities from whatever labels are available, masking missing targets and using source hints to handle different annotation conventions. A non-melody model predicts 160 binary features at 25 Hz (with four slots per rhythm event giving 100-Hz timing), covering quarter-note, eighth-note, and downbeat probabilities, meter denominator support, median tempo bins, relative tempo bins, chord pitch-class probabilities for root/chroma/extension/bass, 24 key supports, section boundary and 23 section label scores. A second, separately trained melody model operates on subdivisions of the predicted beat grid, conditioned on a vocal or full-melody hint, and outputs a categorical distribution over 128×7 pitch–interval categories, 24 durations, one rest, and five auxiliary slots.

Step 3: Decode with musical structure. Frame-wise probabilities are converted into discrete decisions by a six-stage decoder, where each stage constrains the next: (1) pool the median-tempo predictions to select a single global tempo anchor from bins in 60–240 BPM; (2) jointly decode eighth-note pulse positions and their intervals, which set local tempo, using dynamic programming with a log-ratio penalty on adjacent pulse intervals; (3) jointly decode metrical positions and meter changes to build the beat grid; (4) decode keys and section labels at the measure level, restricted to downbeats; (5) decode chords with boundaries aligned to beats and spell them enharmonically under the local key; (6) decode the melody from the Markovian predictions, where each onset's score depends on the preceding pitch through seven signed-semitone interval buckets (ranges (-∞,−18], [−17,−12], [−11,−6], [−5,5], [6,11], [12,17], [18,∞)) — this is what suppresses local octave errors like C4–E4–G4 becoming C5–E4–G4.

Step 4: Distill into one autoregressive model. SheetSage2-AR uses a MERT-v2-FullSong (MERT2-FS) encoder on 24-kHz mono audio and an autoregressive decoder that predicts each token from the audio and preceding tokens, with greedy decoding restricted by an event grammar. It is trained only on pseudo-labels produced by the Prober's structured decoders, so it learns the complete lead-sheet representation as a single sequence-to-sequence task and needs no task-specific dynamic programming at inference. The event language combines absolute beat timestamps quantized into 10-ms bins with relative positions on a sixteenth-note grid; a deterministic notation builder converts the sequence into ABC notation or MIDI.

Why This Matters

Impact on research. The paper attacks two persistent bottlenecks in unified music transcription at once: data scarcity (by showing that automatically labeled, rendered MIDI can substitute for curated audio–score pairs) and sequence coherence (by demonstrating that structural constraints like a shared tempo anchor and interval-dependent pitch scoring measurably reduce metrical-level and octave mixing). The finding that a structured teacher can be distilled into a plain autoregressive student without inference-time dynamic programming is relevant beyond music — it is a sequence-level distillation result about retaining constrained-decoding quality in a simpler runtime.

Real-world applications:

• Assistive score creation for musicians and producers, converting a recording into an editable lead sheet in ABC or MIDI form.
• Music education and analysis, providing chord, key, beat, downbeat, and section annotations for teaching or study of popular-song repertoire.
• Pluggable rhythm/harmony front-ends for downstream systems, since the Prober and AR models produce beat grids, chords, keys, and sections that other tools can consume.
• Synthetic supervision as a general recipe, useful wherever labels are scarce but symbolic data is plentiful — the pipeline itself (curate seeds, fine-tune a symbolic labeler, bootstrap, render) is portable.

Industry relevance. Model weights and inference code are publicly available at https://huggingface.co/m-a-p/SheetSage2, which lowers the barrier to adoption in DAWs, karaoke and accompaniment tools, sample-pack and library cataloging, and rights-adjacent search. The paper's Impact Statement is explicit that automatic transcription can facilitate copying compositions from recordings, that a transcribed lead sheet does not grant permission to reproduce or distribute a work, and that transcriptions can contain errors and should not be treated as authoritative scores.

Future Directions

• Relaxing the fixed musical assumptions. The paper's own Limitations section points to the sixteenth-note grid (which only approximates triplets and finer subdivisions), a single tempo anchor per recording (unsuited to pieces with large tempo changes), a limited meter inventory, 12-tone equal temperament, and a chord vocabulary restricted to triads, suspended, sixth, and seventh chords (so extended and altered chords such as ninths cannot be represented).
• Broadening genre coverage. The lead-sheet format and training data target popular music with a steady beat; the authors state SheetSage2 is not designed for classical music, which often requires several voices and expressive tempo, or for free-rhythm music without a metrical grid.
• Testing whether explicit constraints become unnecessary with more data. The conclusion suggests that with more training data, some explicit musical constraints may become less necessary — an open empirical question about the trade-off between structure and scale.
• Improving synthetic supervision quality. The authors call for advances in symbolic music analysis and generation to produce better, more diverse synthetic supervision, since annotated recordings remain scarce.

Target Audience

Researchers and graduate students in music information retrieval, audio signal processing, and sequence modeling who work on transcription, structured decoding, or distillation; practitioners building music analysis or score-generation tools who want a reproducible baseline with public weights; and music technologists interested in how synthetic MIDI-derived supervision can substitute for scarce human annotations. Readers need comfort with dynamic programming decoders and autoregressive modeling to follow the methodology, though the high-level narrative (synthetic data, structured decoding, distillation) is accessible to a broader technical audience.

Authors’ abstract

Transcribing music into a human-readable score requires a coherent understanding of rhythm, harmony, melody, and form. Two obstacles limit this goal: annotated recordings are scarce, and accurate local predictions can still produce inconsistent musical sequences. We present SheetSage2, a unified music transcription framework that combines synthetic data, task-specific structured decoding, and autoregressive distillation. Automatically annotated MIDI, rendered into audio, provides scalable supervision across music understanding tasks. Task-specific structured decoders integrate complementary musical cues and their temporal dependencies to produce musically coherent scores. Autoregressive distillation further retains transcription accuracy without task-specific dynamic programming at inference. Across eight benchmark collections, a single SheetSage2-AR model exceeds the listed prior systems on 12 of 15 benchmark--metric pairs in our evaluation, substantially improving over SheetSage1 and surpassing task-specific models on several benchmarks. Model weights and inference code are publicly available.