Toward Part-Aware Choral Transcription with singing voice assignment

AI summary

Overview

Research area: Music information retrieval (MIR) — specifically automatic music transcription (AMT) for choral singing, with joint vocal-part (SATB) assignment. Published under eess.AS (arXiv:2610.09861v1).

Technical level: Advanced. The paper assumes familiarity with neural AMT architectures (CRNN encoders, BiGRUs, onset/offset/frame decoding), note-level evaluation metrics, and combinatorial assignment procedures.

Scope (one sentence): The paper introduces PawCT, an end-to-end neural framework that jointly identifies which of the soprano, alto, tenor, and bass (SATB) parts are active in mixed choral audio and transcribes each active part into its own note-level MIDI track, and it compares this against part-agnostic transcription and two-stage post-hoc voice assignment.

What This Paper Is About

Choral music is normally written and rehearsed as four separate lines (soprano, alto, tenor, bass), but existing note-level choral transcription systems collapse everything into a single merged note track, which is of limited use for rehearsal, teaching, or rebuilding a score. The problem is hard because the four parts share very similar vocal timbre, their pitch ranges overlap, and choral singing has soft onsets and smooth pitch glides, so telling "which note belongs to which part" is genuinely ambiguous. The authors' goal is to make part assignment part of transcription itself rather than something bolted on afterwards.

Key Contributions

1. A part-aware formulation of choral AMT. The authors define the task as jointly detecting which SATB parts are active and producing a separate note-level track per active part (up to four labeled tracks), in contrast to the previous single merged track. They state this is, to their knowledge, the first end-to-end neural framework for this task.

2. PawCT, a structured end-to-end architecture. The model combines a shared CRNN encoder with part-specific onset, offset, and frame heads, a part-presence head that gates inactive parts to empty tracks, union-level supervision built by max-pooling the predicted parts, and structured training targets derived from a range prior (RP) and an ordered-continuity (OC) extension.

3. Two voice-assignment priors used as training targets. RP assigns notes by register using SATB pitch ranges — S: [60, 88], A: [55, 79], T: [48, 72], B: [40, 67] — with penalties for out-of-range pitch, distance from the range center, and disagreement with an available source label. OC extends this with within-part melodic continuity and offset-overlap penalties, plus a one-to-one assignment per onset group of up to four notes that preserves pitch order.

4. A controlled comparison against post-hoc assignment. The paper also presents PagCT (a part-agnostic counterpart producing one merged track) and a two-stage pipeline in which PagCT output is passed to Post-VA, a symbolic BiLSTM classifier, so that joint and post-hoc voice assignment can be compared directly.

Main Findings

• Part-agnostic transcription improves substantially. On the YouChorale test set, PagCT reaches a 50-ms onset note F1 of 0.382 and a 100-ms F1 of 0.573, compared with 0.237 and 0.347 for the previous choral-specific state-of-the-art model (Yu et al.), a 61.2% relative improvement at 50 ms. Without pitch-transposition augmentation, PagCT's 50-ms note F1 is 0.333.

• Frame accuracy does not imply note accuracy. MuScriptor reaches a frame F1 of 0.730, close to PagCT's 0.738, but only 0.084 note F1 at 50 ms and 0.231 at 100 ms. Onsets and Frames shows the opposite imbalance — 50-ms note precision 0.450 versus recall 0.178 — which the authors read as conservative detection of soft vocal onsets. MT3 and YourMT3+ both reach 0.127 note F1 at 50 ms.

• Joint modeling beats post-hoc assignment. On YouChorale, PawCT-RP-OC achieves a macro part-aware note F1 of 0.225 at 50-ms tolerance, versus 0.175 for the two-stage PagCT + Post-VA pipeline and 0.165 for the adapted choral baseline (36.4% relative gain). Its note-level VA Rate — the percentage of part-agnostic performance retained after assignment, F1_avg / F1_agn × 100% — is 58.90%, versus 45.81% for PagCT + Post-VA.

• Union supervision matters. Removing the union loss drops macro part-aware note F1 from 0.217 to 0.190, and the authors note the decline appears across every SATB part.

• Range alone is not enough; continuity is. Adding RP raises average frame F1 to 0.510 but lowers note F1 to 0.209, which the paper attributes to vocal range helping track pitch activity without resolving assignment ambiguity. Adding OC recovers note F1 to 0.225 (average frame F1 0.503).

• VA Rate must be read alongside absolute accuracy. The adapted YourMT3+ baseline retains 62.20% of its part-agnostic note F1, higher than PawCT-RP-OC's 58.90%, but its absolute macro part-aware note F1 is only 0.079. The adapted choral baseline shows the same pattern (69.62% VA Rate, 0.165 note F1).

• Cross-dataset results are mixed but favor the end-to-end model. Trained only on YouChorale, PawCT-RP-OC scores 0.174 average note F1 on CSD and 0.313 on Cantoría, against 0.118 and 0.115 for PagCT + Post-VA; plain PawCT scores 0.161 and 0.271. Performance is worse than on YouChorale for CSD but better for Cantoría, which the authors partially attribute to Cantoría's more uniform complete-SATB structure, while cautioning that recording conditions and annotation protocols may also contribute.

Methodology in Plain English

The researchers take a high-resolution piano transcription design as their template and adapt it to four voices. Audio is converted into a log-Mel spectrogram (16 kHz sampling, 229 bins, NFFT 2048, 160-sample hop), which passes through four convolutional blocks (each two 3×3 convolution–batch-norm–ReLU layers plus frequency pooling), a projection layer, and a two-layer bidirectional GRU with 256 units per direction, giving a frame-by-frame representation of size B × T × 512.

From that shared representation, each of the four parts gets its own sigmoid heads predicting onsets, offsets, and preliminary frames over 88 MIDI pitches. Those preliminary frame rolls are concatenated with the (detached) onset and offset rolls across all parts and refined by a one-layer BiGRU, then projected to 4 × 88 dimensions. Separately, a presence head mean-pools over time and runs a two-layer MLP (256 then 4 units) whose sigmoid outputs gate each part's rolls, so an inactive part yields an empty track.

The distinctive part is how training targets are built. Rather than adding new differentiable losses, the authors use RP and OC as assignment procedures that convert ground-truth note annotations into part-labeled targets. RP scores each note against each part by how far it falls outside that part's range, a weak pull toward the range center, and a penalty for contradicting an available source label. OC adds a penalty for large jumps from the most recent note in that part and a penalty for temporal overlap with it, then solves a small one-to-one assignment for each onset group of up to four notes, preserving pitch order; groups larger than four notes fall back to independent minimum-cost assignment.

Training combines three terms: a masked binary cross-entropy over the part-wise frame/onset/offset predictions, a union loss where the predicted parts are max-pooled and compared against targets built from merged ground-truth notes, and a presence-head BCE, with λ_pres = 0.2. Assignment-cost weights are (λ_part, λ_cont, λ_ov) = (2, 0.35, 4), part-wise loss weights (1, 2, 1), and union weights (1, 1, 1). Optimization uses AdamW with batch size 8, initial learning rate 10⁻⁴ decayed by 0.9 every 10k iterations, up to 200k iterations with early stopping after 30k without validation improvement. Post-VA is trained separately for up to 80 epochs with batch size 32 and patience 10.

Notes are decoded from onset, offset, and frame predictions with an onset-and-frame decoder, using inference thresholds of 0.03 (frame), 0.01 (onset), and 0.005 (offset) chosen on the YouChorale validation set and applied unchanged to CSD and Cantoría. Training data is YouChorale: 22 hours 25 minutes of a cappella recordings, 452 recordings from 261 songs, of which 418 (92.48%) contain all four parts and 34 contain a subset. Pitch-transposition augmentation of one to three semitones in either direction expands the training set by a factor of seven, to roughly 129 hours 38 minutes.

Why This Matters

For research, the paper reframes choral transcription from "find the notes" to "find the notes and say who sings them," and shows that doing both jointly outperforms doing them in sequence. It also introduces a metric (VA Rate) that makes explicit a trade-off the field could easily misread: a system can retain a high fraction of its part-agnostic accuracy while being poor in absolute terms.

Real-world applications implied by the paper's stated motivations:

• Rehearsal tools — singers could isolate and practice their own part from a recording of the full choir.
• Music education — students could see SATB lines separated rather than as an undifferentiated cluster of notes.
• Score reconstruction — recovering part-labeled symbolic scores from existing choral recordings.
• Archival and cataloguing — producing searchable, part-labeled representations of recorded choral repertoire.

Industry relevance spans music-education software, practice and rehearsal apps, and score-editing or notation platforms that need symbolic, part-labeled input rather than audio. The paper's own emphasis on cross-dataset robustness (CSD, Cantoría) speaks directly to whether such systems could survive deployment on recordings that differ from the training corpus.

Future Directions

• Domain adaptation and decoding calibration. The authors explicitly call for this, given that YouChorale-trained models degrade on CSD (0.174 average note F1 for PawCT-RP-OC) relative to YouChorale (0.225) while improving on Cantoría (0.313).
• Disentangling why Cantoría transfers better. The paper offers the more uniform complete-SATB structure of Cantoría as a partial explanation but acknowledges recording conditions and annotation protocols as confounds — a question left open.
• Making assignment priors learnable. RP and OC currently construct training targets rather than acting as differentiable losses; whether these constraints could be optimized directly inside the network is unaddressed.
• Extending beyond four-note onset groups. Groups larger than four notes fall back to independent minimum-cost assignment, and the framework is defined specifically for SATB, leaving richer divisi textures and non-SATB vocal ensemble configurations as open territory.

Target Audience

Researchers and graduate students in music information retrieval working on automatic music transcription, multi-pitch estimation, or voice separation will get the most from this paper, particularly those already familiar with piano AMT architectures who want to see those ideas extended to vocal ensembles. It is also relevant to practitioners building choral rehearsal, music-education, or score-digitization tools who need to know how far part-aware transcription currently reaches, and to anyone interested in the general problem of assigning concurrent sources to labeled streams when timbral cues are weak. Readers without a background in neural AMT will find the architecture and assignment-cost sections dense, though the task framing and the part-aware versus part-agnostic comparison in Figure 1 are accessible without it.

Authors’ abstract

Single-instrument automatic music transcription (AMT) has advanced substantially, yet choral applications require soprano, alto, tenor, and bass (SATB) to be transcribed as separate parts. Recent note-level choral AMT instead produces a single merged note track, limiting rehearsal, education, and score reconstruction. To address this limitation, we introduce Part-aware Choral Transcription (PawCT), to our knowledge the first end-to-end neural framework that identifies active SATB parts from choral audio and transcribes each into a separate note-level track. PawCT combines part-specific onset, offset, and frame prediction with part-presence estimation, union-level supervision, and structured training targets using a range prior (RP) based on SATB pitch ranges and its ordered-continuity (OC) extension, which adds within-part melodic continuity and cross-part pitch ordering. On YouChorale, PawCT-RP-OC achieves a macro part-aware note F1 of 0.225 at a 50-ms onset tolerance, outperforming an adapted choral baseline (0.165) by 36.4% relative and a two-stage post-hoc assignment pipeline (0.175). Its part-agnostic variant, PagCT, achieves a 50-ms onset F1 of 0.382, compared with 0.237 for the previous state-of-the-art choral AMT model. Cross-dataset evaluations on CSD and Cantoria further assess performance under dataset shift. These results demonstrate the benefit of jointly modeling note transcription and vocal-part assignment. Code and demos are available at https://hanyu-meng.github.io/Paw_Choral_AMT_Demo/.