Diptych: Scoped, AI-Interpreted Comparison for Reference Listening in Music Production

AI summary

Overview

• Research area: Human–Computer Interaction, specifically AI-assisted music production and reference listening (the practice of comparing a work in progress against a finished commercial recording). The work sits at the intersection of HCI, music information retrieval (MIR), and audio engineering.
• Technical level: Intermediate. The interaction design and study method are accessible to a general HCI reader, but the feature taxonomy assumes familiarity with production and audio-analysis vocabulary (LUFS, chroma, Tonnetz, formants, jitter/shimmer).
• Scope, in one sentence: The paper presents DipTych, a system that keeps the choice of what to compare in the user's hands while an AI interprets structured audio features within that chosen scope, evaluated in a within-participants study with 12 musicians and a source-blinded expert validation panel of 4 listeners.

What This Paper Is About

Reference listening means holding your track against a commercial recording to judge what to change. Existing tools make this comparison at the level of the complete track, so they silently decide which passages are "the same" — a decision that is itself part of the listening. The paper's goal is to build and study a system where the user defines the comparison scope (whole tracks, or two independently selected segments that may differ in start time, length, or structural label), and the AI returns an interpretation tied to that scope, alongside raw feature evidence the user can inspect.

Key Contributions

1. An interaction design that keeps comparison scope under user control, supporting both whole-track comparison and independent selection of segments from each track, without requiring a shared timestamp, duration, or structural label.
2. An empirical account of how musicians' comparative judgments change when they can set that scope and inspect structured audio analysis alongside AI interpretation, from a within-participants study with 12 musicians using a retain/revise/withdraw/uncertain protocol.
3. An independent expert perceptual reference: 4 expert listeners performed blind A/B listening on the same tracks and then rated participants' anonymized claim units without knowing their source (before or after system use).
4. Design principles for AI-assisted music comparison, the central one being that being correct is not enough — a comparison must also fit the scope, be specific, and point toward a decision the user can act on.

Main Findings

• Additional differences surfaced and largely supported: Participants used the system to surface additional differences, nine of ten of which received at least partial expert support.
• Usability: Participants reported good usability. The paper states that the System Usability Scale (SUS) was administered, but the truncated content provided does not report the numeric SUS score.
• Clarity of next steps: Participants reported greater clarity about possible next steps (whether to change something, try something, or leave it alone).
• Disagreement treated as data: Participants were explicitly instructed that AI output could be incomplete, inaccurate, or inappropriate, and could mark initial judgments as retained, revised, withdrawn, or still uncertain, as well as record agreement or disagreement with the system view.
• Correctness and usefulness come apart: The paper argues an AI verdict about the wrong comparison is not rescued by being accurate; a comparison earns its place when it moves the user toward a decision.
• Eight-dimensional feature vocabulary: The system exposes 59 audio features grouped into eight production-relevant categories rather than a single collapsed similarity score, extending the disentanglement argument made by MERIT (published work cited in the paper) from a retrieval embedding space into features a producer can read and check.
• Scope-specific interpretation: The AI interpretation describes only the currently selected scope; changing the selection produces a new interpretation for the new pair.

Methodology in Plain English

System build. The target and reference tracks are independently resampled, structurally segmented, and described with a shared feature set. The feature category set was chosen where three independent lines of evidence converge: the vocabulary mix engineers already use, MIR facet organization (rhythm, harmony, timbre, mood), and prior taxonomies of production knowledge. Structural segmentation builds a self-similarity matrix from each track's chroma features, applies a Foote-style checkerboard kernel to get a novelty curve, and uses its peaks as candidate section boundaries; segments shorter than approximately eight seconds are merged into neighbours. Labels such as Intro, Verse, Chorus, Bridge, Outro, and Solo are assigned by rules based on position and repetition patterns, and the paper notes these are navigation cues rather than a trained classifier of musical form. The comparison engine computes differences for the selected scope and sends them plus a glossary of feature definitions to a self-hosted language model via vLLM's OpenAI-compatible endpoint, using schema-constrained decoding and a low sampling temperature. Track features are reused when scope changes.

Feature extraction. 59 features are extracted with librosa and related libraries, resampling to 22.05 kHz for spectral and harmonic analysis and 8 kHz for pitch tracking. The pipeline uses YIN/pYIN pitch tracking, chroma features, spectral descriptors, MFCCs, onset and beat tracking, and RMS and loudness statistics, computed for the full track and each structural segment and stored as one JSON file per track with an overall_song block and a segments array. Supporting libraries named in the tables include librosa, scipy.signal, Beat This!, music2emo, pyloudnorm, numpy, parselmouth, and Whisper.

Study. A researcher-facilitated, single-session, within-participants study, deliberately staged as a before-and-after comparison rather than separate conditions, with 12 musicians. Participants used headphones and calibrated playback volume, completed a background questionnaire on musical background and reference-track experience, and did the same tutorial plus a practice task on a separate song pair (excluded from analysis). Each participant then completed one formal comparison on a predetermined target–reference pair supplied by the research team; the task did not require uploading their own music or making edits in a DAW. They first made unaided judgments in a basic A/B player with no visualizations or AI output, recording the differences heard, the dimension and direction, an approximate time location, the audible basis, confidence, and possible next steps. They then explored the full system and revisited every initial judgment, marking it retained, revised, withdrawn, or still uncertain, and could add new differences and revised next steps. Post-task measures included system-effectiveness items, direct helpfulness and trust ratings, the System Usability Scale, system-specific interaction-clarity items, open-ended feedback, and logs of task time, playback and switching actions, segment selections, and expansions of evidence.

Expert lane. In a separate validation lane, 4 expert listeners first did blind A/B listening on the same materials without access to DipTych or participant responses. Participants' claims from before and after system use were segmented into anonymized claim units and rated without revealing their source. The provided content is truncated mid-sentence at the point describing exactly how each claim was assessed, so the full rating scheme is not reported here.

Why This Matters

The paper reframes a design assumption baked into most comparison tools: that deciding what to compare is the tool's job. It argues instead that this decision is part of the listening itself, and that tools should support the reading of a difference without taking over the selection of it. It also gives builders of AI feedback systems a concrete test — a comparison can be accurate and still tell the user very little — and shows how a source-blinded expert panel can be used to check whether system-surfaced differences are genuinely audible.

Real-world applications:

• Mixing and mastering workflows: letting an engineer hold their chorus against a reference's chorus, or a verse against a chorus, when the two tracks place those sections at different times and durations.
• Audio engineering education: giving learners a structured vocabulary (eight categories, 59 features) to turn "something feels wrong here" into a specific, checkable judgment.
• Audio tool and plugin design: an alternative to whole-track reference matching and to fully automatic style-transfer mixing, keeping scope and judgment with the producer.
• Cross-domain comparison tools: the authors explicitly point to video editing (comparing rough-cut alternatives against source footage) and writing (revising drafts against model texts) as instances of the same pattern.

Industry relevance: the paper speaks directly to the design of reference-comparison plugins and analysis tools (iZotope Ozone and Logic Pro are named as examples of whole-track tools) and to reference-conditioned mixing research such as Diff-MST, which predicts mixing-console parameters for up to twenty tracks from a reference song with manual adjustment only after an initial mix is generated. The critique is that these either do not expose local comparison scope or produce an output instead of presenting a comparison for inspection.

Future Directions

• Isolate the AI's contribution: the study evaluates changes following exposure to the complete system, not the AI component alone, so separating the effect of the AI verdict from the feature tables and timelines is an open question.
• Scale and duration: the study is single-session with 12 musicians; whether scope-first comparison changes real production decisions over longer sessions and larger samples is untested here.
• Ecosystem integration: participants used a research-supplied target–reference pair and did not upload their own music or work in a DAW, so integration with real project files and studio workflows remains to be studied.
• Generalize the scope-first pattern: the authors position music as one case of reference-based creative work and invite testing of the same split between choosing what to compare and reading the difference in video editing and document revision.
• Guard against over-reliance: the paper cites evidence that convincing LLM explanations can increase reliance even when the response is incorrect, raising the question of how to present scope-tied interpretations without encouraging users to accept them wholesale.

Target Audience

HCI researchers working on human–AI interaction and creative tools; MIR and intelligent-music-production researchers interested in interpretable audio features and reference-conditioned systems; designers and engineers building comparison, feedback, or AI-assistance features into audio software; and music production educators interested in structured vocabularies for talking about sound. The paper is also relevant to researchers outside music who study AI-assisted comparison and revision in video, writing, or design, since the design principles are framed as transferable.

Authors’ abstract

Reference listening is a common strategy in music production, but current comparison tools often obscure a key human judgment: deciding what should be compared. We present Diptych, an AI-assisted system that lets users define comparison scope across whole tracks or independently selected segments, while inspecting structured audio features and scope-specific AI interpretations. We evaluated Diptych in a within-participants study with 12 musicians, complemented by source-blinded ratings from four expert listeners. Participants used the system to surface additional differences, nine of ten of which received at least partial expert support, and reported good usability and greater clarity about possible next steps. These findings suggest that AI support for creative comparison should prioritize user-defined scope, inspectable evidence, and actionable guidance, while avoiding authoritative judgments that exceed what the evidence can support.