SongSong: A Time Phonograph for Chinese SongCi Music from Thousand of Years Away

AI summary

Overview

• Research area: Generative music / singing voice synthesis for culturally specific ancient music (cs.SD), with a companion dataset contribution.
• Technical level: Advanced. The method combines autoregressive Transformers, diffusion-based acoustic modeling (DiffSinger-style shallow diffusion), a FastSpeech2-based variance/acoustic encoder, a Wavenet denoiser, and an Encodec + autoregressive token model for accompaniment (initialized from 1.5B-Musicgen-melody).
• Scope in one sentence: The paper introduces SongSong, described as the first music generation model capable of restoring Chinese SongCi to music, and OpenSongSong, a 29.9-hour dataset of ancient Chinese SongCi music, and compares SongSong against the commercial systems Suno and SkyMusic.

What This Paper Is About

Chinese SongCi poems were originally meant to be sung, but their original musical forms largely disappeared because ancient music scores were lost. Existing music generation models are trained mainly on modern pop songs and therefore struggle to produce music in ancient styles, and no open dataset of ancient Chinese SongCi music previously existed.

The goal of this work is to build a model that turns SongCi lyric input into complete music (singing voice plus accompaniment) in a style consistent with SongCi traditions, and to release a large annotated SongCi music dataset so the task can be trained and evaluated at all.

Key Contributions

1. SongSong model: The authors propose SongSong, which they state is the first music generation model that can perform SongCi music to their knowledge. It predicts a melody from the input SongCi, generates the singing voice and accompaniment separately from that melody, and combines all elements into the final piece.
2. OpenSongSong dataset: A comprehensive ancient Chinese SongCi music dataset totaling 29.9 hours, built to address the shortage of publicly available ancient Chinese music datasets and to support SongCi restoration.
3. Experimental comparison: The paper reports experiments comparing SongSong with the commercial systems Suno and SkyMusic under zero-shot and few-shot conditions, evaluated with both objective and subjective metrics, plus an ablation study against GPT-SoVITS (GSV).
4. User-editable output configuration: SongSong produces a config file that users can access and modify to adjust both the generated melody audio and the voiceover audio at the same time, intended to give flexible control over the output.

Main Findings

• OpenSongSong scale and coverage: OpenSongSong contains 29.90 hours of SongCi music from 89 singers, with pitch spanning G1–C6, no missing phonemes among those occurring at least 20 times, MIDI music score information, and text, phoneme, and pitch annotations aligned to the singing voice. For comparison, Opencpop contains 5.25 hours (1 singer, pitch D#3–D#5, missing "van", "ve", "vn") and M4Singer contains 29.77 hours (20 singers, pitch G1–G5). The paper states OpenSongSong has more singers and a broader pitch range than M4Singer, at comparable size.
• Test set: 50 songs not in training were selected from the SongCi collection used to develop OpenSongSong, segmented to remove silence, yielding 85 utterances totaling 1.8 hours from a male singer. Nine experts from music academies performed the subjective evaluation.
• SongCi-style metrics: In the zero-shot test, SongSong scored SCS 78.44, AC 65.54, and PA 75.37, versus Suno (SCS 35.30, AC 19.28, PA 62.59) and SkyMusic (SCS 25.37, AC 45.69, PA 54.81). In the few-shot test, SongSong scored SCS 75.19, AC 69.76, PA 78.48, versus Suno (SCS 42.19, AC 28.90, PA 48.85) and SkyMusic (SCS 46.67, AC 52.60, PA 59.04).
• Few-shot reference did not fix style mismatch: The paper reports that Suno and SkyMusic have poor prompt acceptance ability and largely generate pop-style music with instruments such as guitars and keyboards instead of the requested guzheng, even when SongCi audio is provided as a reference. SkyMusic's pronunciation accuracy rose after few-shot while Suno's fell.
• Ordinary-music metrics favored the commercial systems: SongSong scored lower on MS, ER, MC, and SQ than Suno and SkyMusic in both settings (zero-shot SongSong: MS 56.67, ER 46.26, MC 53.93, SQ 55.56; Suno: MS 65.48, ER 68.85, MC 63.19, SQ 62.96; SkyMusic: MS 63.96, ER 65.48, MC 59.11, SQ 72.67). The authors attribute this to smaller model size and training data, the relatively fixed sentence structure and monotonous style of SongCi, and the commercial systems' use of repetition, contrast, and modulation to build more complex pop arrangements.
• Voice naturalness: The paper states Suno and SkyMusic are not lower than SongSong on VN, with SongSong at VN 65.56 (zero-shot) and 72.81 (few-shot), Suno at 65.37 and 71.04, and SkyMusic at 72.89 and 72.74.
• Objective metrics: Zero-shot FAD_vgg values were SongSong 5.41, Suno 5.74, SkyMusic 7.88; few-shot values were SongSong 5.41, SkyMusic 6.31, Suno 7.92. FAD_pann values were close across systems (zero-shot 3.35e-4 to 3.87e-4; few-shot 3.42e-4 to 5.55e-5 range across entries). The paper's text describes SongSong's FAD_vgg as "relatively high" while the reported table values are the lowest of the three in both settings, so the text and the table do not align on this point; the stated conclusion is that SongSong is comparable to the others on FAD_pann.
• Ablation against GPT-SoVITS (GSV): Using 20 unused SongCi audio clips and their lyrics, with GSV given one SongCi piece for few-shot and two music experts rating the outputs, SongSong outperformed GSV on FAD_vgg (7.22 vs 20.30) and on SongCi subjective metrics (PA 81.25 vs 45.25, VN 70.00 vs 69.25, SCS 75.50 vs 46.25). FAD_pann was 5.05e-4 for SongSong and 4.93e-4 for GSV. Experts reported GSV could sing with some rhythm but could not completely sing the input SongCi, with missing words and repeated generation.
• Case study: Mel-spectrogram comparison shows the original SongCi recording is repetitive over time; Suno's output varies strongly in frequency and loudness and lacks mid- and high-frequency bands; SkyMusic reflects the overall SongCi style but has abrupt high-frequency changes; SongSong's spectrogram shows clear temporal repetition consistent with SongCi rhythm.
• Not reported: The paper does not report inference latency, model parameter counts for SongSong overall, training compute cost beyond the hardware used, or deployment results.

Methodology in Plain English

The authors avoid trying to learn a direct mapping from lyrics to melody, because that would need a large amount of paired lyric-melody data that does not exist for SongCi. Instead they break the task into a chain of smaller steps.

First, a module takes the SongCi text and predicts a rhythmic description for each word, covering tonality, chords, rhythm patterns, and cadences. This rhythm sequence is treated as an intermediate representation that bridges lyrics and melody, following the idea behind the TeleMelody system. Second, a second module with the same architecture converts that rhythm into a sequence of notes, where each note is described by bar, position, pitch, and duration. The notes are turned into a melody MIDI file plus a config file.

Third, the singing voice is generated from the melody. The model uses phonetic units rather than raw characters, so it must infer how long each phoneme lasts and what the pitch contour (F0) is. A duration predictor built from one-dimensional convolutions estimates phoneme durations, and a diffusion model implemented with Wavenet infers F0. Both feed an acoustic encoder whose output conditions a second diffusion model that reconstructs the Mel spectrum of the singing voice. To speed up inference and improve quality, the authors adopt the shallow diffusion mechanism from DiffSinger: a ConvNeXt-based auxiliary decoder first estimates the spectrum at an intermediate denoising step, and denoising starts from there rather than from pure noise.

Fourth, accompaniment is generated. Following an approach described as being in the style of Copet et al., continuous audio features are quantized using the Encodec encoder, text tokens and audio tokens are fed to an autoregressive Transformer to predict accompaniment tokens, and the Encodec decoder converts those tokens back into a continuous Mel spectrogram. The module is initialized from 1.5B-Musicgen-melody.

Finally, the singing voice audio, melody audio, and accompaniment audio are merged into the complete SongCi piece.

Training setup: The lyric-to-rhythm and rhythm-to-melody modules each use 4 encoder and 4 decoder Transformer layers with 4 attention heads and 256 linear units. The acoustic and variance encoders follow FastSpeech2 with 4 feed-forward Transformers. The duration predictor has 5 convolutional layers with kernel size 3 and input/output size 512; the auxiliary decoder has 6 ConvNeXt layers with kernel size 7 and size 512; the denoiser and F0 predictor are Wavenet-based with 20 convolutional layers of kernel size 3, with denoiser input/output sizes 512 and 1024 and the F0 predictor at half that. Training used the largest-corpus speaker from OpenSongSong, about 3.5 hours, with the Adam optimizer at a learning rate of 5e-4 on GeForce RTX-3090 hardware for up to 160,000 steps.

Evaluation setup: Subjective metrics were music structure (MS), equipment richness (ER), motivation continuity (MC), sound quality (SQ), pronunciation accuracy (PA), voice naturality (VN), conformity with the SongCi style (SCS), and accompaniment conformity (AC). Objective evaluation used Frechet Audio Distance with VGGish and PANN embeddings. The zero-shot prompt given to Suno and SkyMusic was "Chinese classical style, Song Dynasty, Guzheng, male voice."

Why This Matters

Impact on research: The work targets a low-resource, culturally specific music domain that mainstream music generation models handle poorly, and it supplies both a dataset (29.9 hours, 89 singers, full phoneme coverage, aligned text/phoneme/pitch annotations, MIDI) and a pipeline designed around an intermediate rhythm representation. It also provides a documented negative result: an LLM-style token model (GSV) trained on the same data fails at lyric-melody alignment and omits or repeats words, which supports the authors' argument that a melody-generation stage matters in low-resource settings.

Real-world applications:

• Restoration and re-performance of lost ancient Chinese music forms for cultural heritage and archival projects.
• Teaching materials for Chinese literature and music education, letting students hear SongCi as sung rather than only recited.
• Creative content production for traditional-style music, film, documentary, or stage work, using the editable config file to adjust melody and voiceover.
• A training and benchmarking resource for singing voice synthesis, accompaniment generation, and lyrics-melody alignment research on non-pop repertoires.

Industry relevance: The comparison against Suno and SkyMusic frames a concrete commercial gap: general-purpose, prompt-driven music models produce pop-style arrangements and ignore stylistic instructions such as the guzheng-only prompt, and they mishandle rare characters common in classical Chinese text. For developers building culturally specific or style-constrained music tools, the paper suggests that domain data plus an explicit melody stage can outperform scale alone on style adherence and pronunciation accuracy, while scale still wins on sound quality and perceived musical richness.

Future Directions

• Sound quality: The authors state that the lack of high-quality Song Dynasty music recordings prevents the trained model from generating music with very high sound quality, which the ablation and main results reflect in SQ and FAD_vgg differences.
• Model and data scale: The paper attributes SongSong's weaker MS, ER, and MC partly to its smaller model size and training data capacity relative to Suno and SkyMusic, suggesting larger models and more SongCi audio as a direction.
• Controllability and style following: The finding that Suno and SkyMusic largely ignore stylistic prompts raises the question of how to enforce instrument, style, and structure constraints in generated music, including through SongSong's user-editable config file.
• Alignment for long or rare-word lyrics: GSV's missing-word and repeated-generation behavior, and the commercial systems' inaccurate singing of rare characters, point to lyric-melody alignment for classical Chinese text as an unresolved problem.

Target Audience

Researchers and practitioners in music generation, singing voice synthesis, and diffusion-based audio modeling who need either a new low-resource domain dataset or an architecture template for style-constrained music generation. Also relevant to digital humanities and cultural heritage researchers working on Chinese classical poetry and ancient notation, and to engineers building music tools for non-Western or historically specific repertoires who want a documented baseline against Suno and SkyMusic.

Authors’ abstract

Recently, there have been significant advancements in music generation. However, existing models primarily focus on creating modern pop songs, making it challenging to produce ancient music with distinct rhythms and styles, such as ancient Chinese SongCi. In this paper, we introduce SongSong, the first music generation model capable of restoring Chinese SongCi to our knowledge. Our model first predicts the melody from the input SongCi, then separately generates the singing voice and accompaniment based on that melody, and finally combines all elements to create the final piece of music. Additionally, to address the lack of ancient music datasets, we create OpenSongSong, a comprehensive dataset of ancient Chinese SongCi music, featuring 29.9 hours of compositions by various renowned SongCi music masters. To assess SongSong's proficiency in performing SongCi, we randomly select 85 SongCi sentences that were not part of the training set for evaluation against SongSong and music generation platforms such as Suno and SkyMusic. The subjective and objective outcomes indicate that our proposed model achieves leading performance in generating high-quality SongCi music.