Voice Conversion and Speech-to-Speech Translation

Certainty is something a listener hears before parsing a single word. In 2018 Xiaoming Jiang and Marc D. Pell put that to a panel. They recorded 151 short English sentences. Each was produced in a scenario in which the speaker was very confident, almost confident but not 100% sure, very unconfident, or neutral. Eighteen native Canadian-English listeners then judged every utterance on a 5-point confidence scale. The agreement was strong enough to use as a filter. A clip was kept as a stimulus only if at least 13 of the 18 listeners — more than 72% — agreed that a level of confidence was conveyed.

Two findings sit on top of each other. Certainty is audible, and it survives an unfamiliar accent. The abstract says so: “We demonstrate that native Canadian-English listeners can recognize confident and doubtful expressions in foreign- and regional-accented speakers. A stronger impression of confidence was shown towards the native speakers.” But who carries the sentence changes how certain it sounds. A linear mixed-effects model found a significant effect of speaker accent on perceived confidence, F(2, 471) = 68.48. The same intended certainty was read differently depending on whose voice delivered it.

That accent, and the prosody wrapped around it, is exactly what a voice conversion or a speech-to-speech translation system replaces. A hedge can leave the source speaker as a hedge and reach the listener as a claim, with every word and every name intact. A review that checks only whether the words and the names came through would pass that output without hesitating. That is the difficulty the rest of the lesson keeps returning to. The evidence anyone thinks to collect is evidence about words. This failure lives entirely in what the words were wrapped in.

Two operations are being asked for here, and they are worth naming apart. Voice conversion changes selected vocal attributes while attempting to preserve linguistic content. Speech-to-speech translation changes language, and may also preserve speaker identity, timing, emotion, or style, through a cascade or through a direct model.

Both attract the same instruction, and the instruction is not a complete brief. “Preserve everything except language” says nothing about the fact that content, identity, emotion, timing, accent, disfluency, uncertainty, background, and turn structure can conflict with one another. Some of them have no direct equivalent in the target language at all.

The Voice Conversion Challenge 2020 measured one of those conflicts across a whole field of systems. It received 33 submissions, 3 of them baselines, and ran crowd-sourced listening tests on all of them. The organizers' result: “In particular, speaker similarity scores of several systems turned out to be as high as target speakers in the intra-lingual semi-parallel VC task. However, we confirmed that none of them have achieved human-level naturalness yet for the same task.” Identity was preserved to the level of the target speaker. Naturalness was not — in the same systems, on the same task. The cross-lingual task, the one that matters for translation, scored lower on both, with the best systems above 4.0 MOS.

So the attributes are not jointly satisfiable. A brief that promises all of them is deciding nothing. Two questions follow it, and they are not the same question. Whether the meaning arrived intact is one. Whether this speaker's voice may carry it at all is the other, and that one needs a signature rather than a score. Neither answer, on its own, should stand in for the complete release decision.

Which is why the order of work is fixed. Write the preservation contract first — the list of attributes that must remain, may change, or must be suppressed. The next decision, cascade or direct model, spends whatever that contract assumed. Evaluating semantics and interaction comes last. It is the point where those assumptions meet an actual conversation.

The four steps are these. Write the preservation contract, stating which attributes must remain, change, or be neutralized. Choose cascade or direct model, comparing ASR–translation–TTS modularity with end-to-end speech translation. Control identity and style through explicit conditioning, consent, and limits on voice preservation. Then evaluate semantics and interaction, testing meaning, names, numbers, prosody, latency, speaker similarity, and repair.

Reverse the order and the evaluation still runs. It simply judges the system against a contract nobody wrote. That is how an output whose words are all correct comes back marked correct while the speaker's perceived confidence has moved — the effect Jiang and Pell sized at F(2, 471) = 68.48, and no word-level check records it.

When that contract becomes a dubbing agreement, it turns on four terms. Used loosely, they leave two things unclear: which stage of the system owes the evidence when something goes wrong, and who may authorize the transformation in the first place.

One of those terms has stopped being an accuracy question. In 2021 a group led by Luisa Bentivogli compared state-of-the-art cascade and direct speech translation systems on English–German, English–Italian and English–Spanish. The comparison used professional post-edits and manual annotation. The paper asks in its title whether the differences between the two paradigms still make a difference, and answers: “Our multi-faceted analysis on one of the few publicly available ST benchmarks attests for the first time that: i) the gap between the two paradigms is now closed, and ii) the subtle differences observed in their behavior are not sufficient for humans neither to distinguish them nor to prefer one over the other.” If annotators can neither tell the two apart nor prefer either, the architecture choice is no longer about output quality. It is about which stages remain open to inspection — that is, about who can answer for the mistake afterwards.

The contract is not the same document in every setting. Four of them pull it in different directions, and each asks for other units and another kind of proof.

The privacy setting has the sharpest published numbers, and they are humbling. The first VoicePrivacy Challenge drew 45 participants from 13 countries, representing 25 teams, and yielded 16 eligible submissions. Anonymization performance then collapses as the attacker gets better informed. Many systems exceed 50% equal error rate against an ignorant attacker. The best fall to 33–43% against a lazy-informed attacker. Against a semi-informed attacker, whose speaker-verification model has been retrained on anonymized data, they fall to 16–26%. The same converted audio, three different privacy figures, and nothing changed but what the attacker knows. The organizers' own summary, published in 2022: “Anonymization is also achieved only partially and always at the cost of utility; no single system gives the best performance for all metrics and each system offers a different trade-off between privacy and utility, whether judged objectively or subjectively.”

Whichever of the four you are working in, the contract has to be specific enough that another team can use it to demonstrate error compounding across a cascade. It also has to say what inspectability is being bought with, because output quality is no longer what separates the two architectures.

Work through four steps. List the source attributes: content, speaker, language, style, timing, emotion, and background. Choose preservation priorities, resolving conflicts rather than promising all attributes — the Voice Conversion Challenge 2020 is the evidence that they cannot all be had at once. Create contrast cases that test negation, politeness, uncertainty, names, and culture-specific expressions. Then design human review, providing source audio, transcript, translation, and generated output together.

Write down what your preservation contract assumes. Write down one counterexample. A hedge delivered with assertive prosody will do, or an accent swap of the kind that moved perceived confidence by F(2, 471) = 68.48 in the Jiang and Pell panel. Then write down what evaluating semantics and interaction is supposed to trigger when it meets that counterexample.

The two questions from earlier become two families of measurement, and neither family carries a release on its own. The first asks whether the meaning arrived. The second asks whether it was allowed to arrive in that voice. Two more complete the portfolio: how errors compound across the stages of a cascade, and whether a user can tell which stage went wrong and repair it.

A community evaluation shows what those families look like once they have units. The IWSLT 2023 evaluation campaign covered 9 shared tasks and attracted 38 submissions from 31 teams: “The shared tasks address 9 scientific challenges in spoken language translation: simultaneous and offline translation, automatic subtitling and dubbing, speech-to-speech translation, multilingual, dialect and low-resource speech translation, and formality control.” Note that dubbing and speech-to-speech translation are scored as separate challenges, not as one. The simultaneous track ran under a single published latency budget: 2 seconds of Average Lagging for speech-to-text, 2.5 seconds of starting offset for speech-to-speech. The offline track drew 10 teams and 37 runs across cascade and direct systems.

That is the standard to copy: a number, a unit, a threshold agreed in advance. Run an output whose words are all correct and whose prosody has moved through the four families. It comes back as a pass on the first line and a failure further down. That combination is the thing the portfolio exists to catch, and no single number can express it.