Corpus of Cross-lingual Dialogues with Minutes and Detection of Misunderstandings

AI summary

Overview

Research area: Natural language processing — specifically automatic speech recognition (ASR), simultaneous speech translation (SST), cross-lingual meeting summarization ("minuting"), and automatic detection of misunderstandings in dialogue.

Technical level: Intermediate. The paper is a corpus/resource paper with an accompanying LLM evaluation experiment; it is readable without deep technical background but uses standard speech-processing and MT terminology.

Scope in one sentence: The paper releases InCroMin, a 5-hour corpus of cross-lingual meetings mediated by an automatic simultaneous speech translation tool, together with written minutes and a first study on annotating and automatically detecting misunderstandings.

Authors and affiliation: Marko Čechovič, Natália Komorníková, Dominik Macháček, and Ondřej Bojar, Charles University, Faculty of Mathematics and Physics, Institute of Formal and Applied Linguistics (ÚFAL), Prague, Czechia. Posted as arXiv:2512.20204v1 [cs.CL] on 23 Dec 2025 under CC BY 4.0.

What This Paper Is About

Automatic speech translation could let people who share no common language hold real meetings, but evaluating such systems requires realistic data. Existing evaluation setups rely on simplified material, such as read speech instead of spontaneous speech, or isolated sentences retrieved from monologues, and therefore miss the spontaneity, interactivity, terminology, missing sentence segmentation, sentences completed by another speaker, and real recording conditions of actual cross-lingual dialogue.

The authors build and release InCroMin, a corpus of genuine meetings in which participants with no shared language were facilitated by an automatic simultaneous speech translation tool, and they add an initial study of when and why misunderstandings occur in such settings — including a test of whether a large language model can detect them automatically.

Key Contributions

1. The InCroMin corpus. Five hours of audio from 10 meetings with 24 participants speaking 12 languages, with ASR output and gold (corrected) transcripts, automatic and corrected English translations, and written English minutes for each meeting. A planned release of 10 further meetings with 20 participants using 4 more languages is announced.
2. An annotation protocol and dataset for misunderstandings. A scheme that identifies spans where misunderstanding occurred, records whether the speakers were aware of it, and classifies the cause as translation error, delay, technical problem, or genuine misunderstanding.
3. An evaluation of an LLM as an automatic misunderstanding detector. Gemini 1.5 Pro was run in a private Google Cloud environment over the English transcripts, and its output was validated by the human annotators, yielding 47% precision and 77% recall.
4. User feedback on the state of the art. Questionnaire responses from participants on usability, subjective slowdown, tiredness, and the specific failure modes of the translation tool.

Main Findings

• Corpus composition: The first release contains 10 meetings, 24 participants, 12 languages, 296 minutes of audio, and 26,854 tokens. One meeting has 5 participants, one has 3, and the rest have 2.
• Incomplete corrections at publication: 15 of 24 corrected transcripts (63%) and 13 English translations (54%) were ready; the rest are promised for future releases.
• Language imbalance: Czech accounts for 45% of the corpus (133 minutes of 296) because more Czech-speaking volunteers were available. Russian is the next largest at 34 minutes and 3,356 tokens; the smallest entries are Portuguese at 5 minutes and Spanish at 3 minutes.
• Causes of misunderstanding: Across the 14 annotated meetings, two annotators together marked 222 misunderstandings. Translation errors accounted for 36.5%, genuine misunderstandings 29.7%, system delays 14.4%, and technical errors 19.4%.
• Awareness: 57.1% of misunderstandings were acknowledged by the speakers, and 42.9% were not. The conclusion states that more than 40% of misunderstandings went unrecognized by the parties.
• Czech vs. non-Czech meetings: In the 9 meetings with Czech speakers there were 8% fewer translation-error misunderstandings, 8% fewer genuine misunderstandings, and 14% more technical-error misunderstandings than in the 5 meetings without Czech. The Czech meetings also showed 17% greater awareness of misunderstandings, which the authors hypothesize stems from shared background knowledge among a professor and his students.
• Inter-annotator agreement: Counting per meeting and per category, correlation was 0.94 for bad translation, 0.90 for unrecognized misunderstandings, 0.88 for misunderstandings found by Gemini, 0.83 for acknowledged misunderstandings, 0.80 for misunderstandings not found by Gemini, 0.72 for technical problems, 0.71 for Gemini true positives, 0.55 for genuine misunderstandings, and 0.49 for ASR delay. Agreement on Gemini false positives was −0.13, traced to 5 meetings where one annotator rejected 4 Gemini suggestions and the other rejected none.
• Gemini performance: The confusion matrix reports 122 items both Gemini and annotators considered true, 137 that Gemini flagged but annotators did not, and 36 annotator-identified misunderstandings Gemini missed. This yields an estimated precision of 47% and recall of 77%.
• Systematic limitation of LLM detection: Gemini typically detects one misunderstanding in a large text chunk where annotators find several, which is why the authors state their method cannot be used to find false positives, and they warn against generalizing from this small dataset.
• Usability: Across 45 questionnaire answers, 15.6% said the tool was usable already, 35.6% said it was usable but tedious, 44.4% said it would be usable if certain things were fixed, and 4.4% (2 respondents) said no tool could fix it and they would rather avoid multilingual meetings.
• Subjective slowdown and tiredness: Most participants reported a slowdown around a factor of 2, though many calls reached 70–80% of normal call "speed". Most reported no extra tiredness, with striking exceptions such as being 3 times as tired. Both measures were self-assessed only.
• Reported failure modes: Wrong target language of machine translation even when set correctly, errors from audio segmentation and voice activity detection, mistranslation of dialogue-management or meta-language (such as who speaks next), incorrect handling of questions and affirmative sentences, spurious short affirmative sentences triggered by any sound, and unclear turn-taking or speaker separation in the output.
• Tooling setup: Meetings ran on the Fairmeeting videoconferencing platform with the Minuteman tool, which uses Whisper Large-v3 for simultaneous translation into English and was extended with the NLLB model (facebook/NLLB-200-distilled-600M) for multi-target translation; NLLB supports 200 languages.

Methodology in Plain English

The authors recruited volunteers from their own personal and professional networks and grouped them so that at least two groups at each meeting used languages the other group did not know, with a shared topic to discuss, such as a student project consultation or a casual chat. They asked participants to avoid any common language even when they shared one, meaning the cross-linguality was arranged rather than a genuine need. Participants were asked to hold 30-minute meetings and then fill in a feedback questionnaire taking about 15 minutes. Two meetings were simulated interviews between a refugee and an integration center officer, run by two actual integration center officers, because obtaining consent from authentic refugees was not feasible.

All meetings were held online. Because Minuteman records a separate audio track for each participant, speaker identity came for free and no automatic diarization was needed. The raw data went through retrieval, synchronization and trimming to the meeting portion, automatic transcription and translation, voluntary or paid correction by a fluent annotator, deidentification (names, locations and organizations replaced with placeholders, and audio segments overwritten with silence), creation of English minutes, and misunderstanding annotation.

For language statistics, the team applied Silero Voice Activity Detection to mark voiced and silent segments, splitting silent intervals in half and assigning each half to the adjacent language, and used the SacreMoses tokenizer — corrected transcripts where available, ASR otherwise — to count tokens.

For the misunderstanding study, annotators worked on the revised English translations because they were not proficient in all 12 languages. They identified markables, delimited spans, labelled each as acknowledged or unrecognized, and assigned one of four reasons: translation error, delay, technical problem, or genuine misunderstanding. Gemini 1.5 Pro was then given a simple prompt and the timestamped conversation, one meeting at a time, and the annotators validated its output after completing their own annotation. Gemini was chosen over GPT-3.5 turbo and Llama 3 because running those in a privacy-preserving manner proved difficult and Gemini 1.5 Pro had the longest context.

Why This Matters

Impact on research. The paper supplies evaluation data for ASR, simultaneous speech translation, quality estimation, and automatic minuting in a setting that existing corpora do not cover: spontaneous, interactive, multi-party dialogue with no shared language and no sentence segmentation. It also opens misunderstanding detection as a measurable task, with a baseline and an annotation protocol others can reuse or contest.

Real-world applications:

• International business meetings where participants speak different languages and need an automatic interpreter.
• Public-service and integration-center interviews, such as the refugee-support scenario the authors explicitly simulate.
• Academic collaboration, student project consultations, and supervision across language borders.
• Multilingual conferences, remote teamwork, and any customer-facing conversation where a live translation layer mediates.

Industry relevance. The findings point to concrete product problems: incorrect target-language selection, unclear turn-taking, absent speaker separation, and confusing presentation of whether speech has been processed. The paper also suggests a feature direction — a monitor that alerts participants when a misunderstanding is likely and proposes a clarification question — which would require detector precision well above the 47% reported here before it could be shipped without generating false alarms.

Future Directions

• Complete the planned release. The second batch of 10 meetings, 20 participants, and 4 additional languages (300 minutes) is pending deidentification, along with the outstanding transcript and translation corrections.
• Improve LLM misunderstanding detection. The central open problem is granularity: Gemini finds one misunderstanding per text chunk where humans find several, which makes false-positive detection infeasible. Better prompting, smaller chunks, or different models are natural follow-ups.
• Target the unrecognized cases. Since more than 40% of misunderstandings went unnoticed by the participants themselves, the authors identify this as a priority — an automatic alerting system would matter most precisely where humans fail to notice.
• Address dialogue-management and interface failures. Users reported mistranslated meta-language (who speaks next), unstable turn-taking, unsynchronized audio and translation, and clumsy interfaces, pointing to work on dialogue management, output presentation, and translation quality.

Target Audience

Researchers and engineers working on speech translation, simultaneous interpreting systems, meeting summarization, and multilingual dialogue evaluation will get the most from this paper, since InCroMin and the misunderstanding annotations are directly usable as benchmarks. It is also relevant to practitioners building cross-lingual meeting tools who want a documented list of real user-facing failure modes, and to annotation and corpus-construction researchers interested in protocols for subjective phenomena such as misunderstandings.

Authors’ abstract

Speech processing and translation technology have the potential to facilitate meetings of individuals who do not share any common language. To evaluate automatic systems for such a task, a versatile and realistic evaluation corpus is needed. Therefore, we create and present a corpus of cross-lingual dialogues between individuals without a common language who were facilitated by automatic simultaneous speech translation. The corpus consists of 5 hours of speech recordings with ASR and gold transcripts in 12 original languages and automatic and corrected translations into English. For the purposes of research into cross-lingual summarization, our corpus also includes written summaries (minutes) of the meetings. Moreover, we propose automatic detection of misunderstandings. For an overview of this task and its complexity, we attempt to quantify misunderstandings in cross-lingual meetings. We annotate misunderstandings manually and also test the ability of current large language models to detect them automatically. The results show that the Gemini model is able to identify text spans with misunderstandings with recall of 77% and precision of 47%.