Lipi-Ghor: A Large-Scale, Diarized, Multi-Domain Bengali Speech Corpus and a Scalable Pipeline for Building It

AI summary

Overview

Research area: Speech processing and natural language processing for low-resource languages (Bengali ASR, speech corpus construction, speaker diarization). Technical level: Intermediate. Scope: The paper introduces Lipi-Ghor, an 885-hour speaker-diarized, multi-domain Bengali speech corpus built from YouTube, together with a reusable four-stage construction pipeline and ASR benchmarks on a human-verified gold split.

What This Paper Is About

Bengali is spoken by over 230 million people, but open Bengali speech datasets are small, restricted to short read or crowd-sourced utterances, and lack speaker labels and long-form audio. The authors build a much larger corpus by harvesting Bengali YouTube videos that already carry Bengali caption tracks, automatically labeling who is speaking when, and packaging everything into a speaker/transcript/timestamp format. They then measure how well existing speech recognition systems handle this more realistic, messy Bengali speech, and whether the corpus is good enough to train better ones.

Key Contributions

1. A replicable construction pipeline. Four stages — domain-aware channel curation, Bengali caption-track filtering, audio and caption extraction, and SSTT annotation — described as language-agnostic except for the manually curated channel list.
2. The Lipi-Ghor corpus. 885 hours of speech from 998 unique videos across 596 unique channels and 17 domains, annotated in SSTT (Speaker, Speech, Transcription, Timestamp) format by aligning automatic diarization with Bengali YouTube captions. The authors state it is the first Bengali corpus to combine large-scale coverage with speaker-diarized, long-form audio.
3. A human-annotated gold evaluation split. 500 clips drawn by stratified sampling across all 17 domains, corrected by 17 volunteer native Bengali-speaking annotators over approximately 30 hours of manual work, giving human-verified transcript references instead of noisy captions.
4. Benchmarks and analysis. Four off-the-shelf ASR systems evaluated zero-shot and two models fine-tuned on the corpus, with per-domain and per-duration breakdowns of where recognition degrades.

Main Findings

• The corpus is large but heavily skewed. Audiobooks (254.37 hours), talkshows (142.40 hours) and movies (122.18 hours) dominate; three domains account for 58.6% of audio and seven of seventeen reach 83.0%. The skew reflects where Bengali caption coverage is densest on YouTube, not an ideal training balance.
• Speech is mostly auto-captioned. Of the 998 videos, 896 (89.78%) use auto-generated captions and only 102 (10.22%) use manually created captions, which is why the authors call the corpus weakly supervised and position it as pre-training data rather than a benchmark.
• Diarization reveals unusual conversational structure. The corpus contains 984,647 speaker-attributed segments and 9,056 speaker labels. The median recording has 4 speakers (IQR 1–12), 25.9% are single-speaker narration, and 31.0% have ten or more speakers. Median segment duration is 1.54 seconds and 62.6% of segments fall under two seconds; mean speech ratio is 0.76, with 639 hours covered by at least one speaker segment (the text reports the remainder as 246.34 hours, the summary table as 243 h).
• Zero-shot performance spans a wide range (30.51% to 84.53% WER). Hishab TITU Conformer-L reaches 30.51% WER / 18.23% CER, Meta MMS 1B 43.06% / 21.16%, Seamless M4T-v2 66.71% / 45.54%, and Whisper large-v3 84.53% / 72.92%.
• Fine-tuning on the corpus beats every off-the-shelf system. Faster Whisper Medium fine-tuned on roughly 58 hours of Lipi-Ghor data reaches 20.70% WER / 11.90% CER, and Fast Conformer reaches 24.67% / 15.56% — both below the best zero-shot result of 30.51%.
• External benchmark results are competitive. On FLEURS Bengali, Faster Whisper Medium scores 21.7% and Fast Conformer 20.03%; a Moonshine-Base model with a transplanted BanglaBERT WordPiece vocabulary reports 21.54% WER / 10.79% CER.
• Errors concentrate in emotive and dialectal content. Across models, WER is lowest on audiobooks and news and highest on drama, comedy and regional dialects. Hishab TITU rises from 21.02% on audiobooks to 57.01% on Regional/Dialect, where Seamless M4T-v2 exceeds 100% WER (113.70%). Whisper large-v3 stays near 85% almost everywhere, failing largely independently of domain.
• Shorter segments correlate with higher error, but only suggestively. Across seven domains, median segment duration and WER give a Spearman correlation of −0.75; extending turn rate to all seventeen domains weakens to +0.42 with p = 0.09. Cartoons are a counterexample: 28.5 turns per minute yet still easy to recognize.
• Utterance length has only a modest effect. Because diarization already chops long recordings into short turns, only the very-long quartile shows a consistent uptick (Hishab TITU goes from 28.46% in the long quartile to 32.60% in the very long quartile).
• Caption references carry real noise. Disagreement between YouTube captions and human-corrected gold references ranges from 0.00% WER/CER for Comedy/Satire to 28.57% WER / 25.85% CER for Cartoon, with audiobooks at 2.57% / 1.12%.
• The diarizer has systematic artifacts. No recording is annotated with more than two simultaneous speakers (682 recordings peak at two, 337 at one), and detected overlap falls as speaker count rises — a sign of over-segmentation. 21.8% of the 9,056 speaker labels cover under ten seconds of speech, and the mean of 8.9 speakers per recording is inflated relative to the median of 4.

Methodology in Plain English

Instead of crawling YouTube blindly, the authors hand-picked Bengali channels per domain, prioritizing high-volume "anchor" channels and then adding smaller ones for diversity. They kept only videos that already had a Bengali caption track, checking availability without downloading audio through their own toolkit and splitting video IDs into batches run across Kaggle, Google Colab and GitHub Codespaces to work around rate limits. Audio and Bengali subtitles were then pulled with yt-dlp, auto-translated caption tracks were discarded, and cross-domain duplicates removed — 1,019 extracted recordings became 998 unique videos. Each file was run through a commercial diarization model (Precision-2 from pyannoteAI, built on pyannote.audio) to mark who spoke when, and that output was aligned with captions to produce segment-level SSTT annotations. Everything was packaged into three splits: raw diarization output, diarization aligned with transcripts (the 885-hour corpus), and 364 videos pre-chunked into ASR-ready segments. For evaluation, annotators used a custom web tool that shows the video alongside a waveform and a colored speaker ribbon, letting them correct transcripts, split or merge segments, and reassign speakers.

Why This Matters

Impact on research. Bengali has over 230 million speakers yet its open speech resources are small, short-form and undiarized — Common Voice Bengali at roughly 40 hours, OOD-Speech at 23 hours of test data, and existing multilingual efforts like YODAS contributing only 27 hours for Bangla. Lipi-Ghor offers training-scale, long-form, speaker-attributed audio, and shows that targeted channel curation, not exhaustive crawling, is the real lever for low-resource corpus building. The per-domain and per-duration analyses also give the field a map of where Bengali ASR actually breaks.

Real-world applications.

• Training and adapting Bengali ASR for call centers, broadcast monitoring, and public-service transcription.
• Searching and indexing long-form Bengali media — drama, sermons, talkshows, news archives — with speaker-attributed transcripts.
• Meeting and interview transcription that needs to distinguish speakers, since 31.0% of recordings carry ten or more speakers.
• Speech analytics for regional and dialectal content, where current systems fail most sharply (Seamless M4T-v2 at 113.70% WER on Regional/Dialect).

Industry relevance. The results show a mid-sized open model fine-tuned on 58 hours of this corpus (20.70% WER) outperforming a Bengali-specific commercial-style system used zero-shot (30.51%), which is a strong argument that domain-matched, weakly supervised data can substitute for expensive licensed corpora. The three-split packaging and shared SSTT schema also mean teams can swap gold clips for noisy ones without changing their training pipeline.

Future Directions

• Improve dialect coverage. Regional/Dialect is represented by only 4 videos and 1.29 hours, which the authors explicitly call insufficient for dialect-robust training, yet it is where all models fail worst.
• Move beyond a two-speaker overlap ceiling. The diarization output never contains more than two concurrent speakers, and over-segmentation inflates high speaker counts; better diarization, or human-verified speaker labels at scale, would fix a known weak point.
• Reduce reliance on a closed model. Precision-2 is commercial and API-gated, so the diarization stage is not fully open; an open replacement would make the pipeline end-to-end reproducible.
• Validate transfer to other languages. The authors present cross-lingual applicability as a hypothesis only, noting that 3–5 rich channels per domain yielded 20–50 hours and that the real bottleneck is domain-aware channel discovery.

Target Audience

Speech and NLP researchers working on low-resource and Indic languages; engineers building ASR training pipelines from web data; dataset curators interested in weakly supervised, diarized corpus construction; and practitioners who need Bengali speech tooling and want a realistic picture of current model performance before committing to a system.

Authors’ abstract

Bengali, spoken by over 230 million people, remains severely under-resourced in spoken language research. Existing open Bengali ASR corpora are confined to short, read or crowd-sourced speech and lack speaker diarization and long-form audio. We present Lipi-Ghor, a large-scale weakly-supervised Bengali speech corpus comprising 885 hours of audio sourced from 998 YouTube videos across 596 unique channels, spanning 17 diverse domains including audiobooks, drama, religious sermons, news, and regional dialects. Each recording is annotated in SSTT format (Speaker, Speech, Transcription, Timestamp), which combines automated diarization with Bengali YouTube caption tracks. To our knowledge, Lipi-Ghor is the first Bengali speech corpus to pair large-scale coverage with speaker-diarized, long-form audio annotations, both of which are absent from existing open Bengali resources. We also present the construction methodology as a replicable pipeline for any language with a sufficient captioned YouTube presence. Targeted channel curation across a handful of rich domains yields corpus scales that were previously out of reach for low-resource languages. We benchmark four widely-used ASR models under zero-shot and fine-tuned conditions. All four degrade consistently on emotive, dialectal, and long-form content. Because caption targets are noisy, we also release a human-annotated gold evaluation split, verified by 17 volunteer annotators over 30 hours of human-in-the-loop correction and stratified across all 17 domains. We release our original contributions (video-ID lists, diarization output, pipeline code, and the gold annotations) under CC-BY-4.0, and distribute caption-derived transcripts for research use only; no raw audio is redistributed.