A Benchmark for Bangla Task-Oriented Dialogue Understanding

AI summary

Overview

Research area: Natural Language Processing — multilingual and low-resource task-oriented dialogue understanding, with a focus on Bangla, English, and romanized Bangla (Banglish).

Technical level: Intermediate. Readers should be comfortable with transformer encoders (BERT-family models), intent classification, slot filling, and the idea of dialogue state tracking, but no specialized mathematics is required.

Scope: The paper introduces BanglaTOD, a parallel tri-lingual, multi-turn task-oriented dialogue benchmark for Bangladesh-specific service domains, and uses it to benchmark three pretrained encoders on joint intent classification and slot filling under context-aware and context-free conditions.

What This Paper Is About

Task-oriented dialogue systems must recognize what a user wants (intent), extract the details they provide (slots), and keep track of context across turns — but almost no such resource exists for Bangla, and none covers the mixed Bangla / English / Banglish communication common in Bangladesh. The authors build BanglaTOD, a manually constructed dataset of 209 multi-turn dialogues whose every turn is aligned across Bangla, English, and Banglish, with annotations for intents, token-level slots, state updates, and accumulated dialogue states. They then use it to measure how well mBERT, BERT-base-uncased, and BanglaBERT handle dialogue understanding across these three representations, and how much dialogue history matters.

Key Contributions

1. A turn-aligned multilingual dialogue resource. BanglaTOD contains 209 dialogues across nine Bangladeshi service domains and 64 intents, with parallel Bangla, English, and Banglish utterances annotated for intent, slots, and dialogue states, yielding 4,689 language-specific turn records.
2. Context-aware multilingual baselines. mBERT, BERT-base-uncased, and BanglaBERT are each fine-tuned on pooled data from all three representations and evaluated separately on Bangla, English, and Banglish for joint intent classification and slot filling.
3. Analysis of language and context effects. The paper reports cross-language performance gaps, an ablation showing the contribution of preceding dialogue turns, and a qualitative error analysis of recurring intent confusions and slot extraction failures.
4. A benchmark for future dialogue state tracking. Because each turn includes state update records and the accumulated dialogue state, the dataset is positioned to support downstream dialogue state tracking research, not just utterance-level NLU.

Main Findings

• mBERT is the strongest overall backbone. Overall domain accuracy 0.859 ±.007, intent micro-F1 0.601 ±.012, intent macro-F1 0.408 ±.015, intent weighted-F1 0.623 ±.012, slot entity-level micro-F1 0.330 ±.003, macro-F1 0.194 ±.003, weighted-F1 0.302 ±.006, and joint exact match 0.219 ±.010 (mean ± standard deviation over seeds 42, 43, 44).

• mBERT is relatively stable across languages. Its intent micro-F1 ranges from 0.573 (Banglish) to 0.625 (English), with Bangla at 0.605.

• BERT-base-uncased is much better on English and Banglish than on Bangla. Intent micro-F1 is 0.620 on English and 0.600 on Banglish but only 0.435 on Bangla; overall domain accuracy is 0.822 and overall joint exact match is 0.189 ±.001.

• BanglaBERT is strongest on Bangla but transfers poorly. Its highest intent micro-F1 is 0.659 on Bangla, dropping to 0.232 on English and 0.249 on Banglish; overall domain accuracy is 0.675 and overall joint exact match is 0.123 ±.007.

• Domain classification is far easier than intent classification. For mBERT, domain accuracy reaches 0.859 while intent accuracy is 0.601; the same gap appears across all backbones.

• Rare intents drag down macro-F1. For mBERT, intent macro-F1 is 0.408 versus weighted-F1 of 0.623, indicating difficulty with infrequent intent categories.

• Slot filling is the hardest task. Even the strongest model reaches only 0.330 entity-level F1. Errors frequently involve omitted slots and incomplete multi-word spans.

• Dialogue history matters substantially for intent. Removing preceding turns reduces intent accuracy by 19.0–25.7 percentage points: mBERT 0.605 → 0.356 (+0.249), BERT-base 0.578 → 0.321 (+0.257), BanglaBERT 0.385 → 0.195 (+0.190). Token-level slot macro-F1 drops much less: mBERT 0.250 → 0.215 (+0.035), BERT-base 0.216 → 0.179 (+0.037), BanglaBERT 0.128 → 0.106 (+0.021).

• Per-intent performance varies widely. At the seed-42 checkpoint, hotel_book_cancel scores 1.000 (mBERT), 1.000 (BERT), and 0.882 (BanglaBERT), while several government-service intents perform poorly: nid_card_service 0.250 / 0.154 / 0.000, police_services 0.000 / 0.154 / 0.267, govt_fin_assist 0.333 / 0.231 / 0.100, land_records 0.588 / 0.286 / 0.190, and tax_services 0.000 / 0.476 / 0.286.

• Specific intent confusions recur. mBERT predicts cash_out_information as transaction_support in three of four instances in each language representation, and BERT-base confuses tour_cancellation with modify_booking across all three representations.

• Short, structured slot values are recovered; broad and multi-word ones are not. For mBERT at seed 42, B-booking_id reaches 1.000 precision and 1.000 recall (support 9) and B-date_of_birth reaches 1.000/1.000 (support 3), while B-appl_status is 0.000/0.000 (support 3), B-query has 0.208 precision (support 15), and B-service has 0.276 precision (support 12). The continuation label I-requ_info reaches only 0.448 precision and 0.277 recall (support 47); BanglaBERT and BERT-base predict O for 47 and 30 of 65 requested_information tokens, respectively.

Methodology in Plain English

The authors first curated the data. Four annotators wrote the dialogues through pairwise role-playing (one playing the user, one the system) to imitate realistic service interactions, spanning 2 to 10 turns, with follow-up requests that refer back to information given earlier. Nine service domains were chosen for everyday relevance: Education (50 dialogues, 9 intents), Government Service (32, 19), Healthcare (25, 5), Mobile Financial Services (25, 5), E-commerce (25, 10), Restaurant (16, 5), Tourist (14, 5), Travel (12, 4), and Hotel (10, 2). All dialogues and annotations were manually reviewed. Every dialogue was then written out in Bangla, English, and Banglish with the turns aligned, and each turn annotated with intents, token-level slots (type, textual expression, canonical value, and token boundaries), turn-level state updates, and the accumulated dialogue state. The dataset ships as JSON and is publicly available on Hugging Face.

The 209 dialogues were split by dialogue into 137 training, 36 validation, and 36 test dialogues, with intent-based stratification and all three parallel representations of a dialogue kept in the same partition to prevent cross-language leakage.

For the experiments, three encoders — mBERT, BERT-base-uncased, and BanglaBERT — were each fine-tuned on pooled Bangla, English, and Banglish data and then evaluated separately per language. The joint model uses a two-layer intent classifier (768 → 256 → 64) and a linear BIO slot classifier, with domain predictions derived from the predicted intents. For each target utterance, the input concatenates up to three preceding turns from the same dialogue and language, separated by [SEP] tokens, with a maximum input length of 256 subtokens (no truncation was observed). Slot supervision is restricted to the first subtoken of each word in the target utterance, with context, special tokens, and continuation subtokens masked out of the slot loss.

Training used AdamW with a weighted joint loss of L_int + 1.5 L_slot, a learning rate of 1.5 × 10⁻⁵, layer-wise learning rate decay of 0.95, weight decay 0.01, a cosine scheduler with 10% warm-up, batch size 16, up to 100 epochs with patience 8, dropout 0.15, gradient clipping 1.0, and class-weighted cross-entropy with label smoothing 0.1 for the intent loss. Model selection used 0.5 · Acc_int + 0.5 · F1_slot (token, macro), and every configuration was run with seeds 42, 43, and 44, with results averaged across seeds.

Evaluation covers domain accuracy, intent accuracy (equivalent to micro-F1 for single-label classification), intent macro-F1 and weighted-F1, entity-level slot precision/recall/F1 based on exact span and type matches, and joint exact match, which requires both a correct intent and an exact match of the complete slot-label sequence. A separate ablation retrains each backbone without preceding turns to isolate the value of dialogue history.

Why This Matters

Impact on research. BanglaTOD fills a gap the authors identify explicitly: existing multilingual resources such as MultiATIS++, MTOP, and GlobalWoZ give limited coverage of Bangladesh-specific conversational settings, and prior Bangla intent classification and slot filling work relied on translated datasets that lack a realistic local ontology. Because every turn is aligned across three representations of the same conversational content, the benchmark lets researchers compare language and script effects while holding content constant — something translated corpora cannot do. The inclusion of state updates and accumulated dialogue states also opens a path to dialogue state tracking in a low-resource language.

Real-world applications:

• Mobile financial services, where the dataset's intents include cash_out_information and transaction_support — precisely the pair mBERT most often confuses.
• Government service chatbots handling NID card service, land records, tax services, police services, and government financial assistance, which the paper identifies as the most difficult intent inventory.
• Healthcare appointment and inquiry assistants, one of the nine domains covered.
• E-commerce, restaurant, hotel, travel, and tourist booking assistants, including cancellations and booking modifications.

Industry relevance. Organizations serving Bangladeshi users — banks, mobile financial service providers, telecoms, and contact-center platform vendors — routinely handle conversations that mix Bangla script, English, and Banglish. The paper's results indicate that no single off-the-shelf encoder handles all three representations well: mBERT is the most balanced, BERT-base-uncased is strong on English and Banglish but weak on Bangla (intent micro-F1 0.435), and BanglaBERT is strong on Bangla (0.659) but transfers poorly to English (0.232) and Banglish (0.249). The finding that removing dialogue history costs 19.0–25.7 percentage points of intent accuracy is a direct practical signal that production systems should retain multi-turn context rather than classify utterances in isolation.

Future Directions

• Hierarchical intent modelling, which the authors name as future work, to better handle the domain-intent structure where domain accuracy (0.859 for mBERT) far exceeds intent accuracy (0.601).
• Banglish normalization, to reduce the spelling and script variation the authors cite as an open problem and as a limitation of the current role-played data.
• End-to-end dialogue state tracking, leveraging the state update and accumulated state annotations that the baselines in this paper do not yet evaluate.
• Broader and more natural data. The limitations section notes uneven coverage across domains, intents, and slot types, and that role-played conversations may not capture the linguistic diversity, informal expressions, and spelling variation of real interactions — particularly naturally occurring code-switching, which the aligned parallel representations do not fully reflect.

Target Audience

Researchers and practitioners working on multilingual or low-resource natural language understanding, especially those building conversational systems for South Asian languages; NLP groups studying cross-lingual transfer and script/romanization effects; and developers at banks, mobile financial services, telecoms, and chatbot vendors who need an evaluation set for Bangla, English, and Banglish dialogue understanding. Graduate students looking for a well-scoped, publicly available benchmark with clear baselines and analyzed error patterns will also find it useful.

Authors’ abstract

Task-oriented dialogue understanding remains underexplored for Bangla, particularly in multilingual settings involving English and romanized Bangla (Banglish). We introduce BanglaTOD, a parallel tri-lingual benchmark comprising 209 multi-turn dialogues across nine Bangladesh-specific service domains and 64 intents. Each conversational turn is aligned across Bangla, English, and Banglish, yielding 4,689 language-specific turn records annotated with intents, token-level slots, state updates, and accumulated dialogue states. We establish context-aware baselines for joint intent classification and slot filling using mBERT, BERT-base-uncased, and BanglaBERT. Cross-language analysis reveals substantial performance differences across models and linguistic representations, while removing dialogue history reduces intent accuracy by 19.0--25.7 percentage points. BanglaTOD provides a foundation for multilingual dialogue understanding and future dialogue state tracking research in Bangladesh-specific conversational applications. The dataset is publicly available at \href{https://huggingface.co/datasets/afri7here/BanglaTOD}{Hugging Face}.