Investigating the Role of Reasoning-Language Alignment in Monolingual Retrieval-Augmented Generation

AI summary

Overview

Research area: Natural Language Processing; specifically multilingual retrieval-augmented generation (RAG), reasoning-trace language control, and benchmark construction.

Technical level: Intermediate. Readers should be comfortable with RAG pipelines, agentic tool-calling loops, chunking strategies, and basic paired significance testing, but the paper's central question is stated plainly and the experimental design is easy to follow.

Scope: The paper builds a fully monolingual German RAG testbed over the tabletop role-playing game The Dark Eye and uses it to test whether forcing an LLM to reason in the language of the query and retrieved evidence helps or hurts, compared to forced English, forced French, unconstrained reasoning, and no reasoning at all.

What This Paper Is About

Current reasoning models are trained to reason mostly in English, and prior work (Qi et al., 2025) showed that forcing a model to reason in a different language degrades accuracy even when that language matches the prompt — but that study used only short prompts. This paper asks whether the same holds for RAG, where the model reasons over a large amount of retrieved evidence written in the target language, and whether aligning the reasoning language with that evidence helps. Since no suitable testbed existed, the authors build one: a fully monolingual German corpus and QA benchmark over a fictional world the model provably does not know, so that retrieval is genuinely required.

Key Contributions

1. A structure-preserving German knowledge base pipeline. The authors release a semi-automatic tool that converts the densely formatted, multi-column PDFs of The Dark Eye "green series" region sourcebooks (G01, G05, G08) into a hierarchical German knowledge base with section-level chunks annotated with parent/child links. Because the books are copyrighted, they release the pipeline rather than the knowledge base itself.

2. A curated German QA benchmark. 585 human-checked single-passage questions (LLM-generated with a generator/critic loop and then human-reviewed) plus 30 hand-written multi-hop questions that require synthesis across sections and draw on all three books.

3. A controlled study of reasoning-language alignment in monolingual RAG. The authors vary only the reasoning language (forced German, forced English, forced French, unconstrained, no reasoning) while holding the query, corpus, and answer language fixed as German, and cross this with two chunking strategies (SimpleChunks vs. SectionChunks).

4. An examination of when reasoning helps. By comparing one-hop and multi-hop questions, the paper identifies a task-complexity threshold below which reasoning traces hurt rather than help.

Main Findings

• Retrieval is necessary for this testbed. Without retrieval, the small model scores 1.335 and the large model 1.323 on the 585-question single-passage set, and 1.189 on the multi-hop set — near the minimum of the 1–5 scale. Every retrieval configuration beats no retrieval with p < 10⁻⁷⁰, and the two no-retrieval models are statistically tied (p = .83). The larger model hallucinates too, though with entities that exist elsewhere in Aventuria rather than invented ones.

• Aligning the reasoning language with the data helps, independent of proficiency. On SectionChunks, forced German scores 3.701 versus 3.491 for forced French, a gap of 0.21 that is statistically significant (Holm-corrected p = .019), with German winning on 182 questions against 138 for French. According to the Qwen3 report, the model is more proficient in French (average benchmark 71.8) than German (68.2), so the gain comes from alignment with the data, not language ability. The same ordering holds on the multi-hop set (German 2.889 vs. French 2.778).

• Richer, structure-aware context reduces the cost of forced reasoning. Moving from SimpleChunks to SectionChunks, unconstrained reasoning improves the least (3.636 → 3.718, Δ = 0.08, not significant at p = .26), forced English improves more (3.467 → 3.670, Δ = 0.20), and forced German improves the most (3.409 → 3.701, Δ = 0.29, p < 10⁻⁴). With plain fixed-size chunks, forced German was the worst of its group; with section-based chunks it rises above forced English.

• English bias is narrowed but not overturned. On the single-passage set, unconstrained English (3.718), forced German (3.701), and forced English (3.670) are statistically tied (all pairwise p > .5). On the multi-hop set forced English (3.000) sits slightly above forced German (2.889). Forced German matches the model's native English reasoning but never surpasses it, echoing Qi et al. (2025) that prompt hacking gives only limited control.

• Reasoning helps only above a complexity threshold. Disabling reasoning is best on the single-passage set (3.903), significantly above every reasoning setting on that set (Holm-corrected p ≤ .04), but it is the worst retrieval configuration on the multi-hop set (2.778), tied with French. The authors suggest a system could estimate how many retrieval hops a question needs and enable reasoning only above that threshold.

• The judge metric is usable for ranking but not for near-ties. A manual re-rating of 50 answers (five per configuration) showed exact agreement with the LLM judge in 66% of cases, agreement within one point in 90%, and disagreements of two or more points in 10%. Four of the five large gaps were cases where the judge gave the minimum score of 1 to an answer a human rated 3 or 4. Appendix C documents one such case: the system answers Westwinddrache and quotes the source sentence, yet receives a judge score of 1 and a human score of 4.

• The judge shows a length bias, but it works against the headline result. Pooled over the eight retrieval configurations (4,680 graded answers), mean answer length falls as the rating rises, from up to 113 words at the low ratings to 63 at rating 5 (Spearman ρ = −0.39, p < 0.001). "No reasoning" produces the shortest answers (64 words on average), so part of its single-passage advantage may be the judge rewarding brevity. However, forced German answers are slightly longer than forced French ones (85 vs. 82 words), so the bias runs against the German advantage — and the German-over-French ordering also holds on the multi-hop set, which is rated by humans only.

• Agentic RAG introduces a language-leakage failure mode. Models forced to reason in French sometimes issue their retrieval tool calls in French against a German knowledge base. Because the reasoning language can leak into tool calls and hurt retrieval itself, the choice of reasoning language matters more here than in prompt-only reasoning, where the model's language cannot affect the retrieval step.

Methodology in Plain English

The authors needed a setting that was fully monolingual in a non-English language and genuinely dependent on retrieval. They chose The Dark Eye (German: Das Schwarze Auge), a German tabletop role-playing game first published in 1984 whose fictional world Aventuria is documented across decades of German sourcebooks but is too niche for LLMs to know from pretraining. They used three region-description books (G01, G05, G08) totaling about 1.13M tokens — far beyond the model's context window, so retrieval is unavoidable.

Because off-the-shelf PDF converters (Docling, Marker, pdfalto) all failed to recover the reading order and section boundaries from the dense two- and three-column layouts, they built a semi-automatic pipeline: convert to ALTO XML with pdfalto, split into YAML by page and block, turn heading-font blocks into Markdown headings, repair missing words by matching against pdfalto's token stream using Levenshtein distance (asking for confirmation when more than 20% of a block changes), have a human resolve heading levels where the automatic and Marker outputs disagree, manually fix reading order, and finally load each heading-plus-text into a chunk linked to its parent and child sections.

Two chunking schemes are compared. SectionChunks follows the book's own hierarchy (738, 665, and 843 chunks for G01, G05, G08; mean 501 tokens; 2,246 total). SimpleChunks uses a uniform fixed size set to the mean SectionChunks size (736, 730, 784 chunks; 501 tokens each; 2,250 total), so both hold the same 1.13M tokens at comparable granularity.

The RAG system is a standard agentic loop. Given a German query, the model either calls a tool or answers. Two tools read the knowledge base: a semantic search that embeds the query, retrieves the top 30 chunks, reranks them with Qwen3-Reranker-0.6B, and returns the top 3; and a direct chunk lookup by ID for walking to parent or child sections. A blacklist prevents repeating chunks. Embeddings come from Multilingual-E5-large. The loop runs on Qwen3-30B-A3B on a single 24 GB GPU, with a larger Qwen3-235B-A22B-Thinking model used only as a no-retrieval baseline and as the judge (running it in the full loop was too expensive). Decoding uses the recommended settings: temperature = 0.6, top_p = 0.95, top_k = 20, min_p = 0, with a fixed seed.

To control the reasoning language, the authors follow the prompt-hacking method of Qi et al. (2025): the system prompt instructs reasoning in the target language and the trace is prefixed with a fixed sentence in that language (e.g., German "Ich muss zunächst durchgehen, was ich über die Anfrage des Nutzers bereits weiß."). The final answer is always required to be in German. French serves as a control — a high-resource language the model is not trained to reason in, and one that does not match the German data, which separates alignment from raw ability.

Single-passage answers are rated 1–5 by the large Qwen3 model given the question, gold answer, and source section. Multi-hop answers are rated 1–5 blindly by three human experts with deep Aventuria knowledge. Configurations are compared with the Wilcoxon signed-rank test on paired per-question ratings, corrected with the Holm method across the ten pairwise comparisons.

Why This Matters

Impact on research. The paper isolates an axis that prior multilingual RAG work does not: earlier studies vary the language of the query or the documents, whereas this one holds query, corpus, and answer all German and varies only the language the model reasons in. It also extends the prompt-only finding of Qi et al. (2025) into the RAG setting, showing that the failure mode there is not just degraded answers but potentially degraded retrieval, since the reasoning language can leak into tool calls. The released testbed and benchmark give the community a controlled, fully monolingual setting in which reasoning over target-language evidence can be studied.

Real-world applications:

• Enterprise and public-sector RAG over non-English corpora. Organizations deploying assistants over German, French, or other non-English document collections need to know whether forcing on-language reasoning helps or hurts — this paper suggests it helps, but that on-language reasoning alone still will not beat a model's native English reasoning.
• Retrieval infrastructure for niche or proprietary domains. The structure-preserving PDF pipeline (hierarchical chunking plus parent/child links for citation down to a specific section and page) is directly reusable for any corpus where headings carry meaning the surrounding text does not.
• Adaptive reasoning budgets. The finding that reasoning helps only above a complexity threshold suggests a practical design: estimate how many retrieval hops a query needs and enable reasoning only past that point.
• Evaluation of LLM-as-a-Judge pipelines. The documented length bias and the asymmetric over-penalization of answers the judge deems hallucinated are directly actionable for anyone using an LLM judge to rank systems.

Industry relevance. Companies serving non-English users with RAG assistants face a concrete architectural decision — which language the model reasons in — that this paper shows is not neutral. The paper's conclusion points to fine-tuning models for native multilingual reasoning rather than relying on inference-time prompting, which is a product-level as well as a research-level implication.

Future Directions

• Test other models and language pairs. All system runs use one model family (Qwen3) and one language pair (German vs. English, with French as a control). Whether the effects transfer to other models and to truly low-resource languages is an open question.

• Move beyond prompt-based control. The authors use prompt hacking only and do not fine-tune the model to reason in the target language, which prior work (Qi et al., 2025) suggests would help more. Whether fine-tuning closes the gap to native English reasoning is unresolved.

• Find the complexity threshold empirically. Reasoning helps on multi-hop questions and hurts on one-hop questions, but the authors state that pinning down the exact threshold needs a benchmark graded by hop count, which they do not have.

• Quantify the language-leakage failure mode. The observation that French reasoning sometimes produces French retrieval queries against a German knowledge base is reported only qualitatively; measuring how often this happens and how much it costs retrieval is left to future work.

• Fully re-rate the single-passage set with humans. The judge validation covers 50 sampled answers, and the authors note a possible bias in which the English-reasoning judge misses errors in non-English reasoning. A complete human re-rating is left open.

Target Audience

Researchers and practitioners working on multilingual NLP, retrieval-augmented generation, and reasoning models — particularly those deploying RAG systems over non-English corpora. It is also useful for teams building domain-specific knowledge bases from structured documents, and for anyone who relies on LLM-as-a-Judge evaluation and wants a concrete demonstration of where that metric misleads. Readers interested in tabletop role-playing game tooling may also find the Aventuria knowledge base relevant, though the authors are explicit that the game world is a means of constructing a knowledge source the model does not know, not the application being built.

Authors’ abstract

Reasoning traces improve large language models (LLMs), but current models are trained to reason mostly in English. It has been shown that forcing a model to reason in another language degrades accuracy, even when the reasoning language matches the language of the prompt -- but only for a setting where the model reasons over a short prompt. Here, we ask whether the same holds for retrieval-augmented generation (RAG), where the model must read and integrate a large amount of retrieved evidence in the target language. To study this, we build a fully monolingual German RAG question-answering testbed over the fictional world of the tabletop role-playing game The Dark Eye, a domain that is richly documented in German but too niche for the model to answer from memory, so that it has to rely on retrieval. Varying the forced reasoning language of an agentic RAG system on this testbed, we find that aligning the reasoning language with the language of the query and the retrieved documents helps. Forced German reasoning outperforms forced French, although the model benchmarks higher in French, so the benefit comes from alignment and not from language proficiency. The advantage grows when the retrieved context is richer and structure-aware. However, forced German only reaches the level of the model's native, unconstrained English reasoning without surpassing it, showing that native multilingual reasoning is needed. We publicly release the testbed and QA benchmark.