Mostly Reading: What Open-Book Scores Measure in Document-Level LoRA Adapters

AI summary

Overview

Research area: Natural language processing — parameter-efficient fine-tuning (LoRA), adapter composition and retrieval, and evaluation methodology for document-level knowledge in language models.

Technical level: Intermediate. The paper assumes familiarity with LoRA fine-tuning, adapter merging, and standard QA metrics (token-level F1), but its central argument is conceptual rather than mathematical.

Scope: A 150-document study on Llama-3.1-8B-Instruct showing that open-book evaluation of per-document LoRA adapters mostly measures a shared reading behaviour rather than stored document content, and that a cloze probe which withholds the document reveals a very different picture.

What This Paper Is About

Researchers have proposed storing one LoRA adapter per document so that adapters can be merged or retrieved as a knowledge bank. The usual way to score such an adapter is open-book: put the document in the prompt and measure answer quality. The problem is that this score cannot distinguish an adapter that holds the document's content from one that has merely learned to extract answers from whatever text it is shown. The paper's goal is to separate these two abilities by swapping which adapter answers a fixed document, and by adding a cloze probe that asks for the same facts with the document withheld.

Key Contributions

1. A training-text ablation that isolates reading from holding. Holding each document's question-answering pairs fixed while varying the text trained beside them, across five conditions (natural article, style-normalized rewrite, synthetic fill-in-the-blank, no context, swapped context).
2. A cloze probe as the content-sensitive measure. Each held-out fact is rewritten as a declarative sentence with one blank, presented without the document and in a surface form no adapter was trained on.
3. A substitution and decomposition analysis. Measuring how much of a specialist's advantage a random adapter, a same-domain adapter, or a retrieval-selected adapter recovers, on both probes.
4. A composition study across 56 collections comparing √lerp, update merging, TIES, joint training, and random adapters at collection sizes N ∈ {2, 4, 8, 16, 25, 32}, plus a catalogue of four silent evaluation and training errors that each produced a clean but wrong result.

Main Findings

• Open-book scores are mostly a shared reading skill. Pooled over the 26 collections of 16 documents or more (595 document rows), a random adapter trained on another document recovers 82% (CI 77–87) of the specialist's open-book advantage over the base model, but only 2% (CI −19 to 17) of its cloze advantage — a share not distinguishable from zero.
• Content does reach the weights. Specialists beat the base model in cloze recall by 0.094. A cloze gain clear of zero appears exactly where an adapter read the scored document's full text (natural, style-normalized, and swapped context scored on the document it read).
• Closed-book and cloze rank the training conditions in nearly opposite orders. The conditions that lead closed-book are those that read least of the scored document: swapped-on-source 0.456, synthetic 0.443, no context 0.426, against 0.399 for natural and 0.320 for the base model. On cloze, the no-context adapters fall below the base model by 0.132 (CI 0.087–0.176).
• An adapter can read nothing and still score near a specialist open-book. The no-context adapter reaches 0.773 open-book F1, only 0.049 (CI 0.025–0.074) behind the natural adapter at 0.822 and ahead of both rewrites.
• A foreign adapter recovers most of the open-book advantage. Over 120 source–target pairs, a same-domain adapter recovers 88% (CI 77–101) of the specialist's advantage; the interval reaches 100%, so the non-transferring part cannot be distinguished from zero. No-context adapters recover 78% (CI 68–90).
• Weight geometry tracks what was read, not what was asked. Pairwise Frobenius distances between adapter updates correlate with sentence-embedding distances between documents, but the swapped-context condition shows the geometry follows the donor text (ρ = +0.217) rather than the source of the questions (ρ = +0.033), even though swapped adapters' updates travel as far as natural ones (21.09 vs 21.10). Over all layers, 0.708 of an adapter's six nearest neighbours share its domain against a chance rate of 0.161.
• Weight-space retrieval does not find content-bearing adapters. In a bank of 120 specialists with 30 query documents (five per domain) whose own adapters the bank never holds, weight-space selection is not distinguishable from random on either probe: the pre-registered cloze contrast is +0.020 (CI −0.038 to +0.075), text embedding gives −0.001 (CI −0.060 to +0.061), and open-book the weight-space selection gives −0.024 (CI −0.061 to +0.013). Weight space places its choice in the query's domain for 26/30 queries versus 5/30 for random. The 30-query test cannot, however, exclude an advantage of about four fifths of what the query's own adapter gains over a random one (0.507 vs 0.415 cloze recall).
• Merging preserves the reading skill and loses the content. At 16 documents or more, correctly scaled merges recover 87% to 89% of the specialists' open-book advantage but only 12% to 16% of their cloze advantage, with intervals including zero — against 94% (CI 70–139) for joint training.
• Open-book retention of merges is high and close to random. At rank 8 over the nine cells with N ≥ 4, √lerp preserves 0.95–1.01, update merging 0.93–0.99, and TIES 0.94–1.00 of the specialists' open-book score, against 0.88–0.97 for random. The paper declines to test merges against parity with specialists because a retention near one says little when the random baseline already recovers about four fifths of the advantage.
• Scaling dominates the merging operator. Averaging each factor at 1/N instead of N^(−1/2) lowers open-book retention by 0.176 to 0.348, every interval excluding zero; at matched own-term weight, the cross terms cost at most 0.035 of open-book retention.
• Joint training keeps what merges lose. Pooled over collections of 16 documents or more, joint training exceeds the three merges by 0.038 to 0.045 of the specialists' open-book score and by 0.172 to 0.183 of their cloze recall; it exceeds random on cloze retention by 0.150 (CI 0.087–0.212) against 0.043 to 0.052 for the merges.
• Four silent errors produced clean, wrong results. (a) A TRL max_length default of 1024 tokens left 44% of documents with no answer token in any training example, reversing the comparison between training conditions (no-context scored 0.168 above natural under the default, 0.049 below it at 2048 tokens; ρ = −0.943, p = 0.017). (b) A leftover repetition penalty of 1.1 that had placed update merging's best cell at 1.006 instead of 0.994. (c) A 1/N² own-term scaling error that looked like a cost of merging, with retention falling from 0.82 at four documents to 0.62 at 32. (d) A precision mismatch from merging adapters into a bf16 base, which lowers the specialists' cloze gain by 0.045 (CI 0.011–0.080).

Methodology in Plain English

The team took 150 English Wikipedia articles, 25 each from biology, computer science, economics, history, law, and medicine, and had Claude Sonnet 4.6 generate 12 question-answering pairs per article, validated by string match against the source. Eight pairs per article were used for training and four held out (three factual and one true/false).

They trained one LoRA adapter per article on Llama-3.1-8B-Instruct, loaded in 4-bit NF4 with bf16 computation as in QLoRA, at rank 8 on all seven projection matrices, for 500 steps, capped at 2048 tokens.

The central trick is to fix the scored document and change which adapter answers it. If a specialist's open-book advantage came from holding its document, a foreign adapter should not reproduce it; if it came from a general disposition to extract answers from supplied text, it should. They quantify each result as a share of the specialists' advantage over the base model, and as a retention (a ratio of scores rather than advantages), using 95% percentile bootstrap intervals from 10,000 paired resamples, with the document as the resampling unit except for composition (collection), retrieval (query), and transfer (source–target pair).

To test whether content is in the weights at all, they add a cloze probe: each held-out fact is rewritten as a declarative sentence with one blank, presented without the document and in a surface form no adapter saw in training, scored by recall of the reference fill. They also examine the geometry of the weight updates, comparing pairwise update distances to sentence-embedding distances between documents, and they run a retrieval experiment in which a bank of 120 specialists is searched by weight-space proximity or text embedding for 30 held-out query documents.

Finally they build a composition grid of 56 collections at sizes from 2 to 32 documents, under within-domain and cross-domain regimes, comparing three merging methods, joint retraining on the union, and a random out-of-collection adapter averaged over five draws.

Why This Matters

Impact on research. Open-book scoring is the standard way adapter-based knowledge banks are evaluated, including in parametric retrieval-augmented generation. If that score mostly reflects a behaviour every adapter shares, then reported gains for adapter banks, adapter routing, and adapter merging overstate how much document content is actually being stored or preserved. The paper gives the field a concrete diagnostic (withhold the document), a baseline discipline (average random adapters over several draws), and a cautionary catalogue of four failures that produced plausible results without raising any error.

Real-world applications:

• Enterprise document question answering. Teams building per-document or per-policy adapter banks can check whether their adapters store content or just learn to read supplied context, which determines whether they need retrieval at all.
• Knowledge-base maintenance and audit. A cloze-style probe provides a cheap way to test whether a stored adapter actually contains a fact, independent of whether the document is available at inference time.
• Model merging in deployment. Practitioners deciding between merging adapters and retraining jointly now have measured evidence: at 16 documents or more, merges kept 87–89% of the open-book advantage but only 12–16% of the cloze advantage, while joint training kept 94%.
• Evaluation and benchmarking. The four silent errors — a truncation default, a leftover decoding penalty, a scaling bug, and a precision mismatch — are directly transferable lessons for anyone running LoRA experiments.

Industry relevance. The work comes from a Bosch Center for Artificial Intelligence collaboration with LMU Munich, and it targets the practical question of whether adapter banks are a viable storage medium. Its recommendations — score with and without the document, baseline against random adapters, and hold precision and decoding constant across every arm of a table — are cheap to adopt and directly affect whether an internal evaluation is trustworthy.

Future Directions

• Does the effect hold beyond one model and one configuration? Every result except the geometry rests on Llama-3.1-8B-Instruct at one rank and one training schedule on 150 encyclopedic articles; the authors state they cannot say how the shares move with model size, adapter capacity, or genre.
• Where does joint training stop keeping pace? Composition was tested only on collections fixed in advance and no larger than 32 documents, so both the size at which a rank-8 joint adapter stops matching specialists and the incremental case — where composition would have to beat retraining on each new document — remain untested.
• Can retrieval over adapters be made to work? With 30 queries the confidence interval still admits an advantage of about four fifths of the query's own adapter's margin over a random one, so a larger, better-powered retrieval study is the obvious next step.
• Do the findings transfer to unseen text? Wikipedia articles have probably been seen in pre-training, and the base model already reaches 0.81 of the specialists' cloze recall over the composition grid's rows; whether a random adapter recovers as little of the cloze advantage on genuinely unseen documents is untested.

Target Audience

Researchers and engineers working on parameter-efficient fine-tuning, adapter banks, model merging, and parametric retrieval-augmented generation. It is especially valuable for practitioners who design evaluation harnesses for fine-tuned models, and for anyone who has reported strong open-book numbers for per-document adapters and wants to know what those numbers actually measure. Readers need working familiarity with LoRA and standard QA metrics, but no deep mathematical background.

Authors’ abstract

Single-document LoRA adapters have been proposed as units of knowledge to be merged and retrieved; yet open-book scoring, with the document in the prompt, can credit a document's own adapter (its specialist) for reading that document as readily as for holding it. Training one adapter for each of 150 Wikipedia articles on Llama-3.1-8B-Instruct, we find that open-book scores mostly measure a reading behaviour every adapter shares: a random adapter trained on another document recovers 82 % of the specialist's open-book advantage over the base model but only 2 % of its advantage on a cloze probe that asks for the same document's facts without providing the source text, a share we cannot distinguish from zero. Content does reach the weights, since the specialists beat the base model in cloze recall by 0.094, and the geometry of their updates follows the documents they read; yet, in a test of 30 queries, adapters chosen by weight-space proximity or text embedding are not distinguishable from random ones on either probe, although that test cannot exclude an advantage of about four fifths of what a query's own adapter gains over a random one. At 16 documents or more, correctly scaled merges recover 87 % to 89 % of the specialists' open-book advantage but only 12 % to 16 % of their cloze advantage, with intervals that run from below zero to at most 32 %, against 94 % for joint training.We also report four silent errors, each of which produced a clean, wrong result.