RAG Corpus Design, Chunking, and Metadata

A policy document contains definitions, exceptions, footnotes, and an approval date. If ingestion slices it every five hundred characters, the exception may be separated from the rule it modifies.

Chunking is not merely a performance setting. It defines the units that can be retrieved, cited, permissioned, and retired. Published systems have already made every decision in this lesson, in the open: how many units one corpus becomes, whether a smarter splitter repays its cost, what a scanner does to a number before any parser sees it, and what a regulator expects when a source is withdrawn.

The best strategy depends on source structure and query type. The trade is measurable rather than a matter of taste.

Granularity multiplies the corpus. One English Wikipedia dump, dated 2021-10-13, was re-indexed at three granularities: 41,393,528 passages averaging 58.5 words, 114,219,127 sentences averaging 21.0 words, and 256,885,003 propositions averaging 11.2 words. The 2024 paper that did it, “Dense X Retrieval: What Retrieval Granularity Should We Use?”, published the counts and the arithmetic behind them. “We decomposed 6 million pages into 41 million passages, 114 million sentences, and 257 million propositions. On average, a passage contains 6.3 propositions, and a sentence contains 2.3 propositions.” The finer unit was not merely more numerous. With the unsupervised retrievers SimCSE and Contriever, indexing by propositions raised averaged Recall@5 by +12.0 and +9.3 points — 35.0% and 22.5% relative improvement — across five open-domain QA datasets. One dump, three corpora, three different sets of things that can be cited.

Semantic chunking is the row where intuition is most often wrong. A 2024 study put the question in its title, “Is Semantic Chunking Worth the Computational Cost?”, and tested it on three retrieval-related tasks: document retrieval, evidence retrieval, and retrieval-based answer generation. The abstract answers it. “The results show that the computational costs associated with semantic chunking are not justified by consistent performance gains.” Treat a semantic splitter as a claim to be tested against fixed-size baselines on your own corpus, not as a default upgrade.

Returning a larger parent is not free either. Where the evidence sits inside the context window changes how well the model uses it. “Lost in the Middle” measured that in 2024 on multi-document question answering and key-value retrieval: “In particular, we observe that performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models.” Padding a window with parent text or overlapping near-duplicates can bury the answer-bearing span in the position where it is used worst.

The vector or keyword text is only one field in a trustworthy index. Source identity, version and time, structural location, authority and scope, access policy, and transformation lineage all have to survive ingestion, because none of them can be reconstructed from an embedding afterwards.

The access-policy field is the one with a numbered failure attached to it. On 11 June 2025 Microsoft issued CVE-2025-32711, “M365 Copilot Information Disclosure Vulnerability”. The official description is one sentence: “Ai command injection in M365 Copilot allows an unauthorized attacker to disclose information over a network.” Microsoft classified it as CWE-74 injection and rated it Critical, with a CVSS 3.1 base score of 9.3 (vector CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:C/C:H/I:L/A:N), recording no public disclosure and no observed exploitation. NIST's National Vulnerability Database carries the same description verbatim and scores it 7.5, High. Two scoring bodies, one lesson for corpus design. Whatever sits inside an assistant's retrieval scope is reachable by injected input. A permission that was attached to the source document and dropped at chunking is not a bookkeeping lapse. It is the exposure surface.

Inspect the sources, and how they were parsed, before changing prompts. The most instructive case is one where the damage happened before any parser ran. A construction plan listed three rooms of 14.13, 21.11 and 17.42 square metres. Scanned on a Xerox WorkCentre 7535, all three came out as 14.13. David Kriesel documented this in July and August of 2013: WorkCentre and ColorQube machines were silently substituting digits through JBIG2 pattern-matching compression, with OCR deliberately switched off. His own description of the mechanism: “This is not an OCR problem (as we switched off OCR on purpose), it is a lot worse – patches of the pixel data are randomly replaced in a very subtle and dangerous way: The scanned images look correct at first glance, even though numbers may actually be incorrect.” Xerox principal engineer Francis Tse confirmed the “character substitution issue” in a company blog post on 6 August 2013. On 12 August Xerox confirmed the defect was an eight-year-old software bug affecting all compression modes, and patches began rolling out on 22 August. A retrieval system fed those pages would have cited a document that looked perfect and said something false.

Treat ingestion as a versioned data product: inventory the sources, parse against fixtures, choose the retrievable unit deliberately, attach metadata and ACLs, deduplicate versions, validate a sample against the originals, and publish a manifest that versions the corpus, parser, chunker, embedder, index, and deletion status.

The deduplication and manifest steps are where an obligation sits, not just a preference. When the US Federal Trade Commission settled with Everalbum, Inc., deletion had to reach further than the source files. The Commission's own published analysis of the order says what had to go: “Part III of the proposed order requires Respondent to delete (A) photos and videos of Ever app Users who requested deactivation of their accounts, (B) face recognition data that it created without obtaining Users' affirmative express consent, and (C) models and algorithms it developed in whole or in part using images from Users' photos.” That analysis was published for public comment in the Federal Register on 25 January 2021, and the settlement was finalised in May 2021. Chunks, embeddings, and cached parent documents are exactly the derived representations that clause reaches. A corpus release that cannot say which records came from which source, in which version, cannot execute a deletion of that shape at all.

No search can surface evidence that ingestion did not preserve. The same 6 million pages became 41,393,528 passages or 256,885,003 propositions, depending on one decision taken before any query arrived. A scanner turned 21.11 into 14.13 before any parser opened the file. A permission dropped at chunking became a disclosure path with a Critical rating. A deletion order reached the models built from the deleted images. Each of those is an ingestion-time fact that ranking cannot revisit.

The next lesson compares sparse, dense, and hybrid retrieval. Those methods score the records created here. They cannot reconstruct missing provenance or repaired semantics afterward.