DMAP: A Distribution Map for Text

AI summary

Overview

Research area: Natural Language Processing — statistical text analysis using large language models, machine-generated text detection, and calibration of instruction-tuned models.

Technical level: Intermediate. The paper is written for readers comfortable with probability distributions, hypothesis testing, and language model internals, but the core idea is explained with concrete diagrams and worked examples. Formal proofs are deferred to an appendix.

Scope in one sentence: The paper introduces DMAP, a method that converts any text — through any language model — into a standardized set of samples on the unit interval that encode both token rank and token probability, and demonstrates three applications: validating reported decoding parameters, re-examining the "probability curvature" assumption behind machine-generated text detectors, and exposing statistical fingerprints of synthetic fine-tuning data in downstream models.

What This Paper Is About

Standard ways of measuring text with a language model — average per-token log-likelihood, log-rank, and perplexity — reduce a rich sequence of next-token probability distributions to a single number. This loses context: whether a token probability of 0.1 or a rank of third is "surprising" depends on how many reasonable alternatives the model was choosing between at that moment. The paper calls this the contextualization problem and proposes DMAP, a mathematically grounded recipe that maps a text onto a density function on [0, 1], producing a uniform distribution exactly when text was generated by pure sampling from the evaluator model, and a characteristic, visually readable shape otherwise (head bias, tail bias, or tail collapse).

Key Contributions

1. The DMAP algorithm. A simple method that represents a text via a language model as a set of samples in [0, 1] that jointly encode rank and probability information. Each token defines an interval whose length is its conditional probability and whose left endpoint is the total probability mass of all tokens judged more likely; a point is drawn uniformly from that interval. An entropy-weighted variant removes the sampling randomness, weights each token by the entropy of its conditional distribution (clipped at λ = 2 for stability), and makes the method deterministic and more sensitive. The method is open-sourced at https://github.com/Featurespace/dmap, computationally efficient, and runs on consumer hardware with small models such as OPT-125m.

2. Validating generation parameters. DMAP can detect incorrect or inconsistent generation settings in published data — top-k, top-p, temperature, or the identity of the language model itself — both qualitatively (by eye) and quantitatively (via a chi-squared uniformity test). The authors report discovering a major data error affecting most of the top papers in zero-shot machine-generated text detection: at the time of their writing, HuggingFace enabled top-k by default with k = 50, making it easy to accidentally leave top-k on while reporting pure-sampling experiments.

3. Informing machine-generated text detector design. Using DMAP, the authors identify a design weakness in existing zero-shot detectors built on the probability curvature thesis and propose alternative principles for detector design.

4. Revealing statistical signatures of post-training data. DMAP shows how statistical fingerprints of fine-tuning data persist in the outputs of instruction-tuned models, including without access to internal model probabilities for the generating model.

Main Findings

• Pure sampling produces a uniform DMAP distribution. Proposition 3.1 proves that when a text is generated by pure sampling from a model, applying DMAP with that same model as evaluator yields samples that are i.i.d. uniform on [0, 1]. This result extends to decoding strategies such as top-k, nucleus (top-p) and temperature sampling, provided the same generation/decoding pairing is used in both steps. The plotted examples use k = 40 bins.

• Decoding strategies have characteristic DMAP shapes. On XSum data generated by OPT-125m and evaluated by OPT-125m: pure sampling is close to uniform; top-p = 0.8 sampling is flat over an initial portion of the interval before rapidly dropping off; temperature τ = 0.8 and top-k = 50 sampling both produce head-biased, highly characteristic shapes. These experiments ran 300 texts of around 300 tokens each.

• Three recurring visual patterns. Head bias — tokens the evaluator deems likely are over-represented in the text. Tail bias — unlikely tokens are over-represented; typical when one base model's output is evaluated by another base model of similar entropy. Tail collapse — the bottom portion of the evaluator distribution is strongly under-represented, often seen in human-written text and consistent with the folklore motivation for nucleus sampling.

• Human text versus machine text. Human-written news text from RAID (evaluated by OPT-125m) is generally somewhat surprising to the evaluator, with a sharp drop-off on the right-hand side of the distribution — the bottom five percent of the OPT-125m distribution places too much weight on tokens unrepresentative of human writing. This drop-off is much less pronounced with more modern evaluator models.

• Base models versus instruction-tuned models. Mistral 7B (a base model) under pure sampling (top-p = 1) is tail-biased when evaluated by OPT-125m, as are cross-evaluations among Mistral, Falcon, and Llama 3.1 8B. In contrast, Mistral 7B Instruct and ChatGPT outputs are head-biased. All were evaluated with OPT-125m as the scoring model in Figure 2.

• Probability curvature fails under pure sampling. The paper reports that DetectGPT, Fast-DetectGPT and Binoculars remain effective when top-k sampling generates the text but perform worse than a coin toss under pure sampling, due to inversion of the probability curvature. Inverting classifications is not possible because these detectors are zero-shot with fixed directionality.

• Detector AUROC numbers. Table 1 reports AUROC across XSum, SQuAD and WritingPrompts for top-k = 50 versus top-k = None (pure sampling):

  • Fast-DetectGPT, Llama-3.1-8B: 0.702 / 0.200 (XSum), 0.739 / 0.208 (SQuAD), 0.915 / 0.289 (Writing)
  • Fast-DetectGPT, Mistral-7B-v0.3: 0.770 / 0.276, 0.819 / 0.299, 0.906 / 0.339
  • Fast-DetectGPT, Qwen3-8B: 0.765 / 0.289, 0.612 / 0.320, 0.923 / 0.377
  • DetectGPT, Llama-3.1-8B: 0.606 / 0.408, 0.527 / 0.299, 0.723 / 0.422
  • DetectGPT, Mistral-7B-v0.3: 0.679 / 0.486, 0.586 / 0.365, 0.688 / 0.457
  • DetectGPT, Qwen3-8B: 0.635 / 0.445, 0.463 / 0.380, 0.724 / 0.479
  • Binoculars, Llama-3.1-8B: 0.825 / 0.325, 0.849 / 0.365, 0.942 / 0.410
  • Binoculars, Mistral-7B-v0.3: 0.823 / 0.350, 0.851 / 0.416, 0.931 / 0.404
  • Binoculars, Qwen3-8B: 0.857 / 0.416, 0.752 / 0.467, 0.949 / 0.492

• A worked quantitative example. Evaluating 10,000 tokens of Llama-generated text with Mistral-7B as the evaluation model gives a chi-squared statistic whose p-value is less than 10⁻¹⁰, making it implausible that the text was produced by pure sampling from Mistral 7B.

• Synthetic fine-tuning data leaves a fingerprint. Two sizes of Pythia models were fine-tuned on the OASST2 dataset with responses from humans, from Llama 3.1 8B at temperature 1, and from Llama 3.1 8B at temperature 0.7. The only head-biased model was the one fine-tuned on temperature-sampled data — which was itself the most head-biased fine-tuning data. Human-written instruction fine-tuning data showed a dramatic tail-collapse, larger than in other human-written text. Fine-tuned models showed increased density in the final bin, which the authors suggest could detect mild overfitting during supervised fine-tuning and inform early stopping. All instruction-tuned models were less tail-biased than the original Pythia model.

• Low-entropy positions carry little signal. Restricting DMAP plots to times when the next-token distribution has low entropy produces plots with little useful information, justifying the entropy down-weighting.

Methodology in Plain English

The researchers use an evaluator model — a language model used purely for scoring, not for generating — to look at each token of a text in turn. For each position they ask: how does this token sit inside the model's ranked list of next-token probabilities? They then build an interval whose length equals the token's probability and whose position along [0, 1] is determined by how much probability mass sits above it. A point is drawn uniformly at random inside that interval; the collection of points is the DMAP sample. Sorting these into bins and plotting a histogram immediately shows whether the text leans toward the head of the model's distribution, the tail, or neither.

Because the random draw adds noise, the authors also define an entropy-weighted version that replaces the random point with the expected density across the interval, weighting each token by the entropy of its conditional distribution (capped at λ = 2). This places more weight where the next-token choice is genuinely uncertain and down-weights positions where the token was nearly certain, making the tool more sensitive.

For rigorous rather than visual conclusions, they split [0, 1] into bins using the Terrell-Scott rule with k = (2T)^(1/3) bins, compute bin frequencies, form a chi-squared statistic measuring departure from uniformity, and convert it into a p-value testing whether the text is consistent with a claimed generation method. The paper's central theoretical result is a proof that pure sampling from the evaluator model produces i.i.d. uniform DMAP samples, which is what licenses the uniformity test; the same proof technique extends to decoding strategies such as temperature, top-k and top-p.

Why This Matters

Impact on research. The paper argues that a large fraction of the zero-shot machine-generated text detection literature was run on data inadvertently generated with top-k = 50 rather than pure sampling, and that later work (for example Dubois et al., 2025) reused those texts. It shows that the probability curvature principle underlying DetectGPT, DetectLLM, Fast-DetectGPT and Binoculars does not hold for pure sampling from base models, so detectors that look strong in the literature can fall below chance in an unrestrictive black-box setting. DMAP also provides a reproducible, cheap way to check published claims about generation parameters before they propagate.

Real-world applications

• Dataset and publication integrity: verifying that released text corpora match the decoding parameters their documentation claims, before downstream research depends on them.
• Detector design and red-teaming: identifying when an existing detector's core assumption breaks, and designing detection principles that are robust to base models with careful prompting.
• Model forensics and attribution: the large differences between DMAP plots when generator and evaluator models differ point toward new approaches to language model identification.
• Monitoring fine-tuning and calibration: using the final-bin density and the head-bias/tail-bias signature as cheap signals of overfitting or of synthetic-data contamination during supervised fine-tuning.

Industry relevance. The work is co-authored by the Risk and Security AI Lab at Visa Inc. and uses OPT-125m as the default evaluator specifically because it runs on consumer hardware in a few minutes, making the tool practical outside large compute budgets. Because the method is model-agnostic, it applies across architectures and scales, and it does not require access to internal probabilities from the model that generated the text under analysis.

Future Directions

1. Data curation and efficient fine-tuning — the authors suggest DMAP could guide which training data to keep or drop, or which parameters to fine-tune.
2. Better calibration for instruction-tuned models — rather than temperature scaling to align perplexity with a pre-trained model, future work could calibrate DMAP distributions directly, including calibrating them to match human text patterns as a form of control over model confidence during frontier training.
3. Formalizing context dependence — the distinct DMAP signatures observed across text domains such as poetry, news, and technical writing suggest the method could separate inherent prediction difficulty from model-specific limitations.
4. Model identification and forensic analysis — the dramatic differences in DMAP plots when generator and evaluator differ are presented as a promising lead for language model identification.
5. The paper also leaves open what other quantitative measures of the sample set {x_i} might reveal, beyond the histograms and chi-squared tests used here.

Target Audience

This paper is most useful to researchers and practitioners working on machine-generated text detection, model evaluation and calibration, and statistical methods for analyzing language model outputs; to engineers who need a lightweight, model-agnostic text analysis tool that runs on modest hardware; and to teams in security, risk and trust-and-safety who must verify the provenance and generation settings of text data. Readers who want the mathematical grounding rather than just the applications should be comfortable with probability, the probability integral transform, and basic hypothesis testing.

Authors’ abstract

Large Language Models (LLMs) are a powerful tool for statistical text analysis, with derived sequences of next-token probability distributions offering a wealth of information. Extracting this signal typically relies on metrics such as perplexity, which do not adequately account for context; how one should interpret a given next-token probability is dependent on the number of reasonable choices encoded by the shape of the conditional distribution. In this work, we present DMAP, a mathematically grounded method that maps a text, via a language model, to a set of samples in the unit interval that jointly encode rank and probability information. This representation enables efficient, model-agnostic analysis and supports a range of applications. We illustrate its utility through three case studies: (i) validation of generation parameters to ensure data integrity, (ii) examining the role of probability curvature in machine-generated text detection, and (iii) a forensic analysis revealing statistical fingerprints left in downstream models that have been subject to post-training on synthetic data. Our results demonstrate that DMAP offers a unified statistical view of text that is simple to compute on consumer hardware, widely applicable, and provides a foundation for further research into text analysis with LLMs.