Production NLP Systems and Monitoring

The surrounding architecture determines whether measured model quality survives contact with real traffic. An offline document pipeline, an online prediction service, a search and retrieval service and a generative language application can run identical weights and still be four different systems. Each answers to a different contract with the clock, the index and the reviewer.

That contract is itself a versioned artifact, published and revised like the model. MLCommons released MLPerf Inference v5.0 on 2 April 2025 with a new Llama 2 70B Interactive serving benchmark. One of the submitters described it this way: “This workload is based on the same Llama 2 70B model that was first introduced in MLPerf Inference v4.0, but features more stringent latency constraints of 450 ms TTFT and 40 ms TPOT (25 tokens per second per user).” MLCommons's own writeup records what the earlier budget had been: the v4.0 server latencies “were set … at 2s TTFT and 200 ms TPOT”.

Between v4.0 and v5.0 the model did not change. The 99th-percentile time-to-first-token allowance fell from 2 s to 450 ms, and the 99th-percentile time-per-output-token allowance from 200 ms to 40 ms. A "tight latency budget" is therefore not a property of a model at all. It is a number that someone published, that carries a version, and that can be tightened underneath a service which never redeployed a single weight.

Observability should follow the request across components rather than stopping at the model endpoint.

Each stage owns its own versions, timeouts and failure modes. An incident is diagnosed by naming which stage broke. That is only possible if a single trace carries identifiers from all five: ingress and policy, text preparation, evidence and inference, decision and response, telemetry and feedback.

A tokenizer change can alter sequence length, offsets, truncation, and unknown-token behavior while the model file remains identical. An index refresh can change answers without a model deployment. A label-policy change can invalidate an old threshold.

Record versions for normalization, tokenization, label schema, model, index, prompt or template, post-processing, threshold, and fallback. A release identifier should reconstruct the complete path from request to response.

The unknown-token problem is not hypothetical, and its size has been measured. Sander Land and Max Bartolo searched tokenizer vocabularies for entries the model was never meaningfully trained on, and published the result at EMNLP in 2024. What they found is a prevalence rather than an anecdote: “we detect the presence of such tokens across all tested models, with typically around 0.1–1% of the vocabulary consisting of severely under-trained tokens, although their prevalence varies significantly.” On the open OLMo v1.7 model, 191 of 49,575 tokens passed their verification threshold. A separate 2024 study, AnomaLLMy, reached the same phenomenon from outside a closed system. It reported 413 major and 65 minor anomalous tokens in GPT-4's vocabulary — the cl100k_base token set — through API access alone, for $24.39 in credits. A vocabulary is a component with its own defect rate. A defect rate is something you version and re-audit, not something you assume away.

One tokenizer was built around exactly this obligation. SentencePiece, published in 2018, guarantees that decoding inverts encoding: “Decode(Encode(Normalize(text))) = Normalize(text)”. Its authors, Taku Kudo and John Richardson, call that “lossless tokenization, in which all the information to reproduce the normalized text is preserved in the encoder’s output”. The decision that matters for a release identifier is the next one. “The model file of SentencePiece is designed to be self-contained to guarantee perfect reproducibility of the normalization and subword segmentation.” The tokenizer is a versioned artefact because its authors made it one.

Generic CPU and error-rate dashboards are necessary, but they do not show whether language behavior remains useful.

The size of that blind spot is on the record for shipped products. Magesh, Surani, Dahl, Suzgun, Manning and Ho ran a preregistered benchmark of 202 legal queries against shipped, retrieval-augmented commercial products between 22 March and 22 April 2024. Twenty of the queries were taken verbatim from LegalBench's Rule QA task, twenty verbatim from BARBRI bar-exam prep questions, and 162 were hand-written or adapted. Despite vendor claims of "hallucination-free" citations, the tools hallucinated more than 17% of the time. The two systems failed in opposite directions: “Lexis+ AI provides accurate (i.e., correct and grounded) responses on 65% of queries, while Ask Practical Law AI refuses to answer queries 62% of the time and responds accurately just 18% of the time.”

Note where Thomson Reuters' Ask Practical Law AI lost its accuracy. The study attributes its failures to the narrow document universe it retrieves over, not to the generator sitting on top of it. A refusal rate of 62% and an accuracy of 18% would both look like model behavior on a dashboard that only watched the model. Retrieval-side signals have to be monitored as their own population.

A change may involve code, model, tokenizer, labels, corpus, index, prompt, or policy.

For a whole class of systems, the last step of this sequence is not a good habit but a documented obligation that outlives the release. Article 72 of the AI Act requires providers of high-risk AI systems to establish and document a post-market monitoring system running throughout the system's lifetime, based on a written plan that forms part of the technical documentation under Annex IV. Article 72(2) says what that system has to do: “The post-market monitoring system shall actively and systematically collect, document and analyse relevant data which may be provided by deployers or which may be collected through other sources on the performance of high-risk AI systems throughout their lifetime, and which allow the provider to evaluate the continuous compliance of AI systems with the requirements set out in Chapter III, Section 2.”

The template is still being drafted. Regulation (EU) 2026/1744 of 8 July 2026, the Digital Omnibus on AI, replaced Article 72(3) and applies from 27 July 2026. The provision now in force gives the Commission until 2 September 2027 to adopt guidance, including a template for the post-market monitoring plan, taking utmost account of the opinion of the Board. The five steps below are the shape of the evidence such a plan has to produce, whether or not the template has arrived.

A retrieval system may have to ingest documents hourly while its encoder changes quarterly. A classifier can remain statistically stable even though a policy redefinition makes its labels obsolete.

Assign separate freshness objectives and ownership to source documents, annotations, features, indexes, models, thresholds, and evaluation sets. Rebuild only the components that the evidence justifies.

The immediate goal is containment. The lasting goal is a safer system and a test that would have caught the failure.

Containment is measured with a clock and a blast radius, and there is a canonical measurement of both. Microsoft released the Tay chat bot on Twitter on 23 March 2016. Gina Neff and Peter Nagy date the end of it precisely: “Sixteen hours after Tay started interacting with and learning from Twitter users, Microsoft took Tay offline.” Inside that window Tay had already posted more than 93,000 tweets. A reactivation seven days later failed as well.

Sixteen hours is not slow by the standards of 2016. The point is that the blast radius was 93,000 outputs wide before the first containment action, and that the system had been tested. Peter Lee, Corporate Vice President of Microsoft Research, wrote on 25 March 2016: "We stress-tested Tay under a variety of conditions, specifically to make interacting with Tay a positive experience". He also wrote: "Unfortunately, in the first 24 hours of coming online, a coordinated attack by a subset of people exploited a vulnerability in Tay". Pre-release testing had happened. It did not cover the case that mattered. His closing commitment is the sentence an NLP runbook should be able to produce on demand: "Tay is now offline and we'll look to bring Tay back only when we are confident we can better anticipate malicious intent that conflicts with our principles and values." Containment first. Restoration only against a named, testable condition.

Include every transformation and decision from ingress to final action. Label versioned artifacts, timeouts, caches, external dependencies, logs, privacy boundaries, fallback, and human handoff.

Then choose six quality signals and six operational signals. For each, define its population, aggregation window, alert threshold, likely false alarm, and incident owner. Finish by writing down two numbers your diagram does not yet contain: the latency budget the service is held to, and the retention period each logged field is kept for.