Kierston & KierstonBench: Simulation of Allentown, PA and Evaluation of Language Models in Crime Investigation and Surveillance Tasks

AI summary

Overview

Research area: AI evaluation and LLM agent benchmarking, with a supporting contribution in population/urban simulation — specifically, the use of large language models for criminal investigation and mass surveillance record analysis.

Technical level: Intermediate. The benchmark framing and headline results are accessible to a general AI reader; the engine section (cellular network planning, timing advance, physical cell identity allocation) is technical and aimed at practitioners who want to use or extend the simulator.

Scope: The paper introduces Kierston, a city-scale simulation engine that generates linked investigative records for roughly 125,000 simulated residents of Allentown, Pennsylvania, and KierstonBench, a 32-task-family benchmark that scores language models on record linkage, routine reconstruction, incident investigation and deception assessment through a restricted provider-records interface.

What This Paper Is About

Investigators work by combining records held by separate organizations — a vehicle registration, a cellular event, a bank transaction — and testing whether the resulting account fits the observations. Language models are candidates for querying these sources and writing the analysis code, but there is no way to measure how well they do it: real investigative data lacks ground truth, and independently generated tables cannot show whether records from different providers describe the same underlying behavior.

The paper's goal is to make that measurement possible by simulating a population's daily activities first and deriving every provider record from that shared timeline, so the true events are known. It then wraps the resulting records in a benchmark that gives a model only a restricted, provider-style interface and asks it to reach conclusions the records support — or to abstain.

Key Contributions

1. Kierston, a simulation engine. A city builder and behavior runtime that places approximately 125,000 residents in Allentown, PA using public census, map and infrastructure data, generates their daily activities, and derives telecommunications, financial, administrative, plate-reader and surveillance records from a single behavioral timeline — retaining the activities as ground truth. It includes provider-faithful formats, incomplete observations and measurement error.

2. KierstonBench, a 32-task-family benchmark. Tasks span three groups: Technical Tasks (record linkage, routine reconstruction), Real-Time Investigations (staged incidents with records restricted to what was available shortly afterward), and Deception Tasks (false accounts that must be checked against independent records). Models choose which provider sources to query and receive negative credit for unsupported identifications.

3. An evaluation of sixteen language models. Beyond scores, the authors analyze the models' recorded queries, scripts and answers to examine how evidence collection and interpretation produce their conclusions.

4. A reproducible, portable engine. Thirty-one ordered initialization stages and a nine-pass daily runtime, with keyed random draws so that any resident's records can be regenerated exactly regardless of processing order. A new city is a short profile naming a map extent, a census microdata area and a legal jurisdiction.

Main Findings

• Most models could not do records investigation. In the evaluation of sixteen language models, the majority failed at investigative record work and submitted identifications that the retrieved records did not support.

• A small frontier subset largely succeeded. A few frontier models solved most tasks and abstained when the evidence was insufficient, which the benchmark treats as the correct behavior under negative credit for wrong attributions.

• Deception and assumption failures persisted even in strong models. In some cases the frontier models still fell for common deceptions and everyday assumptions embedded in the tasks.

• No model replaced an investigator. The authors' summary conclusion from the evaluation is that none of the sixteen models substituted for a human investigator.

• The simulated city is realistically instrumented. The cellular planner produces 198 structures (77 registered from FCC data, 91 small cells, 20 monopoles and 10 rooftops) carrying 3,319 cells, giving a median home-to-nearest-site distance of about 420 m — roughly half what registered structures alone would provide.

• Timing advance adds real location signal. Timing advance is quantized to 78.125 m steps, with morphology-dependent excess path length (mean 8, 20, 40 or 60 m), so a stationary handset reports a stable value and adds a radial constraint on top of sector-based localization.

• Outside traffic blocks a shortcut. Without non-resident vehicles, every plate read would belong to a resident and a model could treat unregistered plates as noise; the engine therefore adds commuters, deliveries and through trips.

• Benchmark design deliberately hides evaluation cues. All explicit evaluation markers are removed from what the model sees, motivated by prior work showing that models which recognize an evaluation may behave differently.

• Per-model scores are not in the supplied content. The truncated text ends in Section 3.7; the paper's reported per-model results, task-level scores and ablations are not present in the material provided, so no accuracy figures are quoted here.

Methodology in Plain English

The authors work in two stages that are deliberately kept separate.

First, they build a city. Households are cloned from the American Community Survey public microdata for the Allentown area and placed into real OpenStreetMap buildings. Residents get jobs matched to real worksites within their reported commute distance, schools, social ties, criminal and civil dockets, bank accounts, vehicles with VINs and plates, handsets with real type-allocation codes, and cellular subscriptions. Registered radio masts come from the FCC; smaller sites that fall under the registration threshold are added by a model that ranks demand on a 250 m grid and places rooftops, monopoles and street-pole small cells. Real public data is used for anything a real analyst could look up — a tower, a handset model, a plate series, a road — while anything identifying a person is synthetic.

Second, they run days. Each simulated day proceeds in nine ordered passes: restore the previous day's closing state, plan 911 demand and taxi matching, let each resident's activity layers propose what they want to do, schedule and route it, assign vehicles, settle payments and compute minute-by-minute radio attachment, add outside traffic, record every physical pass over a camera (separately from what the sensor logged), and finally generate calls and messages from the social graph. Because every payment, call, cell attachment and plate read is derived from the same activity timeline, the ground truth is exact.

For the benchmark, the records are re-presented through an interface that mimics separate telecommunications, financial, administrative and surveillance providers, each holding its own identifiers. The model gets a question and initial evidence, chooses which sources to query, and answers without any access to activity histories or incident roles. Wrong identifications cost points, so claiming a match the records do not support is worse than abstaining. The authors then read the models' queries and scripts to understand why they failed, not just whether they did.

Why This Matters

Impact on research. The paper supplies what the field has been missing for this task class: linked records with known ground truth and an interface that prevents the model from peeking. It also separates two things that benchmarks usually conflate — whether a model retrieves the right evidence, and whether it interprets that evidence correctly — by logging queries and scripts alongside answers.

Real-world applications:

• Records-fusion and analytics software testing. The engine can supply test data with known ground truth for systems that join records across providers.
• Deployment planning. It can support planning of plate-reader and cell-site deployments, since it models coverage, morphology classes and site spacing explicitly.
• Analyst training. Simulated cases with known answers let trainees practice cross-source linkage without touching real investigative data.
• Privacy-preserving method development. As with Synthea for health records, synthetic investigative data lets record-based research proceed without exposing real people.

Industry relevance. The engine targets exactly the record types that law-enforcement and intelligence analytics vendors ingest — carrier network events and call detail records, plate reads, bank transactions, and state and federal administrative files — including realistic complications like MVNO roaming, co-located tenants on shared towers, and modelled sites flagged distinctly from registered ones. The authors explicitly note both the engine and the benchmark are dual-use.

Future Directions

• Raising the ceiling on deception and assumption failures. The strongest models still accepted common deceptions, which points to a need for tasks and training that specifically target plausible-but-unsupported narratives rather than retrieval alone.
• Extending beyond the current records depth. Several subsystems are modeled only to benchmark depth — messaging platforms carry sparse traffic compared with carrier SMS and calls, and prescriptions are a static list with no dispensing history. Deepening these would broaden what can be tested.
• Porting to other cities and jurisdictions. The authors frame a city as a short profile (map extent, census microdata area, legal jurisdiction) and record which stock parameters were fitted locally versus inherited from Allentown; validating on new cities is the natural next test of portability.
• Grading the reasoning, not just the answer. Because queries and scripts are logged, a further step is scoring evidence-collection strategy and abstention calibration directly, rather than only final identifications.

Target Audience

AI evaluation researchers and benchmark designers working on agentic tool use and abstention; law-enforcement and intelligence analytics engineers who need ground-truth test data for records-fusion systems; simulation researchers interested in census-based synthetic populations with provider-level instrumentation; and policy or civil-liberties readers who want a concrete picture of what automated record investigation can and cannot currently do. Readers wanting only the capability picture can read the abstract and findings; readers wanting to build a city or extend the benchmark will need the engine sections.

Authors’ abstract

Large language models are candidates for collecting and analyzing records in criminal investigations and mass surveillance. Their capability and accuracy in this work cannot currently be measured. Real deployments lack the ground truth needed to score a model's conclusions, and no benchmark evaluates these capabilities in a simulated environment. We present Kierston, a city-scale simulation engine, and KierstonBench, a benchmark that tests these capabilities against known simulated events. Kierston models approximately 125,000 residents of Allentown, Pennsylvania, generating their daily activities and the telecommunications, financial, administrative and license-plate records that surveillance and investigative systems collect about them. These records derive from a common behavioral timeline and are distributed across sources representing separate providers. Models investigate through a restricted records interface, linking observations across sources without access to the underlying simulation. The benchmark comprises 32 task families covering record linkage, routine reconstruction, incident investigation and assessment of deceptive accounts. In an evaluation of sixteen language models, most could not perform records investigation and submitted identifications the records did not support. A small number of frontier models solved most of the tasks and abstained when evidence was insufficient. In some cases these models still fell for common deceptions and assumptions, and none replaced an investigator.