The Pulse
CTGT Opens Concordance to Trace Inconsistent AI Decisions
CTGT has released Concordance, an Apache 2.0 tool for tracing why an AI application produces different decisions across repeated runs. In the company’s test, one banking manual yielded four different sets of obligations even though the mode
AI.info Team · Written with AI · fact-checked ·
Qwen3.8-27B returned the same answer to 800 of 800 test requests, yet CTGT’s AI pipeline produced four different sets of obligations from the same banking manual. The company says the discrepancy came not from changing model answers but from small changes in the requests its software constructed around them. CTGT released Concordance on October 21 under the Apache 2.0 license to help developers locate where that kind of variation first appears.
The release includes a testing protocol, reference code and experimental data. Concordance compares model requests and responses across repeated runs, then identifies the earliest stage where the outputs diverge. Its published results come from one document and one company-built pipeline, not a survey of enterprise deployments.
Four runs, four obligation lists
CTGT’s test processed a nine-page retail banking manual four times using Qwen3.8-27B served by SGLang with deterministic inference enabled. The runs extracted 32, 31, 33 and 33 obligations, and each produced a different result. Among the missed items was a requirement to escalate an unresolved conflict to the Head of Compliance.
Across the runs, 74 model requests recurred with identical content. The model gave the same answer to each repeated request; the surrounding application changed what it sent. In one comparison, four asset-ID lines differed inside a prompt of 12,525 characters. CTGT says the model then treated five conflict-resolution rules as background information rather than rules to follow.
“Any application that writes upload IDs, unordered collections or shared state into its prompts carries the same exposure, regardless of deterministic inferencing,” write Cyril Gorlla and Andrew Noonan of CTGT Engineering in the research report. Cyril Gorlla and Andrew Noonan, CTGT Engineering
Concordance tracks the first divergence
The tool examines two layers: the inference server’s responses to prompts and the full application pipeline’s final decisions. For pipeline tests, CTGT’s protocol captures each model call, groups identical requests across runs, and reports whether an altered request or a changed answer marks the first divergence. That distinction helps developers tell model-serving variation apart from changes introduced by their own software.
The open-source repository provides a call-comparison script, an inference benchmark, data from the pipeline runs and a small synthetic pipeline designed to reproduce the effect. The synthetic example can run against an OpenAI-compatible endpoint. CTGT estimates it takes under ten minutes with a small model; reproducing the published inference benchmark requires a GPU host and the 27-billion-parameter model.
In the separate inference benchmark, the model’s first 1,000 output tokens matched a batch-size-one control in all 800 requests, with requests run at concurrency 40. That result measures text agreement, not correctness. When CTGT replayed the request behind one incorrect classification eight times, the same wrong answer appeared in all eight replays.
Stable answers can still be wrong
CTGT traced the pipeline variation to three implementation details: upload-specific identifiers in prompts, a 30-item list whose order changed, and a shared topic registry that grew between runs. The company replaced the identifiers with stable aliases, sorted the list and began each test in a fresh organization. Five runs after those changes produced the same 34 obligations.
The result illustrates a limit of reproducibility checks: repeating an error consistently does not make it correct. CTGT says its next measurement challenge is to test whether decisions remain faithful to policies when input wording changes. The current Concordance tests do not establish that the tool prevents policy violations.
The released package leaves out the production system
The evidence is deliberately narrow. CTGT reports one document, one pipeline and a small number of runs, and notes that stored outputs needed normalization of upload-specific IDs for comparison. The company’s production ingestion engine, prompts and captured model calls are not included in the public repository; the toy pipeline demonstrates the mechanism, not the original system.
Concordance therefore gives developers a way to inspect repeatability, not a guarantee that an AI system follows its rules or makes sound decisions. Establishing that still requires tests against expected outcomes and the policies a system is meant to apply. The released repository contains the comparison code, reproduction example and summary data, but not the production pipeline that generated CTGT’s findings.