The Pulse
NeoSigma Finds Jev Cut Agent-Trace Judging Costs 6.2x
NeoSigma says Jev cost $3.23 for 10,000 decisions over 834 distinct frozen agent traces, compared with $19.98 for GPT-5 Mini. In a 100-trace human review, the judges matched 88 and 85 labels, respectively, though the company says the agreem
AI.info Team · Written with AI · fact-checked ·
“The value of a cheap decision model is not that it replaces every LLM judge. It changes how evaluation capacity can be allocated.”
Kartikeya Gupta, Gauri Gupta and Ritvik Kapila, authors of the NeoSigma post
NeoSigma’s comparison of TypeSafe’s Jev decision model with OpenAI’s GPT-5 Mini found that Jev completed 10,000 decisions over 834 distinct frozen agent traces for $3.23, versus $19.98 for GPT-5 Mini. The company published the results on September 26, 2026, arguing that lower-cost, faster verdicts could let teams inspect a larger share of agents’ production behavior.
The experiment tested a bounded task: decide whether an agent completed a user’s request, returning either “pass” or “fail.” NeoSigma describes Jev as a model built to make structured decisions and return probabilities, rather than generate a free-form explanation that another system must parse. The distinction narrows the job; it does not guarantee the answer is right.
One frozen trace set, two judges
NeoSigma ran the judges against 834 frozen traces from Tau2’s retail, airline and telecom tasks. The traces came from 278 public tasks, each run with three fixed seeds; both judges saw the same agent policy, conversation, tool calls, results and final response. The agent runs used GPT-5.6 Terra, and the company says it collected the traces once before freezing them for evaluation.
Each judge completed 10,000 valid decisions, including repeated judgments used to measure consistency. For an accuracy check, a reviewer who could not see the model verdicts labeled a separate sample of 100 traces. The reviewer marked 29 as failures, providing a small human reference set for the comparison.
$3.23 for 10,000 decisions
Jev’s median decision took 231 milliseconds, compared with 7.8 seconds for GPT-5 Mini. In the blinded review, Jev agreed with the human labels on 88 of 100 traces; GPT-5 Mini agreed on 85. NeoSigma says that three-point difference was not conclusive given the sample size.
The failure results favored Jev in that audit: it identified 25 of the 29 human-labeled failures, while GPT-5 Mini identified 17. On repeated judgments of the same 100 traces, Jev changed its verdict on three traces; GPT-5 Mini changed its verdict on 19. Against Tau2’s reward labels across all 834 traces, however, GPT-5 Mini scored slightly higher: 84.4% agreement to Jev’s 83.5%.
The cost comparison reflects the judges as configured in the experiment, including GPT-5 Mini’s generated rationale tokens—not just model input prices. NeoSigma projects that one million decisions at the measured rate would cost about $323 with Jev and $1,998 with GPT-5 Mini; those larger totals are linear projections, not runs the team performed. For 10,000 first-seen traces, the company reports a projected 9.6-fold cost gap, where replay caching has less effect.
A cheaper verdict still needs an escalation path
One airline trace shows why matching a final outcome to the initial request can misgrade an agent. The customer first asked to change a flight, then rejected a $396 upgrade and told the agent to leave the booking unchanged. The human reviewer and Jev marked the exchange a pass; GPT-5 Mini and Tau2’s reward marked it a failure.
NeoSigma’s recommendation is to reserve code for exact checks, use a low-cost decision model for bounded questions across more traces, and send uncertain or high-impact cases to a stronger model or a human. The experiment supports Jev as a candidate for that first-pass role, not as a general replacement for other judges: its human-agreement result was inconclusive, and its 98.99% repeatability fell just short of the team’s descriptive 99% target.