The Pulse
STRATA Benchmark Contains 8,735 Clinical Timeline Hypotheses
A paper presents STRATA, a benchmark that scores the plans, evidence and verdicts behind statistical tests of temporal claims about vaccine adverse-event reports. Its 8,735-hypothesis pool includes a held-out test set of 1,500, and experime
AI.info Team · Written with AI · fact-checked ·
STRATA contains 8,735 hypotheses about patterns in clinical timelines, including a held-out test set of 1,500. The benchmark evaluates whether AI systems can identify the statistical test and the relevant reports for a claim, as well as reach a verdict. A system can return the right answer for the wrong evidentiary reason, so STRATA scores the decisions behind its conclusion, too.
Chengyang He, Wenlong Zhang, Koduvayur Subbalakshmi and Ping Wang of Stevens Institute of Technology describe STRATA and a staged method called ART in a paper submitted to arXiv on October 29, 2026. The work concerns temporal patterns in vaccine adverse-event reports, such as whether one event tends to precede another. The paper’s authors write:
Extensive experiments with five model configurations show that systems fail mostly at choosing which records count, a verdict–evidence gap that accuracy alone conceals.
Chengyang He, Wenlong Zhang, Koduvayur Subbalakshmi and Ping Wang, authors of the paper
STRATA checks the evidence behind a verdict
Many data-analysis evaluations provide the relevant records and score a model’s final answer. STRATA instead asks a system to identify a claim’s null hypothesis, choose a test and retrieve the records that belong in the calculation. A correct verdict based on the wrong records does not count as fully correct.
The benchmark uses 3,166 clinical timelines drawn from adverse-event reports submitted to the U.S. Vaccine Adverse Event Reporting System. The reports cover three COVID-19 vaccines: Janssen, Pfizer and Moderna. STRATA’s full pool contains 8,735 hypotheses in four wording registers; its held-out test set contains 1,500 hypotheses built around 861 distinct facts.
Claims address patterns including event order, co-occurrence, proximity and whether an event appears early or late in a timeline. Each case is labeled supported, refuted or insufficient. Insufficient cases are further classified as lacking enough reports, having no qualifying reports, or producing an inconclusive test. A deterministic procedure calculates the reference evidence sets and verdicts.
ART separates planning, retrieval and testing
ART stands for articulate, retrieve, test. First, a model turns a claim into a plan that specifies the comparison, null hypothesis, statistical test and rule for selecting reports. Next, it writes a database query to retrieve evidence. A deterministic process runs the prescribed test and returns the verdict, rather than leaving the final calculation to the model.
The researchers compare five model configurations and several baselines. With ART, the 31-billion-parameter Gemma 4 configuration reaches 0.981 verdict accuracy; Claude Opus 5.5 reaches 0.991, while GPT-5.6 also scores 0.981. The authors report that the open model comes within 0.01 of frontier models under the same staged setup.
In an ablation, removing ART’s deterministic testing stage reduces verdict accuracy for the tested configurations. The paper also identifies errors in choosing which reports count as a major source of failure. One example shows how a model can give the right verdict while counting only reports that support the claim and omitting the rest of the comparison group.
The reports do not establish vaccine causation
STRATA tests whether systems can analyze patterns in the reports; it does not test whether a vaccine caused an adverse event. The paper says supported verdicts describe regularities in what was reported to VAERS, not causal effects.
The benchmark’s hypotheses are generated from templates, and the system is given a codebook of analysis rules while the mapping from claim wording to analysis plan is withheld. The authors identify clinician-written claims and held-out templates as future tests of whether systems can apply those rules to wording not produced by the templates. They say the full release, including code, follows acceptance.