CordisBench: Can Language Models Reason About Component Lifecycles in Dynamic Agent Harnesses?

AI summary

Overview

• Research area: Natural language processing; specifically the evaluation of large language models as reasoners about software component lifecycles in dynamic "agent harnesses" (arXiv:2609.01600v1 [cs.CL], 01 Sep 2026, by Damien Sileo and Dimitri Kachler, Univ. Lille / Inria / CNRS / Centrale Lille / CRIStAL).
• Technical level: Advanced. The paper assumes familiarity with LLM benchmarking, agent harnesses, dependency injection, lifecycle/teardown semantics, and formal reference semantics.
• Scope: The paper introduces and analyzes CordisBench, a 1,200-question benchmark measuring whether language models can predict the consequences of component lifecycle changes (localization, schedule prediction, guaranteed/reachable conditions, and executable reconfiguration) as the number of relevant interactions scales from 2 to 32.

What This Paper Is About

Dynamic agent harnesses let a language model modify the software that shapes its own execution, for example by adding, removing, or reconfiguring plugins, services, memory policies, and tools. That flexibility creates a reasoning burden: a single local plugin change can propagate through dependencies and trigger cleanup whose effects depend on what else is still running. CordisBench asks whether models can anticipate those consequences themselves, without symbolic assistance or execution feedback, and how reliably that reasoning holds up as more interactions become relevant.

Key Contributions

1. CordisBench itself: a 1,200-question structured-output benchmark derived from 240 independently generated systems, covering localization, schedule prediction, guaranteed conditions, reachable conditions, and (in the Cordis-native setting) reconfiguration.
2. A scaling design: within each task and setting, the benchmark increases the number of relevant interactions (2, 4, 8, 16, 24, or 32) while keeping question form, answer representation, and scoring fixed.
3. A Cordis-native setting checked against execution: model predictions are compared against runtime execution against Cordis 4.0.0-rc.7, and proposed reconfiguration sets are translated into Cordis dispose(...) operations and actually executed.
4. Task-specific metrics and diagnostics: Jaccard similarity for set-valued tasks, per-observable accuracy for prediction, and executed success for reconfiguration, plus shortcut controls (best control reaches only 7.3% whole-answer exact-match accuracy), an output-limit diagnostic, and a fixed-schedule diagnostic.

Main Findings

• Localization holds up better than consequence reasoning: performance on identifying affected components often remains strong as interaction count grows, while final-state prediction and condition reasoning across teardown orders become less reliable.
• GPT-5.6 Luna shows the clearest gap: localization stays near ceiling, but formal reachable-condition Jaccard falls from 91.7% to 14.1% and Cordis-native executed reconfiguration success falls from 62.5% to 25.0%. All of its primary responses parse, so the drops reflect incorrect answers rather than formatting failures.
• DeepSeek V4 Flash is weaker overall: its formal prediction accuracy falls from 81.2% to 57.7%. From size 8 onward it returns all labels on nearly every parsed condition question in both settings, and since each condition question names an equal number of true and false conditions, returning every label earns exactly 50% Jaccard. Its flat set-valued curves therefore reflect a return-all strategy rather than stable partial reasoning.
• Gemini 3.7 Flash remains strong on most Cordis-native tasks, but its output length matters: 29 of its responses reached the 8,192-token output limit, including 28 primary responses. Rerunning only those questions with a 32,768-token limit let all 29 complete and parse.
• The output limit explains only part of the decline: at size 32, formal guaranteed-condition Jaccard rises from 20.2% to 71.2%, reachable-condition Jaccard from 31.1% to 45.0%, and prediction accuracy from 79.8% to 84.0%. It does not explain the remaining reachable-condition decline from 100% at size 2 to 45.0% at size 32, nor the fully parsed GPT-5.6 Luna trends.
• Holding schedule count fixed does not remove the trend: in a separate 240-question diagnostic with exactly two schedules at every size, GPT-5.6 Luna parses every response while guaranteed-condition Jaccard falls from 81.2% at size 8 to 64.4% at size 32, and reachable-condition Jaccard from 91.1% to 69.7%.
• More inference helps, at a cost: on a balanced subset of 78 questions at 16 interactions (26 from each of three replicates), GPT-5.6 Luna goes from no reasoning to the default medium setting and Cordis-native prediction rises from 31.2% to 85.4%, while executed reconfiguration rises from 0% to 50%. At medium effort, GPT-5.6 Luna uses 2,967 reasoning tokens per question on average.
• An exact reference semantics exists: for all 528 Cordis-native questions, the finite reference semantics and Cordis execution agree on every observation and action outcome used for scoring, meaning the controlled lifecycle consequences can be computed deterministically.
• Execution separates failure modes in reconfiguration: Gemini 3.7 Flash succeeds on 92 of 96 questions, with two target failures and two malformed answers. GPT-5.6 Luna reaches the target on 67 of 96 questions but 11 of those plans use more prior disposals than necessary, leaving 56 minimal successes. DeepSeek V4 Flash reaches the target on 33 of 96 questions, but 32 of those plans are nonminimal, leaving one minimal success. The remaining answers include 44 target misses, 17 malformed responses, and two invalid actions. The gap between target attainment and benchmark success is 11.5 percentage points for GPT-5.6 Luna and 33.3 points for DeepSeek V4 Flash, driven by unnecessary prior disposals.
• Counting distinct outcomes remains difficult: accuracy is 26.4% for Gemini 3.7 Flash, 13.9% for GPT-5.6 Luna, and 4.2% for DeepSeek V4 Flash. In the Cordis-native setting, where controlled teardown orders are listed, Gemini answers 27 of 48 correctly, GPT-5.6 Luna 20 of 48, and DeepSeek V4 Flash 6 of 48. In the formal setting, GPT-5.6 Luna and DeepSeek V4 Flash answer none of 96, while Gemini answers 11.
• Strict exact match compounds with output length: primary-task parse rates are 95.5% for Gemini 3.7 Flash, 100.0% for GPT-5.6 Luna, and 93.8% for DeepSeek V4 Flash. GPT-5.6 Luna's Jaccard on formal set-valued tasks falls from 94.7% at size 2 to 50.5% at size 32, while its parsed per-observable prediction accuracy falls from 93.8% to 56.4%.

Methodology in Plain English

The authors built small, fully specified systems where components have dependencies, an initial application state, activation effects, and cleanup effects. The central pattern is the "restore-on-cleanup" effect: each plugin records the state value it sees when it starts and restores that value when it stops. Two such plugins can each behave sensibly alone but produce different final states depending on the order in which they stop, which the paper calls the teardown order.

Because each generated system is finite, the authors enumerate the legal lifecycle continuations that can affect the answer and execute each to completion, producing exact reference answers for localization, schedule prediction, guaranteed conditions, reachable conditions, and outcome counting. The benchmark exists in two forms: a controlled formal setting with a fixed-width vector of integers modulo m and short arithmetic effect programs, and a Cordis-native setting where the same lifecycle patterns are compiled into Cordis plugins and executed against Cordis 4.0.0-rc.7.

Scoring is deterministic and verifiable from structured outputs. The primary metrics are Jaccard similarity for localization and condition sets, per-observable accuracy for schedule prediction, and executed success for reconfiguration; missing or malformed outputs score zero. Three efficiency-oriented models (Gemini 3.7 Flash, GPT-5.6 Luna, and DeepSeek V4 Flash (0731)) were run at temperature zero with reasoning effort set to low and an 8,192-token output limit, with one completion per question, no tools, and no execution feedback. To control for correlated questions from the same generated system, confidence bands use percentile intervals from a cluster bootstrap over systems, and the benchmark was generated independently three times to yield three replicates per task and interaction count.

Why This Matters

Impact on research. The paper separates two capabilities that benchmarks often blur: identifying what a change can affect, and predicting what it will actually leave behind. It also provides a controlled scaling axis (relevant interactions from 2 to 32) and shows that exact symbolic reference semantics agree with runtime execution on all 528 executable questions, giving future work a verifiable target rather than a purely judgment-based one.

Real-world applications.

• Agent platforms where a model adds, removes, or reconfigures plugins, tools, or memory policies mid-run and must not corrupt the harness state.
• Dependency-injection and plugin runtimes that need to predict which dependents are removed and which cleanup handlers fire when a provider disappears.
• Deployment and infrastructure tooling where restoring a captured configuration value on shutdown can race with other components restoring their own captured values.
• Rollback and compensation logic in long-running workflows, where the order of undoing composed actions determines the resulting state.

Industry relevance. The paper's systems argument is that when dependencies and cleanup effects can be represented explicitly, harnesses should compute or verify their mechanical consequences directly instead of spending model inference on lifecycle management. That matters because lifecycle reasoning happens inside an agent run, so inference spent there adds latency and compute to the agent's primary task; the paper notes GPT-5.6 Luna uses nearly 3,000 reasoning tokens per question on the 16-interaction subset at medium effort for instances the finite semantics solves exactly.

Future Directions

• Extending the benchmark beyond the current restricted class to cover failures, irreversible external actions, hot module replacement, and other production concerns that the authors explicitly exclude.
• Testing whether tool access, execution feedback, or retries inside a full agent harness change the difficulty, since the current evaluation isolates lifecycle reasoning with no tools or execution feedback.
• Broadening model coverage: Gemini 3.7 Flash is near ceiling on several Cordis-native tasks and DeepSeek V4 Flash adopts a near return-all strategy on condition tasks at larger sizes, leaving GPT-5.6 Luna with the clearest dynamic range for several comparisons.
• Determining how far the derived schedule space, rather than final-state tracking alone, drives the difficulty of the outcome-count diagnostic, given the large gap between the Cordis-native and formal settings (27 of 48 versus 11 of 96 for Gemini 3.7 Flash).
• Designing harnesses so more lifecycle behavior is amenable to automatic computation or verification, which the authors suggest could improve both reliability and inference efficiency.

Target Audience

Researchers and engineers working on LLM evaluation, agent harness design, and dynamic software reconfiguration will benefit most. It is also relevant to practitioners building plugin or dependency-injection runtimes who need to reason about cleanup ordering, and to formal-methods researchers interested in comparing symbolic reference semantics against model predictions. Readers should be comfortable with benchmark design, structured-output scoring, and component lifecycle terminology.

Authors’ abstract

Dynamic agent harnesses let language models change the software that shapes their own execution. This flexibility brings a new reasoning burden: a local plugin change can propagate through dependencies and cleanup. We introduce CordisBench, a 1,200-question benchmark of this lifecycle reasoning. It combines a controlled formal setting with programs executed against Cordis, a runtime that manages component dependencies and cleanup, and asks models to identify affected components, predict state after a specified teardown order, determine which conditions hold under all or some orders, and choose reconfigurations that succeed when executed. Across these tasks, we evaluate three efficiency-oriented models at low reasoning effort with 2, 4, 8, 16, 24, or 32 relevant interactions, using deterministic task-specific scoring. Models usually handle small systems well but grow less reliable as more interactions become relevant, especially when predicting final state and when reasoning across teardown orders. Additional inference effort recovers marked gains for some models. The cost is nontrivial: on our 16-interaction subset, GPT-5.6 Luna uses nearly 3,000 reasoning tokens per question at medium effort. For these controlled instances, that cost is avoidable: an independent finite reference semantics agrees with Cordis execution on every observation and action outcome used for scoring across all 528 executable questions.