The Pulse
OpenAI Trains GPT-6 Astra on Ironclad Contract Workflows
OpenAI and Ironclad built 11 contracting tasks to train and evaluate OpenAI’s GPT-6 Astra agent. Astra scored 55% on the research evaluation, compared with 41.6% for GPT-5.6 Sol, but OpenAI says the results do not measure customer time savi
AI.info Team · Written with AI ·
OpenAI says GPT-6 Astra scored 55% on a set of 11 contract-workflow tasks, compared with 41.6% for GPT-5.6 Sol, in a research collaboration with contracting software company Ironclad announced October 6. Astra also completed attempts in an estimated average of 19.2 minutes, down from 37 minutes for Sol. OpenAI cautions that those times are simulations, not measured customer savings.
Eleven tasks built around contract work
The collaboration focuses on work that spans a company’s contracting software: setting up agreements, configuring approval paths and adjusting reusable legal terms. The tasks included creating nondisclosure agreements, building procurement approval processes and updating a clause based on a requester’s selected jurisdiction. OpenAI says an experienced user would take about 30 to 40 minutes per task, on average.
Each task was assessed against between 8 and 50 criteria, depending on its complexity. That let the researchers evaluate not just whether an agent completed individual steps, but whether the finished workflow followed the stated business rules. For example, a purchasing process might require Finance approval above a spending threshold, Security review for certain requests and Legal review of nonstandard terms.
Ironclad employees and people at OpenAI who use Ironclad helped identify the tasks. Ironclad also supplied hosted environments of its software for model practice. OpenAI says it created synthetic training tasks from contracts publicly available in the SEC’s EDGAR database, after filtering to remove personal information, and used reinforcement learning with practice and feedback.
A faster estimate, not a customer benchmark
OpenAI compared Astra using its Max reasoning setting with Sol using High reasoning, the settings where each model scored best. Across the 11 tasks, Astra averaged 55.0% against Sol’s 41.6%. The reported estimated time per attempt was 19.2 minutes for Astra and 37.0 minutes for Sol.
In two examples shown by OpenAI, Astra met about 94% of the criteria in an estimated 20 minutes; Sol met about 85% in an estimated 32 minutes. Those demonstrations illustrate performance on individual research tasks, not a separate test of deployed customer work. An internal model used during Astra’s development scored 63.7% on the same task set, OpenAI says, but the company did not say that score was available in Astra.
The company’s caveat matters: its timing figures are simulated estimates based on assumed model processing and generation speeds. OpenAI says the evaluation covers the 11 research tasks, not all Ironclad workflows. The results therefore show a gap on a small, purpose-built assessment—not how much time customers will save in daily operations.
What the data does—and does not—include
OpenAI says it did not use OpenAI customer data, its internal contracts or nonpublic Ironclad customer data or contracts for training or evaluation. The distinction is important to the collaboration’s claim: the tasks represent professional contracting work, but the training set was built synthetically from public contracts rather than drawn from private customer agreements.
Sunita Verma, Ironclad’s chief technology officer, described the target as broader than automating isolated clicks: For AI to be genuinely useful in complex areas of contracting, it has to do more than complete individual actions. Agents need to understand the full contracting lifecycle, including how business workflows connect while preserving the controls teams rely on. Our collaboration with OpenAI brings these real-world challenges into the research process and helps move the technology forward.
Ironclad brings controls into the evaluation
The work gives OpenAI a training and evaluation setting built around software where business rules govern how contracts move through an organization. A workflow can fail even when an agent performs each individual action correctly if it routes an exception to the wrong team or loses a rule partway through the process. OpenAI frames that as a reason human oversight and the contracting platform’s controls remain important.
OpenAI says it is seeking a small number of additional software-company partners to identify professional tasks that agents still cannot reliably complete. For those collaborations, it asks partners to bring concrete examples of failures, people with deep knowledge of the work, a secure testing environment and data that can safely support research. In the Ironclad evaluation, the measurable result is 55.0% across 11 tasks; the company’s own caveat is that its estimated completion times are not customer time savings.