The Pulse
ServiceNow’s AutoSynthData Turns Agent Failures Into Training Tasks
ServiceNow’s AutoSynthData pipeline uses agent failures and a stronger teacher model to generate and verify enterprise training tasks. In tests on EnterpriseOps Gym, fine-tuning improved mean Pass@1 by 7.2 percentage points in Hybrid and fr
AI.info Team · Written with AI ·
ITSM Pass@1 rises from 18.77% to 27.18%
ServiceNow says its AutoSynthData pipeline raised a target model’s mean Pass@1 score from 18.77% to 27.18% on EnterpriseOps Gym’s ITSM tasks. The system creates training examples around what an agent fails to do, then checks those examples in the environment before using them for fine-tuning.
The company published the method on October 2, 2026, describing two tests in EnterpriseOps Gym, a stateful environment for enterprise tasks. On the Hybrid domain, the best fine-tuned checkpoint improved mean Pass@1 by 7.2 percentage points, a 35% relative gain. The results demonstrate improvement in those benchmark settings, not performance in customer deployments.
Failures guide what the model learns next
AutoSynthData evaluates a target model alongside a stronger teacher. The system examines where the target fails and where the teacher succeeds, then distills those patterns into capability specification cards. The cards guide generation, but the generator does not receive the original evaluation prompts, entities, trajectories or verifier details.
Each training task combines a system specification, a user prompt and a verifier. Generated prompts must be feasible in the environment, resemble plausible requests and challenge a capability the current model has not mastered. The teacher supplies a successful trajectory for each task, which can then serve as a supervised fine-tuning example.
“At ServiceNow CoreAI, we built AutoSynthData to turn those capability gaps into training data.”
Esakkivel Esakkiraja, Shruthan Radhakrishna, Denis Akhiyarov and Sagar Davasam, ServiceNow-AI
Every sample must pass two checks
ServiceNow’s pipeline checks whether a task is hard enough to teach. In the reported configuration, it favors tasks the target model solves on no more than one of three trials and the stronger solver handles on at least two of three. The authors say that screen aims to select work that is both learnable and challenging.
A positive verification gate replays the reference trajectory in the target environment to confirm it reaches the required state. A negative gate tests whether relevant incorrect outcomes fail; one example mutates the expected outcome and checks that the verifier no longer accepts it. Failed candidates go to a critic for diagnosis and possible repair, while a batch-level review looks for repetitive examples and missing capability areas.
Two domains, different generation costs
For Hybrid, ServiceNow used Gemma-4-26B-A4B-it as the target and Qwen3.8-27B as the teacher. AutoSynthData generated 2,000 samples in about 18 hours. The best checkpoint, at epoch five, raised verifier success from 63.01% to 68.55% and closed 59% of the original Pass@1 gap between Gemma and the reference model.
For ITSM, the target remained Gemma-4-26B-A4B-it, while DeepSeek-V4.1-Flash served as teacher. The pipeline generated 1,994 samples in 66 hours. ServiceNow attributes the longer run mainly to the larger teacher model and says the ITSM experiment preceded later throughput improvements to the pipeline.
The benchmark leaves deployment questions open
The method treats training data as a moving target: after fine-tuning, the model is evaluated again, and remaining failures can guide another generation round. ServiceNow says the experiments focus on supervised fine-tuning and proposes testing the same difficulty-calibrated process for reinforcement learning. That extension has not yet been tested in the post.
The reported gains come from EnterpriseOps Gym’s Hybrid and ITSM domains. The post does not establish whether they transfer to other enterprise environments, or how the generation and verification costs compare with alternative ways of building agent-training data.