The Pulse
Baidu presents ERNIE Evolve for autonomous AI research
ERNIE Evolve can select research directions, modify code and run experiments autonomously, Baidu says. Baidu reports benchmark gains and performance improvements in three inference tasks for its in-development ERNIE-5.5-Flash.
AI.info Team · Written with AI · fact-checked ·
A system for iterative AI R&D
Baidu says ERNIE Evolve is its first step toward an automated research system for artificial intelligence research and development. The system is designed to work toward explicit objectives under defined constraints, handling both the choice of research direction and the development of implementations through experiments.
Given a code repository, an optimization objective and evaluation criteria, ERNIE Evolve can change code and run experiments autonomously in isolated environments. Its architecture has two nested loops: one selects what to investigate, while the other develops and tests changes within each line of research.
The outer loop assigns proposed directions to parallel research lanes, each with a tree of candidate code versions. Within a lane, the system selects a candidate, specifies a hypothesis, makes a code change and submits the result for evaluation. An evaluator measures performance and checks correctness on real workloads, while a reviewer examines the code changes and results. Only improvements that pass evaluation and review are retained.
Results on public tasks
Baidu tested the system on NanoChat AutoResearch, NanoGPT Speedrun and SOL-ExecBench, tasks covering training algorithms, training speed and GPU kernel optimization. ERNIE Evolve exceeded the best publicly verified results it compared against on NanoChat AutoResearch and NanoGPT Speedrun, Baidu says, but its SOL-ExecBench score was close to, and below, the listed leading result.
On NanoChat AutoResearch, Baidu reports a validation BPB score of 0.8763, against a best public result of 0.8878. On NanoGPT Speedrun Track 1, it reports reaching the target loss in 38.9 seconds, compared with 39.9 seconds for the best public result; on Track 3, it reports 2,590 steps, against 2,690. For SOL-ExecBench, its mean score was 0.819, while the best public result shown was 0.827.
The system used the same research loop across the three tasks, without adapting it to a particular metric or code structure. The optimized results have been open-sourced.
Tests on inference workloads
Baidu also applied ERNIE Evolve to inference optimization for the in-development ERNIE-5.5-Flash. The changes had to pass the same load tests and correctness checks as engineering changes, and work within specified modules and existing accuracy and latency requirements.
For low-latency inference on Kunlunxin P800, four rounds of research raised generation speed to 623.6% of the baseline, Baidu reports, with all correctness checks passed. The work included enabling optimized execution paths and changing host-side scheduling and synchronization. One alternative—replacing specialized communication with general-purpose collectives—reduced speed by 14.0% and was dropped.
For deterministic inference, Baidu reports a 33.7% increase in throughput, a 26% reduction in average time per output token and a 12% reduction in time to first token, with determinism preserved. Requests had to generate identical tokens and log probabilities regardless of which other requests shared their batch, Baidu says.
In a separate GPU task focused on decoding speed, ERNIE Evolve increased speed by a further 14.9%, with accuracy unchanged from the baseline, Baidu says. The deployment work is described by Baidu as its first step toward incorporating automated research into the team’s day-to-day engineering.