The Pulse
Emergence Reports 19.16% on VeriContest’s Verified-Code Test
Emergence says its Craft system passed both proof checks and hidden tests on 19.16% of problems in VeriContest’s LeetCode set. The company also reports 100% on VERINA and up to 100% on an evaluated VeriSoftBench subset using Lean 4.
AI.info Team · Written with AI ·
Emergence said on October 6 that its Craft platform passed both formal-proof checks and hidden tests on 19.16% of problems in the LeetCode set used by the VeriContest benchmark. The company also reported near-perfect results on two Lean 4 proof-generation evaluations, but the figures come from company-run tests and do not establish how the system performs on production software.
Two checks, not just working code
VeriContest evaluates whether generated code both conforms to a formal specification, as checked by a proof, and passes hidden tests that compare its output with expected results. Emergence’s Verified Code Generation component cleared both requirements on 19.16% of problems, the company said. It listed Claude Opus 4.7 at 15.36%, GPT-5.5 at 7.10%, Claude Sonnet 4.6 at 6.67% and Gemini 3.1 Pro at 4.78% on the same measure.
Those percentages describe the combined proof-and-test task, not ordinary code-generation pass rates. The distinction matters: code that passes a benchmark’s tests has not necessarily been proved correct, while a proof establishes correctness only against the formal specification it receives. Emergence’s earlier VCG write-up makes a separate comparison on functional benchmark scores and says its system trails an unverified Claude Opus 4.7 baseline on several tests; the company attributes the gap partly to specification drift and possible benchmark familiarity.
Lean 4 checks the proof itself
Emergence also tested its VCG/Tiny Prover system on VERINA and a selected portion of VeriSoftBench, where Lean 4 checks whether each proof follows its formal rules. The company reported that the system solved all VERINA problems on its first attempt. On the evaluated VeriSoftBench subset, it solved 99% on the first attempt and 100% after a second.
The company’s September 14 technical post specifies that its VeriSoftBench evaluation used the 100-proof Aristotle subset, rather than the full repository-scale benchmark. It describes VERINA as 189 tasks, with the system evaluated on code generation and proof generation against a supplied specification. Those evaluation boundaries make the scores more informative than an unqualified claim of perfect software verification.
A proof can still formalize the wrong requirement
Lean 4 can determine whether an implementation satisfies a mathematical statement. It cannot determine whether that statement captures what a user actually intended. Emergence’s researchers acknowledge that boundary directly: In the benchmark setting we study here, a formal specification is provided as the source of truth. Nghia Nguyen, Aditya Vempaty, Ashish Jagmohan, Prasenjit Dey and Ravi Kokku, Emergence
That caveat sets the scope of the results. A machine-checked proof can rule out certain implementation errors relative to a specification, but it does not certify that the specification is complete, faithful to business intent or appropriate for a real deployment. Emergence’s separate VCG account likewise identifies specification fidelity as a central reliability problem.
Company benchmarks, not deployment evidence
Emergence presents the scores as evidence for its “Neuroformal AI” approach: neural models generate candidate code and proofs, while formal tools check whether stated requirements hold. Its release does not describe an independent reproduction of the evaluations, nor does it show that the benchmark scores translate into fewer failures in customer systems.
The results therefore mark progress on constrained tasks, not a general guarantee for autonomous software work. For the headline VeriContest result, the concrete threshold is demanding: Craft passed both the formal proof and hidden tests on 19.16% of the LeetCode-set problems.