The Pulse
Artificial Analysis Puts Frontier Models on a Real Robot Arm
Artificial Analysis launched AA-Robotics v0.1, a benchmark that tests five frontier models on eight tasks using a physical robot arm. GPT-6 Astra led with a mean score of 69.4, but took about 30 minutes per task against a 20-second human av
AI.info Team · Written with AI ·
Five frontier models scored between 46.3 and 69.4 on a new physical-robot benchmark, while taking about 30 minutes per task that a human completed in an average of 20 seconds. Artificial Analysis published AA-Robotics v0.1 on October 14, testing models as they directly controlled a real six-axis robot arm rather than responding to prompts in a simulated environment.
GPT-6 Astra placed first with a mean score of 69.4 out of 100, followed by GPT-6.1 Sol at 64.6, Gemini 4 Argon at 55.3, Claude Opus 5.5 at 48.8 and Claude Fable 5.1 at 46.3. The results offer a first comparison of how general-purpose language models handle physical tasks, while the long run times and uneven performance show how far the tested systems are from practical, fast manipulation.
Eight tasks, one physical arm
The benchmark gives each model access to a Hiwonder NexArm through a simple HTTP API. Models run in isolated containers, can write and execute code to plan movements, and inspect the workspace through four cameras: three around the table and one mounted on the arm’s wrist. Each attempt is zero-shot, and the final state is graded against the task’s stated pass criteria.
The eight tasks range from pressing a red button and moving a toy crocodile into a marked rectangle to pouring a drink into a cup, sorting blocks into a box, matching shapes, writing “hello world,” building a tower and measuring a box. A fully completed task earns 100 points; partial completion receives credit for progress. GPT models ran in Codex, Claude models in Claude Code and Gemini 4 Argon in Antigravity, all at the high-effort setting.
GPT-6 Astra leads, but not by much
The scores reflect sharply different strengths across tasks. All five models fully completed the button press and crocodile placement. On the more demanding measure-the-box task, none earned a full score: GPT-6 Astra and Gemini 4 Argon scored zero, while GPT-6.1 Sol and Claude Opus 5.5 received 17 points each; Claude Fable 5.1 scored zero.
Artificial Analysis writes: “We find that frontier language models in coding agent harnesses have indeed recently become capable of directly manipulating robot arms for simple tasks.” The benchmark’s examples also show that completing a task once does not always mean knowing when to stop: a model could reach a passing state, continue adjusting an object, then undo its progress.
Time and cost remain the larger gap
Artificial Analysis reports an average human completion time of 20 seconds per task, compared with roughly half an hour for four of the five models. Gemini 4 Argon averaged 52 minutes. The organization describes the models as roughly 100 times slower than the human baseline, though the average masks large differences between tasks and between individual runs.
GPT-6 Astra took about 30 minutes and cost an average of $16.36 per task. GPT-6.1 Sol cost $1.93 per task, more than eight times less, while its mean score was only about five points lower. The company says 70% of model spending went to input context, reflecting the volume of visual information the systems processed.
Grasping still fails more often than it works
Across six tasks involving grasping, the best-performing models successfully held the intended object in about 45% of attempts: GPT-6 Astra reached 45%, and Claude Fable 5.1 reached 44%. Claude Opus 5.5 managed 40%, Gemini 4 Argon 36% and GPT-6.1 Sol 28%. The benchmark counts a grasp as successful only when the intended object stays held through a lift or carry; misses, drops and wrong-object grasps do not qualify.
Those results make the benchmark’s practical limits concrete. The models can perform simple actions with a physical arm, but they often miss objects, disturb nearby pieces or keep acting after a task is already complete. In this first release, the clearest measure of that gap is still the clock: about 30 minutes for a model, against 20 seconds for a person.