The Pulse

Google launches benchmark for visual knowledge-work agents

Google and Artificial Analysis introduced a benchmark for agents working across visually complex files. Google says Gemini 4 Carbon ranked second in quality among tested models and completed tasks at six times lower average cost than the to

AI.info Team · Written with AI · fact-checked · 2026-11-10

A test built around visual work

Google announced on Nov. 10 that it is launching the Multimodal Knowledge Agent benchmark with Artificial Analysis. The test evaluates whether AI agents can reason across visually complex files to complete knowledge-work tasks.

Most knowledge-work benchmarks can be solved with text alone, without needing to look at an image or page layout, Google says. Google points instead to professional tasks involving architectural blueprints, engineering schematics and the preparation of reports, presentations and spreadsheets.

Specialists from 10 work verticals and 83 occupations helped create MKA. The 200-task evaluation set was developed with professionals including architects, biologists, underwriters and data scientists.

Tasks and scoring

The benchmark covers business, construction, education, engineering, finance, healthcare, law and government, logistics and mapping, science, and software and data. Each task is a multi-step project that would take a seasoned professional three to eight hours to complete.

Each task combines a realistic prompt and a bundle of multimodal input files with an expert-written rubric. The rubrics contain roughly 70 yes-or-no checks, weighted 75% for factual accuracy and 25% for visual design.

Artificial Analysis runs each model through the same open-source agent harness, Stirrup, with the same tools, sandbox and turn budget. A grader built on a non-Google model assesses each deliverable, opening files, rendering pages, inspecting spreadsheet formulas and checking the work against the expert rubric.

Google reports that domain experts agreed with the automated grader 96% of the time when they assessed model outputs themselves. Artificial Analysis keeps the 200-task evaluation set private, Google says, to prevent the tasks from leaking into training data or being found by models through search.

A separate set of 10 public tasks is available on Hugging Face for people to inspect the format and test their own agents.

Results and access

Google's technical report covers the benchmark's methodology and results. The report says the tested models differed substantially in efficiency, using between 30 and 130 turns on average to complete each task; the report also says strong perception abilities are needed and describes models' unsuccessful attempts to compensate when vision is removed.

Among the models tested, Gemini 4 Carbon ranked second in quality and completed tasks at six times lower average cost than the top-scoring model, Google said. The model is available through the Gemini API in Google AI Studio.