CogArena: Benchmarking Multimodal Agents on Interactive Behavioral Experiments

AI summary

Overview

Research area: AI agent evaluation and cognitive science (multimodal browser agents, behavioral benchmarking).

Technical level: Intermediate. Readers need familiarity with statistical testing (proportion tests, correlations, bootstrap) and standard experimental paradigms, but the paper is written accessibly for both AI and psychology audiences.

Scope (1 sentence): The paper introduces CogArena, a ten-task benchmark in which AI agents complete classic behavioral experiments through a real web browser, scored at three levels: task completion, accuracy against human baselines, and reproduction of literature-established behavioral signatures.

What This Paper Is About

Claims about AI cognition are usually tested by turning behavioral experiments into text prompts, which strips away the visual stimuli, timing pressure, and interactive interfaces that human participants actually experience. Meanwhile, autonomous browser agents have become capable enough to enroll in online studies and submit plausible-looking data that current detection methods miss. CogArena closes both gaps by making agents take the experiments as a real online participant would, then checking whether their response data show the same statistical fingerprints that decades of human research has documented.

Key Contributions

1. A curated ten-task behavioral benchmark. CogArena v1 covers learning, risk, social/economic decision-making, memory recall, memory recognition, perception, judgment, foraging, compositionality, and moral decision-making. Each task is a standalone jsPsych v8 web application adapted from a vetted research codebase, with 16 to 200 trials and keypress, click, or typed response modalities.

2. A three-level scoring pipeline with fixed statistical tests. Level 1 checks completion with five binary checks; Level 2 compares accuracy to human baselines using normal-CDF scoring on 31 metrics across eight tasks; Level 3 tests 39 pre-specified behavioral signatures with fixed statistical tests (never a language model as judge). A composite 0–100 score uses weights (0.15, 0.35, 0.50).

3. A measured chance floor from a model-free Random Agent. A random-keypress agent was run ten times per task (100 sessions) under identical scoring to anchor interpretation of all three levels.

4. A preliminary four-model evaluation plus an open platform. 400 sessions (4 models × 10 tasks × 10 repeats) through a single scaffold, with tasks, scoring code, a reproduction recipe (release v1.3.0), and a public leaderboard for new submissions.

Main Findings

• Completion is not guaranteed. Only 167 of 400 sessions passed all five Level 1 checks, and 124 sessions had fewer than half their signatures testable, 83 of those with no testable signature at all. The model-free Random Agent completed every session it started (Level 1 = 1.000), while the best model, gemini-3-flash-preview, reached 0.936.

• Most failures were premature declarations of completion. Of the 233 sessions that failed a Level 1 check, 191 ended because the agent declared the task finished before the experiment ended; only 30 ran out of time, 1 ran out of steps, and 11 were stopped by the scaffold after ten failed actions in a row.

• Ranking depends on how sessions are averaged. Over all 100 sessions per model, gemini-3-flash-preview scored 58.6 composite, kimi-k2.5 44.2, gpt-5.4-nano 36.1, and qwen3-vl-235b-instruct 26.4. Over measurable sessions only, gemini scored 58.7 (n = 96), kimi-k2.5 50.7 (n = 76), qwen3-vl-235b-instruct 48.3 (n = 41), and gpt-5.4-nano 45.7 (n = 63) — qwen rises because it rarely finished what it started, and that failure is recorded at Level 1.

• Above-chance but uneven alignment. Over measurable sessions, mean Level 2 was 0.345 versus a chance floor of 0.159, and mean Level 3 was 0.490 versus 0.184.

• Some signatures reproduced almost always, others never. Above-chance phishing discrimination appeared in 90% of testable sessions, preference for saving more people in moral dilemmas in 100%, and cooperating after opponent cooperation in the prisoner's dilemma in 100%. Recognizing previously seen images was reproduced in 0 of 25 testable sessions, the travel-cost residence effect in foraging in 0 of 32, and faster responses at higher motion coherence in 0 of 21 (the authors note the single-screenshot scaffold is a likely cause).

• Finishing does not imply human-like data. Even among sessions passing every Level 1 check, only 40% of testable signatures were reproduced. Agents finished the grid bandit without choosing tiles their own reward history favored in 44% of sessions, and finished foraging without ever showing the travel-cost effect.

• Single sessions are unreliable. In 28 of 40 model–task cells, Level 3 was zero in some sessions and above zero in others; the median within-cell standard deviation of Level 3 was 0.20, and 60 of 139 model–signature combinations passed in some sessions and failed in others. kimi-k2.5 on marbles_risk had a composite of 63.2 ± 24.8.

• Capacities are harder than biases. Agents reproduced capacity signatures (patterns a normative model also predicts) in 31% of testable cases and bias signatures in 66% (36% versus 66% with response-time signatures set aside), meaning the headline Level 3 score is inflated by patterns that good performance does not require.

• Absolute response time is uninformative; relative speed is testable. Agents took up to ten times longer than people (about twelve times the median on random dot motion), largely because each action waits on a model call. The seven response-time signatures test within-session relations only, and agents passed them in 30 of 217 testable cases; response times did vary within sessions (median coefficient of variation 0.27 to 0.58).

• Screenshots matter. Re-running four tasks with screenshots removed so the agent saw only page text and element lists dropped composite scores by 31% to 53%, significantly on three tasks.

• A stronger model does better, but the evidence is thin. The state-of-the-art gpt-5.6-sol, run once per task, scored Level 1 of 0.960 versus 0.750 for the panel and Level 3 of 0.656 versus 0.490, but beat the best panel model on only four of the ten tasks.

• Rankings are robust to scoring choices. The all-session ranking held under three alternative weightings — (0.10, 0.30, 0.60), (0.20, 0.40, 0.40), and (0, 0, 1) — and the measurable-session ranking held under three alternatives to the {0, 0.5, 1} Level 3 grade, shifting the agent–floor gap by at most 0.021.

Methodology in Plain English

The authors rebuilt ten classic psychology and economics experiments as real websites, using the same browser framework (jsPsych v8) that online labs use, complete with instruction screens, practice trials, and main trial blocks. An AI agent opens the page in a browser and plays the part of a participant: it sees rendered screenshots, clicks buttons, and presses keys. The benchmark supplies the experiments and the scoring, but not the agent — whoever submits supplies the "scaffold" that translates model output into clicks and keypresses.

Scoring happens on three rungs. The first asks whether the agent actually got through the experiment, checking that data arrived, at least 80% of trials were completed, timeouts stayed under 20%, inputs were valid, and the experiment reached its end. The second compares task-specific accuracy metrics to the original studies' human participants, placing the agent on a normal curve around the human mean so 0.5 means "average participant." The third is the distinctive one: for each paradigm, the authors pre-specified statistical tests for signatures the human literature reliably finds — like tit-for-tat reciprocity, primacy in serial recall, or coherence-dependent accuracy — and graded each as 1.0 if significant in the expected direction, 0.5 if directionally right but not significant, 0.0 otherwise. Signatures that could not be tested (because a session ended early or responses never varied) were excluded rather than scored zero, and a session counted as "measurable" only if at least half the task's signature weight was testable.

Four economy-tier multimodal models ran through the same Browser-Use scaffold, ten sessions per model–task cell with a fresh random seed each time, for 400 sessions. All model steps used screenshots. Sessions were capped at 1800 seconds and 500 steps per task, and per-trial deadlines were switched off so a slow model would not be penalized as a missed trial. A random-keypress agent provided the chance baseline.

Why This Matters

The paper separates two things that benchmark design usually conflates: an agent getting to the end of a task, and an agent producing data that behaves like a human's. That distinction matters for anyone who uses behavioral data as evidence — either as a measure of machine intelligence or as a scientific resource collected online.

Real-world applications:

• Data integrity in online behavioral research. The results indicate which signatures (preferences especially) current agents reproduce reliably and therefore cannot be used to screen out bots, and which ones (capacities and response-time relations) may still separate people from agents.
• Model and vendor evaluation. Teams choosing a browser agent for form-filling, survey completion, or multi-step web workflows get a task suite that tests sustained multi-trial interaction under deadlines, not just single-page success.
• Cognitive-science benchmarking of AI. Researchers asking whether models behave like people get pre-registered statistical tests rather than ad hoc prompt comparisons.
• Human-factors testing of experiment platforms. Running agents through live study interfaces surfaces broken flows and premature-termination behavior before human participants encounter them.

Industry relevance: browser-agent products are increasingly marketed for autonomous web task completion. CogArena's finding that 191 of 233 failures came from the agent declaring a task finished when it was not is a directly actionable failure mode for anyone shipping these systems, and the benchmark–harness separation lets vendors submit their own scaffold without changing the measurement.

Future Directions

• Collect human data on the platform itself. No human participants have completed the CogArena versions of the tasks; 24 of the 31 Level 2 baselines come from the original studies' released participant data rather than from these implementations. The authors call this their first priority, noting the platform already serves the same experiments to human visitors.
• Broaden the model sample. All four panel models are economy-tier, the state-of-the-art model ran only once per task, and every model used the same Browser-Use scaffold, so results describe model-plus-scaffold combinations rather than models alone.
• Expand and rotate the signature set. The current task set does not test every known phenomenon (the grid bandit, for instance, fixes the number of remaining choices, so exploration under varying choice counts cannot be tested). The authors point to a forty-task catalog already hosted and to per-task signature additions.
• Improve agents on the distinctive failures. The motion-coherence response-time signature (0 of 21) is attributed partly to the scaffold, since a single screenshot carries no motion — an open problem for how browser agents should be given temporally extended visual input.

Target Audience

AI agent and benchmark researchers who need evaluation tasks beyond end-state web success; behavioral and social scientists who collect data online and need to know which measures remain robust against autonomous agents; cognitive scientists studying machine cognition who want literature-grounded statistical tests rather than prompt-level accuracy; and product teams building or procuring browser-automation agents who want to know where these systems silently declare victory.

Authors’ abstract

Claims about the cognitive capabilities of large language models are typically tested by translating behavioral paradigms into text prompts, bypassing the visual interfaces and time pressure through which human participants actually engage with experiments. At the same time, AI agents can now complete online behavioral tasks and produce human-like data, which raises concerns that online datasets in the social and behavioral sciences are increasingly open to contamination. We introduce CogArena, a benchmark of ten experimental paradigms from the social and behavioral sciences. Agents complete the experiments through a standard web browser, working with the same visual stimuli and response interfaces that online participants use. A three-level scoring procedure evaluates whether the agent completed the task, how accurately it performed, and whether its data show the behavioral signatures each paradigm reliably produces in people. Across four multimodal models, each run ten times per task, we find that finishing an experiment and reproducing its signatures are different things. Many sessions did not finish, and those that did reproduced some signatures reliably and others not at all. CogArena provides the tasks, scoring code, and a public leaderboard for new submissions.