The Pulse
LiveGeo tests find AI agents struggle as geographic conditions change
A new benchmark tests whether AI agents can revise geographic plans as simulated city conditions change. Researchers found that performance fell sharply in dynamic settings compared with matched static tasks.

AI agents performed markedly worse on geographic tasks when simulated city conditions changed: mean success fell from 19.1% on matched static tests to 5.7% and 4.0% in two dynamic settings, relative declines of 70.3% and 79.3%, researchers report in a new preprint. The researchers introduced LiveGeo, a benchmark for testing whether agents can check old information and revise plans as their surroundings evolve, on November 12.
The authors evaluated nine proprietary and open-source models using the same agent framework and tool-access protocol. They found that strong performance on fixed tasks did not necessarily translate into reliable decisions when the simulated world changed.
A city that changes over time
LiveGeo runs on GeoWorld, a simulation platform grounded in real urban geography. The platform maintains road networks, information about places and regional events, while allowing external conditions to change on their own timelines and keeping records of world states.
The simulated world draws on six real cities, with 88 regions, 1,376 points of interest and 107,372 cached routes. Its cities are Shanghai, Tokyo, Singapore, London, New York and Cape Town.
LiveGeo contains 1,237 verifiable task instances across seven categories and gives agents access to 31 types of tools. The released repository describes the set as 1,088 static tasks and 149 matched dynamic tasks.
From finding places to changing plans
The atomic tasks cover identifying a place from visual or textual clues, finding a destination that meets requirements, and deciding whether a journey is feasible. More complex tasks require agents to coordinate people with different starting points and schedules, plan sequences of activities, account for risks, or apply local geographic context.
Matched dynamic tasks start with the same request and initial world state as their static counterparts. In the static setting, task-relevant conditions stay fixed; in task-only dynamics, events written for the task take effect as virtual time advances. Shared dynamics also introduces other city events on the same timeline, requiring the agent to determine which changes matter to its plan.
Results and failure modes
The strongest agent reached 38.0% pass@1 on the paper’s place-grounding and place-discovery tasks, according to the researchers. Performance dropped further on compositional tasks, which require agents to satisfy multiple geographic constraints together.
For example, Claude Sonnet 5 scored 22.8% on the static baseline of the dynamic evaluation, falling to 2.0% under shared dynamics and 2.7% under task-only dynamics. Gemini 3.7 Flash scored 26.2% on the static baseline, 13.4% under shared dynamics and 14.1% under task-only dynamics.
The authors identified unsupported geographic claims in static settings and failures to monitor changes in dynamic settings as major bottlenecks. Agents often failed to obtain updated information after relevant world changes, the paper says.
Among runs that observed every necessary event, 36.4% under task-only dynamics and 38.6% under shared dynamics correctly reported the relevant changes but still reached incorrect decisions. Their preprint presents the findings as an evaluation in simulated geographic environments, rather than a test of agents operating in actual cities.
Available research tools
The GeoWorld repository includes a bundled sample city that can be opened in World Studio without API credentials. It also describes how researchers can run the benchmark and build task generators using the platform’s cities, tools and event replay.
The sample contains recorded places, routes and service records from London. Recorded task examples and model results can also be inspected without model credentials.
Sources
Related stories
- Ai2’s XYBench tests whether AI answers address users’ real goals
- MatrAIx opens proposals for long-horizon personal agent benchmark
- New risk benchmark finds newer AI models tend toward neutrality
- Meshy says print-preparation tool reduced defects in 3D test
- KeyBanc finds faster growth but weaker margins at AI-first firms