Skip to content
AI.info

The Pulse

MatrAIx opens proposals for long-horizon personal agent benchmark

Personal Agents' Last Exam will test AI agents on tasks that change as users' circumstances and permissions evolve. Proposals are open, but the project's evaluation infrastructure is still forthcoming.

MatrAIx opens proposals for long-horizon personal agent benchmark

MatrAIx has opened proposals for an open-source benchmark intended to test whether personal AI agents can handle tasks as users' circumstances, plans and permissions change. The project, called Personal Agents' Last Exam, is an open-source version of the company's PersonalAgentEval; its evaluation infrastructure is still forthcoming.

It was launched on November 15, with separate proposal tracks for environments and tasks. The earlier evaluation released its first version on October 14.

Tasks that change over time

The proposed benchmark is designed around tasks with verifiable outcomes in which an agent must reconcile a person's history with new evidence, coordinate across apps and people, and decide when to act, ask or wait. The project lists challenges including inferring unstated preferences, updating stale memories, respecting spending limits and withdrawn consent, and working through changing interfaces or expired sessions.

One illustrative case involves a request to reorder snacks: the agent would have to account for a daughter's allergy, a recent recall and a box already at daycare. In another, a replacement flight arrives at a different airport, requiring the agent to consider an existing car booking and an unanswered message from the traveler's partner.

The planned environments span areas including shopping, travel, health, work and household utilities. Personal apps such as mail, calendar and messages can connect workflows across those domains.

How contributions will work

Under the environment track, contributors can propose an app or website, its workflows and connections, along with a construction plan and quality checks. Task proposals instead describe a user persona, relevant history, a situation and the personalization challenge, as well as plans for scenarios, grading and model runs.

After a plan is approved and the infrastructure becomes available, contributors are to add implementation and evidence to the same pull request. Reviewers will compare the work with the approved scope and reproduce key results; only a validated merge counts as an accepted contribution.

For tasks, authors are expected to develop an executable scenario, timeline, context, rubrics and a reference solution. The contribution guide also asks them to plan user interactions and world events while allowing for multiple valid agent strategies.

Difficulty requirements

For implementation review, task authors must submit at least one independent run without hints for each designated model from the GPT, Claude and Gemini families, acting as a personal agent. The project recommends five runs per model and gives complete five-run evidence higher review priority.

Each designated model must fail every submitted unhinted run and receive a mean normalized rubric score below 50% for an accepted task. The threshold is below 30% for a flagship task. A separate hint-assisted run for each reference model must reach a 100% rubric score and pass every success gate.

The first infrastructure release will publish the initial model roster, according to the contribution guide.

Sources

Related stories

Explore

More articles