The Pulse
MatrAIx opens proposals for long-horizon personal agent benchmark
Personal Agents' Last Exam will test AI agents on tasks that change as users' circumstances and permissions evolve. Proposals are open, but the project's evaluation infrastructure is still forthcoming.

MatrAIx has opened proposals for an open-source benchmark intended to test whether personal AI agents can handle tasks as users' circumstances, plans and permissions change. The project, called Personal Agents' Last Exam, is an open-source version of the company's PersonalAgentEval; its evaluation infrastructure is still forthcoming.
It was launched on November 15, with separate proposal tracks for environments and tasks. The earlier evaluation released its first version on October 14.
Tasks that change over time
The proposed benchmark is designed around tasks with verifiable outcomes in which an agent must reconcile a person's history with new evidence, coordinate across apps and people, and decide when to act, ask or wait. The project lists challenges including inferring unstated preferences, updating stale memories, respecting spending limits and withdrawn consent, and working through changing interfaces or expired sessions.
One illustrative case involves a request to reorder snacks: the agent would have to account for a daughter's allergy, a recent recall and a box already at daycare. In another, a replacement flight arrives at a different airport, requiring the agent to consider an existing car booking and an unanswered message from the traveler's partner.
The planned environments span areas including shopping, travel, health, work and household utilities. Personal apps such as mail, calendar and messages can connect workflows across those domains.
How contributions will work
Under the environment track, contributors can propose an app or website, its workflows and connections, along with a construction plan and quality checks. Task proposals instead describe a user persona, relevant history, a situation and the personalization challenge, as well as plans for scenarios, grading and model runs.
After a plan is approved and the infrastructure becomes available, contributors are to add implementation and evidence to the same pull request. Reviewers will compare the work with the approved scope and reproduce key results; only a validated merge counts as an accepted contribution.
For tasks, authors are expected to develop an executable scenario, timeline, context, rubrics and a reference solution. The contribution guide also asks them to plan user interactions and world events while allowing for multiple valid agent strategies.
Difficulty requirements
For implementation review, task authors must submit at least one independent run without hints for each designated model from the GPT, Claude and Gemini families, acting as a personal agent. The project recommends five runs per model and gives complete five-run evidence higher review priority.
Each designated model must fail every submitted unhinted run and receive a mean normalized rubric score below 50% for an accepted task. The threshold is below 30% for a flagship task. A separate hint-assisted run for each reference model must reach a 100% rubric score and pass every success gate.
The first infrastructure release will publish the initial model roster, according to the contribution guide.
Sources
Related stories
- LiveGeo tests find AI agents struggle as geographic conditions change
- Cloudflare open-sources experimental Kitesurf browser for AI agents
- Pine AI releases open-source filmmaking studio and adds phone calls
- KeyBanc finds faster growth but weaker margins at AI-first firms
- LangChain puts agent records at the center of LangSmith