MMEnterprise-Bench: Evaluating Enterprise Agents over Evolving Multimodal Records

AI summary

Overview

Research area: Evaluation of LLM-based agents for enterprise question answering — specifically retrieval-augmented multimodal QA over workplace records that change over time.

Technical level: Intermediate. Readers should be comfortable with retrieval-augmented generation (RAG), multimodal LLMs, and standard QA accuracy metrics, but the paper's core ideas are explained without heavy formalism.

Scope (one sentence): The paper introduces MMEnterprise-Bench, a 1,613-question benchmark built on simulated, evolving, multimodal workplace records across 30 projects, and uses it to measure how well 10 proprietary and open-source multimodal LLMs reason over sufficient evidence, resolve cross-source and temporal conflicts, and recognize unanswerable questions.

What This Paper Is About

Enterprise agents answer questions over workspaces where information is spread across chats, emails, notes, documents, and figures, and where that information keeps being shared, corrected, and revised while old versions stay behind. This creates two hard problems: the same fact can exist in inconsistent versions across different sources (cross-source conflict), and a question may require the state of a record at a particular moment rather than its latest version (temporal conflict). The paper's goal is to build a benchmark that isolates exactly these difficulties and to measure how far current multimodal LLMs fall short on them.

Key Contributions

1. A new benchmark of 1,613 questions grounded in evolving multimodal workplace records. MMEnterprise-Bench simulates 30 collaboration projects seeded from public papers and reports (12 research papers and 18 government publications), yielding a retrieval corpus of 7,140 textual and visual records. Crucially, earlier records and document versions are retained rather than overwritten, producing 287 record versions and preserving superseded values as retrievable evidence.

2. A three-part capability taxonomy spanning eight task types. Questions are organized into Evidence Reasoning (1,102 questions, 68.3%; Extraction, Attribution, Computation, Comparison, Interpretation), Conflict Resolution (424 questions, 26.3%; Cross-source and Temporal), and Unanswerability (87 questions, 5.4%).

3. A large-scale evaluation of 10 enterprise agents. Four proprietary models (GPT-5.6 Sol, GPT-5.6 Luna, Gemini 3.8 Flash, Gemini 3.1 Pro) and six open-source models (Qwen3.5-27B/9B/4B/2B, InternVL3.5-8B, Kimi-VL-16B-A3B-Instruct) are tested in a RAG setting at k=10 and k=50 retrieved records, plus analyses at k=100 and with a larger retriever.

4. Evidence-grounded diagnosis of where models fail. Accuracy is broken down by the number, source type, and modality of gold evidence records, and a retrieval analysis separates "evidence was retrieved" from "evidence was used," showing that higher evidence recall does not reliably produce higher answer accuracy.

Main Findings

• Even the strongest model is far from reliable. GPT-5.6 Sol achieves the best overall accuracy of 52.6% with 10 retrieved records and 68.0% with 50. Gemini 3.8 Flash reaches 63.9% and Gemini 3.1 Pro 62.5% at k=50; the strongest open-source model, Qwen3.5-27B, reaches 50.3% at k=50 and 38.6% at k=10, a substantial gap from the top proprietary models.

• Conflict Resolution is the weakest category by a wide margin. At k=10, accuracy across models ranges from 11.8% to 36.3% on cross-source conflicts and from 13.2% to 33.5% on temporal conflicts. GPT-5.6 Sol itself gets 36.3% on cross-source and 32.5% on temporal questions at k=10, versus 67.5% on Extraction.

• Multiple visual records are especially hard. GPT-5.6 Sol scores 27.5% on questions requiring multiple images, compared with 83.3% on questions grounded in a single textual record.

• More evidence records make every model worse. At k=10 and with one required gold record, GPT-5.6 Sol reaches 83.3% on text-only questions and 52.9% on image-only questions; with two or more required records those drop to 51.5% and 27.5%, and mixed text-and-image evidence with 2+ records gives 25.6%.

• More retrieval helps strong models, not weak ones. Going from k=10 to k=50 lifts GPT-5.6 Sol from 52.6% to 68.0% and Qwen3.5-27B from 38.6% to 50.3%, whereas Qwen3.5-2B and Kimi-VL show little change.

• Higher evidence recall is not the bottleneck. Raising k from 50 to 100 with the 2B retriever raises recall from 57.0% to 64.8%, yet Qwen3.5-2B accuracy falls from 25.0% to 22.2%. With the 8B retriever, recall rises from 59.6% to 67.3% while InternVL3.5-8B drops from 38.0% to 37.6%. The larger 8B retriever gains 1.7%, 2.6%, and 2.5% recall at k=10, 50, and 100 respectively, but at k=100 GPT-5.6 Sol improves only from 75.9% to 77.2% while Qwen3.5-4B declines from 46.9% to 46.6%.

• Strong reasoning does not imply knowing when to abstain. On unanswerable questions at k=10, GPT-5.6 Sol scores 65.5%, below Gemini 3.8 Flash at 87.4%, Qwen3.5-27B at 75.9%, and Qwen3.5-9B at 73.6%.

• Qualitative failures are concrete. In one cross-source case, an interview count is corrected from 43 to 34 in a revised memo that also adds a row with value 23, while a Notion entry retains 32; GPT-5.6 Sol answers 34 instead of the requested 23. In a temporal case, a deadline revised from September 1 at 20:00 to September 2 at 18:00 causes the model to return the revised deadline when asked for the deadline immediately preceding the revision.

Methodology in Plain English

The researchers first picked 30 seed documents — public papers and government reports covering domains including artificial intelligence, healthcare, economics, cybersecurity, transportation, and environmental policy — and extracted their text, tables, and figures. Each seed became the technical backbone of a simulated "project." An LLM then planned a chronological chain of collaborative events for each project: someone shares a result, someone spots an error, someone issues a correction, a document gets revised. Those events were realized as realistic records — Slack messages, Notion entries, emails, meeting notes, PDF pages, slides, and standalone images — with team roles, communication channels, access relationships, and deadlines constraining who did what, when, and where.

The key design choice is that revisions never deleted the originals. Old values stayed in the workspace, so the same fact can appear inconsistently across sources and across time — exactly the conditions that generate cross-source and temporal conflicts. All projects share one workspace of 7,140 records (5,725 textual, 1,415 visual; about 2.66 million tokens) that serves as the retrieval corpus.

Questions were generated by a mix of rule-based templates (where the answer was directly derivable from structured project data) and LLM generation with task-specific instructions (for questions needing reasoning over record content and relationships). The LLM produced the question, reference answer, step-by-step solution, and supporting records together. Unanswerable questions were made by perturbing answerable ones — altering a meaningful entity or condition and keeping only candidates that could not be answered from the full project records.

Quality control ran in stages: LLM-based checks verified that records were consistent with source information and planned events, and that cited evidence was sufficient to derive the reference answer, flagging ambiguous questions, wrong answers, and answer leakage. For human validation, three annotators blind-reviewed 100 items (50 LLM-accepted, 50 LLM-rejected). None of the 50 LLM-accepted items was rejected by any annotator, and all 29 human-rejected items were also rejected by the LLM — 71% of items were rejected by no annotator and 29% by at least one.

For evaluation, Qwen3-VL-Embedding-2B retrieves the top-k records from the workspace, ordered by timestamp and ID, and these are given to the model as context. GPT-5.6 Luna serves as the automatic judge, given the question, reference answer, and model response.

Why This Matters

Impact on research. Existing enterprise QA benchmarks are largely text-only, and the ones that do introduce near-duplicates or correction chains generally accept only the up-to-date value as correct. MMEnterprise-Bench is, by the paper's comparison table, the only benchmark that simultaneously covers workplace settings, multimodal records, cross-source conflict, temporal conflict, unanswerability, and real-document grounding. It reframes "retrieval quality" as an insufficient explanation for enterprise agent failure and puts temporal state recovery on the evaluation agenda.

Real-world applications:

• Internal knowledge assistants that must answer "what is the current number?" versus "what was the number before the correction?" over wikis, chat, and shared drives.
• Compliance and audit tooling, where reconstructing the state of a record at a specific time is the actual requirement, not reading the latest revision.
• Financial and policy analysis over reports and spreadsheets that get re-issued, where superseded figures must not silently leak into answers.
• Abstention-aware copilots that flag when the workspace simply does not contain the evidence needed, rather than fabricating an answer — relevant to any deployment where wrong answers carry cost.

Industry relevance. The paper's most actionable signal for practitioners is negative: scaling the retriever or the number of retrieved chunks is not a reliable path to better answers. Organizations deploying enterprise RAG should expect the marginal return on retrieval investment to vary sharply with model strength, and should invest in conflict-aware reasoning and calibrated abstention instead.

Future Directions

• Teaching models to use evidence, not just find it. The decoupling between evidence recall and answer accuracy — recall rising while Qwen3.5-2B and InternVL3.5-8B accuracy falls — suggests a distinct "evidence utilization" capability that current RAG pipelines do not train for.
• Conflict-aware and time-aware retrieval. Retrieval currently uses an embedding model over a timestamp-ordered context. Whether query-time reasoning about record provenance, revision links, or explicit version graphs could close the large gap on Cross-source and Temporal questions is left open.
• Closing the unanswerability gap. The finding that the strongest overall model is not the best at abstaining raises the question of how to train or prompt for calibrated abstention without sacrificing answerable-question accuracy.
• Beyond simulation. The authors acknowledge that simulated records grounded in 30 public documents may not capture every aspect of real enterprise environments, and that LLM-based generation and validation may leave residual errors. Extending to real, messier corpora with genuine access controls is a natural next step.

Target Audience

Researchers and engineers working on enterprise RAG, agentic document QA, and multimodal LLM evaluation will get the most from this paper, particularly those designing benchmarks for temporal reasoning or conflict resolution. Product teams building workplace copilots will find the negative retrieval results and the abstention analysis directly relevant to design decisions. Readers new to the area can use the task taxonomy in Table 1 as a compact primer on what makes enterprise QA different from ordinary document QA.

Authors’ abstract

Enterprise agents answer questions over workplace records that evolve through team collaboration. As information is shared, corrected, and revised across documents, chats, and figures, multiple versions of the same information accumulate in the workspace. Reliable question answering therefore requires agents to reason over sufficient evidence, resolve cross-source and temporal conflicts, and recognize when the available evidence is insufficient to establish an answer. To systematically evaluate these capabilities, we introduce MMEnterprise-Bench, a benchmark comprising 1,613 questions grounded in evolving multimodal records from simulated collaboration projects, covering three categories: Evidence Reasoning, Conflict Resolution, and Unanswerability. We evaluate 10 proprietary and open-source multimodal LLMs and find that even the strongest models struggle when evidence contains cross-source or temporal conflicts or spans multiple images. Moreover, increasing the retrieval budget or employing a larger retriever improves evidence recall but does not consistently translate into higher answer accuracy. These findings highlight the persistent challenges of reasoning over evolving workplace records and establish MMEnterprise-Bench as a testbed for developing more reliable enterprise agents. Code and data are available at https://mlvlab.github.io/MMEnterprise-Bench/.