MITRA: An AI Assistant for Knowledge Retrieval in Physics Collaborations

AI summary

Overview

Research area: Information retrieval and applied natural language processing — specifically Retrieval-Augmented Generation (RAG) for internal knowledge management in a large high-energy physics collaboration (cs.IR).

Technical level: Intermediate. The paper assumes familiarity with RAG concepts such as embeddings, vector databases, reranking, and large language model quantization, but explains each design decision in accessible terms.

Scope: The paper presents and evaluates a prototype on-premise RAG assistant, MITRA, that answers natural-language questions by retrieving text from internal CMS analysis documentation, comparing its retrieval performance against an Okapi BM25 keyword baseline.

What This Paper Is About

Large scientific collaborations such as the Compact Muon Solenoid (CMS) at CERN accumulate enormous volumes of internal documentation — analysis notes, internal wikis, procedural guidelines — that traditional keyword search handles poorly because it depends on exact phrasing matches and misses semantic context. MITRA is a prototype conversational assistant that retrieves and reranks relevant passages from that documentation so that both new and experienced researchers can get cited, context-aware answers about specific physics analyses. The paper's central goal is to show that a fully on-premise RAG pipeline can outperform keyword search on realistic, paraphrased queries while keeping proprietary collaboration data private.

Key Contributions

1. A modular, automated ingestion pipeline for pulling internal analysis documents from web-interface databases and extracting high-fidelity text, designed to support extensibility and version-aware updates.
2. A two-tiered database design that first identifies the relevant analysis from an abstracts database and then locks onto a full-text database for that single analysis, preventing context confusion between different analyses.
3. A fully on-premise deployment model in which the embedding model and the LLM both run on local collaboration GPU hardware, guaranteeing that no proprietary research data leaves the collaboration's secure network and avoiding ongoing per-token API costs.
4. A retrieval benchmark against Okapi BM25 on two expert-designed query sets, showing MITRA's advantage on realistic, paraphrased queries and reporting rank-aware metrics (MRR, NDCG) in addition to P@k and R@k.

Main Findings

• Strong performance on exact-phrasing queries for both systems: On Set 1 queries (which use the exact phrasing found in the source documents), BM25 and MITRA both reached P@1 of 1.00, R@1 of 0.85, P@3 of 0.40 and R@3 of 0.90. BM25 was slightly better at P@5 (0.32 vs. 0.24) and R@5 (1.00 vs. 0.90), consistent with its strength in exact keyword matching.
• Large advantage on realistic paraphrased queries: On Set 2 queries (where users paraphrase, e.g. asking for "transverse momentum requirement" when the document says "p_T cut"), MITRA achieved P@1 of 0.75 versus 0.13 for BM25 and R@1 of 0.66 versus 0.03. At k=3, MITRA reached P@3 of 0.33 and R@3 of 0.81 versus 0.25 and 0.56 for BM25; at k=5, P@5 of 0.20 and R@5 of 0.81 versus 0.18 and 0.59.
• Rank-aware metrics confirm better ordering: On Set 1 both systems had an MRR of 1.00, with NDCG@3 of 1.00 for both and NDCG@5 of 1.00 for MITRA versus 0.98 for BM25. On Set 2, MITRA reached MRR 0.81 and NDCG@5 0.88, compared with MRR 0.35 and NDCG@5 0.59 for BM25 (NDCG@3: 0.91 for MITRA versus 0.67 for BM25).
• The paper notes a caveat about P@k and R@k: Because most queries have only one relevant answer, P@5 would ideally be 0.2, and Precision naturally decreases while Recall naturally increases as k grows. The authors argue P@k alone can be misleading because it is rank-agnostic — a system ranking the correct answer first scores the same P@5 as one ranking it fifth — which is why they added MRR and NDCG.
• Robustness against out-of-context questions: When locked onto a dark matter search analysis and asked "How many Higgs bosons were discovered in this search?", the system did not hallucinate; it inferred from retrieved passages that the document was unrelated to Higgs bosons and told the user the analysis was a dark matter search.
• Latency and throughput: The prototype exhibits low per-query latency — described as a few seconds "as can be seen in the demo" — and can process simultaneous user requests independently. No formal latency or throughput benchmark is reported; the authors list that as future work.

Methodology in Plain English

MITRA has an offline pipeline that builds the knowledge base and an online process that answers questions (illustrated in Figure 1 of the paper).

Building the knowledge base. Analysis documents live in databases with web interfaces, so the team wrote Selenium browser-automation scripts that log in, navigate to the right sections, and download the notes, typically as PDFs. Rather than standard PDF-to-text libraries such as PyPDF or PDFPlumber, which struggle with complex layouts, they use OCR engines such as Surya and Tesseract because these preserve the distinction between main content, figure captions, page numbers, line numbers, and tables. The extracted text is chunked by paragraph, on the assumption that paragraph boundaries align with logical separation of ideas.

Encoding and searching. Each chunk is encoded into a 768-dimensional vector space using the Dense Passage Retrieval model facebook/dpr-question_encoder-multiset-base via the Transformers library, and the vectors are stored in an open-source vector database, Chroma DB. A user question is encoded with the same model and compared against the stored passages using cosine similarity to retrieve the top-k candidates, where k is tunable; a larger k costs more computation in the next stage. Those candidates are then rescored by a cross-encoder, cross-encoder/ms-marco-MiniLM-L-6-v2, which is more accurate but too slow to run over the whole database — hence the two-stage retrieve-then-rerank design. The final ranked passages become context for the LLM, which is explicitly prompted to ground its answers strictly in the retrieved context to reduce hallucination.

Two-tiered context control. Because questions like "What is the most important background?" have different answers for, say, a Higgs to di-muon analysis versus a dark matter search, MITRA first searches a database containing only abstracts to identify the single most relevant analysis, then asks the user to confirm that choice — a human-in-the-loop validation step. Once confirmed, the conversation locks onto a second database containing the full chunked text of only that analysis (combining information from multiple documents about it), and all subsequent queries in that session go there. Starting a new conversation lets a user switch analyses.

Private generation. The embedding model and the LLM run on local collaboration GPU servers, for example an NVIDIA Tesla T4 with 15GB of memory. The generator is a 4-bit quantized version of Mistral-7B (a 7.2B parameter model), served locally with Ollama and integrated through the LangChain framework. Users interact through a Streamlit web interface.

Evaluation setup. The team compared MITRA against Okapi BM25, a widely used term-frequency/inverse-document-frequency ranking function that excels at exact keyword matches but has no inherent semantic understanding. Two query sets were designed by domain experts: Set 1 uses the exact phrasing found in the source documents; Set 2 uses common synonyms and paraphrases. Performance was measured with P@k, R@k, MRR, and NDCG. The number of queries in each set is not reported in the paper.

Why This Matters

Impact on research. The paper argues that a significant amount of expert effort goes into producing detailed collaboration documents that are underused because retrieval is inefficient. A system that reliably finds semantically correct passages can speed up onboarding of new PhD students and analysis-group members, help experts quickly grasp the nuances of a measurement, and unlock the value of the collaboration's collective knowledge. Preserving privacy matters because the corpus contains unpublished results and proprietary analysis details that cannot be sent to external API services.

Real-world applications:

• Internal question answering over analysis notes, wikis, and procedural guidelines for large experimental collaborations such as CMS.
• Onboarding assistance that lets new collaboration members ask natural-language questions instead of learning the exact terminology of the documents.
• Privacy-preserving enterprise search for any organization with sensitive internal documents and existing GPU infrastructure.
• Context-sensitive retrieval where the same question has different correct answers depending on which sub-project or document set is relevant — the general problem the two-tiered database design addresses.

Industry relevance. The paper positions its approach against the ATLAS collaboration's concurrent system, which relies on external, API-based services such as OpenAI's GPT-4o mini, citing cost-versus-quality considerations. The authors argue that while per-query API costs are low for individuals, cumulative operational expense for a collaboration of thousands of members over several years can become substantial, and that an on-premise stack avoids both per-token costs and data-egress risk. The pattern — local models, local vector stores, browser-automation ingestion, and human-in-the-loop context confirmation — is directly transferable to enterprises weighing hosted LLM APIs against self-hosted deployment.

Future Directions

1. Broaden the knowledge base to include multiple document types beyond the current analysis notes.
2. Build a larger evaluation framework covering the generation step, including quantitative LLM-as-a-judge metrics for faithfulness and answer relevancy, plus a formal user study with multiple domain experts measuring performance and inter-annotator agreement. The current evaluation benchmarks only retrieval, which the authors call the critical first step for mitigating hallucination.
3. Run a formal performance benchmark quantifying query latency and throughput under realistic load, and deploy on a production-grade GPU cluster to handle high-concurrency needs; the authors note that scaling to higher concurrency appears to require more compute capacity rather than architectural redesign.
4. Productionize the system by adopting a high-performance pluggable inference engine such as vLLM or llama.cpp, implementing strict network and access-control policies, and adding multi-turn conversational support.
5. Evolve MITRA into a proactive research agent capable of summarizing recent analysis updates, comparing methodologies between two related measurements, identifying strategies to handle disagreements between experimental data and Monte Carlo simulations, finding signal models with similar topology, and finding gaps in current search spaces.

Target Audience

This paper benefits researchers and engineers working on RAG systems and applied IR, particularly those deploying private, on-premise assistants over large proprietary document corpora. It is also relevant to scientific collaboration members and computing coordinators who manage internal documentation and are considering alternatives to API-based assistants, as well as to industry practitioners evaluating self-hosted versus hosted LLM deployment for sensitive internal knowledge bases. The paper is written at an intermediate level: it is accessible to readers who know the basics of embeddings, vector search, and LLM prompting.

Funding note: The authors acknowledge support from the U.S. DOE, Office of Science, Office of High Energy Physics, under Award No. DE-SC0017647.

Authors’ abstract

Large-scale scientific collaborations, such as the Compact Muon Solenoid (CMS) at CERN, produce a vast and ever-growing corpus of internal documentation. Navigating this complex information landscape presents a significant challenge for both new and experienced researchers, hindering knowledge sharing and slowing down the pace of scientific discovery. To address this, we present a prototype of MITRA, a Retrieval-Augmented Generation (RAG) based system, designed to answer specific, context-aware questions about physics analyses. MITRA employs a novel, automated pipeline using Selenium for document retrieval from internal databases and Optical Character Recognition (OCR) with layout parsing for high-fidelity text extraction. Crucially, MITRA's entire framework, from the embedding model to the Large Language Model (LLM), is hosted on-premise, ensuring that sensitive collaboration data remains private. We introduce a two-tiered vector database architecture that first identifies the relevant analysis from abstracts before focusing on the full documentation, resolving potential ambiguities between different analyses. We demonstrate the prototype's superior retrieval performance against a standard keyword-based baseline on realistic queries and discuss future work towards developing a comprehensive research agent for large experimental collaborations.