Local Hybrid Retrieval-Augmented Document QA

AI summary

Overview

Research area: Natural Language Processing, specifically retrieval-augmented generation (RAG) and privacy-preserving enterprise question answering over documents.

Technical level: Intermediate — readers benefit from familiarity with retrieval concepts (dense embeddings, BM25, hybrid fusion) and RAG pipelines, though the paper explains its architecture in accessible terms.

Scope: A single-author paper presenting a fully local (on-premises) question-answering system that combines BM25 sparse retrieval with dense BGE embeddings, generates answers with Llama 3.2 via Ollama, and quantifies hallucination with an LLM-as-Judge protocol across three English benchmarks.

What This Paper Is About

Organizations that handle sensitive documents face a trade-off: cloud-based AI question answering offers strong accuracy but sends proprietary data to external providers, while local-only processing protects privacy but often performs poorly. This paper builds and evaluates a RAG system that runs entirely on local infrastructure — document parsing, embedding, retrieval, and LLM inference — without internet access, and tests whether it can achieve competitive accuracy on legal-, scientific-, and conversational-style documents. The goal is to show that privacy and performance do not have to be mutually exclusive for enterprise deployment.

Key Contributions

1. A fully local, three-component RAG architecture (HTML/CSS/JavaScript frontend, Flask-based client API, and a server handling retrieval, RAG orchestration, and LLM inference) in which no document content or credentials leave the host.
2. A hybrid retrieval configuration combining BM25 and dense BGE embeddings, tuned across 10 weight configurations (10%–100% sparse weight) to identify an optimal 30% sparse / 70% dense balance.
3. A reliability assessment using an LLM-as-Judge framework over more than 1,500 query-answer pairs, with bootstrap-derived 95% confidence intervals for core metrics (MRR, Recall@10, Hallucination Rate).
4. A documented execution environment and reproducibility package (fixed random seeds, cached embeddings and indices, versioned configs, and public source code) reporting performance on consumer-grade hardware.

Main Findings

• Optimal hybrid balance: Across all three datasets, the 30% sparse / 70% dense configuration emerged as optimal, balancing semantic understanding with lexical precision. Natural Questions validation confirmed this weighting.
• Hybrid gains are consistent but not uniform: On SQuAD, hybrid matched dense MRR (0.805) while improving Recall@10 from 0.959 (dense) to 0.974 and Answer Coverage from 0.976 to 0.980. On MS MARCO, hybrid reached Recall@10 of 0.620 versus 0.605 for dense and 0.480 for sparse, but its MRR (0.250) was lower than dense alone (0.315). Natural Questions hybrid reached MRR 0.813, Recall@10 0.978, and Answer Coverage 0.987.
• Sparse retrieval alone is weak on real search queries: BM25-only MRR was 0.103 on MS MARCO versus 0.717 on SQuAD, illustrating sensitivity to real-world query phrasing.
• Weight sensitivity differs by dataset: SQuAD remained resilient up to 50% sparse weight; MS MARCO showed steep degradation as sparsity increased; Natural Questions exhibited a balanced optimum.
• Low hallucination rates: LLM-as-Judge evaluation over 500 queries per dataset found a 0.8% hallucination rate on SQuAD (faithfulness 4.93, confidence 4.87, success 99.2%) and 6.2% on MS MARCO (faithfulness 4.79, confidence 4.71, success 96.8%). Natural Questions hallucination results were pending.
• Tight confidence intervals on retrieval: Recall@10 was 0.974 [0.971, 0.977] for SQuAD, 0.620 [0.610, 0.630] for MS MARCO, and 0.978 [0.973, 0.982] for Natural Questions, using 1,000 bootstrap resamples.
• GPU acceleration matters: Measured 4.2× speedup for embedding 1,000 document chunks and approximately 3× reduction in inference latency versus CPU-only execution on an NVIDIA RTX 4050.
• Latency and scalability: Median query latency was 2.0 seconds for a single user (P90 3.4 seconds); under 10 concurrent users, median latency rose to 6.8 seconds. Processing 100MB PDFs took roughly 4 minutes on CPU versus 1 minute with GPU.
• Evaluation cost: Hallucination judging cost approximately $5 per 500 judged queries using Gemini (approximately $0.01 per call, versus $0.03 for GPT-4), with automatic budget controls defaulting to a $50 stop limit.
• Judge reliability check: 50 double-coded samples at temperature 0 vs 0.2 yielded 96% agreement on the hallucination label, 82% exact agreement on faithfulness, and 100% agreement within ±1.

Methodology in Plain English

The system is split into three parts that talk to each other over local TCP sockets using a simple JSON protocol. A web frontend handles document upload and chat; a Flask-based client API validates and forwards requests; and a server does the heavy lifting. All credentials live only server-side in environment variables, so a compromised client cannot reach them.

For retrieval, the system does two things at once. It converts text chunks into dense vectors using the BGE embedding model and searches them with FAISS for semantic similarity, and it builds an in-memory BM25 index for precise keyword matching. A LangChain EnsembleRetriever combines the two score lists using a weighted linear mix. The authors swept 10 weighting options (10% to 100% sparse, in 10% increments) and found 30% sparse / 70% dense worked best overall.

Documents (PDF, CSV, JSON) are split with RecursiveCharacterTextSplitter at 300–400 tokens with 30–50 token overlap. Retrieved chunks are assembled into a prompt alongside chat history and the user's question, then sent to Llama 3.2 running via Ollama on localhost:11434, configured with temperature 0.6, top-p 0.95, top-k 40, and a 1024-token output limit.

For evaluation, the team used three benchmarks: SQuAD v1.1 (10,570 items, Wikipedia), MS MARCO v2.1 (9,706 items, Bing), and Natural Questions (3,610 items, Google). Metrics covered coverage (Recall@K), ranking quality (MRR, mean/median rank), answer fidelity (Exact Match, Answer Coverage), and reliability. For hallucination, they drew stratified samples of 500 queries per dataset (uniform across question-length tertiles) and had Gemini act as an offline judge — never a generator — scoring each answer for groundedness, faithfulness (1–5), and confidence (1–5). The authors are explicit that production answers are always generated locally by Llama 3.2; Gemini's API is used only to grade outputs at scale.

Why This Matters

Impact on research: The paper does not claim a new retrieval algorithm — it explicitly states its hybrid fusion uses standard linear weighting. Its contribution is demonstrating that established hybrid techniques can be assembled into a privacy-preserving local deployment with competitive benchmark numbers, plus a transparent accounting of confidence intervals, cost, and hardware requirements. It also surfaces an honest negative result: hybrid fusion did not beat dense-only MRR on MS MARCO.

Real-world applications:

• Banks reviewing proprietary contracts and financial documents without sending text to external providers.
• Hospitals querying clinical or administrative records while keeping patient data on-premises.
• Law firms searching case files and internal memos under data-sovereignty constraints.
• Any regulated organization that needs conversational document search but cannot use cloud LLM APIs.

Industry relevance: The system runs on consumer-grade hardware (an RTX 4050 laptop GPU with 6GB VRAM and 32GB system RAM), making it a plausible starting point for small and mid-sized enterprises. The documented evaluation cost (~$5 per 500 judged queries) and budget controls give a concrete sense of what offline quality assurance costs. The paper also frames the trade-off clearly: local processing reduces external exposure but enables internal data mining, so access logging and rate limiting are recommended alongside deployment.

Future Directions

1. Stronger retrieval fusion: The authors defer comparison with cross-encoder rerankers and dynamic or query-specific hybrid methods (naming DAT and MoR) because the current approach uses fixed linear weighting.
2. Multi-judge and human validation: Hallucination judgments currently rely on a single LLM-as-Judge; planned work includes multi-judge majority voting and human-annotated validation subsets. The Natural Questions hallucination results were still pending at publication.
3. Exhaustive chunking sensitivity: The 300–400 token window with 30–50 token overlap was chosen empirically under compute constraints; broader analysis (128–1024 tokens with varying overlaps) was not explored.
4. Security hardening and language coverage: Planned additions include RBAC, PII redaction, retrieval poisoning detection via embedding-space anomaly scoring and signed chunk manifests, and multilingual support through retrievers such as mBGE with language detection and dynamic routing.
5. Scalability: The synchronous, single-node design causes tail latency under load (6.8 seconds median at 10 concurrent users); asynchronous batching and sharded indices are proposed.

Target Audience

This paper is most useful to engineers and architects building enterprise document search on private infrastructure, particularly in regulated sectors such as healthcare, finance, and law. It also suits applied NLP researchers who want a concrete, reproducible reference point for hybrid retrieval performance on SQuAD, MS MARCO, and Natural Questions under fully local constraints. Readers seeking novel retrieval algorithms will find less here than those looking for a well-documented deployment recipe with honest limitations, cost figures, and a public code repository (commit 30c52ff).

Authors’ abstract

Organizations handling sensitive documents face a critical dilemma: adopt cloud-based AI systems that offer powerful question-answering capabilities but compromise data privacy, or maintain local processing that ensures security but delivers poor accuracy. We present a question-answering system that resolves this trade-off by combining semantic understanding with keyword precision, operating entirely on local infrastructure without internet access. Our approach demonstrates that organizations can achieve competitive accuracy on complex queries across legal, scientific, and conversational documents while keeping all data on their machines. By balancing two complementary retrieval strategies and using consumer-grade hardware acceleration, the system delivers reliable answers with minimal errors, letting banks, hospitals, and law firms adopt conversational document AI without transmitting proprietary information to external providers. This work establishes that privacy and performance need not be mutually exclusive in enterprise AI deployment.