Comparing the Performance of LLMs in RAG-based Question-Answering: A Case Study in Computer Science Literature

AI summary

Overview

Research area: Natural Language Processing / Large Language Models — specifically Retrieval Augmented Generation (RAG) applied to question-answering over scientific literature.

Technical level: Intermediate. Readers should be comfortable with concepts such as embeddings, vector stores, similarity search, prompt engineering and attention mechanisms.

Scope (one sentence): The paper benchmarks four open-source 7-billion-parameter LLMs against OpenAI's GPT-3.5 on question-answering over recent computer science abstracts, both with and without RAG support, using accuracy, precision, cosine similarity, expert and AI ranking, latency and cost.

What This Paper Is About

Off-the-shelf LLMs can only answer from knowledge frozen at training time, so they hallucinate or fail on recent findings; GPT-3.5, for instance, was trained on data up to January 2022. The authors build a RAG pipeline over a corpus of 2023–2024 computer science abstracts so that an LLM can retrieve and cite up-to-date, domain-specific evidence before answering.

The goal is a direct comparison of how five LLMs — GPT-3.5, Mistral-7b-instruct, LLaMa2-7b-chat, Falcon-7b-instruct and Orca-mini-v3-7b — perform on the same computer science question-answering task, and what trade-offs in answer quality, latency and cost they carry.

Key Contributions

1. A RAG-enabled question-answering pipeline for computer science literature built from a purpose-built dataset of 4,929 abstracts (1,121 on large language models, 1,810 on edge computing, 1,998 on quantum computing) extracted from Springer and IEEE papers published between 2023 and 2024.
2. A head-to-head comparison of four open-source LLMs and GPT-3.5 under identical configurations (temperature 0.01, max_tokens 2000, quantised open-source models, LangChain chains), covering both RAG-enabled and non-RAG (ChatGPT) answering.
3. A custom evaluation set of 30 question–answer pairs authored by two human experts — ten each for quantum computing, large language models and edge computing — evaluated by accuracy and precision for binary questions and by cosine similarity, Gemini ranking and human-expert ranking for long answers.
4. Practical evidence on side parameters of deployment: per-model average latency, per-model cost, and demonstration that prompt formatting (the Mistral instruction template) materially changes answer quality.

Main Findings

• GPT-3.5 with RAG is the strongest overall performer. It recorded the highest binary-question accuracy and precision at 0.9048 and 0.9048, and the best long-answer results, confirming its standing among the models tested, at a cost of $0.000643 for answering 30 questions with a maximum of 2000 tokens per run.

• Mistral-7b-instruct is the best open-source model. Among open-source LLMs it had the highest binary-question accuracy and precision (0.8571 and 0.8571; the paper cites 0.857 in the discussion), compared with GPT-3.5+RAG's 0.9048, and was rated excellent on 6 out of 9 long answers by both Gemini and the human expert. Despite only 7 billion training parameters versus GPT-3.5's 175 billion, the gap is small.

• Orca-mini-v3-7b follows closely on binary questions. It reached an accuracy of 0.8095 and precision of 0.8095.

• LLaMa2-7b-chat and Falcon-7b-instruct lag and show hallucination. Both scored 0.6190 accuracy (0.619 for Falcon as tabulated) with precision of 0.6842, indicating factually incorrect yet confidently stated answers even with RAG.

• Without RAG, performance collapses. ChatGPT (standalone GPT-3.5) was the worst on binary questions with accuracy 0.4761 and precision 0.625, and the lowest cosine similarity on long answers at 0.0977.

• Cosine similarity scores were low across the board. Table 3 reports GPT-3.5+RAG 0.2291, Mistral+RAG 0.2339, Falcon+RAG 0.2326, LLaMa2+RAG 0.2263, Orca+RAG 0.1806 and ChatGPT 0.0977; the prose in Section 4 instead cites 0.4479 for GPT-3.5+RAG and 0.2754 for Mistral-7b-instruct, which differ from the table values.

• Ranking counts in Table 3 are inconsistent with the stated question count. The paper describes 21 binary questions and 9 long-answer questions out of 30 pairs, but each model's Gemini-rank and human-rank columns in Table 3 sum to 20 (for example, GPT-3.5+RAG: Gemini 3 poor / 5 average / 12 excellent; human 8 / 3 / 9).

• Latency and cost trade-off is the main practical distinction. GPT-3.5+RAG averaged 1.7413 seconds per answer at $0.000643, versus Orca+RAG at 99.2167 seconds, Mistral+RAG at 105.9543 seconds, Falcon+RAG at 107.31642 seconds and LLaMa2+RAG at 107.4748 seconds, all at no financial cost. LLaMa2-7b-chat had the highest average latency among the open-source models.

• Prompt format matters for some open-source models. Mistral-7b-instruct and Falcon-7b-instruct required their instruction templates; Mistral's template is shown as [INST] instructions [/INST] answer [INST] Follow-up instructions [/INST]. Given an unstructured prompt, Mistral answered "I cannot directly answer that question with a simple yes or no", whereas with the template it correctly identified a lightweight unified collaborated relinquish (LUCR) edge gateway architecture study.

• Attention design helps explain the open-source gap. Llama 2 uses vanilla multi-head attention (MHA), Falcon uses multi-query attention (MQA) with lower memory load but reduced output quality, and Mistral uses grouped query attention (GQA) plus sliding window attention for higher throughput and better handling of longer sequences.

• AI-generated rankings are unstable. Gemini's rankings changed with slightly different instructions and sometimes changed when the same question was repeated; the single human expert's quantum computing judgment may carry some bias, though the two sets of rankings aligned closely.

Methodology in Plain English

• Building the corpus. Because of limited access to full papers and limited computing resources, the authors collected only abstracts — 4,929 of them — from Springer and IEEE journal papers published in 2023–2024 across three trending areas: large language models, edge computing and quantum computing. Titles, authors, publication dates and keywords were stored alongside abstracts for possible metadata-based retrieval.

• Cleaning the text. HTML artifacts, colons, semi-colons, extra whitespace, duplicates, null values and inconsistent casing were removed using Python's Regular Expressions (re) library. The cleaned abstracts were stored as individual JSON files.

• Turning text into vectors. Abstracts were split into chunks of at most 1024 characters with a 200-character overlap, using LangChain's "RecursiveCharacterTextSplitter", splitting on double line breaks and full stops followed by a space. Chunks were embedded with the pre-trained sentence transformer SPECTER (allenai-specter), which encodes scientific documents using the SciBERT transformer and takes the final [CLS] token representation as the abstract's representation. Embeddings were stored in the FAISS (Facebook AI Similarity Search) vector store from META.

• The QA pipeline. A user query is embedded with SPECTER and matched against the vector store using similarity search with a threshold of 0.6 and k = 10, returning the top 10 most similar chunks. If a chat history exists, the question and history are first passed to an LLM to produce a refined search query. Retrieved chunks plus the user query are passed to the LLM to produce a concise answer. Three LangChain chains were maintained: a format chain ("create_stuff_documents_chain"), a history chain ("create_history_aware_retriever") and a QA chain ("create_retrieval_chain").

• Models and settings. Five models were tested, with quantised versions of the open-source LLMs run locally. All trials ran on a MacBook Pro with an M2 processor and 16GB of RAM; temperature was 0.01 for fact-based answering and max_tokens was 2000.

• Evaluation. Binary questions were scored on accuracy (correct answers over total questions) and precision (correct answers over confidently answered questions), where models had to answer "yes", "no" or "do not know". Long answers were scored by average cosine similarity between the generated answer and an answer candidate, plus a three-level (poor, average, excellent) usefulness rating from Google's Gemini and from a human expert in quantum computing.

Why This Matters

Impact on research. The paper claims there was little to no prior evidence evaluating RAG-incorporated open-source versus closed-source LLMs specifically in the computer science domain — most domain-specific RAG work has concentrated on biomedical and legal applications. It provides a reproducible, domain-specific benchmark design (dataset construction, retrieval parameters, evaluation across quality, latency and cost) that other researchers can adapt, and it demonstrates that open-source models can approach proprietary performance.

Real-world applications:

• Research assistants that let computer scientists query recent publications, including follow-up questions, without reading every paper.
• E-learning and scholarly search platforms that answer student or researcher questions with references to recent literature.
• Customer support and enterprise QA systems where a company's own documentation is used as the retrieval source.
• Privacy-sensitive deployments, where a GPU-accelerated local open-source model keeps data on premises.

Industry relevance. The study frames model choice as a trade-off triangular decision: accuracy above cost favours the GPT family, data privacy favours a GPU-accelerated local open-source model, and personal use favours something like Mistral-7b-instruct with RAG under existing infrastructure. It also shows that latency in local open-source deployment can be reduced by accelerating CPU to GPU, while subscription-based LLMs avoid that infrastructure burden entirely.

Future Directions

• Apply systematic prompt engineering, since multiple prompts had to be developed before settling on one, and the chosen prompt may not be optimal; format sensitivity was demonstrated for Mistral-7b-instruct and Falcon-7b-instruct.
• Extend retrieval beyond vector stores to alternative data retrieval mechanisms, and use full-text research articles instead of only abstracts.
• Expand beyond computer science into other domains, and move beyond the single human expert to a pool of experts whose rankings are averaged to reduce bias in human evaluation.
• Re-evaluate emerging LLMs as the field evolves, and investigate the instability of AI-based ranking tools such as Gemini, whose judgments varied with instruction wording and repeated queries.

Target Audience

• NLP and RAG practitioners choosing an LLM for a domain-specific question-answering system, particularly those weighing open-source against proprietary options.
• Researchers and graduate students building retrieval-augmented systems over scholarly literature who need a template for dataset construction and multi-metric evaluation.
• Engineers and product teams in education technology, research tooling or enterprise support who must justify a model choice on the basis of accuracy, latency, cost and data privacy.
• Lecturers and students studying applied GenAI, since the paper documents the full pipeline from data collection through chunking, embedding, retrieval and evaluation in accessible terms.

Authors’ abstract

Retrieval Augmented Generation (RAG) is emerging as a powerful technique to enhance the capabilities of Generative AI models by reducing hallucination. Thus, the increasing prominence of RAG alongside Large Language Models (LLMs) has sparked interest in comparing the performance of different LLMs in question-answering (QA) in diverse domains. This study compares the performance of four open-source LLMs, Mistral-7b-instruct, LLaMa2-7b-chat, Falcon-7b-instruct and Orca-mini-v3-7b, and OpenAI's trending GPT-3.5 over QA tasks within the computer science literature leveraging RAG support. Evaluation metrics employed in the study include accuracy and precision for binary questions and ranking by a human expert, ranking by Google's AI model Gemini, alongside cosine similarity for long-answer questions. GPT-3.5, when paired with RAG, effectively answers binary and long-answer questions, reaffirming its status as an advanced LLM. Regarding open-source LLMs, Mistral AI's Mistral-7b-instruct paired with RAG surpasses the rest in answering both binary and long-answer questions. However, among the open-source LLMs, Orca-mini-v3-7b reports the shortest average latency in generating responses, whereas LLaMa2-7b-chat by Meta reports the highest average latency. This research underscores the fact that open-source LLMs, too, can go hand in hand with proprietary models like GPT-3.5 with better infrastructure.