Towards AI Evaluation in Domain-Specific RAG Systems: The AgriHubi Case Study

AI summary

Overview

Research area: Natural Language Processing, specifically retrieval-augmented generation (RAG) for low-resource languages and domain-specific AI evaluation in agriculture.

Technical level: Intermediate. The paper combines system engineering (RAG pipelines, embeddings, vector search) with human-centered evaluation, and explains both without heavy mathematical formalism.

Scope: This paper presents AgriHubi, a Finnish-language agricultural decision-support RAG system built over eight development iterations and evaluated through two structured user studies, with a focus on how model choice and iterative refinement affect answer quality, latency, and linguistic accuracy.

What This Paper Is About

General-purpose large language models are poorly suited to specialized domains like agriculture, where they produce confident but weakly grounded answers, and this problem is worse for low-resource languages such as Finnish, where high-quality domain documentation exists but general models struggle to use it. The authors build AgriHubi, a RAG system that pairs Finnish agricultural documents with open PORO family models, and they evaluate it in real use across two rounds of user testing. The goal is to provide empirical evidence on how to adapt RAG architecture, select models, and refine a deployed system in a low-resource language context.

Key Contributions

1. A domain-adapted RAG system for Finnish-language agricultural decision support that integrates Finnish agricultural PDF documents with open PORO family models and a FAISS-based retrieval index.
2. A complete, described system architecture spanning data ingestion and preprocessing, embedding and indexing, generative model integration, a multilingual (Finnish, Swedish, English) Streamlit user interface, and a SQLite feedback loop that logs queries, retrieved chunks, responses, ratings, and timestamps.
3. A documented eight-iteration development process (January to August 2025) in which user feedback drove changes to retrieval strategy, chunking logic, similarity thresholds, model selection, and interface design.
4. A two-round user evaluation comparing model generations with five-point Likert ratings (67 ratings in April 2025 with PORO-34B; 47 ratings in August 2025 with PORO-2-70B, and in some cases PORO-2-8B), supplemented by internal performance evaluations across LLaMA, Viking-33B, PORO-34B, PORO-2-8B, and PORO-2-70B.

Main Findings

• Answer quality improved across evaluation rounds: In April 2025 (67 ratings, PORO-34B), 46% of answers fell into the lowest two rating categories and only 3% received the top score. In August 2025 (47 ratings, PORO-2-70B and in some cases PORO-2-8B), low ratings (1–2) dropped to 38% and top ratings (score 5) rose to 21%.

• The full rating distribution shifted, not just the extremes: Mid-range ratings of 3 increased from 25% to 32% between rounds, while ratings of 4 declined from 25% to 9%. Ratings of 1 and 2 each moved from 15 (22%) and 16 (24%) in April to 9 (19%) and 9 (19%) in August.

• System-level refinements mattered more than model scaling alone: The authors report that improvements in retrieval quality, preprocessing, and context construction had a greater impact on answer quality than switching to larger models.

• A consistent accuracy-latency trade-off exists: PORO-2-70B produced the most accurate and detailed answers but responded more slowly, especially under shared GPU load, and testers often associated slower responses with lower trust. PORO-34B gave the first stable Finnish output but often produced incomplete answers, while PORO-2-8B improved fluency with moderate latency.

• Viking-33B was discontinued: Tested during iterations 4–6, its responses were short and unstable, so it was removed. LLaMA was used only as an early baseline and struggled with Finnish agricultural terminology.

• Iteration 8 changes produced a visible score jump: Replacing filename-based retrieval with full-chunk retrieval, increasing maximum answer length from 700 to 2000 tokens, improving error handling, and fully localizing the interface into Finnish coincided with a clear increase in evaluation scores in the second round, especially for completeness and clarity.

• Qualitative feedback matched the quantitative trend: First-round users described answers as incomplete, overly general, terminologically inconsistent, or slow; second-round users described the system as clearer and more reliable, with improved terminology, more natural Finnish, and better use of retrieved sources.

• Swedish support remained limited: Sparse training data and inconsistent document coverage led to unstable responses for Swedish queries, reducing trust among bilingual users.

• Evaluation was small-scale by design constraints: The 67 and 47 response samples were insufficient for fine-grained statistical analysis, and differences in question types between rounds may have introduced minor scoring bias.

Methodology in Plain English

The authors built a RAG pipeline with four parts: a document store, a retriever, a generative model, and a user interface. Agricultural PDFs were extracted (with OCR fallback for scanned documents), segmented into chunks, and tagged with metadata. Chunks were embedded using OpenAI's text-embedding-ada-002 and stored in a FAISS index using L2-normalized vectors and cosine similarity. When a user submits a query, the system embeds it, retrieves the highest-scoring chunks, groups them into a context window, and sends them with the query to a selected model (Llama 3.2, PORO-34B, PORO-2-8B, or PORO-2-70B) through the Farmi / GPT-Lab APIs. Responses stream back through a Streamlit interface, and every query, retrieved passage, response, rating, and timestamp is stored in a local SQLite database (chat_history.db).

The system was developed in eight iterations between January and August 2025, using feedback logged in that database to guide changes. Evaluation used a five-point Likert scale (1 = poor, 5 = excellent) in two rounds, April and August 2025, with question topics kept similar to support direct comparison. Four metrics were used: user-perceived quality via Likert ratings, end-to-end response latency, factual accuracy and completeness, and fluency of Finnish agricultural terminology. Internal evaluations additionally assessed user feedback stability, qualitative errors, latency, and domain coverage across all model backends.

Why This Matters

Impact on research: The paper addresses a documented gap noted by the authors — that most agricultural RAG systems focus on English-language data and that theoretical RAG research is abundant while empirical studies of domain-specific systems deployed in real-world settings are scarce. It also responds to criticism that evaluation practices in this area are inconsistent and often rely on basic metrics such as Exact Match rather than assessing retrieval quality or grounding effectiveness.

Real-world applications:

• Agricultural decision support for Finnish farmers, providing grounded answers from national documentation such as research reports, regulatory documents, advisory guidelines, and environmental records.
• Public advisory and policy communication, aligned with Finnish national initiatives the authors cite, including Hiilestä kiinni and Ruokavirasto, which emphasize responsible data use and practical decision support.
• Bilingual or multilingual advisory services, given the interface's support for Finnish, Swedish, and English, while also illustrating the difficulties of extending support to a second language.
• Low-cost domain dataset construction, building on cited work showing hybrid human-in-the-loop workflows can reach a cost of about $0.25 per 100 pages.

Industry relevance: The study shows that a domain-specific RAG system can run without local GPUs by hosting PORO models (PORO-2-70B, PORO-2-8B, PORO-34B) on GPT-Lab / Farmi servers, and that the accuracy-latency trade-off of larger models directly affects user trust. For organizations deploying generative AI in specialized domains, the finding that retrieval, preprocessing, and context construction mattered more than model scaling is a practical design signal, as is the observation that a feedback loop stored in SQLite guided most design decisions.

Future Directions

1. Integrating external, real-time data sources: The authors propose connecting AgriHubi to sensor-based and API-driven inputs such as weather services, soil measurements, and environmental monitoring systems, so the retrieval layer can reason over current conditions rather than only static documents.

2. Adding longer-term system memory: Persisting interaction history, retrieved context, and past recommendations would allow continuity across sessions and cumulative reasoning over time, which the authors argue could improve consistency and reduce repeated explanations in seasonal or longitudinal agricultural scenarios.

3. Evolving from question answering toward decision support: Future extensions could support lightweight planning tasks, scenario exploration, and recommendation workflows such as crop management suggestions or policy-compliant guidance — which the authors note will require careful data governance, controlled update cycles, and explicit handling of uncertainty when multiple data sources conflict.

4. Improving multilingual balance and evaluation rigor: Swedish support remained limited due to sparse training data and inconsistent document coverage, and the authors state that the lack of a consistent evaluation process across iterations made structured comparison difficult, pointing to the need for more systematic evaluation practices.

Target Audience

This paper benefits researchers and practitioners working on retrieval-augmented generation, applied natural language processing for low-resource languages, and multilingual large language model deployment. It is also relevant to software engineering researchers interested in iterative, feedback-driven development of AI systems, and to developers and product teams in agri-tech, public-sector advisory services, or any domain-specific setting who need a concrete example of how to architect, deploy, and evaluate a RAG system outside English-language benchmarks.

Authors’ abstract

Large language models show promise for knowledge-intensive domains, yet their use in agriculture is constrained by weak grounding, English-centric training data, and limited real-world evaluation. These issues are amplified for low-resource languages, where high-quality domain documentation exists but remains difficult to access through general-purpose models. This paper presents AgriHubi, a domain-adapted retrieval-augmented generation (RAG) system for Finnish-language agricultural decision support. AgriHubi integrates Finnish agricultural documents with open PORO family models and combines explicit source grounding with user feedback to support iterative refinement. Developed over eight iterations and evaluated through two user studies, the system shows clear gains in answer completeness, linguistic accuracy, and perceived reliability. The results also reveal practical trade-offs between response quality and latency when deploying larger models. This study provides empirical guidance for designing and evaluating domain-specific RAG systems in low-resource language settings.