A Graph-based RAG for Energy Efficiency Question Answering

AI summary

Overview

• Research area: Natural Language Processing / Retrieval-Augmented Generation, applied to the energy sector (Energy Efficiency question answering).
• Technical level: Intermediate — the paper assumes familiarity with LLMs, RAG, embeddings, and Knowledge Graphs, but the architecture is described step by step.
• Scope (1 sentence): The paper builds a graph-based RAG system that automatically extracts a Knowledge Graph from Italian energy guidance and regulation documents, answers user questions in Italian and English by navigating that graph, and validates the answers using domain experts and the RAGAs framework on 101 question-answer pairs.

What This Paper Is About

Energy users need reliable, locally-grounded answers about energy efficiency, regulations, and incentives, but general-purpose LLMs tend to hallucinate on this kind of domain-specific and jurisdiction-specific content. The authors propose a graph-based Retrieval-Augmented Generation architecture that extracts a Knowledge Graph from domain documents and then reasons over it to answer questions. Their goal is to test whether this graph-based retrieval actually produces answers that domain experts judge valid, including when the question and answer language differs from the source documents, which are in Italian.

Key Contributions

1. A three-part graph-based RAG architecture for Energy Efficiency: a Knowledge Extractor that pulls entity-relationship-entity triples from domain PDFs and web pages, a Knowledge Base holding the KG plus auxiliary user-metadata tables, and a Retrieval & Generation component that answers questions with citations to sources.
2. An automated, prompt-based KG construction pipeline: documents are cleaned, chunked, and parsed with an LLM into triples, with optional domain-expert guidance and an optional ontology filter; node identities are unified by hashing names.
3. A local, entity-anchored retrieval and reasoning procedure: the system embeds the question, finds the most similar Entity objects by cosine similarity, then expands to their outgoing and incoming relationships and their associated text chunks before generating an answer.
4. A human-based multilingual validation study: 101 question-answer pairs (Italian and English, Italian and Swiss contexts, plus country-agnostic recommendations) assessed by 4 domain experts using the RAGAs properties of faithfulness, answer relevance, and context relevance, plus an ablation experiment against an LLM-only configuration.

Main Findings

• Overall answer validity: The system achieved an overall validity score of 75.2 ± 2.7% across the 101 questions asked in both Italian and English (202 evaluations, assessed by n = 4 domain experts).
• Best performance: Answers to country-agnostic questions in Italian reached 81.0 ± 4.1%, the highest cell in the results table.
• Language difference: Italian-language responses scored 77.4 ± 2.9% versus 73.0 ± 2.5% for English responses, a gap the authors attribute to translation, reporting a 4.4% accuracy loss due to translation.
• Context-country breakdown: Overall scores were 73.4 ± 0.9% for Italian-context questions (25 items), 71.2 ± 2.1% for Swiss-context questions (25 items), and 78.1 ± 3.0% for country-agnostic questions (51 items).
• Language-by-context detail: For English questions, Italian-context items scored 73.6 ± 1.0%, Swiss-context items 67.7 ± 2.4%, and country-agnostic items 75.2 ± 2.9%. For Italian questions, the corresponding figures were 73.3 ± 0.8%, 74.4 ± 1.9%, and 81.0 ± 4.1%.
• Latency and call volume: Each question was answered in 19.08 ± 4.48 seconds on average, using a fixed 2 LLM calls and an average of 3.55 ± 1.01 embedding calls per question.
• Ablation shows retrieval is necessary: Without any persistent memory or retrieval component, the LLM-only configuration answered general EE questions but produced excessively long answers and inaccuracies (e.g., claiming winter causes more frequent baths or showers), and answered specific regulation questions mostly incorrectly — for example stating the maximum deductible spending limit in Italy for 2025 was 8,000 euros instead of 5,000 euros, and suggesting the user verify online.
• Country-specific questions are harder: The authors suggest the slightly lower scores on country-specific questions stem from laws and articles containing complex information tied by temporal or spatial constraints, which are harder to extract and manage than general recommendations.

Methodology in Plain English

The researchers assembled Italian-language source websites covering energy consumption, energy efficiency, regulations, and incentives. They cleaned the text of page numbers and HTML tags, split it into chunks of size 1000 with overlap 200, and used an LLM-based triple extractor (LangChain's LLMGraphTransformer with gpt-4o-mini) to pull out entity-relationship-entity triples. Entity and relationship names were normalized (underscores replaced by spaces, lowercased, first letter capitalized), and nodes were identified by MD5 hashes of their names so that duplicate concepts merged into single nodes.

The graph was initialized with a simple ontology in an ONTO namespace defining OWL classes such as Entity, Relationship, Property, Document, and Chunk, along with object properties (hasSource, hasTarget, hasRelationship, hasChunk) and datatype properties (hasName, hasContent, hasValue, hasValueEmbedding). Documents and chunks became nodes too, and embeddings were computed from entity names, relationship names, and chunk text using OpenAI's text-embedding-3-small.

At question time, the system extracts triples from the question with the same LLM approach, embeds the question and the extracted entity names, and uses cosine similarity to find the most similar Entity objects. In the reported experiment it kept k = 12 entities with a similarity threshold t = 0.5, then retrieved c = 5 chunks and i = 10 incoming and o = 10 outgoing relationships. These were serialized into a prompt together with the original question and user metadata, and gpt-4o-mini generated the final answer with citations to source page links. The design ranges reported as empirically selected are 3 ≤ k ≤ 15, 0.5 ≤ t ≤ 0.75, 5 ≤ o, i ≤ 10, and 5 ≤ c ≤ 10.

The validation used 101 question-answer pairs: 25 on Italian regulations and incentives, 25 on Swiss regulations and incentives, and 51 recommendations applicable to both Italy and Switzerland. Questions and reference answers existed in both Italian and English. Domain experts with access to ground-truth answers scored each response against the three RAGAs properties, lowering the score proportionally when a property was not met. The architecture was deployed inside the ENERGENIUS Guru Decision Support System as part of the ENERGENIUS European project.

Why This Matters

Impact on research. The paper provides one of the few empirically validated, human-assessed evaluations of graph-based RAG in the energy efficiency domain, and it explicitly notes that no relevant prior literature existed on improving EE for users and households with a graph-based RAG approach. It also isolates the value of retrieval by contrasting the graph-based system with an LLM-only ablation on the same 101 questions.

Real-world applications:

• Household and consumer advisory tools that explain energy-saving behaviors and appliance choices in the user's own language.
• Decision support for navigating national and regional incentive schemes, such as the Italian furniture and household appliance bonus or Swiss cantonal building rules.
• Small and medium-sized enterprises seeking guidance on energy consumption and efficiency, the use case targeted by the related Energy Chatbot work.
• Multilingual public-sector or utility help desks that must answer the same regulatory question for users writing in different languages than the source documents.

Industry relevance. The system was integrated into a real Decision Support System within the ENERGENIUS European project (Horizon Europe Grant Agreement No 101160720), and its citation-bearing answers are aimed at transparency and accountability of knowledge sources — properties that matter for regulated energy advice. Reported average response time of 19.08 ± 4.48 seconds and a bounded number of LLM calls per question also give a rough sense of operational cost.

Future Directions

1. Expanded model comparison: The authors state that testing the architecture should involve a variety of different LLMs and text embedding systems, since the current study used only gpt-4o-mini and text-embedding-3-small.
2. Broader multilingual validation: To better validate multilingualism, they call for tests in a wider range of languages beyond the Italian and English question-answer pairs and Italian source documents used here.
3. Handling temporally and spatially constrained content: Performance was slightly lower on country-specific questions involving laws and articles, which the authors attribute to complex temporal or spatial constraints — an extraction and reasoning problem the current design does not specifically address.
4. Continued data collection during project maturity: Because ENERGENIUS is in its early stages and real usage data are still being collected, the authors plan to continuously gather and analyze data to enhance the system and confirm these preliminary findings.

Target Audience

Researchers and practitioners working on RAG architectures, Knowledge Graph construction from documents, and LLM evaluation methodology will find the architecture description and the human-based RAGAs validation protocol most useful. The paper is also relevant to energy-domain professionals, policy-oriented product teams, and builders of multilingual advisory systems who need to judge whether graph-based retrieval is worth the added pipeline complexity over vector-only RAG — the ablation results give a concrete answer for domain-specific questions. Readers without background in LLMs, embeddings, or ontologies would need to consult the cited background literature first.

Authors’ abstract

In this work, we investigate the use of Large Language Models (LLMs) within a graph-based Retrieval Augmented Generation (RAG) architecture for Energy Efficiency (EE) Question Answering. First, the system automatically extracts a Knowledge Graph (KG) from guidance and regulatory documents in the energy field. Then, the generated graph is navigated and reasoned upon to provide users with accurate answers in multiple languages. We implement a human-based validation using the RAGAs framework properties, a validation dataset comprising 101 question-answer pairs, and domain experts. Results confirm the potential of this architecture and identify its strengths and weaknesses. Validation results show how the system correctly answers in about three out of four of the cases (75.2 +- 2.7%), with higher results on questions related to more general EE answers (up to 81.0 +- 4.1%), and featuring promising multilingual abilities (4.4% accuracy loss due to translation).