Offline Language-Based Support System for Tactical Decision-Making in Air Combat Operations

AI summary

Overview

Research area: Natural Language Processing — retrieval-augmented generation (RAG) applied to military decision support and pilot training.

Technical level: Intermediate. The paper describes a modular engineering pipeline (embedding, vector search, local LLM inference, GUI) rather than introducing new model architectures or algorithms.

Scope: The paper presents MATE (Mission Assistant for Training and Evaluation), a fully offline, open-source RAG tool that answers natural-language tactical questions from air combat manuals, and reports an exploratory expert evaluation by three Brazilian Air Force fighter pilots using three open-weight language models.

What This Paper Is About

Air combat tactics are documented in long, complex manuals that pilots and mission planners need to consult quickly, often in environments with no internet access or where connecting to cloud services is not permitted. Most existing language-model question-answering tools depend on cloud services, which makes them unsuitable for aircraft, secure simulators, or air-gapped defense settings. The goal of this work is to build and evaluate a question-answering assistant that runs entirely on local hardware using open-source components, retrieving answers from doctrinal manuals and generating responses with locally hosted open-weight models.

Key Contributions

1. An offline, modular RAG architecture for tactical documents. MATE combines document indexing, semantic retrieval over a FAISS vector index, local LLM inference via the Ollama API, a Tkinter interface, and packaging for air-gapped deployment — all built from open-source components with no internet connection required at runtime.

2. A working implementation adapted to sensitive-document constraints. The system processes local manuals (.txt, .pdf, .docx), splits them into overlapping segments, and keeps all embeddings, indexes, and models on local storage, explicitly targeting use in classified or disconnected military environments.

3. An expert validation with operational fighter pilots. Three Brazilian Air Force fighter pilots rated 15 answers (five tactical questions × three open-weight models) plus five tool-level aspects on a 5-point Likert scale, providing domain-expert judgment rather than automated benchmark scores.

4. Identification of retrieval, not generation, as the bottleneck. The evaluation isolates retrieval relevance and context integration as the weakest rated aspects, and the authors trace this to concrete pipeline parameters such as the 4,000-character context limit.

Main Findings

• Overall answer quality was rated adequate: Average ratings across the five tactical questions ranged from 4.00 to 4.40 on a five-point scale (Llama 3.1 8B: 4.13; Qwen 2.5 14B: 4.00; Mistral NeMo 12B: 4.40).

• Mistral NeMo 12B scored highest: It obtained the top average and matched or exceeded the other models on every question. The largest model gap appeared in Q4 (innovation of the Double Attack), where Llama 3.1 8B received 3.33 and Mistral NeMo 12B received 4.67.

• One question was rated 3.00 by every pilot for every model: Q3, on the weaknesses of the Fighting Wing doctrine, received 3.00 ± 0.00 across all three models. Because the pattern is model-independent, the authors attribute it to the retrieved content rather than to generation.

• Best-scoring question: Q5 (Loose Deuce vs. Double Attack) received 5.00 ± 0.00 from Llama 3.1 8B, 4.67 ± 0.58 from Qwen 2.5 14B, and 5.00 ± 0.00 from Mistral NeMo 12B.

• Response time was rated maximally: All pilots gave 5.00 ± 0.00 for response time compared to manually locating the information in the reference text.

• Retrieval-related aspects were the weakest: Relevance of retrieval scored 3.67 ± 0.58 and integration of retrieved context scored 3.67 ± 0.58, versus 4.33 ± 1.15 for conceptual/terminological accuracy and 4.33 ± 0.58 for clarity and coherence.

• A concrete cause of retrieval loss is identified: With segments of 1,200 characters and a 4,000-character context limit, only about three of the five retrieved segments reach the model in full, so lower-ranked relevant passages may be truncated or omitted.

• The panel profile: Three fighter pilots with an average of 6.7 years of operational experience and 933 flight hours, reporting moderate familiarity with LLMs (occasional use and basic knowledge), mostly through ChatGPT, Gemini, and Llama.

• The authors explicitly frame results as exploratory: The panel was small, questions covered a single chapter of one manual, and the number of answers per model does not support statistical comparison between models or a meaningful inter-rater agreement analysis.

Methodology in Plain English

The researchers built a five-part pipeline. First, document indexing: tactical manuals stored locally are converted to plain text and cut into overlapping chunks of 1,200 characters with 250 characters of overlap, so no idea is lost at a boundary. Each chunk is converted into a 384-dimensional numerical vector (an embedding) by the multilingual SentenceTransformers model paraphrase-multilingual-MiniLM-L12-v2, normalized and stored in a FAISS inner-product index so that search behaves like cosine similarity.

Second, semantic retrieval: a user's question is embedded with the same model, the five most similar chunks (k = 5) are retrieved and concatenated in order of similarity into a context capped at 4,000 characters.

Third, language generation: the context plus the question, wrapped in a system instruction, is sent to a locally running open-weight model through the Ollama API. The instruction told models to answer directly and concisely, strictly from the reference chapter, using standard aviation terminology, and to say explicitly when a question was out of scope.

Fourth, a Tkinter interface with a model selector, Ask/Clear/Stop controls, keyboard shortcuts, and error handling talks to a FastAPI backend over local HTTP. An optional Whisper-based voice module exists but was disabled for the evaluation.

Fifth, offline deployment: everything is local, and PyInstaller can package the system as a standalone executable for air-gapped use.

For evaluation, the authors could not use classified material, so the knowledge base was Chapter 5, "Section Tactics: Two-versus-One," from Robert L. Shaw's Fighter Combat: Tactics and Maneuvering (1985). Five questions were written about mutual support and the evolution from Fighting Wing to Double Attack to Loose Deuce. Each pilot rated all 15 answers and five tool-level aspects on a 1-to-5 scale and could add free-text suggestions.

Why This Matters

Impact on research: The paper offers a small but concrete expert-grounded data point that open-weight RAG pipelines running fully offline can produce tactically adequate answers, and it localizes the failure mode in the retrieval stage rather than the generator. It also demonstrates an evaluation pattern — domain experts scoring answers against a public manual — that other sensitive-domain NLP projects could reuse when classified data cannot be shared.

Real-world applications:

• Onboard mission systems: offline question answering about tactical doctrine where connectivity is unavailable or prohibited.
• Flight simulators: a training aid that lets pilots query procedures and tactics during simulated missions.
• Training environments and mission planning: fast lookup during planning cycles and post-mission debriefings.
• Secure document access more broadly: the modular design is stated to be adaptable to different document types beyond air combat manuals.

Industry relevance: The work is a template for deploying RAG in air-gapped, regulated, or classified settings using only open-source and open-weight components — a configuration relevant to defense contractors, simulator manufacturers, and any enterprise that cannot send documents to a cloud provider.

Future Directions

• Broader evaluation: more pilots, more questions, additional tactical scenarios and manuals, enabling inter-rater agreement analysis and statistical comparison between models.
• Retrieval quality: improving how manuals are segmented, how relevant passages are selected, and how much context is passed to the model — directly addressing the 3.67 ratings and the truncation caused by the 4,000-character cap.
• Response time and model capacity: using more capable hardware to support larger models.
• Interactivity: adding dialogue history for follow-up questions and integrating the tool with simulation environments.
• Voice interface: optimizing the Whisper-based voice module, which the authors report is currently too heavy for routine use in flight or simulator training.

Target Audience

Defense and aerospace engineers building decision-support or training tools for disconnected environments; NLP practitioners interested in applied RAG system design and evaluation with small expert panels; military instructors and simulator developers exploring AI-assisted pilot training; and researchers studying human-in-the-loop evaluation of language models in high-stakes, safety-critical domains.

Authors’ abstract

An offline language-based support tool is presented to assist pilot training and tactical decision-making in air combat operations. The system, named MATE (Mission Assistant for Training and Evaluation), applies retrieval-augmented generation (RAG) to answer natural language questions using content extracted from doctrinal and tactical manuals. Built entirely with open-source components, it performs text preprocessing, embedding generation, semantic search, and language generation locally, ensuring full offline functionality. The tool was evaluated by three Brazilian Air Force fighter pilots, who rated the answers produced with three open-weight language models (Llama 3.1 8B, Qwen 2.5 14B, and Mistral NeMo 12B) to five tactical questions derived from a publicly available air combat manual. Average ratings ranged from 4.00 to 4.40 on a five-point scale, with Mistral NeMo 12B obtaining the highest score, and the pilots rated the response time of the tool as fully adequate compared to manual search. The relevance of the retrieved content was identified as the main point for improvement. Potential applications are discussed in onboard mission systems, flight simulators, and training environments.