NOTAM-Evolve: A Knowledge-Guided Self-Evolving Optimization Framework with LLMs for NOTAM Interpretation

AI summary

Overview

Research area: Natural Language Processing applied to aviation safety — specifically structured information extraction and reasoning over Notices To Airmen (NOTAMs), combining large language models, knowledge-graph retrieval, and iterative preference-based self-optimization.

Technical level: Advanced. The paper assumes familiarity with supervised fine-tuning, Direct Preference Optimization (DPO), retrieval-augmented generation, TableRAG, curriculum learning, and knowledge-graph querying.

Scope: The paper formalizes "Deep Parsing" of NOTAMs as a dual challenge of dynamic knowledge grounding and schema-based inference, and introduces NOTAM-Evolve — a three-stage self-evolving LLM framework — alongside a 10,000-sample expert-annotated benchmark, reporting a 30.4% absolute accuracy gain over the base model (arXiv:2511.07982v1 [cs.CL], 11 Nov 2025).

What This Paper Is About

NOTAMs are official, time-sensitive aviation bulletins written in condensed telegraphic language with specialized abbreviations and nonstandard syntax; with over one million active NOTAMs issued annually worldwide, misreading them can mean missed runway closures, inoperative navigation aids, or restricted airspace. Existing automated systems typically stop at what the authors call "Shallow Parsing" — rule-based pattern matching or traditional named entity recognition — which extracts surface text but not the operational conclusions a dispatcher actually needs.

The paper's goal is to formalize and solve the harder "Deep Parsing" task, which requires grounding a NOTAM's textual references in external, time-varying aeronautical data (for example, mapping airport code ZBAA to its active runway list) and then applying static domain schemas to infer operational status (for example, reasoning that a given reduced length corresponds to a "Basic Approach Lighting System (BALS)"). NOTAM-Evolve is proposed as a framework that lets an LLM teach itself this skill largely without hand-written reasoning traces.

Key Contributions

1. Problem formalization and dataset. The paper formally defines structured NOTAM interpretation as a deep parsing challenge requiring both dynamic knowledge grounding and schema-based inference, and releases a benchmark of 10,000 globally sourced, expert-annotated NOTAMs.

2. Self-evolving framework. It proposes NOTAM-Evolve, which lets an LLM autonomously master complex NOTAM interpretation through iterative preference optimization and consensus-based inference, without requiring extensive manual supervision.

3. Knowledge-graph-enhanced retrieval (KG-TableRAG). A domain-specific knowledge graph is layered onto TableRAG so that the model can resolve implicit relationships — such as a runway's dependent lighting systems or navigation aids — that flat table retrieval would miss.

4. Empirical performance leap. Experimental validation reports a 30.4% absolute accuracy improvement over the base LLM, establishing what the authors describe as new state-of-the-art performance on this task.

Main Findings

• 30.4% absolute accuracy improvement over the base model. NOTAM-Evolve reaches an average (AVG) score of 0.762 versus 0.458 for its base model, DeepSeek-R1-Distill-Qwen-7B.

• Per-task results for NOTAM-Evolve (ours): Light 0.620, Area 0.725, Runway 0.836, Taxiway 0.868, AVG 0.762.

• It beats other open-source LLMs and the best SFT baseline. For comparison: Mistral-7B AVG 0.535, Llama3.1-8B-instruct AVG 0.450, qwen2.5-7B AVG 0.624, and qwen2.5-7b-instruct (SFT) AVG 0.744. Traditional methods score far lower (Regex Template Rule-based Matching AVG 0.425; UIE AVG 0.350).

• It approaches commercial models despite a much smaller parameter count. GPT-4o achieves AVG 0.785 and DeepSeek-R1 achieves AVG 0.828, versus NOTAM-Evolve's 0.762. On individual categories NOTAM-Evolve exceeds GPT-4o on Taxiway (0.868 vs 0.914 — no, GPT-4o is higher there); the notable case is Runway, where NOTAM-Evolve reaches 0.836 against GPT-4o's 0.770. The paper stresses that aviation's operational, security, and cost constraints often preclude closed-source third-party APIs.

• SFT alone can destroy reasoning ability. DeepSeek-R1-Distill-Qwen-7B (SFT) collapses to AVG 0.212 (Light 0.18, Area 0.226, Runway 0.236, Taxiway 0.204), which the authors connect to prior findings that fine-tuning on data lacking Chain-of-Thought rationales can impair reasoning.

• Ablation: both retrieval and multi-view inference matter, and they are complementary. With both components, AVG is 0.762. Removing KG-TableRAG (-KG) drops performance by 2.2% to 0.740, especially on knowledge-dependent tasks like Q-code mapping. Removing Multi-View Inference (-Multi-View) produces a larger 4.1% decline to 0.721. Removing both falls to the lowest level, 0.690.

• Multi-view inference with rewriting and voting adds about 5% accuracy. The paper reports this figure directly from its experimental validation of the technique, which generates N=5 semantically equivalent NOTAM variants and applies majority voting.

• Iterative self-optimization improves every category. Across three iterations: Light 45 → 54 → 62; Area 63 → 72 → 73; Runway 78.8 → 84.2 → 83.6; Taxiway 64.6 → 81.2 → 86.8. The paper highlights Taxiway rising from 64.6% to 86.8% and Light from 45% to 62%.

• Complexity grows sub-quadratically in practice. Theoretical preference pairs scale as roughly 9K²t²(1−η), giving 2,415 / 5,915 / 11,320 pairs across iterations 1–3, while effective pairs are 1,449 / 3,549 / 6,792. Wall-clock time is 0.58 h, 1.5 h, and 3.2 h, with scale factors of 1.0×, 2.6×, and 2.1×. Three mechanisms suppress theoretical O(t²) scaling to observed 2.3× average growth: error threshold filtering removes 40% of low-difficulty samples, curriculum sampling reduces effective batch size by 38%, and accuracy saturation limits error generation through (1−η) decay. The error suppression term (1−η) decreases from 0.55 to 0.38 as global accuracy moves from 45% to 62%.

• Convergence is fast. The authors state the framework achieves commercial SOTA-level NOTAM parsing accuracy within 3–5 iterations without model distillation, and that convergence is achieved in 3 iterations at 62% accuracy, with total wall-clock time ranging from 35 minutes to 3.2 hours on NVIDIA A800 GPUs.

• Case study: hierarchical inference over unmentioned components. For a NOTAM stating "CHOISEUL L BAY AIRPORT CLOSED TO ALL OPERATIONS" at airport AGGC, baseline systems lacking airport–runway structure may output {"airport": "AGGC", "runway": "", ...}. KG-TableRAG queries the knowledge graph with "AGGC" and retrieves that "RWY 07R" belongs to that airport; the LLM then infers the runway must also be closed and outputs {"airport": "AGGC", "runway": "RWY 07R", ...}.

Methodology in Plain English

The framework has three connected stages.

Stage 1 — Knowledge-grounded retrieval (KG-TableRAG). Aviation operational data lives in periodically updated tables, so the model needs current facts. But plain table retrieval fails when relationships are implicit. NOTAM-Evolve therefore first has the LLM write a Cypher query against a domain-specific knowledge graph. The graph results are concatenated with the original query to form an enriched query, which is then used to retrieve from the operational tables. That retrieved information is combined with the original NOTAM to form the final model input.

Stage 2 — Self-optimizing model refinement. Starting from an untuned open-source LLM, the annotated dataset is split (for example, 8:2 into train and test). In each iteration, the current model generates responses for the training inputs; these are compared against ground truth and labeled correct or incorrect, and stored in a response pool. An error rate ξ(x) is estimated as the fraction of the last K′ generated responses that were incorrect. Correct pairs become supervised fine-tuning (SFT) data and the model is updated by minimizing standard negative log-likelihood loss. Then a preference dataset is built from triples of (input, correct response, incorrect response). Inputs with a high error rate (ξ(x) ≥ τ) get dynamic augmentation into N_aug semantically preserving variants, each contributing additional preference triples. During DPO training, a weighted curriculum schedule α_e = min(e/E, 1) shifts sampling from uniform toward error-weighted using w_e(x), with β_weight controlling emphasis. The DPO loss uses the SFT-updated model as the policy and the model from the start of the iteration as the reference. The loop stops when test-set accuracy reaches a target η.

Stage 3 — Multi-view inference with rewriting and voting. Because the base model produces inconsistent predictions on edge cases, the input NOTAM is paraphrased into N=5 semantically equivalent variants under controlled rewriting that preserves aviation terminology (e.g., "RWY" abbreviations), spatiotemporal constraints, and safety-critical numeric values. Paraphrasing combines lexical substitution (e.g., "CTAM" ↔ "Controller Advisory Message"), syntactic restructuring through voice alternation, and contextual expansion with optional ICAO phraseology clarifications. Each variant is processed independently and a majority vote determines the final structured output.

Implementation used the DeepSeek-R1-Distill-Qwen-7B model, fine-tuned with the Unsloth framework and its recommended configurations, on a single NVIDIA A800-80GB-PCIe GPU. A prediction counts as correct only if it exactly matches the ground truth in both format and all annotated field values. All LLMs were evaluated on identical inputs — the same domain-specific prompts and the same information retrieved by the knowledge-grounded retrieval module.

The dataset was built by randomly sampling from global NOTAM traffic for the entire year 2024. Annotation was non-extractive and inferential: annotators supplied semantically correct values for each field even when the value was never stated explicitly (for example, assigning "Closed" to a Runway Status field that is only implied). Two expert dispatchers annotated independently, achieving a Krippendorff's Alpha of 0.96; all discrepancies were resolved by a third senior expert to produce the gold-standard dataset. Dataset statistics are 10,000 total samples, 39.2 average words per notice, 8.1 average valid days, Asia as the top region at 38.8%, and "Movement Area (M)" as the top Q-Code at 49.8%, with evaluation splits of Light (1,000), Area (4,000), Runway (2,500), and Taxiway (2,500).

Why This Matters

Impact on research. The paper reframes NOTAM processing as a reasoning problem rather than a sequence-labeling problem, and shows that a 7B open-source model can approach commercial GPT-4o and DeepSeek-R1 performance (AVG 0.762 vs 0.785 and 0.828) through knowledge grounding plus self-evolution rather than distillation. It also contributes a labeled benchmark with a reported Krippendorff's Alpha of 0.96, which is a rare asset for a domain where data is sparsely labeled and regionally fragmented. The self-evolving loop is notable for reducing reliance on human-annotated reasoning traces.

Real-world applications:

• Flight dispatch and pre-flight briefing, where a runway closure inference like the AGGC/RWY 07R case prevents routing aircraft toward unavailable infrastructure.
• Air traffic management and NOTAM office automation, converting raw bulletin streams into structured operational status fields.
• Airline operations control and delay mitigation, since misinterpretation of closures or inoperative navigation aids translates into costly delays or safety incidents.
• Safety-report triage and safety management systems in aviation organizations, given the framework's knowledge-grounding and iterative-refinement design.

Industry relevance. The authors argue explicitly that efficiency is not the only point: the operational, security, and cost constraints of aviation often preclude closed-source third-party APIs, making a small open-source model that reaches comparable accuracy strategically important for deployment.

Future Directions

• Computational scalability. The authors acknowledge that iteration costs grow progressively, similar to reinforcement learning paradigms; more efficient optimization strategies are listed as future work.
• Ground-truth quality ceilings. The inherent complexity of NOTAMs makes perfectly accurate annotation difficult, which may cap achievable performance; the authors propose LLM-assisted annotation combined with expert validation as a remedy.
• Multilingual and real-time extension. Extending the framework to multi-lingual NOTAMs and real-time operational scenarios is explicitly named as future work.
• Open questions the paper leaves. The knowledge graph's construction details and scale are deferred to supplementary material, and the framework is only validated on the authors' own 10,000-sample benchmark — whether the 30.4% gain transfers to independently collected NOTAM corpora remains untested.

Target Audience

Aviation NLP and information-extraction researchers; applied machine learning engineers working on retrieval-augmented generation, knowledge graphs, or DPO-based self-training pipelines; aviation safety and air traffic management practitioners evaluating automation of NOTAM handling; and industry teams at airlines, ANSPs, and aviation data providers who need deployable open-source models under security and cost constraints. Readers need a working grasp of LLM fine-tuning and preference optimization to follow the methodology sections, though the problem framing and results are accessible to aviation domain specialists.

Authors’ abstract

Accurate interpretation of Notices to Airmen (NOTAMs) is critical for aviation safety, yet their condensed and cryptic language poses significant challenges to both manual and automated processing. Existing automated systems are typically limited to shallow parsing, failing to extract the actionable intelligence needed for operational decisions. We formalize the complete interpretation task as deep parsing, a dual-reasoning challenge requiring both dynamic knowledge grounding (linking the NOTAM to evolving real-world aeronautical data) and schema-based inference (applying static domain rules to deduce operational status). To tackle this challenge, we propose NOTAM-Evolve, a self-evolving framework that enables a large language model (LLM) to autonomously master complex NOTAM interpretation. Leveraging a knowledge graph-enhanced retrieval module for data grounding, the framework introduces a closed-loop learning process where the LLM progressively improves from its own outputs, minimizing the need for extensive human-annotated reasoning traces. In conjunction with this framework, we introduce a new benchmark dataset of 10,000 expert-annotated NOTAMs. Our experiments demonstrate that NOTAM-Evolve achieves a 30.4% absolute accuracy improvement over the base LLM, establishing a new state of the art on the task of structured NOTAM interpretation.