CodeGraph: Open-Taxonomy Knowledge Graph for Source Code with Wikidata Grounding

AI summary

Overview

Research area: Software engineering and information retrieval — specifically large-scale semantic annotation of source code, knowledge-graph construction, and entity linking to Wikidata.

Technical level: Intermediate. The paper assumes familiarity with knowledge graphs, LLM inference pipelines, and entity linking, but presents the architecture and evaluation protocol in largely self-contained terms.

Scope: The paper describes a pipeline that annotates roughly 145 million source files with high-level software-engineering concepts using a code-specialised LLM, grounds that vocabulary in Wikidata, and materialises the result as a typed property graph called CodeGraph.

What This Paper Is About

Public code archives such as GitHub and the Software Heritage Archive hold billions of files, but the engineering knowledge buried in them — which algorithms a file implements, which programming paradigms it follows, which design patterns it instantiates, and which application domain it serves — is not exposed by existing tools, which are limited to syntactic or token-level analysis. The authors build a pipeline that uses a code-specialised LLM to extract those four semantic categories from whole source files, links the extracted concepts to Wikidata identifiers through a three-stage procedure, and assembles everything into a large typed property graph. They then apply it at corpus scale to Stack-Edu and report the resulting graph statistics alongside a calibrated precision protocol.

Key Contributions

1. The first open-taxonomy LLM annotation pipeline for source code that scales to the full Stack-Edu corpus. It processes approximately 167 million files across 14 programming languages and 125 billion tokens, extracting algorithms, paradigms, design patterns, and application domains as metadata. The taxonomy is deliberately open: the model emits free-form concept names that are reconciled a posteriori rather than constrained to a predefined label set.

2. A three-stage Wikidata grounding procedure. A deterministic SPARQL stage with a class-hierarchy filter resolves unambiguous concepts; a Deep Research Agent backed by a separate open-weight model (Qwen3.6-27B) handles ambiguous or empty results; and a hierarchy-rollup stage imports the parent-of closure of each resolved QID via transitive traversal, producing a dual conceptual layer — a local corpus-defined vocabulary and a global Wikidata-grounded taxonomy.

3. A massive open-taxonomy knowledge graph. CodeGraph contains 158,202,834 nodes and 1,020,887,480 edges, of which 144,910,008 are :File nodes. The consolidated semantic layer comprises 37,412 domains, 19,660 algorithms, 5,955 design patterns, and 193 programming paradigms, with 19,807 distinct Wikidata entities imported as :WikidataEntity nodes.

4. A calibrated quality-assurance protocol. A small adjudicated human gold set characterises task difficulty and benchmarks a pool of candidate LLM verifiers; the best-performing verifier (Devstral-2 2512) is then deployed as an automated per-annotation filter over the full corpus, with both extractor annotation and verifier decision preserved in the released graph.

Main Findings

• Scale relative to prior work. CodeGraph is reported as two orders of magnitude larger than GraphGen4Code, the closest existing code knowledge graph, which covers approximately 1.3 million Python files and is limited to syntactic artifacts (classes, functions, data- and call-flow edges). Table 1 positions CodeGraph at 145 million files, 14 languages, with semantic metadata (algorithms, paradigms, patterns, domains) and Wikidata grounding — the only resource in the table marked as having all three of semantic metadata, Wikidata, and knowledge graph.

• Composition and density. The graph's average degree is 12.91, consistent with a large but sparse typed network. The semantic layer is enriched with 268,394 :AlgorithmCategory nodes and 12,941,405 :AlgorithmComplexity nodes attached to algorithm concepts.

• Grounding coverage is near-complete. Of the consolidated vocabulary, 36,879 of 37,412 domains, 17,492 of 19,660 algorithms, 5,666 of 5,955 design patterns, and 188 of 193 paradigms are linked to Wikidata, indicating the extracted concepts align almost entirely with established encyclopaedic entries.

• Language distribution is skewed. Java (30.6%), Python (16.9%), and C++ (11.2%) together account for 58.7% of files. The top domain labels are web development, database, and data science, with a long tail of specialised concepts.

• Human annotation is intrinsically difficult. On a stratified sample of 127 source files, 17 annotators produced 292 judgements across 6 programming languages and 4 semantic categories: 68.4% Okay, 17.7% NotOkay, 13.9% Unsure. Fleiss' κ was 0.153 overall, with domain at 0.213, algorithm at 0.200, design pattern at 0.024, and paradigm at −0.111.

• Verifier selection favoured recall. Across 1,536 model–human comparison pairs, the six candidate verifiers spanned F1 from 84.9% to 88.5%. Devstral-2 2512 was retained with 80.6% accuracy, 82.1% precision, 96.0% recall, and 88.5% F1. On 256 valid human–LLM comparisons it reached 77.3% raw accuracy, with the strongest category-level Cohen's κ for domain (0.353). Adjudication of 50 disagreement cases upheld the model in 33 (66.0%) versus the human in 17 (34.0%).

• Silver-tier acceptance varies by axis. On a 10,000-file subset, paradigm scored highest (93%), followed by algorithm and domain (both 88%) and design pattern (82%). Cross-language acceptance stayed within a narrow 87–90% band (C 90%, Go 89%, Java 90%, JavaScript 90%, Python 87%, Rust 88%).

• Two precision bounds. The verifier's overall acceptance of roughly 88% is systematically higher than the human Okay rate of 68.4%, so the authors present 68% as a conservative lower bound and 88% as a loose upper bound on plausible precision.

• Wikidata-restricted verification is stricter, not contradictory. Restricting the verifier to annotations successfully linked to a Wikidata entity causes algorithm, design pattern, and domain acceptance to drop markedly, while paradigm remains most reliable. The authors' working hypothesis is a granularity mismatch: algorithm and design pattern labels typically describe constructs at the level of a single function or class, so asking whether the whole file instantiates the corresponding Wikidata concept penalises acceptance.

Methodology in Plain English

The pipeline starts from Stack-Edu, a curated subset of The Stack v2 produced by a learned quality classifier that retains files with pedagogical value. It spans 15 mainstream languages — 14 programming languages plus Markdown — with approximately 167 million files, 125 billion tokens, and about 150 GB on disk. The roughly 22 million Markdown files are excluded from semantic extraction because they contain documentation or mixed prose rather than executable code, leaving about 145 million files that become the graph's :File nodes.

Each remaining file is passed through Qwen3-Coder-30B-A3B-Instruct, an open-weight mixture-of-experts model with 30 billion total parameters and about 3 billion active parameters per forward pass. Rather than asking one prompt for everything, the authors issue a battery of separate structured prompts targeting four orthogonal axes: application domains, algorithms (with category and, when inferable, asymptotic complexity), programming paradigms, and design patterns. Each prompt enforces a strict JSON schema and instructs the model to return empty arrays when a file offers no support for that axis, converting silent omissions into explicit negatives. Inference runs through vLLM with continuous batching and tensor parallelism at a context window of up to 16,384 tokens, deployed on a data-parallel Ray + vLLM topology. Sampling used the model's recommended settings of temperature 0.7 and top-p 0.8. The extraction stage consumed approximately 400,000 GPU-hours on 256 NVIDIA A100 GPUs within the LEONARDO supercomputer.

Extracted labels are then consolidated: separators and stray punctuation are stripped, casing normalised, and frequent variants above a recurrence threshold collapsed into a single canonical entry. The result is materialised as a directed typed property graph in which :File nodes carry language, origin, and quality metadata, concept nodes are keyed by their canonical normalised form, and edges are typed by the semantic relation they encode (IMPLEMENTS, BELONGS_TO_DOMAIN, USES_PARADIGM, APPLIES_PATTERN) while retaining the provenance of the originating annotation. A derived layer groups :Algorithm nodes under :AlgorithmCategory parents emitted by the extractor, and links them to :AlgorithmComplexity nodes that normalise heterogeneous complexity strings such as "O(n log n)", "linear in input", and "quadratic time" into a shared asymptotic vocabulary.

Grounding to Wikidata happens in three stages. Stage 1 matches each canonical concept through a SPARQL full-text query over labels and aliases with a class-hierarchy filter using wdt:P31 (instance of) and wdt:P279 (subclass of); a single unambiguous candidate is linked directly via a MATCHES_WIKIDATA edge. Stage 2 hands ambiguous or empty candidate lists to a Deep Research Agent implemented on Qwen3.6-27B, which inspects the candidates and the original concept, consults linked Wikidata descriptions, and either selects the best-matching QID or reports that none is appropriate. Stage 3 traverses the wdt:P279 chain for two hops from each resolved QID, materialising the direct parent and grandparent as additional :WikidataEntity nodes connected through WIKIDATA_PARENT edges. For example, a file annotated with the domain neuroscience (Q207011) is linked upward through life sciences (Q864928) to science (Q336).

For quality assurance, manual annotation of the full corpus is ruled out: at a conservative two minutes per file–axis pair, coverage would require roughly 9,500 person-years at 2,000 working hours per year. The authors instead build a gold set from 127 source files with 292 judgements from 17 annotators, each pair independently judged by three annotators and disagreements resolved by majority vote, then benchmark six code-oriented LLMs against it under a uniform protocol. The best verifier is deployed over the entire corpus, tagging every file–annotation pair Okay or NotOkay.

The pipeline source code and a small reproducibility sample are released on GitHub, and the full CodeGraph is available on HuggingFace.

Why This Matters

Impact on research. The paper shows that high-level semantic properties of code — what a program does and what engineering conventions it follows — can be extracted and grounded at a scale previously limited to syntactic artifacts. It supplies a public, Wikidata-anchored reference layer against which other corpora, models, and annotation methods can be compared under a shared identifier space, and it contributes a reusable precedent for calibrating LLM annotation quality using a small human gold set plus a verifier. The three-stage grounding design is also a general pattern: deterministic resolution where possible, expensive reasoning reserved for the ambiguous long tail.

Real-world applications:

• Semantic code search: retrieving or filtering snippets by the algorithms, paradigms, or design patterns they embody rather than by lexical matches on identifiers or comments, and navigating the Wikidata hierarchy to broaden or narrow results along the concept tree.
• Automated algorithm selection: the paper motivates this as a long-standing learning problem typically addressed by training selectors over hand-curated portfolios; a graph linking files to algorithm nodes and complexity classes is a structural step toward corpus-derived portfolios.
• Cross-corpus comparison: anchoring concepts such as machine learning or distributed systems to specific QIDs removes residual lexical ambiguity and makes comparisons across repositories meaningful.
• Educational and corpus-provenance analysis: because Stack-Edu is filtered for pedagogical value, the domain and algorithm distribution gives a structured view of what kinds of programming practice that corpus represents.

Industry relevance. The graph is built from an open-weight model, which the authors emphasise makes the extraction step fully reproducible and lets the same weights be audited and reused. Both the pipeline and the graph are released publicly, so engineering organisations can adopt the annotation schema and grounding procedure for their own private repositories, and model and tooling vendors get a benchmark for how well code-specialised LLMs perform as zero-shot semantic annotators rather than assistants.

Future Directions

• Function-level evaluation. The authors explicitly leave a more precise characterisation of the Wikidata-restricted acceptance drop to future work, suggesting function-level evaluation as a possible resolution of the hypothesised granularity mismatch between whole-file verification and constructs that live inside a single function or class.

• Strengthening design-pattern and paradigm claims. Extreme disagreement among human annotators (design pattern κ = 0.024, paradigm κ = −0.111) means these categories cannot currently support strong quantitative claims; the paper leaves open how to make them measurable.

• Investigating the verifier's false-positive behaviour. The gap between the ~88% verifier acceptance rate and the 68.4% human Okay rate is attributed partly to false positives and partly to the conversion of human Unsure judgements into Okay decisions; a more precise characterisation is not reported.

• Extending beyond the two illustrated use cases. The paper states the use cases are not exhaustive, naming semantic code search and automated algorithm selection as illustrations of what the typed structure could support; whether corpus-scale graph retrieval actually outperforms dense embedding baselines on such tasks is not evaluated in the content available.

Target Audience

Researchers and practitioners in software engineering, information retrieval, and knowledge-graph construction who are interested in large-scale semantic annotation of code. It is especially relevant to teams building code search or retrieval systems, to those applying LLMs as annotators under open taxonomies, to anyone working on entity linking against Wikidata at scale, and to corpus maintainers who need a calibrated protocol for measuring annotation precision without exhaustive human labelling. Readers seeking detailed per-query retrieval evaluations or function-level quality analyses will find those outside the scope reported here.

Authors’ abstract

Public software repositories, like GitHub and Software Heritage Archive, store billions of files, yet extracting their implicit engineering knowledge ---i.e., the algorithms they implement, the paradigms they follow, the patterns they instantiate, and the application domains they serve--- remains challenging, as current tools are constrained to syntactic and token-level analysis. We present a pipeline for building an open-taxonomy semantic annotation of source code using a code-specialised Large Language Model. The extracted entities are grounded in Wikidata through a three-stage linking procedure: a deterministic SPARQL stage handles unambiguous entities, a Deep Research Agent resolves the residual long tail, and a hierarchy-rollup stage imports the parent-of closure of each resolved Wikidata identifier. The resulting annotations are materialised as a source-code-specific open-taxonomy knowledge graph. We further introduce a calibrated quality-assurance protocol that quantifies annotation precision by combining a small human gold set with an LLM-as-a-judge filter. We applied our pipeline to the 167 million files of the Stack-Edu corpus, creating the first known large-scale open-taxonomy knowledge graph for source code. Our graph, named CodeGraph, contains approximately 158 million nodes, which include around 145 million files, about 63,000 extracted concept entities (such as algorithms, paradigms, design patterns, and application domains), and roughly 19,800 grounded Wikidata entities. Furthermore, CodeGraph features approximately 1 billion typed edges that connect files to their respective concepts, link these concepts to their grounded Wikidata identifiers, and relate them to their parent categories, covering 14 programming languages.