HGAdapter: Hypergraph-based Adapters in Language Models for Code Summarization and Clone Detection

AI summary

Overview

Research area: Natural language processing applied to source code, specifically parameter-efficient fine-tuning of pre-trained language models (PLMs) for code understanding and generation.

Technical level: Intermediate. Readers should be comfortable with concepts such as pre-trained language models, tokenization, adapters/PEFT, graph and hypergraph neural networks, and metrics like BLEU-4 and F1. The paper explains its own architecture in enough detail that a motivated reader can follow the core idea without prior hypergraph expertise.

One-sentence scope: The paper proposes three kinds of "high-order" correlations among code tokens (AST family, lexical, and line), builds a generator that extracts them, and inserts a hypergraph-based adapter (HGAdapter) into existing PLMs to improve code summarization and clone detection.

What This Paper Is About

Transformer-based language models represent code as pairwise relationships between tokens through self-attention, which the authors argue misses "high-order" correlations where several tokens act together as one structural unit. The paper identifies three such groupings inside source code and asks whether encoding them as hyperedges — edges that can connect more than two entities — can improve a language model's performance on code tasks. The goal is a lightweight, insertable adapter module that adds this structural signal to a frozen PLM without full fine-tuning.

Key Contributions

1. Three proposed high-order correlations in code tokens: AST family correlation (tokens under a shared abstract syntax tree parent node, e.g. a, +, b forming an addition operation), lexical correlation (a single identifier such as SimpleCalculator split by a tokenizer into multiple sub-tokens), and line correlation (tokens sharing a line of original code).

2. A tokens and hyperedges generator that parses code with tree-sitter, tokenizes with the target PLM's own tokenizer, and records token-to-hyperedge mappings in COO format along with hyperedge types.

3. HGAdapter, a novel adapter module derived from an improved hypergraph neural network architecture. The improvements include a simplified attention mechanism, heterogeneous linear transformations keyed to hyperedge type, and integration with adapter tuning so that PLM parameters stay frozen while only adapter parameters are updated.

4. Empirical validation on two public benchmarks — code summarization on CodeSearchNet across six languages and code clone detection on BigCloneBench — with ablation studies isolating each of the three correlation types and a parameter-count comparison.

Main Findings

• Code summarization gains over full fine-tuning: In overall BLEU-4, HGAdapter improves over full fine-tuned PLMs by 2.33 for RoBERTa, 1.99 for CodeBERT, 2.01 for GraphCodeBERT, 1.91 for UniXcoder, 1.87 for TinyLlama-Math&Code, and 1.76 for Qwen2.5-Coder-0.5B.

• Code summarization gains over standard adapter tuning: Improvements of 2.31 (RoBERTa), 2.08 (CodeBERT), 2.05 (GraphCodeBERT), 2.01 (UniXcoder), 1.45 (Code Llama 7B), 1.92 (TinyLlama-Math&Code), and 1.84 (Qwen2.5-Coder-0.5B).

• Code summarization gains over the structural adapter: HGAdapter surpasses the structural adapter by 1.68 (RoBERTa), 1.52 (CodeBERT), 1.49 (GraphCodeBERT), 1.58 (UniXcoder), 1.01 (Code Llama 7B), 1.60 (TinyLlama-Math&Code), and 1.63 (Qwen2.5-Coder-0.5B). The authors interpret this as evidence that treating the three structural signals as high-order correlations beats treating them as pairwise relationships.

• Encoder vs. decoder asymmetry: HGAdapter produced larger improvements over standard adapters for encoder-based models (RoBERTa, CodeBERT, GraphCodeBERT) than for decoder-based models (Code Llama 7B, TinyLlama-Math&Code, Qwen2.5-Coder-0.5B). The authors suggest this is because encoder models extract richer contextual information, and that Code Llama 7B's large parameter count already provides strong code feature extraction, leaving less room to improve.

• Smaller datasets benefit more: The authors report relatively greater improvements on the Ruby and JavaScript datasets, which they attribute to those datasets being smaller in scale, so that full fine-tuning or general adapter tuning may yield insufficient training effectiveness.

• Highest reported overall BLEU-4: Code Llama 7B with HGAdapter at 21.53 overall, followed by UniXcoder with HGAdapter at 21.06.

• Code clone detection F1 gains over full fine-tuning: 1.28 for CodeBERT, 1.23 for GraphCodeBERT, and 1.12 for UniXcoder.

• Code clone detection F1 gains over standard adapter tuning: 1.31 for CodeBERT, 1.21 for GraphCodeBERT, and 1.17 for UniXcoder.

• Code clone detection F1 gains over the structural adapter: 1.01 (CodeBERT), 0.80 (GraphCodeBERT), and 0.89 (UniXcoder).

• Precision/recall balance differs by model: For CodeBERT, HGAdapter improved precision by 1.87 and recall by 0.69 over full fine-tuning. For GraphCodeBERT, precision rose by 1.76 and recall by 0.71. For UniXcoder, HGAdapter raised recall by 2.35 but did not raise precision — the authors attribute this to UniXcoder's inherently high-precision baseline.

• Ablation, code summarization (CodeBERT): Removing AST family hyperedges, lexical hyperedges, and line hyperedges caused average decreases of 0.59, 1.31, and 0.78 respectively. Lexical hyperedges had the most significant impact of the three. The p-value was less than 0.05.

• Ablation, code summarization (TinyLlama-Math&Code): Removing AST family hyperedges, lexical hyperedges, and line hyperedges caused overall drops of 0.79, 1.12, and 0.62 respectively.

• Ablation, code clone detection (CodeBERT, BigCloneBench): Removing AST family hyperedges decreased precision by 0.89, recall by 0.74, and F1 by 0.82. Removing lexical hyperedges decreased precision by 0.65, recall by 0.37, and F1 by 0.51. Removing line hyperedges decreased precision by 0.48, recall by 0.44, and F1 by 0.46. Here the p-value was less than 0.01, and AST family hyperedges had the most substantial impact — the authors link this to their role in helping models understand code structure.

• Parameter efficiency: Compared to the PLM itself, HGAdapter accounts for roughly 0.3%–1% of parameters. Against the standard adapter, HGAdapter adds about 8% more parameters for RoBERTa, CodeBERT, GraphCodeBERT, and UniXcoder; 2% for Code Llama 7B; 5% for TinyLlama-Math&Code; and 11% for Qwen2.5-Coder-0.5B — an overall range described as approximately 3%–11% additional parameters.

• Dataset scale used: CodeSearchNet training splits range from 24,927 (Ruby) to 251,820 (Python) samples, with test splits from 1,261 (Ruby) to 14,918 (Python). Average tokens per snippet range from 126.27 (Ruby) to 280.58 (Python), with average hyperedges from 53.27 (Ruby) to 74.45 (JavaScript). BigCloneBench has 901,028 training, 415,416 validation, and 415,416 test samples, averaging 401.84 tokens and 182.11 hyperedges.

Methodology in Plain English

The authors split their approach into two parts.

Part one is a tokens-and-hyperedges generator. Instead of feeding raw code straight to the language model's tokenizer, they first parse the code into an abstract syntax tree using the tree-sitter parser. They walk the tree in postorder. At each leaf node they take the code text, run it through the target PLM's own tokenizer, and assign each resulting token a unique ID. If a leaf's text splits into more than two tokens, they create a new hyperedge for those tokens and label it a "lexical hyperedge." They also note the line number of the leaf's text and group tokens by line, creating "line hyperedges." When they reach a parent node, they collect all tokens returned by its children; if there are more than two, they create a new hyperedge labelled an "AST family hyperedge." The token-to-hyperedge associations are stored in COO format (a sparse representation of coordinate pairs). The output is a token sequence plus the hyperedge IDs and types.

Part two is the HGAdapter itself. It is inserted between PLM layers. All PLM parameters are frozen; only adapter parameters are trained. The adapter takes the hidden state vectors produced by a PLM layer for each token, along with the token IDs, hyperedge IDs, and types. It first projects each hidden state down to a smaller dimension (64, per the training settings) and adds the previous HGAdapter layer's output before applying a ReLU activation. It then projects back up to the model's normal hidden dimension and adds the original hidden state, passing the result to the next PLM layer.

The hypergraph message passing is a two-stage process. First, the adapter aggregates token vectors into hyperedge vectors using a simplified attention mechanism, where the attention score between a token and a hyperedge is computed against a learnable query vector that depends on the hyperedge's type, with softmax taken across the tokens belonging to that hyperedge. Second, each hyperedge vector goes through a linear transformation chosen according to its type (a "heterogeneous" transformation), and then hyperedge vectors are aggregated back to the tokens, again with type-aware attention. The resulting token vector is what gets added into the next layer's computation.

Experimental setup: For summarization they use CodeSearchNet in six languages (Ruby, JavaScript, Java, Go, PHP, Python) with BLEU-4 as the metric, the Adam optimizer at a learning rate of 1×10⁻⁴, batch size 64, and 20 epochs. For clone detection they use BigCloneBench (Java) with precision, recall, and F1, the AdamW optimizer at 5×10⁻⁵, batch size 4 (covering 8 code snippets), and 10 epochs. The best adapter parameters by validation score are saved. The adapter's internal vector dimension is 64, and cross-entropy loss is used throughout. Experiments ran on a machine with 64GB RAM and an RTX 3090 GPU with 24GB. In code clone detection with encoder-only models, the hidden state at the starting position serves as the code representation; two code vectors are concatenated and passed to a fully connected classifier.

Why This Matters

Impact on research: The paper argues that self-attention's pairwise view of tokens is an inherent limitation for code, where meaning often lives in groups (an operation, an identifier, a statement). It offers a concrete, parameter-efficient way to inject group-level structure into a frozen model, and its ablation results claim a measurable effect for each of the three correlation types. It also revisits hypergraph neural networks with a simplified attention scheme and type-specific transformations, which may be useful beyond code.

Real-world applications:

• Automated documentation: generating natural-language descriptions for functions and files to help developers navigate unfamiliar codebases.
• Duplicate and near-duplicate code detection: finding copy-paste or semantically equivalent code that may need consolidation or that may indicate license or maintenance problems.
• Code search and retrieval systems that rely on accurate code representations.
• Legacy codebase maintenance, where summarization and clone detection help teams understand large or poorly documented systems.

Industry relevance: The adapter is lightweight — roughly 0.3%–1% of the PLM's parameter count — and is designed to be inserted into a range of models, including small ones like Qwen2.5-Coder-0.5B (0.5B parameters, HGAdapter contributing 3.1M) and TinyLlama-Math&Code (1.1B parameters, 6.1M). For teams that cannot afford full fine-tuning of large code models, this is a relevant trade-off, though the paper also notes increased training time and inference latency.

Future Directions

• Scaling to larger language models. The Limitations section states that the impact of HGAdapter on larger-scale PLMs remains to be explored.

• Reducing the latency overhead. The authors acknowledge that hypergraph construction and processing steps increased training time and inference latency, despite the module's small parameter footprint.

• Extending beyond complete-code-input tasks. HGAdapter relies on tasks where the full code is available, so it is not directly applicable to tasks without code input, such as code generation.

• Mining additional correlations and combining with other PEFT methods. The authors propose exploring more high-order correlations in code and natural language, and investigating how other parameter-efficient fine-tuning methods could be integrated with HGAdapter or with high-order correlations.

Target Audience

This paper is most useful to researchers and practitioners working on code representation learning, parameter-efficient fine-tuning, and graph- or hypergraph-based models for source code. It will also interest engineers who want to add structural awareness to an existing code language model without full retraining, and graduate students looking for a worked example of translating a graph-theoretic idea into a drop-in adapter module with empirical evaluation across six programming languages and two task types.

Authors’ abstract

Pre-trained language models (PLMs) are increasingly being applied to code-related tasks. Although PLMs have achieved good results, they do not take into account potential high-order data correlations within the code. We propose three types of high-order correlations in code tokens, i.e. abstract syntax tree family correlation, lexical correlation, and line correlation. We design a tokens and hyperedges generator to capture these high-order data correlations. We improve the architecture of hypergraph neural networks and combine it with adapter tuning to propose a novel hypergraph-based adapter (HGAdapter) to fine-tune PLMs. HGAdapter can encode high-order data correlations and is allowed to be inserted into various PLMs to enhance performance. Experiments were conducted on several public datasets, including six languages of code summarization and code clone detection tasks. Our methods improved the performance of PLMs in datasets to varying degrees. Experimental results validate the introduction of high-order data correlations that contribute to improved effectiveness.