AdmTree: Compressing Lengthy Context with Adaptive Semantic Trees

AI summary

Overview

• Research area: Natural Language Processing — long-context inference and context compression for Large Language Models.
• Technical level: Advanced. The paper combines gist-token compression, adaptive segmentation driven by perplexity and entropy, and a binary tree aggregation mechanism with custom attention branches.
• Scope: The paper introduces AdmTree, a hierarchical context compression framework that dynamically segments long inputs and builds a bottom-up semantic binary tree of gist-token summaries, evaluated on LongBench and several auxiliary long-context benchmarks.

What This Paper Is About

LLMs slow down and consume large amounts of memory as input context grows, because self-attention cost scales quadratically with token count. Existing ways to shorten context fall into two camps — explicit methods that delete text (losing fine-grained detail) and implicit methods that encode text into compact latent vectors (losing information at certain positions, a positional bias often described as "lost in the middle"). AdmTree's goal is to compress long context while preserving both global meaning and local detail, and to do so without training a large number of new parameters.

Key Contributions

1. An adaptive, hierarchical compression framework. AdmTree segments input based on information density, inserts gist tokens (⟨GT⟩) to summarize variable-length segments as the leaves of a semantic binary tree, and aggregates those leaves bottom-up.
2. A density-aware budget allocation rule. Segment importance is scored with an entropy-adjusted perplexity measure, and gist-token budgets are redistributed so high-scoring segments receive finer sub-segmentation while the overall compression ratio stays fixed.
3. A lightweight, tunable-only-what-is-needed design. The backbone LLM is frozen; only the gist-token attention heads, the gist-token embedding, and a single-layer self-attention aggregator are trained.
4. Bidirectional aggregation and dynamic update. The tree's self-attention aggregator lets later segments influence earlier ones (mitigating the unidirectional limits of causal LLMs), and the tree can be updated incrementally as new context arrives.

Main Findings

• State-of-the-art on LongBench. AdmTree is evaluated across five task types and more than ten datasets from LongBench on LLaMA-2-7B and Qwen-2-7B. The introduction states it surpasses baseline methods by over 10% while maintaining high inference efficiency.
• LLaMA-2-7B average score. AdmTree reaches an average of 44.1 versus 40.1 for Activation Beacon, 39.8 for the fine-tuned original LLM, and 37.2 for the original LLM.
• Qwen-2-7B average score. AdmTree reaches an average of 49.7 versus 47.2 for Activation Beacon, 47.4 for the fine-tuned original LLM, and 45.7 for the original LLM.
• Per-task gains are largest in question answering. On LLaMA-2-7B, AdmTree scores 36.3 on MultiDoc QA against 27.5 for Beacon; on Qwen-2-7B it scores 45.9 against 40.3. The paper reports that performance gains in QA tasks exceed 327.5% in some tasks, and that AdmTree beats Beacon by up to 20 points in some QA tasks.
• Latency stays comparable. On LLaMA-2-7B, AdmTree reports latency 7.8 and a 3.3× speedup, versus 8.0 and 3.2× for Beacon. On Qwen-2-7B it reports 7.0 and 3.4× versus 7.3 and 3.2× for Beacon.
• Outperforms full fine-tuning. The paper states AdmTree is the only compression-based method that consistently beats the Original LLM-FT baseline across both backbone models.
• Prior probing experiments showed the trade-off. On BookSum multi-granularity summarization, explicit methods (LongLLMLingua) declined as summarization granularity became finer, while on NaturalQuestion multi-document QA with 30 documents, implicit methods (Activation Beacon, SnapKV) performed better when the answer-bearing document was placed at the end.
• Ablation (LLaMA-based AdmTree, Single-Doc score). Full model 36.5; without pre-training 26.6; without fine-tuning 29.3; without the tree structure 28.5; without self-attention in aggregation 29.6; with uniform instead of adaptive leaf construction 34.1; with retrieval of the top 75% of tree nodes 32.8.
• Multi-turn dialogue efficiency (ShareGPT). Over turns averaging 765, 3,006, and 6,491 tokens, AdmTree attains perplexity 4.01, 2.91, and 2.79 — the lowest at every turn, and decreasing as the dialogue grows — with latency 0.37, 0.70, 0.92 and TFLOPs 8.37, 34.09, 75.38.
• Robustness to context length and compression ratio. On the MSC dataset (MemGPT-constructed, processed into 1K, 2K, and 4K token samples, measured with ROUGE-L), AdmTree outperforms all baselines across every context length and compression ratio; several baselines collapse with ROUGE-L below 10 under long context combined with high compression.
• Fine-grained retrieval. On Needle-in-the-Haystack with LLaMA-2-7B, AdmTree retrieves the needle consistently across varying document lengths and needle positions, with input contexts substantially longer than those seen in training.

Methodology in Plain English

The approach has three stages.

1. Building leaf gist tokens. The input is first cut into uniform segments of length n. Each segment is then scored with Score(X_i) = PPL(X_i) · exp(−λ · Entropy(X_i)), combining perplexity with an entropy term. Segments are ranked by this score, and their gist-token budgets are reassigned: the top 25% get n/τ tokens, the middle 25% get n/2τ, and the bottom 50% get n/4τ. The overall compression ratio stays at τ, but the local ratio adapts to how information-dense each region is. Each resulting sub-segment is followed by a special ⟨GT⟩ token added to the LLM vocabulary.

2. Building the semantic tree. The hidden states of the gist tokens become leaves of a binary tree, assembled bottom-up. A parent node is computed from its children by a single-layer self-attention followed by an average. The tree is deliberately allowed to be unbalanced — no padding tokens are used, and an unpaired gist token is simply deferred to the next level of aggregation. The tree is also built incrementally: when a new gist token arrives, the earlier tree is partly reused.

3. Compressing and generating. Text tokens use the frozen LLM's original projection matrices, while gist tokens use newly introduced, separately trained projections, giving two attention branches that are then concatenated into a single sequence. Causal self-attention with relative positional embeddings is applied, and tree nodes are flattened left-to-right, bottom-to-top. Only the gist attention heads, the gist embedding, and the aggregator are trained, via next-token prediction on the original input; the backbone stays frozen. Because the aggregator is small and built once at inference, the cost stays on the order of O((L/τ)²) rather than growing further.

For the main experiments, LLaMA-2 (maximum sequence length 4K) uses ×2 compression for 4K–8K contexts, ×4 for 8K–16K, and ×8 for 16K–32K; Qwen-2 uses a uniform ×4. Results are averaged over three inference runs. Backbones are LLaMA-2-7B-Chat and Qwen-2-7B-Instruct. Training details are stated to be in the appendix, which is not included in the provided content.

Why This Matters

Impact on research. The paper reframes context compression as a multi-dimensional problem: a single model must preserve global gist, local detail, and information at all positions simultaneously. It also points out that tree-node attention scores give interpretability that black-box implicit compression methods lack, and it suggests that Mixture-of-Experts architectures could further specialize compression across task types.

Real-world applications.

• Long-running chat assistants and online dialogue systems where conversation history accumulates turn by turn and cannot be recompressed from scratch each time.
• Retrieval-augmented generation pipelines that feed many documents into a limited context window.
• Multi-document question answering over large corpora where the answer may sit anywhere in the input.
• Long-document summarization at multiple granularities, from whole-book summaries down to paragraph-level detail.

Industry relevance. Memory and latency are direct serving costs. AdmTree claims no additional latency relative to other recursive compression methods while improving accuracy, and it trains only a small set of parameters on top of a frozen backbone — which lowers both the cost of adapting an existing model and the risk of degrading the base model's general abilities. The option to retrieve only the top 75% of tree nodes at inference offers a tunable budget for deployments with tight compute limits.

Future Directions

1. Composition with Mixture-of-Experts. The paper explicitly raises the possibility of integrating MoE architectures to specialize compression across diverse task requirements.
2. More aggressive inference-time tree pruning. Retrieving only the top 75% of tree nodes cost little accuracy; the paper frames this as a starting point for further inference-time optimization under limited compute budgets.
3. Improving the explicit-method / implicit-method split. Since explicit methods remain competitive on summarization while implicit methods lead on QA, an open question is whether a single framework can be tuned or routed per task type.
4. Behavior at extreme lengths. The paper notes that Needle-in-the-Haystack contexts substantially exceeded the lengths seen in training while performance stayed stable, which invites further study of how far this generalization extends and how the compression ratio should be set beyond the ranges tested.

Target Audience

Researchers and engineers working on long-context LLMs, context compression, efficient inference, and memory-constrained deployment. It is most useful to readers who already understand transformer attention and token-level compression concepts; the ablation and latency tables are directly actionable for practitioners choosing a compression method, while the tree-construction formulation will interest researchers studying hierarchical or structured representations of context.

Authors’ abstract

The quadratic complexity of self-attention constrains Large Language Models (LLMs) in processing long contexts, a capability essential for many advanced applications. Context compression aims to alleviate this computational bottleneck while retaining critical semantic information. However, existing approaches often fall short: explicit methods may compromise local detail, whereas implicit methods can suffer from positional biases, information degradation, or an inability to capture long-range semantic dependencies. We propose AdmTree, a novel framework for adaptive, hierarchical context compression with a central focus on preserving high semantic fidelity while maintaining efficiency. AdmTree dynamically segments input based on information density, utilizing gist tokens to summarize variable-length segments as the leaves of a semantic binary tree. This structure, together with a lightweight aggregation mechanism and a frozen backbone LLM (thereby minimizing new trainable parameters), enables efficient hierarchical abstraction of the context. By preserving fine-grained details alongside global semantic coherence, mitigating positional bias, and dynamically adapting to content, AdmTree robustly retains the semantic information of long contexts.