Surfing the Activation Wave: Analyzing the Information and Geometry of LLM Hidden States for Speculative Decoding

AI summary

Overview

Research area: Natural Language Processing / efficient LLM inference — specifically speculative decoding and the analysis of internal representations (hidden states) of transformer decoder layers.

Technical level: Advanced. The paper assumes familiarity with transformer architectures, speculative decoding, information theory (Shannon entropy, matrix entropy), and representation-similarity measures (centered kernel alignment).

One-sentence scope: The paper builds a principled, information- and geometry-based framework for choosing which layers' hidden states to use in multi-layer speculative decoding, replacing fixed position-based heuristics.

What This Paper Is About

Speculative decoding speeds up LLM generation by using a smaller draft model to propose tokens that a larger target model then verifies — and many such methods reuse hidden states from several layers of the target model rather than just its final layer. The problem is that deciding which layers to draw from is a combinatorial choice, so existing methods fall back on fixed, position-based rules rather than a principled selection. This paper asks which layers actually carry useful, non-redundant information, and proposes selection algorithms grounded in that analysis.

Key Contributions

1. A two-criteria framework for layer selection. The authors characterize candidate hidden-state sets along two axes: informativeness, measured with matrix-entropy analysis, and diversity (low redundancy), measured with centered kernel alignment (CKA).

2. A layer-wise empirical study of information and geometry. They examine Shannon entropy per layer, aggregate entropy when hidden states are concatenated, and how pairwise CKA changes with depth, reporting that decoder representations form distinct informational and geometric "compartments."

3. Entropy- and CKA-based selection algorithms. Motivated by those observations, they propose algorithms that pick hidden-state sets that are simultaneously highly informative and minimally redundant.

4. Token-entropy filtering. They introduce a preprocessing step that discards hidden states from tokens with low predictive entropy, which is claimed to improve the quality of the statistics used for layer selection.

Main Findings

• Layers encode complementary information, but existing selection is heuristic: the abstract states that decoder hidden states across layers carry complementary information, while speculative decoding methods typically use a small set of layers picked by fixed, position-based heuristics.

• Exhaustive search is infeasible: selecting an optimal layer subset is combinatorial, so the authors note that brute-force evaluation is not practical even for moderately deep models — which is the motivation for a criterion-based approach.

• Representations form distinct compartments: through layer-wise Shannon entropy, concatenation-aggregate entropy, and pairwise CKA trends across depth, the study reports distinct geometric and informational regions within decoder representations.

• Selected sets beat fixed defaults in EAGLE3-style decoding: the authors report that their selected layer sets outperform fixed baseline choices, including the default [2, N//2, N-3] rule, in EAGLE3-style speculative decoding.

• The framework also explains why defaults can work: beyond outperforming the fixed rule, the analysis is said to clarify why certain default layer choices remain competitive rather than being arbitrary.

Note on results: the abstract reports the direction of the comparison (selected sets outperform the fixed baselines) but gives no speedup factors, acceptance rates, accuracy numbers, model sizes, or dataset details. Those specifics are not in the abstract.

Methodology in Plain English

The researchers treat layer selection as a search problem they cannot solve by brute force, so they replace exhaustive testing with measurable proxies for "goodness" of a layer set.

• Measure how much information a layer carries. They analyze hidden-state matrices using entropy-based quantities — first Shannon entropy layer by layer, then the aggregate entropy of concatenating hidden states from multiple layers, to see how much new information each added layer brings.
• Measure how similar layers are to each other. Using centered kernel alignment, they compare hidden-state representations pairwise across depth, which reveals which layers effectively duplicate one another and which are genuinely distinct.
• Turn the measurements into selection rules. They use these two signals to design algorithms that pick layer sets scoring high on informativeness and low on redundancy.
• Clean up the statistics with token filtering. Because some tokens are easy to predict and contribute little signal, they filter out hidden states from low-predictive-entropy tokens before computing the selection statistics.
• Evaluate in a real decoding setup. The chosen layer sets are tested inside EAGLE3-style speculative decoding and compared against fixed baseline layer choices, including the common [2, N//2, N-3] rule.

Why This Matters

Impact on research. The paper reframes multi-layer hidden-state use in speculative decoding from an ad hoc engineering choice into an analyzable question about information content and representational geometry. It offers a reusable methodology — entropy plus CKA — for studying which internal layers of a model are worth reading, and the abstract positions the work as a foundation for extending similar analysis to related applications.

Real-world applications:

• Faster LLM serving: more effective draft-model layer selection can increase the token acceptance rate in speculative decoding, directly reducing latency for interactive chat and API inference.
• Cost reduction at scale: faster per-token generation lowers GPU time and energy per request for high-volume deployments.
• On-device and edge inference: better draft/target alignment helps memory- and compute-constrained settings where a full speculative pipeline may be too expensive.
• Deployment diagnostics: the entropy/CKA analysis can be used to reason about why a particular layer configuration performs the way it does before committing to it in production.

Industry relevance. Speculative decoding is a mainstream inference-acceleration technique in production LLM stacks. A principled, analyzable procedure for choosing which target-model layers to expose to a draft model is directly applicable to inference-serving frameworks, and it reduces reliance on hand-tuned per-model heuristics that must be re-derived whenever the model depth changes.

Future Directions

• Scaling the selection procedure: how well the entropy- and CKA-based algorithms hold up on much deeper models, where the combinatorial space is larger and the reported compartment structure may shift.
• Transferability of selected layer sets: whether layer sets chosen for one model, task, or prompt distribution transfer to others, or whether selection must be re-run per deployment.
• Beyond speculative decoding: the abstract explicitly frames the analysis as a foundation for extending similar layer-information analysis to related applications — for example, other forms of multi-layer feature reuse or interpretability work.
• Validating the informational and geometric compartments: whether the observed entropy and CKA structure is a general property of decoder representations or an artifact of the particular models and setups studied.

Target Audience

Researchers and engineers working on LLM inference efficiency, speculative decoding, and draft-model design; practitioners tuning decoding pipelines for latency or throughput; and interpretability-adjacent researchers interested in layer-wise information content and representational geometry. The paper is best suited to readers comfortable with transformer internals and information-theoretic/representation-similarity measures; the abstract does not indicate that it is written as an introductory treatment.

Authors’ abstract

Decoder hidden states from different layers of large language models (LLMs) often encode complementary information. However, existing speculative decoding methods typically rely on a small set of target-model layers chosen using fixed, position-based heuristics. Selecting an optimal subset of layers is inherently a combinatorial problem, making exhaustive evaluation infeasible even for moderately deep models. In this work, we present a principled framework for multi-layer hidden-state selection for speculative decoding based on two complementary criteria: informativeness, measured through matrix-entropy analysis, and diversity, quantified using centered kernel alignment (CKA). We investigate layer-wise Shannon entropy, aggregate entropy under hidden-state concatenation, and pairwise CKA dynamics across depth, revealing distinct geometric and informational compartments within decoder representations. Motivated by these findings, we propose entropy and CKA-based layer selection algorithms that favor hidden-state sets that are both highly informative and minimally redundant. As part of this framework, we introduce token-entropy filtering, which removes hidden states associated with low predictive-entropy tokens, thereby improving the quality of layer-selection statistics. In EAGLE3-style speculative decoding, our selected sets outperform fixed baseline choices, including the default $[2, N//2, N-3]$ rule, while providing a clearer explanation of why some defaults can remain competitive. Our results provide a practical recipe for multi-layer hidden-state selection for speculative decoding, insights into the geometry and information in LLM decoder states, and a foundation for extending similar analysis to related applications.