ModHiFi: Identifying High Fidelity predictive components for Model Modification

AI summary

Overview

Research area: Model compression and model editing for deep neural networks — specifically structured pruning and classwise unlearning of open-weight models without access to training data, loss functions, or gradients.

Technical level: Advanced. The paper builds on Lipschitz continuity arguments, constrained binary quadratic optimization, and a custom fidelity metric derived from layer-wise reconstruction error.

Scope: The paper proposes a single theoretical and algorithmic framework (ModHiFi) for identifying high-fidelity ("HiFi") model components using only distributional access via synthetic data, and applies it to structured pruning (ModHiFi-P) and classwise unlearning (ModHiFi-U).

What This Paper Is About

Open-weight models are widely available, but their original training data and loss functions usually are not. That makes it hard to modify them for tasks like pruning away unnecessary parameters or unlearning a specific class. The authors ask which components of a trained network actually drive its predictive performance, and they answer this without gradients, without ground-truth labels, and with only synthetic data as a proxy for the training distribution.

Key Contributions

1. Local-to-global error bound (C1). Theorem 3.6 shows that for Lipschitz-continuous networks, the global prediction error is at most linear in the per-channel local reconstruction errors. Corollary B.4 argues that well-trained Transformers, contrary to prior claims in the literature, do exhibit Lipschitz continuity, so the bound applies to CNNs, ViTs, and LLMs alike.

2. Subset Fidelity as an importance measure (C2). Rather than scoring single components, the authors define Subset Fidelity, which measures how well a subset of input contributions can reconstruct a layer's output after optimal weight compensation. Lemma 3.2 establishes boundedness (between 0 and 1) and monotonicity. Computing the optimal subset is NP-complete (equivalent to a Maximum Clique problem), so Theorem 3.9 proves that selecting singletons with the highest fidelity scores is optimal when the features are uncorrelated.

3. The ModHiFi algorithm and its two variants (C3). ModHiFi-P prunes by retaining HiFi components, and ModHiFi-U unlearns by zeroing HiFi components computed from the forget class only. Neither requires gradients, a loss function, or the original training data.

4. Unified treatment of pruning and unlearning. The paper frames both tasks as instances of a single mask-selection problem (Equation MODIFY), with structured pruning as one instantiation (Equation STRUCT-PRUNE) and classwise unlearning as another (Equation UNLEARN).

Main Findings

• Small HiFi sets exist. Across all evaluated models, each layer typically contains fewer than 20% of input channels that achieve subset fidelity of at least 0.8 (Figure 1 and Observation 1).

• HiFi components carry predictive weight. On ResNet-50 with CIFAR-10, perturbing 20% of the HiFi components with zero-mean Gaussian noise of standard deviation 0.01 drops accuracy by about 12%, while perturbing 80% of the non-HiFi components identically drops accuracy by only 1%. At 50% of components and noise standard deviation 0.02, accuracy drops by 85% for HiFi components versus about 1.4% for non-HiFi components.

• Pruning results on ImageNet ResNet50 (Table 1). The unpruned model reports 76.1 accuracy. One ModHiFi configuration reaches 76.70 accuracy at 2.17x FLOP reduction and 1.69x CPU / 1.70x GPU speedup; a second reaches 73.82 accuracy at 3.66x FLOP reduction, 3.05x parameter reduction, 2.42x CPU speedup and 2.38x GPU speedup. For comparison, DFPC (54) reports 73.80 accuracy at 3.46x FLOP reduction with 2.37x CPU and 2.38x GPU speedup, and DFPC (30) reports 75.9 accuracy at 1.98x FLOP reduction. The abstract states ModHiFi-P achieves an 11% speedup over the current state of the art on ImageNet models.

• Pruning results on CIFAR-10 ResNet50 (Table 2). The unpruned model reports 94.99 accuracy. ModHiFi reaches 91.02 accuracy at 4.07x FLOP and 5.36x parameter reduction, versus L2 pruning with synthetic tuning at 90.12 accuracy at 4.07x FLOP and 4.71x parameter reduction, and plain L2 pruning at 15.91 accuracy at 4.07x FLOP and 4.71x parameter reduction.

• Language model results (Table 3, Llama-2-7B). The dense model reports WikiText perplexity 5.12 and average task accuracy 69.00. At 10% sparsity, ModHiFi-P-Alpaca reports perplexity 6.36 and average accuracy 65.96, versus SliceGPT at 6.46 and 59.96. At 20% sparsity, ModHiFi-P-Alpaca reports 9.38 perplexity and 60.62 average, versus ShortGPT at 14.32 and 59.95, and SliceGPT at 8.13 and 51.39. At 30% sparsity, ModHiFi-P-WikiText reports 11.53 perplexity and 48.63 average, versus SliceGPT at 10.96 and 45.82, and ShortGPT at 33.21 and 53.25. Calibration used WikiText-2 with 128 samples of length 1024; no post-pruning recovery fine-tuning was applied by any algorithm.

• Unlearning without fine-tuning. ModHiFi-U achieves complete unlearning on all CIFAR-10 classes without fine-tuning and runs 10 times faster than the baseline in the general editing framework. The truncated Table 4 shows the ResNet-50 base at 94.99 forget and 94.99 retain accuracy, Gradient Ascent at 6.59 forget / 93.44 retain in 30 seconds, and Jia et al. [33] at 3.54 forget / 94.14 retain; the timings and remaining rows for other algorithms were not included in the available content.

• Swin-Transformer limitation and recovery. Without fine-tuning, ModHiFi-U on Swin Transformer fails to achieve state of the art. Only three epochs of fine-tuning on synthetic samples produce a drastic improvement, and after 10 epochs of fine-tuning the forget accuracy is superior to Jia et al. [33], who use full training, when synthetic samples are used. Both forget and remain accuracy are superior when training samples are used. Comparable observations are reported for VGG-19.

• Calibration data quality matters. ModHiFi-P-Alpaca outperforms ModHiFi-P-WikiText on Llama-2-7B. The accuracy-sparsity tradeoff degrades with lower-quality synthetic samples, but less than L2 pruning followed by fine-tuning on synthetic samples.

Methodology in Plain English

The starting point is to stop thinking about individual neurons and instead ask how well a group of components can reproduce a layer's output. For each output channel, the authors compute a fidelity score: take a subset of the input contributions, allow each to be rescaled by an optimal compensation term, and measure how much of the channel's output energy the subset recovers. A score near 1 means the subset explains the output almost completely.

The theory links this local number to global behavior by showing that the model's overall prediction error is bounded by a linear function of these local reconstruction errors, provided the network is Lipschitz continuous. The authors argue that well-trained Transformers satisfy this, which extends the argument beyond CNNs.

Scoring all possible subsets is combinatorial and equivalent to a Maximum Clique problem, so the practical shortcut is to score single components and keep the top ones. Theorem 3.9 justifies this when input contributions are uncorrelated — an assumption that rarely holds exactly but works well empirically. Since the closed form requires only expected inner products of activations and outputs, everything can be estimated with forward passes over synthetic samples, with no gradients and no labels. For vision models the scalar coefficient is estimated directly from batched forward passes; for LLMs a Cholesky-based heuristic is used.

Pruning keeps the high-fidelity components and zeroes the rest, then applies the closed-form compensation to the surviving weights. Unlearning inverts the logic: fidelity scores are computed using only samples from the forget class, and those high-fidelity components are zeroed out.

Why This Matters

Impact on research. The paper offers a theoretical bridge from layer-local reconstruction error to global predictive error, which is unusual for this literature, and it argues against the common claim that Transformers are not Lipschitz continuous. It also provides one framework that handles both pruning and unlearning, which prior work treated as separate problems with separate machinery.

Real-world applications:

• Deploying large vision and language models on hardware with tight compute or memory budgets by removing entire channels and features.
• Removing a specific class or content type from a deployed model to satisfy privacy or data-removal requests.
• Modifying third-party or downloaded open-weight models when the original training corpus is unavailable for licensing or privacy reasons.
• Editing models in settings where synthetic data is already the standard stand-in for real data, as in modern language modeling pipelines.

Industry relevance. The method needs only forward passes over synthetic or calibration data, so it fits workflows where gradients and labels are unavailable or expensive. For LLMs, the use of a calibration set of 128 samples of length 1024 is consistent with existing practice, and the reported ImageNet speedups and CIFAR-10 unlearning results target practical deployment constraints rather than benchmark-only gains.

Future Directions

• Extending the analysis to attention specifically; the current work focuses on the feed-forward network and explicitly leaves attention-specific analysis for future work.
• Exploring other measures of distributional similarity beyond the expected squared difference used in the Subset Fidelity definition.
• Closing the gap on Swin Transformers, where ModHiFi-U without fine-tuning does not reach state of the art and needs fine-tuning to become competitive.
• Addressing the gap between the uncorrelated-feature assumption of Theorem 3.9 and real networks, where correlated features are the norm, plus the NP-hardness of exact subset selection.

Target Audience

Researchers and practitioners in model compression, model editing, and trustworthy machine learning who work with open-weight models and cannot access original training data or loss functions. It also suits theoretically inclined readers interested in Lipschitz-based error propagation and in resource-constrained or privacy-motivated model modification.

Authors’ abstract

Open weight models, which are ubiquitous, rarely provide access to their training data or loss function. This makes modifying such models for tasks such as pruning or unlearning, which are constrained by this unavailability, an active area of research. Existing techniques typically require gradients or ground-truth labels, rendering them infeasible in settings with limited computational resources. In this work, we investigate the fundamental question of identifying components that are critical to the model's predictive performance, without access to either gradients or the loss function, and with only distributional access such as synthetic data. We theoretically demonstrate that the global error is linearly bounded by local reconstruction errors for Lipschitz-continuous networks such as CNNs and well-trained Transformers (which, contrary to existing literature, we find exhibit Lipschitz continuity). This motivates using the locally reconstructive behavior of component subsets to quantify their global importance, via a metric that we term Subset Fidelity. In the uncorrelated features setting, selecting individual components based on their Subset Fidelity scores is optimal, which we utilize to propose ModHiFi, an algorithm for model modification that requires neither training data nor access to a loss function. ModHiFi-P, for structured pruning, achieves an 11\% speedup over the current state of the art on ImageNet models and competitive performance on language models. ModHiFi-U, for classwise unlearning, achieves complete unlearning on CIFAR-10 without fine-tuning and demonstrates competitive performance on Swin Transformers.