Retention-Constrained Post-Training Quantization of Cellpose-SAM for Stem Cell Microscopy

AI summary

Overview

• Research area: Post-training quantization and evaluation methodology for biomedical segmentation foundation models, applied to stem-cell (iPSC) microscopy.
• Technical level: Intermediate. The quantization concepts (bit widths, weight-only vs. activation quantization, per-tensor vs. grouped quantization) are approachable for a reader with basic machine-learning background; the statistical protocol (cluster bootstrap, pre-specified margins) is explained in enough detail to follow without prior specialization.
• Scope: A deployment-oriented evaluation of compressed Cellpose–SAM on a stratified 176-field, three-modality microscopy panel under a pre-specified retention criterion.

What This Paper Is About

Lab-bench and incubator-adjacent imaging hardware usually has no GPU, so a roughly 305 M-parameter segmentation foundation model such as Cellpose–SAM must be compressed before it can be deployed on it. The paper's argument is that picking a compression scheme by a single accuracy number is the wrong question for regulated stem-cell imaging; instead it asks which schemes are safe on which imaging modalities, under a margin fixed in advance and with an audit trail an independent reviewer can re-derive. The goal is a reproducible, modality-stratified retention protocol rather than a leaderboard of compression methods.

Key Contributions

1. A pre-specified retention protocol built on four elements: a criterion defined at a downstream metric (paired instance F1 at IoU 0.5, with AP endpoints) rather than an output-tensor proxy; a margin of −0.02 mean change from FP32 fixed before hold-out data were examined; per-(scheme, imaging modality) verdicts using a 2,000-draw cluster-bootstrap 95% interval so a passing verdict on one modality cannot obscure a failing verdict on another; and released audit artifacts.
2. A sensitivity-guided mixed W4/W8 scheme in which the four highest-sensitivity operators, identified by a one-operator-at-a-time W4 perturbation over 100 eligible Linear/Conv2d operators on a development split, are retained at INT8 while the rest use W4, delivering 6.76× weight-storage reduction with 0/176 observed catastrophic fields.
3. A modality-stratified empirical panel of 176 hold-out fields across BBBC038 (n=70), BBBC039 (n=34), and NIST iPSC (n=72) spanning low, medium, and high culture density, with 107 experimental units as the resampling atom for cluster-bootstrap intervals.
4. A diagnostic mechanism analysis tying each retention verdict to tensor- and encoder-block-level behavior, including raw-output NRMSE, checkpoint-level NRMSE across 26 architectural checkpoints, a weight-bit sweep, and an activation-bit sweep showing sub-A8 rejection as a categorical decoder failure.

Main Findings

• Weight-only W8A16 preserves accuracy across all three modalities. Paired instance F1 changes are ΔF1 ∈ {+0.0004, +0.0003, +0.0006}, with all confidence intervals contained in the [−0.001, +0.002] band. Weight storage falls from 1,162.07 MiB to 295.78 MiB, a 3.93× reduction.
• W4A16-G64 stays within margin on all three modalities. ΔF1 ∈ {−0.0000, +0.0006, −0.0027}, with the widest arm (−0.0027 [−0.0047, −0.0012] on NIST iPSC) still above the −0.02 threshold. Storage is 168.47 MiB, a 6.90× reduction.
• The sensitivity-guided mixed W4/W8 scheme matches W8A16 retention at deeper compression. ΔF1 ∈ {+0.0017, +0.0004, −0.0012}; storage 171.86 MiB, a 6.76× reduction. Its four INT8 exemptions are blocks.0.mlp.lin2, blocks.23.attn.qkv, neck.2, and out.
• Calibrated W8A16-QDQ-obs (weight-only packing) preserves F1 within the margin on every modality, ΔF1 ∈ {−0.0028, +0.0003, +0.0004}, with packed weight storage identical to W8A16 at 295.78 MiB (3.93×).
• Ternary W2A16-G64 fails catastrophically despite the largest compression. It achieves 12.08× (96.21 MiB) but shows ΔF1 = −0.841 [−0.882, −0.793] on BBBC038 and −0.907 [−0.929, −0.876] on BBBC039. On NIST iPSC the paired drop is only −0.023 [−0.054, −0.002] because the FP32 arm is already floor-limited there, but the compressed model produces essentially no correct instances either.
• Catastrophic-failure rates separate the schemes cleanly. W8A16 and the mixed W4/W8 scheme each have 0/176 failed fields; because the percentile bootstrap is degenerate at zero, the paper quotes a rule-of-three upper bound of 3/107 ≈ 0.028 over experimental units instead of a [0, 0] interval. W8A16-QDQ-obs and W4A16-G64 each have 1/176, rate 0.001 [0.000, 0.001]. Ternary W2A16-G64 has 169/176, rate 0.979 [0.949, 1.000].
• Two schemes tie for tightest retention. W8A16 and the mixed W4/W8 scheme both reach 0/176 observed catastrophic fields, so the strongest claim they support is "no failure observed at this sample size," not "failure rate is zero."
• Activation quantization below 8 bits collapses the decoder. On a broader 245-field whole-graph panel (BBBC038 n=99, BBBC039 n=47, NIST iPSC n=99, distinct from the 176-field retention holdout), A8 retains most fields at 83/245 empty, while A4 and A2 collapse to 245/245 empty — a categorical decoder failure rather than a threshold artifact.
• Ternary INT2 detaches by roughly 10× in raw-output NRMSE from FP32 at the flow and cell-probability taps, matching the 169/176 catastrophic fraction.
• Weight-only damage localizes rather than accumulates across the 26 architectural checkpoints (B1–B26, ViT-L transformer blocks plus patch-embedding and neck taps), and median NRMSE moves smoothly with bit width in the weight-bit sweep at fixed A16.
• The NIST iPSC arm is floor-limited on FP32 itself. The paper states that FP32 does not segment reliably on high-density iPSC in this panel, so the NIST verdicts primarily demonstrate that the protocol flags a floor-limited modality rather than that a compressed model preserves iPSC detection quality.

Methodology in Plain English

The authors start from a released FP32 Cellpose–SAM checkpoint (cpsam_v2, roughly 305 M parameters, 1,162.07 MiB) and apply post-training quantization, which needs no additional labeled training data. They assemble a stratified hold-out panel of 176 fields covering three imaging modalities, with sensitivity analysis, bit allocation, and calibration done on a disjoint development split.

Instead of scoring a compression scheme by one number, they define a retention criterion in advance: for each combination of scheme and imaging modality, compute the mean change from FP32 in paired instance F1 at IoU 0.5, then build a 2,000-draw cluster-bootstrap 95% interval over experimental units. A scheme passes on a modality only if the entire interval stays above the fixed −0.02 margin. Resampling happens at the experimental-unit level (107 units across the panel), so BBBC fields resample individually while NIST tiles resample by their 3 source images. Sample-size design targeted roughly 4% confidence-interval widths.

They also record catastrophic failures — fields whose compressed output falls below an absolute quality floor — as a per-scheme rate with its own bootstrap interval. Diagnostic measurements on raw outputs (NRMSE, cosine similarity, median flow angular error) are reported but explicitly do not gate the accept/reject decision; the verdict is post-decoder.

Six schemes are compared: weight-only W8A16, W4A16-G64 with group size 64, ternary W2A16-G64, calibrated W8A16-QDQ-obs weight-only packing (distinct from the whole-graph W8A8 configurations known to collapse on this decoder), and the sensitivity-guided mixed W4/W8 scheme. The mixed scheme's exemptions came from perturbing one operator at a time to W4 across 100 eligible Linear/Conv2d operators and keeping the four most sensitive at INT8.

Why This Matters

The paper reframes compressed-model deployment as an auditable, modality-specific decision rather than a single accuracy score. For laboratories running iPSC culture, that shift matters because a scheme that silently degrades in the high-density regime could delay passaging decisions or contaminate downstream release-testing counts.

Real-world applications:

• Lab-bench and incubator-adjacent imaging stations without GPUs, where a roughly 305 M-parameter foundation model must fit CPU or edge hardware.
• iPSC process monitoring across the full density trajectory, from sparse post-passage plating through mid-log growth to pre-passage confluence.
• Regulated cell-therapy manufacturing workflows, where an independent reviewer must be able to re-derive a compression verdict on independent data.
• Modality-stratified auditing of other segmentation foundation models (mask-token or U-Net decoders), since the protocol itself is described as the transferable element.

Industry relevance: the paper gives a concrete deployment answer — the mixed W4/W8 scheme halves the storage of W8A16 (6.76× versus 3.93×) with no measurable loss on this panel — while also warning that regulatory release-testing pipelines require a tighter margin than the −0.02 used here.

Future Directions

• Build a NIST-analogue arm where the FP32 reference actually detects instances, assembled from a workflow-specific dataset, since on the present panel NIST is floor-limited and per-modality passing there is necessary but not sufficient for iPSC deployment.
• Re-derive rather than inherit the passing bit widths and operator selections for different decoders, modalities, or margins, because those specifics are properties of Cellpose–SAM's promptless flow-integration decoder.
• Extend the retention protocol to modalities outside the panel — 3D volumetric microscopy, time-lapse phase contrast at extended intervals, and non-nuclear stains — where the current verdicts are explicitly not certified.
• Tighten the margin for regulatory release testing, where the paper states a −0.02 mean F1 drop is not acceptable even though the protocol itself scales.

Target Audience

Readers who benefit most are machine-learning engineers and computational biologists deploying segmentation foundation models on CPU or edge hardware in laboratory and cell-manufacturing settings; regulatory and quality reviewers who need a reproducible audit protocol for compressed models; and methodologists working on evaluation practice for medical imaging, who will find the modality-stratified, pre-specified retention criterion and the cluster-bootstrap design directly reusable. It is also relevant to stem-cell workflow owners who need to understand which quantization schemes are safe for iPSC monitoring and which are not.

Authors’ abstract

Induced pluripotent stem cell (iPSC) culture increasingly relies on segmentation foundation models, yet deployment on laboratory CPUs and edge hardware requires compression schemes that are both efficient and auditable. We present a deployment-oriented evaluation of compressed Cellpose-SAM using a pre-specified retention criterion: the 95% cluster-bootstrap interval of mean change from FP32 must remain above a fixed -0.02 margin for every imaging modality. On a stratified 176-field panel spanning BBBC038 nuclei, BBBC039 U2OS fluorescence, and NIST iPSC images across density regimes, weight-only W8A16 preserves instance F1 across all modalities. A sensitivity-guided mixed W4/W8 scheme, using four INT8 exceptions, achieves a 6.76x reduction in weight storage with no observed catastrophic failures (0/176 fields), matching W8A16 at this sample size. In contrast, ternary weight-only quantization achieves 12.08x compression but fails catastrophically on 169/176 fields. These results demonstrate that compression should be evaluated by modality-stratified downstream retention rather than single-number accuracy, and establish a reproducible protocol for auditing compressed foundation models in regulated stem-cell imaging.