TernScope: Model-Hardware Co-Design for Efficient Ternary Endoscopic Segmentation

AI summary

Overview

Research area: Efficient medical image segmentation and model–hardware co-design (edge deployment of low-bit neural networks for endoscopy).

Technical level: Advanced. The paper assumes familiarity with quantization-aware training, structural re-parameterization, systolic-array accelerators, and standard-cell synthesis.

Scope: TernScope presents a fully ternary-weight (W2A8) endoscopic segmentation network built through branch fusion and progressive quantization-aware training, paired with a Gemmini-based ternary accelerator that is compared against an INT8 baseline at 45 nm.

What This Paper Is About

Edge medical segmentation faces a three-way tension between model size, accuracy, and inference speed: a deployable model must fit in limited memory, delineate lesions accurately, and return predictions quickly. The authors argue that shrinking width or spatial processing alone sacrifices the contextual information segmentation depends on, and that a compact network does not automatically yield fast inference or small hardware. TernScope's goal is to treat predictive quality and deployment cost as joint design objectives, learning a compact segmentation graph whose low-bit operators also simplify the arithmetic a custom accelerator has to execute.

Key Contributions

1. SplitLK block: A split-channel large-kernel design in which each encoder stage allocates only a subset of its channels to spatial convolution (16/16/16/32 processed channels out of 32/64/96/128 total), with the remaining channels bypassing spatial processing and rejoining before a pointwise mixer. Kernels of size 5, 7, and 9 operate at 32×32 resolution or below.

2. Nearest Unifier decoder: A narrow multi-scale fusion module that projects all four encoder scales to 16 channels, aligns them with nearest-neighbor resizing, and combines the three deeper features before fusing with the shallow feature — keeping fused width at 16 channels via addition rather than concatenation.

3. Ternary Progressive Quantization-aware Training (TPQ): A schedule that first fuses RepLK training branches into single deployment convolutions, then adapts the fused graph from deep to shallow encoder stages, through skip projections and decoder, and finally to the input convolution and segmentation head — reaching all 25 convolutions in ternary by 40% of training.

4. A matched model-to-hardware mapping: A 16×16 ternary PE array with a shared Affine & Quantization Engine, synthesized in the same Nangate45 45-nm library and Yosys/ABC flow as an INT8 Gemmini baseline, over the same 25-convolution workload.

Main Findings

• Compact model, competitive accuracy: TernScope-Net uses 0.2483 M parameters and 0.0949 GMACs at W2A8, reaching Dice of 0.8158 on Kvasir-SEG test, 0.7291 on ClinicDB, and 0.5945 on ColonDB, with IoU of 0.7247 / 0.6290 / 0.5015.

• Favorable comparison to U-Net: The model uses approximately 1/125 of U-Net's parameters and 1/508 of its MACs, with in-domain Dice within 0.51 percentage points of U-Net and external Dice higher by 5.00 and 5.85 points on ClinicDB and ColonDB respectively.

• Parameter count alone does not explain the result: EGE-UNet (0.0534 M parameters, 0.0720 GMACs) and Mobile-PolypNet (0.2239 M, 1.1380 GMACs) have fewer parameters yet lower overlap scores, which the authors attribute to retaining spatial context inside a compact W2A8 graph.

• Fusion is lossless in FP32: Branch fusion removes 23.21% of parameters and 10.78% of MACs (0.3233 M / 0.1064 GMACs down to 0.2483 M / 0.0949 GMACs) while preserving identical FP32 Dice and IoU on all three datasets.

• Post-training ternarization collapses: Applying ternary PTQ to the fused model drives Dice and IoU to 0.0002 / 0.0004 / 0.0007 across the three datasets despite unchanged topology.

• TPQ beats direct QAT: Direct ternary QAT yields Dice of 0.8075 / 0.7029 / 0.5636, while TPQ reaches 0.8158 / 0.7291 / 0.5945 — a 2.62- and 3.09-point external Dice gain. TPQ also exceeds the FP32 reference on all three datasets.

• Compute-array savings: Against the original Gemmini under the same 45 nm technology, TernScope reduces compute-array area by 88.46% (0.685 mm² to 0.079 mm²) and synthesized standard-cell area by 70.28% (1.816 mm² to 0.540 mm²); the authors also report estimated logic power reduced by 62.82%.

• Throughput and energy: At 100 MHz on the evaluated workload, TernScope reaches 102.701 FPS versus 11.902 FPS for Gemmini (8.629×), and logic energy cost falls from 14.019 to 0.604 mJ per image, a 95.690% reduction.

• Memory tradeoff: SRAM increases slightly from 320 KiB to 344 KiB, while packed W2 weights occupy 1/4 of the W8 payload.

Methodology in Plain English

The authors build the network first and simplify it second. Large kernels are expensive mainly because convolution cost scales with the square of channel count, so instead of applying a big kernel to every channel, they apply it to a small slice of channels and let cheap pointwise mixing spread information across the rest — processing 32 of 128 channels cuts that spatial term to 1/16 of the full-width version. During training the spatial path carries several parallel branches (RepLK); before deployment, batch normalization is folded into each branch and the branches are summed into one convolution using center-padded kernels, with nonlinearities kept outside the merge. The decoder projects all four encoder scales to 16 channels and adds them rather than concatenating, using nearest-neighbor resizing so no upsampling weights are learned.

Only after this fusion do they quantize, because quantization is nonlinear — quantizing branches that later disappear is not the same as quantizing their sum. The TPQ schedule runs for 200 epochs: the first 5% trains the fused model in W8A8, the next 35% converts weights to ternary progressively from the deepest encoder stages outward through the decoder to the input and head, and the remaining 60% trains the full W2A8 graph. Conversions use scale recalibration and short reduced-learning-rate adaptation, activation scales are frozen after final adaptation, and checkpoints are selected only from the fully W2A8 phase using validation Dice. Training uses Kvasir-SEG only (720/80/200 split), with ClinicDB (612 images) and ColonDB (380 images) used as untouched external test sets.

On the hardware side, a ternary weight in {−1, 0, +1} means the PE either subtracts, skips, or adds the activation, so general multipliers disappear from the product path; wider partial sums preserve range before per-channel scaling and bias addition. A shared Affine & Quantization Engine handles post-processing and two-source feature addition on-chip, offloading work that the evaluated Gemmini mapping leaves to the CPU.

Why This Matters

Impact on research: The paper supplies a concrete link between a quantization schedule and a physical circuit: it shows that progressive, fusion-first ternarization is what makes an all-ternary segmentation graph trainable at all, and then quantifies the resulting area, throughput, and energy changes against a well-known INT8 systolic-array baseline under a matched synthesis flow. It also demonstrates that the "fuse first, quantize second" ordering matters, since PTQ on the fused graph collapses entirely.

Real-world applications:

• Capsule/endoscope edge devices where on-device polyp or lesion delineation must run within tight memory and power budgets.
• Portable or battery-powered point-of-care imaging hardware that cannot rely on a GPU or network backhaul.
• Telemedicine and rural screening settings where inference latency and connectivity are both constrained.
• Low-bit accelerator design more broadly, as a reference point for replacing multipliers with sign-selection datapaths in other dense-prediction workloads.

Industry relevance: Packed ternary weights at 1/4 of the INT8 footprint reduce model storage and weight-transfer pressure, and the reported 8.629× FPS gain and 95.690% per-image logic energy reduction speak directly to the cost and power budgets of medical device manufacturers and edge-AI chip designers.

Future Directions

• Exploit ternary sparsity: The authors name sparsity-aware execution and compressed storage for zero-valued ternary weights as the next step, targeting lower computation and data movement — the paper cites sparse accelerators as motivation but does not implement this.
• Recover the in-domain accuracy gap: TernScope trails U-Net on Kvasir-SEG by the reported 0.51 percentage points while exceeding it on both external sets; whether TPQ, longer training, or distillation (explicitly not used here) closes that gap is open.
• Address the SRAM increase: The design raises SRAM from 320 to 344 KiB relative to the baseline, so buffer sizing and on-chip memory organization remain a design lever.
• Extend the co-design beyond this workload: The comparison is fixed to a single 25-convolution graph, a single 16×16 PE array, and the Nangate45 library at 100 MHz; generalization to other segmentation backbones, other technology nodes, and measured silicon rather than standard-cell estimates is untested.

Target Audience

Graduate students and researchers in efficient medical imaging, quantization, and neural architecture design; hardware architects working on low-bit edge accelerators; and practitioners building deployable endoscopic or point-of-care segmentation systems who need to weigh parameter count, MACs, area, throughput, and energy together rather than in isolation.

Authors’ abstract

Edge medical segmentation must reconcile small model size, high accuracy, and fast inference. TernScope addresses this tradeoff through model--hardware co-design, learning a compact segmentation network whose low-bit operators also simplify hardware execution. Split-channel large-kernel blocks preserve spatial context with limited computation, while branch fusion removes redundant inference paths. Ternary Progressive Quantization-aware Training (TPQ) then adapts the fused network from deep to shallow layers, through the decoder, and finally to the input and prediction layers. All 25 convolutions use ternary weights and 8-bit activations, enabling packed storage and multiplier-free computation. The resulting model uses 0.2483 M parameters and 0.0949 GMACs while retaining competitive segmentation accuracy on in-domain and external datasets. Under the same 45 nm technology, TernScope reduces compute-array area by 88.46% compared with the original Gemmini, a systolic-array accelerator with 8-bit $\times$ 8-bit multipliers. These results demonstrate a path from accurate lightweight medical segmentation to compact low-bit hardware.