NLI:Non-uniform Linear Interpolation Approximation of Nonlinear Operations for Efficient LLMs Inference

AI summary

Overview

Research area: Efficient machine learning systems — specifically software/hardware co-design for accelerating the nonlinear operators (SiLU, RMSNorm, Softmax, Sigmoid) inside large language models.

Technical level: Advanced. The paper combines dynamic-programming algorithm design, FP16 numerical error analysis, RTL circuit design (Chisel), and 28nm synthesis, plus LLM benchmarking. Readers need background in transformer architecture, numerical methods, and digital hardware.

Scope in one sentence: The paper introduces NLI, a calibration-free lookup-table framework that places interpolation cutpoints via dynamic programming and implements them in a low-overhead hardware unit for nonlinear LLM operations.

What This Paper Is About

Linear layers in LLMs have been heavily optimized through low-bit quantization and integer hardware, but nonlinear layers (SiLU, RMSNorm, Softmax) still rely on expensive high-precision floating-point transcendental functions such as exp and sqrt. Prior general-purpose approaches like NN-LUT break down on modern LLMs because their approximations were validated only over narrow input ranges, while LLM activations routinely exceed ±100. The goal of this paper is an approximation scheme that is accurate over the entire FP16 domain, requires no data calibration, and maps cheaply onto hardware.

Key Contributions

1. NLI algorithm (software): A non-uniform linear interpolation framework that recasts cutpoint selection as a discrete dynamic programming problem. Given a fixed nonlinear operator and N sorted candidate points on the FP16 grid, it minimizes an additive per-segment interpolation error over M segments, and because the objective has optimal substructure, Bellman's optimality principle yields the globally optimal partition in 𝒪(M × N²) time. The resulting lookup table is calibration-free — it depends only on the function and numeric settings, not on data distributions — so it is reusable across layers and models.

2. NLI Engine (hardware): A plug-and-play, general-purpose nonlinear computation circuit. Through software-hardware co-design, it uses a two-level address translation module to cut address-conversion overhead, and pipelining to boost throughput. It is synthesized in SMIC 28nm.

3. Two-level LUT layout: A hardware-consistent layout of 2 + 8×32 + 1 = 259 cutpoints — 10 macro-intervals where the first and last are not subdivided and each middle interval is uniformly split into 32 bins. The DP only optimizes the 11 macro endpoints, reducing search time roughly 26× versus optimizing all 259 points directly.

4. Comprehensive evaluation: Software experiments on Llama and Qwen at multiple scales, plus ViT and CNNs; hardware experiments comparing area, power, throughput, and efficiency against two state-of-the-art units (NN-LUT and RI-LUT).

Main Findings

• FP32-level accuracy without calibration: Across Llama3-8B, Llama3-70B, Qwen2.5-7B, Qwen2.5-32B, Qwen1.5-110B, Qwen3-8B, and Qwen3-30B-A3B, NLI matches FP32 on MMLU, GSM8k, HumanEval, the zero-shot average, and Wikitext-2 perplexity. For example, Llama3-8B records 62.16 MMLU / 6.14 perplexity under FP32 versus 62.14 / 6.14 under NLI.

• NN-LUT collapses under LLM outliers: On Qwen2.5-32B, NN-LUT produces 25.51 MMLU, 0 on GSM8k and HumanEval, and a Wikitext-2 perplexity of 70360. On Qwen2.5-7B the NN-LUT perplexity is 28194, and on Qwen3-8B it is 825.31. Figure 1(b) shows a perplexity surge up to 7.0 × 10⁴.

• Extreme activations motivate the design: SiLU inputs in seven representative LLMs frequently exceed ±100. The paper reports that the range [-150, 150] covers ≥99.9% of activations, whereas NN-LUT was designed and validated only for the input domain (-5, 5), with a re-implementation trained on [-10, 10] for SiLU.

• Near-machine-precision error bound: With 259 cutpoints, NLI keeps worst-case absolute error below 1.2 × 10⁻³ across the FP16 domain (shown for exp and rsqrt), reducing worst-case error by several orders of magnitude versus NN-LUT.

• Ablation I — macro-only is insufficient: With only M = 11 optimized endpoints and no micro-bins, Qwen2.5-7B drops to 21.14 MMLU and 0 on GSM8k.

• Ablation II — the two-level layout is a good trade: NLI (259 cutpoints) searches in 610 seconds versus 17000 seconds for a direct DP over 259 non-uniform cutpoints (about 28× slower), with essentially unchanged accuracy, and the unconstrained layout is hardware-unfriendly.

• Ablation III — NLI beats standard heuristics at equal budget: At 259 cutpoints on Qwen2.5-7B, uniform spacing gives 45.91 MMLU / 18.13 GSM8k and a curvature-proportional layout gives 65.74 / 32.58, while NLI gives roughly 70.65 / 43.97, close to FP32's 70.56 / 44.28.

• Comparators reduced from 259 to 10: The two-level address translation replaces 258 sub-intervals and 259 parallel comparisons with 10 macro-interval comparators, using only 10 16-bit registers for precomputed scale factors.

• Hardware is substantially smaller and more efficient: NLI occupies 7787 μm², 34 mW at 1 GHz, versus 23238 μm² / 46 mW for NN-LUT and 23647 μm² / 48 mW for RI-LUT. All three achieve 1 G throughput. Efficiency (throughput/(area×power)) is 3.78 for NLI versus 0.94 and 0.88, i.e. 4.02× higher than NN-LUT and 4.29× higher than RI-LUT. NLI saves 68% and 69% in area versus the other two.

• Storage savings: NN-LUT and RI-LUT must store 512 × 16 bits (256 K values and 256 B values), while NLI stores 259 16-bit cutpoint values plus 10 16-bit scale factors.

• Generality beyond LLMs: Substituting NLI for nonlinear operators in ViT and representative CNNs yields no statistically significant accuracy degradation, with per-model results in Appendix A.5.1 (Table 7).

Methodology in Plain English

The core idea is to replace every expensive nonlinear function with a lookup table plus a straight-line interpolation between stored values. The interesting question is where to put the table's breakpoints.

The researchers start from the standard error bound for linear interpolation: error grows with the square of segment length and with the function's curvature, plus a finite-precision term from storing and multiplying in FP16. Uniform spacing wastes budget in flat regions and under-resolves curved ones; curvature-driven spacing can fail when second derivatives aren't available in closed form; and training/calibration-based fitting ties accuracy to the data span, which is exactly what breaks when LLM activations run past ±100.

Instead, they treat breakpoint placement as a shortest-path-style problem over the sorted list of all finite FP16 values in the function's legal domain. A DP table D[L, k] records the minimum error over the prefix up to point k when k is the L-th cutpoint, and P[L, k] records which predecessor achieved it. Segment cost is the mean relative error between the true function and the line joining the segment endpoints, with a denominator floor of τ = 2⁻¹⁴ (the smallest positive normal FP16 value) so near-zero activations don't blow up the ratio. Boundary terms handle clamping at the left and right ends. Backtracking through the predecessor table recovers the optimal cutpoints.

To keep hardware cheap, they don't let the DP choose all 259 points. They fix a 10-macro-interval skeleton — first and last unsubdivided, eight middle intervals each split uniformly into 32 bins — and let the DP optimize only the 11 macro endpoints. This is about 26× faster to search and maps directly to the address-translation circuit: 10 comparators pick the macro interval, a precomputed scale factor converts the offset into a micro-index via one multiply and a floor, and the LUT returns two adjacent values in one cycle.

The hardware is a four-stage pipeline at roughly 1 GHz (matching typical 28nm DNN accelerators): major-interval select and alignment, micro-address generation, table read and slope preparation, and the final interpolate-and-round step. Everything is written in Chisel and synthesized with Design Compiler under the SMIC 28nm library.

Why This Matters

Impact on research: The paper reframes activation approximation as a globally solvable combinatorial optimization rather than a calibration or heuristic problem, and shows that hardware constraints (comparator count, address logic) can be baked into the algorithm's search space instead of being patched on afterwards. It also exposes a real limitation in prior general-purpose LUT work: narrow validity ranges that silently fail at LLM scale.

Real-world applications:

• Edge and on-device LLM inference, where memory bandwidth, die area, and power are tightly constrained.
• NPU and accelerator design, where nonlinear units are often a throughput bottleneck relative to the linear datapath.
• Mixed-precision inference pipelines, complementing W8A8/W4A8 linear quantization without introducing a calibration step.
• Vision and general DNN deployment (ViT, CNNs), since the framework applies to any nonlinear operator.
• MoE architectures such as DeepSeek-V3, where Sigmoid expert routing can be handled by the same unified machinery.

Industry relevance: The asymmetry the paper cites is stark — on an H100 SXM5, FP16 linear compute power is 1024× that of the special function units, yet at head attention size 128 the linear compute demand is only 256× the nonlinear demand. Closing part of that gap with a small, synthesizable block that is drop-in compatible with existing NPU pipelines is directly actionable for accelerator vendors.

Future Directions

• Extending the DP formulation and the two-level layout to lower-precision domains such as FP8 or integer arithmetic, where the finite-precision term τ and the candidate grid would change.
• Exploring adaptive or per-layer cutpoint budgets instead of a fixed 259-point layout, to see whether allocating cutpoints per operator or per layer improves the accuracy/area trade-off.
• Measuring the NLI Engine after fabrication (the paper reports synthesis results, not silicon measurements), including end-to-end latency and energy inside a full accelerator.
• Broadening coverage to more operators and architectures — the paper notes Sigmoid for MoE routing and mentions Tanh and ArcTan as general-purpose targets, but does not report per-operator accuracy for all of them in the main text.
• Quantifying how NLI interacts with existing weight/activation quantization schemes (SmoothQuant, OSTquant) when both are applied simultaneously.

Target Audience

Hardware architects and accelerator engineers designing NPU nonlinear datapaths; ML systems researchers working on inference efficiency and activation quantization; compiler and kernel developers who would write the custom CUDA/Triton implementations the paper mentions; and graduate students studying software-hardware co-design or numerical approximation for deep learning. Practitioners deploying LLMs on edge devices with strict area and power budgets will find the hardware comparison tables the most immediately useful part.

Authors’ abstract

Large Language Models (LLMs) have demonstrated remarkable performance across a wide range of tasks, but their deployment is often constrained by substantial memory footprints and computational costs. While prior work has achieved significant progress in compressing and accelerating linear layers, nonlinear layers-such as SiLU, RMSNorm, and Softmax-still heavily depend on high-precision floating-point operations. In this paper, we propose a calibration-free, dynamic-programming-optimal, and hardware-friendly framework called Non-uniform Linear Interpolation (NLI). NLI is capable of efficiently approximating a variety of nonlinear functions, enabling seamless integration into LLMs and other deep neural networks with almost no loss in accuracy. NLI ingeniously recasts cutpoint selection as a dynamic-programming problem, achieving the globally minimal interpolation error in O(MxN2) time via Bellman's optimality principle. Based on the NLI algorithm, we also design and implement a plug-and-play universal nonlinear computation unit. Hardware experiments demonstrate that the NLI Engine achieves more than 4x improvement in computational efficiency compared to the state-of-the-art designs.