Surrogate Neural Architecture Codesign Package (SNAC-Pack)

AI summary

Overview

Research area: Automated machine learning and hardware/software codesign — specifically hardware-aware Neural Architecture Search (NAS) for FPGA deployment, applied to a high energy physics (HEP) jet classification task.

Technical level: Intermediate. The paper assumes familiarity with neural architecture search, quantization, pruning, and FPGA concepts such as DSPs, LUTs, FFs, and BRAM, but the pipeline itself is described at a high level.

Scope: The paper presents SNAC-Pack, an open-source framework that integrates a multi-stage NAS pipeline (NAC) with a surrogate hardware resource and latency estimator (rule4ml) so that FPGA resource usage and clock cycles can be used directly as search objectives, and demonstrates it end to end on a jet classification dataset.

What This Paper Is About

Designing neural networks that run efficiently on real hardware is hard because existing Neural Architecture Search methods usually optimize accuracy alone or rely on rough proxy measures like bit operations (BOPs), which do not accurately reflect how a model will behave once synthesized on a chip. The authors build SNAC-Pack to let the search process optimize for predicted FPGA resource utilization and latency alongside accuracy, without running time-intensive synthesis for every candidate model. They then test whether the resulting model, when actually synthesized, is competitive with a model found by optimizing BOPs and with an established baseline architecture.

Key Contributions

1. An integrated framework (SNAC-Pack) that couples Neural Architecture Codesign (NAC) multi-stage search with the Resource Utilization and Latency Estimator (rule4ml), enabling multi-objective optimization over accuracy, FPGA resource utilization, and latency.
2. Removal of the synthesis bottleneck: by using rule4ml's surrogate model, candidate architectures can be scored on hardware-relevant metrics without full hardware synthesis for each candidate.
3. An end-to-end demonstration on a high energy physics jet classification task, going from a user-defined search space through global search, local search (quantization-aware training plus iterative magnitude pruning), and finally hls4ml synthesis on a Xilinx Virtex UltraScale+ VU13P FPGA.
4. A direct comparison against a BOPs-based search run with the same number of trials and epochs, producing Pareto fronts under both objective sets and synthesized hardware results for the selected models.

Main Findings

• Accuracy: The Optimal SNAC-Pack model reached 63.84% accuracy, compared with 63.81% for the Optimal NAC model and 63.77% for the Baseline. The threshold for selecting architectures for local search was an accuracy greater than 0.638.
• Estimated metrics from global search (Table 2): SNAC-Pack reported 8,352 BOPs, 3.12 estimated average resources, and 72.24 estimated clock cycles. For comparison, Optimal NAC reported 7,904 BOPs, 3.60 estimated average resources, and 62.69 estimated clock cycles, while the Baseline reported 25,916 BOPs, 7.10 estimated average resources, and 183.74 estimated clock cycles.
• Synthesized hardware results (Table 3, Xilinx Virtex UltraScale+ VU13P): Optimal SNAC-Pack measured 140 ns (24 clock cycles) latency, 70 ns (12 clock cycles) initiation interval, 0 DSPs, 57,728 LUTs (3.34%), 12,605 FFs (0.36%), and 0 BRAM. Optimal NAC measured 125 ns (25 clock cycles) latency, 60 ns (12 clock cycles) II, 0 DSPs, 54,075 LUTs (3.13%), 12,016 FFs (0.35%), and 8 BRAM (0.3%).
• Baseline hardware reference: The pruned-and-quantized baseline measured 105 ns (21 clock cycles) latency, 5 ns (1 clock cycle) II, 262 DSPs (2.1%), 155,080 LUTs (9.0%), 25,714 FFs (0.7%), and 4 BRAM (0.1%).
• Comparable but not superior: The authors state the SNAC-Pack model performs similarly to the NAC model and the baseline in accuracy and is comparable on other criteria, describing the outcome as slight underperformance that still establishes the potential of hardware-awareness in NAS.
• Resource estimation needs refinement: The mismatch between estimated clock cycles (SNAC-Pack 72.24 versus NAC 62.69) and the actual synthesized clock cycles (24 versus 25) is presented as an indicator that resource estimation itself needs improvement.
• Search configuration used: 500 global search trials, 5 epochs per trial, evolutionary population size 20, NSGA-II as the search algorithm, batch size 128 for all training.
• Local search configuration used: 5-epoch warm-up, then 10 iterations of iterative magnitude pruning at 10 epochs each, pruning 20% per iteration, with quantization-aware training at 8-bit precision; selected architectures ended at approximately 50% pruning and 8-bit precision.
• Synthesis configuration used: hls4ml with io_parallel io_type, latency strategy, and a reuse factor of 1.

Methodology in Plain English

The authors start from NAC, which searches for network designs in two stages. In the first, "global" stage, the user defines a space of possible models — how many layers, how many neurons per layer, which activation functions, whether to use batch normalization, and training hyperparameters such as learning rate, L1 regularization, and dropout. A multi-objective evolutionary algorithm (NSGA-II) samples architectures from this space and evaluates them, producing a Pareto front of models that trade off objectives against one another.

The key change SNAC-Pack makes is which objectives are used. Instead of optimizing only accuracy, or accuracy plus BOPs, the search can use any combination of accuracy, BOPs, and the metrics predicted by rule4ml — a surrogate model that estimates FPGA resource utilization (BRAM, DSPs, FFs, LUTs), initiation interval, and latency. In this work the search objectives were estimated average resource utilization, estimated clock cycles, and accuracy. Using a surrogate avoids having to synthesize every candidate architecture, which would be far too slow.

Once the global search yields a Pareto front, an architecture is chosen based on the user's accuracy and resource constraints. That model then enters the second, "local" stage, where it is compressed through quantization-aware training and iterative magnitude pruning, producing another Pareto front from which a model with a particular bit precision and sparsity is chosen. Finally, the optimized model is passed to hls4ml, which generates high-level synthesis code for FPGA deployment.

To test this, the authors applied it to a five-class jet classification problem (light quark, gluon, W boson, Z boson, top quark) using the hls4ml LHC dataset, taking the 8 constituents with the greatest transverse momentum per jet. They searched for a multi-layer perceptron and compared against an 8-constituent MLP baseline described as one of the state-of-the-art architectures for this task. They also ran the same search with NAC optimizing only BOPs and accuracy, using the same number of trials and epochs, to serve as a direct comparison.

Why This Matters

Impact on research. The paper argues that proxy metrics such as BOPs are an imperfect stand-in for real hardware behavior, and it provides a working, open-source pipeline that substitutes a trained surrogate estimator for those proxies inside the search loop. It also reports an honest negative-ish result: the hardware-aware search produced a model that matched but did not beat the BOPs-optimized model on synthesized hardware, and the surrogate's clock-cycle estimates diverged from measured values. That combination of a released framework plus a documented gap between predicted and measured behavior is useful for anyone working on hardware-aware NAS.

Real-world applications (as motivated or exemplified by the paper):

• High energy physics trigger and data-taking systems at the Large Hadron Collider, where models must classify collision events within tight latency and resource budgets. Jet classification is the specific demonstration.
• Resource-constrained edge deployment, the general setting the introduction names as the motivation for automated architecture design and compression.
• Real-time data reduction, consistent with the DOE project acknowledged for supporting the authors ("Real-time Data Reduction Codesign at the Extreme Edge for Science").
• FPGA-accelerated inference workflows in which designers need to explore accuracy-versus-resource tradeoffs without paying the cost of synthesizing every design they consider.

Industry relevance. FPGA deployment pipelines such as hls4ml already sit at the boundary between machine learning and hardware engineering. A tool that folds silicon-level cost estimates into architecture search addresses a practical bottleneck for teams that must fit models into fixed LUT, DSP, FF, and BRAM budgets — including those on small FPGAs where DSP allocation (0 for the NAC and SNAC-Pack models, versus 262 for the baseline) matters.

Future Directions

• Improve the surrogate estimators. The paper explicitly notes that resource estimation can be refined so that search discovers lower-latency architectures that use fewer true resources, citing the observed gap between estimated and synthesized clock cycles.
• Incorporate additional surrogate models trained on large datasets, which the authors name as a future enhancement that could make SNAC-Pack a stronger alternative to the BOPs proxy.
• Broaden beyond the single demonstrated task. The paper presents one jet classification MLP; extending the framework to other architectures (beyond MLPs) and other tasks is left open.
• Tighten the coupling between predicted and measured hardware behavior, so that a search objective agreed with synthesized latency and utilization more reliably; the current selection criterion was accuracy-based (greater than 0.638) rather than validated against measurement.

Target Audience

Researchers and engineers working on Neural Architecture Search, hardware-aware machine learning, and FPGA deployment — particularly those in high energy physics or other latency- and resource-constrained scientific computing settings. It is also relevant to practitioners who use hls4ml or similar HLS toolchains and want automated design-space exploration, and to readers interested in the honest reporting of where surrogate-based optimization still falls short of measured hardware performance. Readers without background in NAS or FPGA terminology will find the pipeline concept accessible but the metric comparisons dense.

Authors’ abstract

Neural Architecture Search is a powerful approach for automating model design, but existing methods struggle to accurately optimize for real hardware performance, often relying on proxy metrics such as bit operations. We present Surrogate Neural Architecture Codesign Package (SNAC-Pack), an integrated framework that automates the discovery and optimization of neural networks focusing on FPGA deployment. SNAC-Pack combines Neural Architecture Codesign's multi-stage search capabilities with the Resource Utilization and Latency Estimator, enabling multi-objective optimization across accuracy, FPGA resource utilization, and latency without requiring time-intensive synthesis for each candidate model. We demonstrate SNAC-Pack on a high energy physics jet classification task, achieving 63.84% accuracy with resource estimation. When synthesized on a Xilinx Virtex UltraScale+ VU13P FPGA, the SNAC-Pack model matches baseline accuracy while maintaining comparable resource utilization to models optimized using traditional BOPs metrics. This work demonstrates the potential of hardware-aware neural architecture search for resource-constrained deployments and provides an open-source framework for automating the design of efficient FPGA-accelerated models.