wa-hls4ml: A Benchmark and Surrogate Models for hls4ml Resource and Latency Estimation

AI summary

Overview

Research area: Machine learning for hardware design automation — specifically, surrogate models that estimate the FPGA resource usage and latency of neural network accelerators generated by the hls4ml toolchain, packaged with a public benchmark and dataset. The work sits at the intersection of high-level synthesis (HLS), edge/embedded ML, and scientific computing (high-energy physics instrumentation).

Technical level: Advanced. The paper assumes familiarity with FPGA resource types (LUTs, FFs, DSPs, BRAM), HLS synthesis flows, graph neural networks, and transformer architectures.

Scope: The paper introduces the wa-hls4ml dataset (683,176 synthesized neural network designs), a standardized benchmark for evaluating resource/latency predictors, and two new surrogate model architectures (a GNN and a transformer) compared against an MLP baseline.

What This Paper Is About

Predicting how much FPGA hardware a neural network will consume — and how fast it will run — normally requires running hardware synthesis, which takes hours. That slow feedback loop makes it hard to iterate on accelerator designs quickly. This paper builds a large open dataset of already-synthesized neural networks plus a benchmarking procedure, and trains neural surrogate models that predict resource usage and latency directly from a model's architecture description, cutting the design loop from hours to seconds.

Key Contributions

1. An open dataset of over 680,000 synthesized neural networks. The training, validation, and test sets total 683,176 synthetic samples: 608,679 fully connected networks, 31,278 one-dimensional convolutional networks, and 43,219 two-dimensional convolutional networks, all converted with hls4ml and synthesized through AMD Vitis targeting Xilinx FPGAs. Each sample stores the model architecture, hls4ml conversion parameters, post-logic-synthesis resource and latency numbers, and synthesis metadata; a companion dataset holds the full projects, logs, and intermediate representations.

2. A formal benchmark for resource and latency estimation. The benchmark defines predefined evaluation metrics (R², SMAPE, RMSE, plus relative percentage error box plots), splits (478,220 training, 102,472 validation, 102,484 test, and 887 exemplar samples), and submission guidelines covering required reports, recommended architecture descriptions, and optional code and weight sharing.

3. An exemplar test set of realistic scientific models. 887 samples from seven real application models — Jet, Top Quarks, Anomaly, BiPC, CookieBox, AutoMLP, and Particle Tracking — synthesized across varying precisions, strategies, reuse factors, target boards, clock targets, and Vivado versions, to test extrapolation beyond the synthetic training distribution.

4. Two new surrogate model architectures. A graph neural network built from five stacked GATv2Conv layers with five attention heads each, and a transformer built from two encoder blocks, both operating on an 18-dimensional per-layer feature vector and predicting LUTs, FFs, DSPs, BRAM, latency in clock cycles, and initiation interval (II).

Main Findings

• Surrogate model accuracy: The paper reports that the models generally predict latency and resources for the 75th percentile within several percent of the synthesized resources on the synthetic test dataset. Per-variable metric tables for the GNN and transformer are not contained in the paper content provided here.

• Domain-specific beats general-purpose: A prior general-purpose GNN approach (Wu et al., 2022), adapted with linear correction factors to the neural network task, achieved SMAPE values of 34.30%, 36.03%, and 31.26% for DSPs, LUTs, and FFs respectively on simple 2-layer MLPs — less precise than the GNN- and transformer-based surrogate models introduced here. Many of that model's raw predictions were negative because the evaluation dataset lies outside its training domain, though predictions still correlated with ground truth.

• Bit operations correlate unevenly: Bit operations (BOPs) approximate resource and timing values more closely for fully connected models than for convolutional models. Within convolutional models the correlation is positive but weaker than for fully connected ones, and slightly better for timing than for resources.

• Reuse factor creates distinct populations: Fully connected models form distinct populations when grouped by reuse factor, and those populations are highly correlated with timing and resource information, with higher reuse factors producing larger latencies and lower resource usage in most cases. This trend is weaker for convolutional models, where model size and complexity matter more than reuse factor.

• Exemplar and test distributions do not overlap well: The distributions of resource and latency features for the exemplar models and the test dataset do not overlap strongly, which the authors identify as room to improve architecture diversity in both sets in future work.

• Datasets before this work were smaller or closed: Comparable prior efforts use 40,000 samples (Wu et al., 2022), 42,000 (HLSyn), 2,465 (Jamal et al., 2023), 1,300 (Dai et al., 2018), and 15,000 (rule4ml). Several of these are not open source or not open datasets.

Methodology in Plain English

The team generated a very large number of small neural networks in Keras and QKeras, varying layer counts, activation functions, neuron counts, weight/bias precisions, reuse factors, and implementation strategies. Fully connected models used hls4ml's resource or latency strategy with io_parallel wiring; convolutional models used the resource strategy with io_stream wiring. Two- and three-layer fully connected models came from a grid search, while deeper fully connected and convolutional models were generated randomly.

Each model was converted to HLS with hls4ml and run through synthesis using AMD Vitis 2023.2 and 2024.2, targeting the AMD Xilinx Alveo U250 board. Jobs ran in parallel on the National Research Platform Kubernetes Hypercluster (Ubuntu 20.04.4 LTS containers, 3 virtual CPU cores and 16 GB RAM per pod, with AMD tools mounted through a Ceph-based persistent volume) and on the Texas A&M ACES HPRC Cluster (Vitis 2024.2, 2 virtual CPU cores and 32 GB RAM). Projects, reports, logs, and a JSON summary of C-synthesis and logic-synthesis resources were collected per sample.

For prediction, each layer of an input model becomes a node with an 18-dimensional feature vector covering input/output dimensions, precision, reuse factor, strategy, layer and activation type, filters, kernel size, stride, padding, batch normalization, and I/O type. Numerical features are z-score normalized using training statistics; categorical features are one-hot encoded. The graph connects nodes in dataflow order, adds self-loops so attention can consider each layer's own features, and appends one-hot global features to every node.

The GNN stacks five GATv2Conv layers with five attention heads each, using attention weights to learn which layer connections matter, followed by layer normalization, ELU activation, dropout, and residual connections. Node embeddings are pooled and concatenated with global features before a final MLP produces predictions. It was implemented in PyTorch and PyTorch Geometric, trained with MSE loss and the AdamW optimizer for 200 epochs on an NVIDIA A10 GPU, using 70% of the total samples.

The transformer treats each layer as a token using the same 18-dimensional features, but without graph connections or global features, and uses two sequential encoder blocks.

The baseline is an MLP adapted from the open-source rule4ml tool: ordinal encoding of categorical inputs, trainable embedding layers, a dense block over numerical features, concatenation, and a final dense block. One MLP is trained per target variable for 200 epochs with the Adam optimizer, minimizing a mean squared logarithmic loss.

Why This Matters

Impact on research: Resource and latency estimation via surrogate models compresses a step that takes hours into seconds, letting researchers explore far more accelerator design configurations than synthesis-in-the-loop workflows allow. The dataset additionally stores the full synthesis chain from ML model to HLS representation to register-transfer level (RTL), with logs, which the authors argue enables applications beyond this paper's scope.

Real-world applications:

• Internet of Things (IoT) devices, where FPGA inference must fit tight power and area budgets.
• Autonomous vehicles, which need low-latency, resource-efficient inference at the edge.
• Scientific sensing and real-time instrumentation, including particle physics experiments such as jet classification, top quark tagging, anomaly detection in audio, image compression encoders, real-time detector data acquisition (CookieBox), and charged particle tracking in silicon detectors.
• Embedded AI generally, where designers need early feedback on hardware requirements before committing to synthesis runs.

Industry relevance: The benchmark is designed as an open, ongoing community resource with a defined submission process: contributors must report predicted values per FPGA metric and box plots comparing predictions to actuals, are strongly encouraged to share architecture details and trained weights, and may propose benchmark enhancements. hls4ml is an open-source framework, and the predictor is intended to help hls4ml users optimize models for FPGA deployment without waiting on vendor tool runs — the paper contrasts this with Vivado/Vitis HLS native estimates, which are only available after running C-synthesis and can be time-consuming.

Future Directions

• Grow and diversify the dataset. The authors state the goal is to continue growing and extending the dataset over time; they specifically flag that the exemplar and test distributions do not overlap strongly and that model architecture diversity should improve.

• Train other approaches on wa-hls4ml. The authors suggest training the Wu et al. (2022) general-purpose HLS estimation model on the wa-hls4ml dataset to see whether its accuracy improves in this domain.

• Evolve the surrogate models. The paper treats the GNN and transformer as an initial iteration beyond the rule4ml baseline and expects continued development beyond this work.

• Widen benchmark participation. The submission process is intended to be open and ongoing with no fixed release schedule, updated to reflect significant contributions, and the benchmark is designed to be applicable beyond resource and latency estimation.

Target Audience

Researchers and engineers working on FPGA-based machine learning acceleration, HLS design-space exploration, and ML-assisted hardware design will get the most from this paper, particularly those who need a large, open, synthesized dataset or a standard benchmark for comparing resource and latency predictors. It is also relevant to high-energy physics and other scientific communities deploying real-time inference at the edge, and to ML researchers interested in graph neural networks and transformers applied to structured hardware-design problems. Readers without background in FPGA resource terminology or HLS tooling will find parts of it difficult, since the dataset schema, metric definitions, and model architectures assume that context.

Authors’ abstract

As machine learning (ML) is increasingly implemented in hardware to address real-time challenges in scientific applications, the development of advanced toolchains has significantly reduced the time required to iterate on various designs. These advancements have solved major obstacles, but also exposed new challenges. For example, processes that were not previously considered bottlenecks, such as hardware synthesis, are becoming limiting factors in the rapid iteration of designs. To mitigate these emerging constraints, multiple efforts have been undertaken to develop an ML-based surrogate model that estimates resource usage of ML accelerator architectures. We introduce wa-hls4ml, a benchmark for ML accelerator resource and latency estimation, and its corresponding initial dataset of over 680,000 fully connected and convolutional neural networks, all synthesized using hls4ml and targeting Xilinx FPGAs. The benchmark evaluates the performance of resource and latency predictors against several common ML model architectures, primarily originating from scientific domains, as exemplar models, and the average performance across a subset of the dataset. Additionally, we introduce GNN- and transformer-based surrogate models that predict latency and resources for ML accelerators. We present the architecture and performance of the models and find that the models generally predict latency and resources for the 75% percentile within several percent of the synthesized resources on the synthetic test dataset.