Real-Time Semantic Segmentation on FPGA for Autonomous Vehicles Using LMIINet with the CGRA4ML Framework

AI summary

Overview

• Research area: Computer vision / hardware acceleration — real-time semantic segmentation for autonomous driving, deployed on FPGA using the LMIINet architecture and the CGRA4ML framework.
• Technical level: Advanced. It assumes familiarity with quantization-aware training, FPGA resource budgeting (LUT/FF/BRAM), coarse-grained reconfigurable arrays, and segmentation metrics.
• Scope (one sentence): The paper re-architects a lightweight CNN–Transformer segmentation model to fit a reconfigurable FPGA backend, then measures accuracy, frame latency, and resource use on a Xilinx ZCU104 board against GPU and prior FPGA baselines.

What This Paper Is About

Semantic segmentation labels every pixel in a driving scene (road, vehicle, pedestrian, sky), but high-accuracy models are usually too heavy and power-hungry for embedded automotive ECUs, while prior FPGA implementations tend to give up accuracy to reach real-time speed. The authors take the lightweight LMIINet model and rewrite the parts that hardware cannot easily support, then map the result onto a coarse-grained reconfigurable array (CGRA4ML) that can stream data off-chip, so that a modern CNN–Transformer hybrid fits on a mid-range FPGA. The goal is to hit roughly 90% pixel accuracy and 45% mIoU while still running in real time on battery- and thermally-constrained vehicles.

Key Contributions

1. A quantized, FPGA-compatible variant of LMIINet that keeps global-context modeling through a simplified Flatten Transformer and channel attention, redesigned around hardware-friendly operations (depthwise-separable and 1×1 convolutions, simplified skip connections, nearest-neighbor upsampling).
2. A CGRA4ML-based mapping that uses operator fusion, streaming dataflow, and on-chip buffering (with off-chip storage for skip connections) to achieve deterministic latency at batch size b=1.
3. A four-phase quantization-aware training schedule and measurement protocol that separates accuracy effects from pure throughput, allowing a fair comparison against the ENetHQ FPGA design and GPU baselines.
4. An end-to-end ZCU104 implementation demonstrating real-time performance with competitive mIoU on Cityscapes, plus a discussion of the migration path to an ASIC for lower unit cost and energy.

Main Findings

• Accuracy preserved under 8-bit quantization: The quantized LMIINet reached approximately 90.0% pixel accuracy and 45.2% mIoU on the Cityscapes validation set, which the authors describe as on par with the unquantized LMIINet. The peak validation mIoU of 0.452 was reached at epoch 239.
• Real-time operation on ZCU104: Latency was approximately 50 ms per frame (50.1 ms reported in the comparison table), corresponding to about 20 FPS (19.95 FPS in the table), measured at 200 MHz with a 5 ns clock period using cycle-accurate Verilator simulation.
• Layer-level utilization varies widely: In the layer-wise Verilator table, several convolutional layers reach roughly 99.3–99.5% processing-element utilization, while lighter layers drop as low as 2.3% (layer 13) and 9.3% (layer 11). Reported per-layer memory ranges from 0.7 MB (layers 5 and 7) up to 33.0 MB (layer 13).
• Higher accuracy than the prior FPGA baseline: Against ENetHQ on FPGA (81.1% pixel accuracy, 36.8% mIoU), the LMIINet design reaches 90.0% pixel accuracy and 45.0% mIoU, at the cost of higher latency (50.10 ms vs 4.9 ms at batch 1 / 3.06 ms at batch 10) and lower throughput on the paper's table (19.95 FPS vs 32.7 FPS at batch 1 / 327 FPS at batch 10).
• Heavier resource footprint than ENetHQ: On the reported synthesis figures, LMIINet uses 206,830 LUTs (89.77%), 196,980 FFs (42.75%), and 6.0 BRAM (1.92%), versus ENetHQ's 342.0 (37%) LUTs, 87,059 (16%) FFs, and 224.5 (25%) BRAM. The ENetHQ figures were measured on a ZCU102, whereas this design targets a ZCU104.
• Power compared indirectly: The authors state that power consumption was not directly measured; they infer much lower power than the GPU baseline's ~225 W TDP from resource utilization and clock frequency.
• Multi-phase training stabilized optimization: Figures 5–7 are described as showing a sharp loss drop in phase 1, stabilization during the frozen-decoder phase 2, a further loss reduction in phase 3 after regularization removal, and a plateau in phase 4; Figure 7 is described as reaching approximately 45% mIoU.
• Reported epoch counts are not fully consistent in the paper: Training is described as spanning 240 epochs, the four-phase schedule in Table I lists epochs 0–50, 50–110, 110–170, and 170–218, Figure 5's caption refers to 218 epochs, and qualitative results are described as coming after 240 epochs of training.

Methodology in Plain English

The team started from LMIINet, a lightweight encoder–decoder model that mixes convolutional blocks with a reduced Transformer for global context. Because the CGRA4ML toolchain cannot implement every original operation, they simplified the network: they added a strided 3×3 downsampling stem, used progressive encoder filters of 24, 48, 96, 128 channels, switched the decoder to nearest-neighbor upsampling with depthwise–pointwise refinement, replaced full multi-head attention with a flattened local attention module built from 3×3 dilated convolutions (dilation rate 2), cut attention heads from 16 to 8 and key dimensions from 128 to 64, simplified channel attention to global average pooling plus two 1×1 convolutions with sigmoid scaling, and reduced the segmentation head to upsampling, concatenation, and a single 1×1 convolution with softmax over the last three decoder features.

They then trained the modified model on Cityscapes with quantization-aware training at 8-bit activations and weights over 240 epochs, batch size 2, initial learning rate 7×10⁻⁴, and SGD with momentum 0.9 and Nesterov, decaying the learning rate by a factor of 0.1 at epochs 110 and 170. Training followed four phases that toggled dropout, data augmentation, decoder freezing, and auxiliary supervision, with auxiliary supervision at 1/8 scale.

The trained model was converted by CGRA4ML into SystemVerilog RTL for a (16×96) processing-element array — 1,536 PEs — clocked at 200 MHz, with 8-bit operands, 32-bit accumulation, 16-bit bias, a 256-depth RAM for weights, a 32,768-depth RAM for edge data and skip connections, up to 256 input channels, a 128-bit AXI interface with a maximum burst of 16 beats, and support for kernel sizes up to 9×9. Correctness and timing were verified in Verilator simulation, then synthesized on the ZCU104, with latency derived by multiplying layer cycle counts by the 5 ns clock period.

Why This Matters

• Research impact: It is an early demonstration that a CNN–Transformer hybrid segmentation model — not just a compact CNN — can be mapped onto reconfigurable logic with off-chip streaming, and it exposes concrete framework gaps (for example, the missing upsample2d layer in CGRA4ML) that other hardware researchers can address.
• Support for hardware–software co-design: The specific design choices — tiling, reuse, loop unrolling, operand bit-widths, on-chip buffer topology — are described as transferring directly to an ASIC netlist, giving a validated blueprint rather than a one-off prototype.

Real-world applications cited or implied by the paper:

• Embedded perception for autonomous vehicles, where pixel-wise road, vehicle, and pedestrian segmentation feeds planning and control under hard deadlines.
• Multi-camera or stereo perception stacks, using CGRA4ML's reconfiguration to time-multiplex layers while holding deterministic per-stream deadlines at 20 FPS monocular feeds.
• Fanless or lightly cooled automotive edge deployments, where the paper argues the board's low inferred device power fits thermal budgets that a ~225 W desktop GPU does not.
• Over-the-air field updates, since FPGA reconfigurability allows patches while preserving a verified timing budget (the paper mentions ISO 26262 processes).

Industry relevance: The paper targets exactly the segment where GPU modules are considered too power-hungry and non-deterministic for automotive ECUs. It argues that FPGA-validated designs lower the risk of later ASIC tape-out by fixing the micro-architecture in advance, and its authors report competitive accuracy with the GPU baseline's 90.0% pixel accuracy and 45.0% mIoU while remaining within real-time latency limits.

Future Directions

1. Add the missing CGRA4ML layer support: Implementing 2D upsampling (upsample2d) natively would directly improve decoder quality, and the paper notes the framework currently lacks it.
2. Implement the omitted block-based operations: Channel shuffle and learnable channel coefficients were not implemented in this version of the architecture; adding them could improve feature extraction and local/global feature interaction.
3. Raise the clock frequency: The Verilator simulation is said to suggest that a higher clock could reduce latency further, with careful tuning needed to avoid simulation instability.
4. Build a segmentation-specific library and explore scaling: A dedicated semantic-segmentation library inside CGRA4ML, lighter model variants, or deployment on larger FPGA chips (which would permit use of hls4ml for better memory access performance) are put forward as paths to better efficiency. Improving mIoU on small and distant instances is also flagged as an open limitation of the current 45% result.

Target Audience

This paper is most useful to FPGA and edge-AI hardware engineers, graduate researchers in efficient deep learning and automotive perception, and system architects evaluating accelerators for ADAS or autonomous driving platforms. It is also relevant to practitioners working with quantization-aware training and hardware description generation toolchains, and to readers already familiar with Cityscapes segmentation benchmarks who want to understand the resource-versus-accuracy trade-off between fully on-chip pruned designs like ENetHQ and streaming designs like this CGRA4ML mapping. Readers without a hardware background may find the resource and dataflow details dense, though the motivation sections (Sections I–II) are comparatively accessible.

Authors’ abstract

Semantic segmentation has emerged as a fundamental problem in computer vision, gaining particular importance in real-time applications such as autonomous driving. The main challenge is achieving high accuracy while operating under computational and hardware constraints. In this research, we present an FPGA-based implementation of real-time semantic segmentation leveraging the lightweight LMIINet architecture and the Coarse-Grained Reconfigurable Array for Machine Learning (CGRA4ML) hardware framework. The model was trained using Quantization-Aware Training (QAT) with 8-bit precision on the Cityscapes dataset, reducing memory footprint by a factor of four while enabling efficient fixed-point computations. Necessary modifications were applied to adapt the model to CGRA4ML constraints, including simplifying skip connections, employing hardware-friendly operations such as depthwise-separable and 1A-1 convolutions, and redesigning parts of the Flatten Transformer. Our implementation achieves approximately 90% pixel accuracy and 45% mean Intersection-over-Union (mIoU), operating in real-time at 20 frames per second (FPS) with 50.1 ms latency on the ZCU104 FPGA board. The results demonstrate the potential of CGRA4ML, with its flexibility in mapping modern layers and off-chip memory utilization for skip connections, provides a path for implementing advanced semantic segmentation networks on FPGA for real-time applications to outperform traditional GPU solutions in terms of power efficiency while maintaining competitive accuracy. The code for this project is publicly available at https://github.com/STAmirr/ cgra4ml_semantic_segmentation