Differentiable Weightless Controllers: Learning Logic Circuits for Continuous Control

AI summary

Overview

Research area: Machine learning for continuous control — specifically reinforcement learning (RL) policy architectures that can be compiled to digital hardware, sitting at the intersection of deep RL, quantization/compression, weightless (lookup-table) neural networks, and FPGA deployment, with an interpretability angle.

Technical level: Advanced. The paper assumes familiarity with reinforcement learning (SAC, DDPG, PPO), surrogate-gradient training of discrete networks, and FPGA synthesis terminology (LUTs, flip-flops, BRAM, DSP slices, clock cycles).

Scope in one sentence: The paper introduces Differentiable Weightless Controllers (DWCs), a symbolic-differentiable architecture that replaces dense matrix multiplications in RL policies with sparse boolean lookup tables, and evaluates them on five MuJoCo continuous-control benchmarks plus an FPGA implementation on an Artix-7 device.

What This Paper Is About

Standard deep neural network controllers rely on large numbers of compute-intensive multiply-and-accumulate operations, which makes them difficult to run efficiently on resource-constrained platforms such as UAVs and mobile robots. The authors ask whether continuous control policies can instead be learned as discrete boolean logic circuits rather than real-valued weight-based networks, while still being trainable end-to-end by gradient descent. Their goal is a policy that matches the returns of full-precision and quantized neural network baselines, but compiles to FPGA-compatible circuitry with few- or single-clock-cycle latency and nanojoule-level energy cost per action.

Key Contributions

1. The DWC architecture. A symbolic-differentiable design that extends Differentiable Weightless Networks (DWNs) from classification to continuous control, adding an adaptive input encoding that converts real-valued observations into binary vectors and a trainable output decoding that converts binary outputs into real-valued actions of suitable range and scale.

2. Successful RL training to parity. The authors show DWCs can be trained with standard gradient-based RL algorithms, reaching parity with floating-point policies (full-precision or quantized) in most experiments across five MuJoCo tasks.

3. Orders-of-magnitude efficiency gains. When compiled to FPGA hardware, DWCs achieve few- or even single-clock-cycle latency and nanojoule-level energy per action, with zero BRAM and zero DSP usage in the computational core.

4. Partial interpretability. The sparse, discrete, symbolic structure allows direct inspection of which input dimensions influence control decisions and which thresholds are relevant.

Main Findings

• Returns comparable to full-precision on four of five tasks. Median returns over 10 trained models: Ant — FP 5.598251k, Quant 4.716681k, DWC 5.677335k; Hopper — FP 2.796813k, Quant 1.930725k, DWC 3.119865k; Humanoid — FP 6.185556k, Quant 5.954057k, DWC 6.140962k; Walker2d — FP 5.043828k, Quant 4.656477k, DWC 5.024956k. The authors state DWC returns are comparable to FP and at least as good as Quant across four of the five tasks, all except HalfCheetah.

• HalfCheetah is the clear exception. FP median return is 11.528543k, Quant is 10.465013k, and DWC is 7.548604k. The authors note that 7.5k models are not performing badly — they learn to run — but do so more slowly than the FP models at 11.5k. HalfCheetah is described as the task most resistant to network reduction and quantization, presumably because it is capacity-limited.

• A scaled-up DWC closes most of the HalfCheetah gap. With 16,384 LUTs per layer and inputs quantized to 255 levels (instead of 63), median return is above 10.3k, matching the quantized baseline's 10.4k median and falling within the uncertainty region of the floating-point policy's 11.5k median. This configuration requires 32k lookups and several popcounts, versus over 70k multiply-accumulate operations for the two baselines. Because making the interconnect learnable at this width would require 16,384² × k = 1.6B parameters (with k = 6), the authors initialize that interconnect randomly and keep it fixed for this size only.

• Extreme efficiency on FPGA. For the standard D_ℓ = 1024 setup, the text reports latency of only 2 or 3 clock cycles, throughput at the maximum 10⁸ actions per second, and energy around 2 nanojoule per action. Table 2 lists latencies of 20 to 30 µs, power between 0.206 and 0.228 W, and energy per action between 2.06 × 10⁻⁹ and 2.28 × 10⁻⁹ J, with 0 BRAM and 0 DSP for all five tasks.

• Smaller models are cheaper still. With layer width D_ℓ = 256, the authors report single-clock-cycle latency, still maximum throughput, and energy per action reduced by approximately a factor of 2 (Table 2 lists 1.02 × 10⁻⁹ to 1.16 × 10⁻⁹ J, latencies of 10 µs, and 0 BRAM and 0 DSP).

• DWCs dominate the quantized baseline in latency and energy. The quantized networks of Kresse & Lampert (2026) show latencies from 21 cycles to over 24,000 cycles, throughput between 4.1 × 10³ and 4.8 × 10⁶ actions per second, and energy per action between 2.8 × 10⁻⁵ and 6.5 × 10⁻⁸ Joule. Table 2 lists their latencies from 210 µs to 243,230 µs and includes BRAM (0 to 15) and DSP (0 to 45) usage.

• Noise robustness is competitive. Under zero-mean Gaussian noise with standard deviations σ ∈ {0.1, …, 0.5} injected into normalized observations, DWCs show robustness comparable to the quantized network, and generally the same level as or better than floating-point networks. The exception is the compact D_ℓ = 256 models, which perform worse on Humanoid under high noise.

• Capacity ablations show saturation, not overfitting. Sweeping layer widths over {128, 256, 512, 1024, 2048, 4096}, four of five tasks are relatively stable, with per-layer sizes above 256 generally exhibiting returns similar to the floating-point baseline — indicating these tasks are not capacity-limited and no overfitting emerges. HalfCheetah is again the exception, with median return increasing monotonically with layer width.

• LUT input count matters mainly for HalfCheetah. Varying k over {2, 3, 4, 5, 6} at constant layer width D_ℓ = 1024, capacity effects limit returns only for HalfCheetah; other returns remain largely unaffected, suggesting k can be chosen based on available hardware (FPGAs commonly support k = 4 or k = 6; custom ASICs may prefer smaller values).

• Learned connectivity highlights important inputs. For Humanoid (averaged over trained D_ℓ = 1024 models), on average 275 out of 376 observation dimensions receive a connection, and performance still matches the FP model, suggesting unconnected dimensions are not necessary for good control. Torso velocity observations receive the highest number of connections, consistent with a reward function heavily dependent on forward velocity.

• Threshold usage is bimodal. Averaged across environments, connections cluster in two modes slightly left and right of the center — index 31 out of 63, corresponding to a normalized observation value of 0 (index 0 corresponds to −10, index 63 to 10). Hopper, Walker2d, and HalfCheetah show heavier tails than Ant and Humanoid; a comparison plot for D_ℓ = 128 DWCs shows lighter tails, with the maximum number of possible first-layer connections reduced to k × D_ℓ = 768.

Methodology in Plain English

The authors keep the reinforcement learning algorithm and the actor-critic setup essentially unchanged, and swap only the policy network for a circuit-like structure. The constraints are therefore all about representation and training, not about the RL objective.

The pipeline works as follows. Each real-valued observation dimension is normalized with a running mean and standard deviation and clipped to the range [−10, 10]. It is then converted into a bitvector by thermometer encoding: a bank of thresholds is placed using stretched-Gaussian quantiles, selected so the first and last thresholds land exactly at ±10 with an additional threshold at 0, producing a fixed-width binary pattern per dimension. These bitvectors are concatenated across dimensions to form the network input.

That binary input passes through two layers of multi-input boolean lookup tables, in which each LUT receives a small number of bits (k = 6 in the standard configuration) selected from the previous layer and outputs a single bit. Selecting which bits feed which LUT is the learned interconnect, determined by straight-through estimation; the LUT contents themselves are the learned tables. Unlike prior work that trained only the first layer's interconnect, the authors make the later layer's interconnect learnable as well, following the finding in Kresse et al. (2025a) that training accuracy consistently improves.

Because the forward pass is discrete, training uses surrogate gradients — specifically the extended finite-difference (EFD) estimator from Bacellar et al. (2024), which aggregates contributions from all address locations. For output, the final bits are partitioned into disjoint groups, one per action dimension. Each group is reduced by a popcount-style group sum, normalized to [−1/2, 1/2], and passed through a learnable per-dimension affine transformation whose scale is parameterized in log form to keep it positive; for SAC, a tanh is applied afterward. This head is fully differentiable and integrates with policy-gradient objectives. At deployment, the affine mapping plus tanh collapses into a single SRAM lookup from popcount to control word.

The setup uses D_ℓ =

Authors’ abstract

Controlling autonomous systems under real-world conditions often requires policies that can be evaluated with low latency and minimal energy consumption. Unfortunately, these conditions are at odds with the use of high-precision deep neural networks as controllers. In this work, we introduce Differentiable Weightless Controllers (DWCs), a symbolic-differentiable architecture that learns flexible, non-linear, yet highly efficient control policies. DWCs can be trained end-to-end via gradient-based techniques, yet compile directly into FPGA-compatible circuits with few- or even single-clock-cycle latency and nanojoule-level energy cost per action. Across five MuJoCo benchmarks, including high-dimensional Humanoid, DWCs achieve returns competitive with standard deep policies (full-precision or quantized neural networks). Furthermore, DWCs exhibit structurally sparse and interpretable connectivity patterns, enabling direct inspection of which input values influence control decisions.