Neural networks for neurocomputing circuits: a computational study of tolerance to noise and activation function non-uniformity when machine learning materials properties

AI summary

Overview

Research area: Neuromorphic/analog computing for machine learning, specifically the robustness of neural networks implemented in dedicated analog neurocomputing circuits when applied to materials property prediction.

Technical level: Intermediate. The core ideas are accessible with a basic understanding of neural networks, but the framing around analog circuit behavior, activation function variation, and materials informatics assumes some familiarity with ML practice.

Scope: A computational study of how circuit noise and non-uniform neuron activation functions affect neural network accuracy across several materials property prediction tasks, and how architecture and training choices influence that robustness.

What This Paper Is About

Analog neurocomputing circuits promise high-throughput, low-power machine learning, and they can be deployed where digital computers are impractical — remote locations, small mobile or autonomous devices, and extreme environments. The catch is that real circuits introduce noise and produce neuron activation functions whose shapes vary from neuron to neuron because of dispersion in device characteristics (transistors, diodes). This paper asks how much that noise and shape variation actually hurt prediction accuracy, and whether the damage depends on how the network is built and trained.

Key Contributions

1. A systematic computational study of hardware imperfection. The authors examine the impact of both circuit noise and activation function (NAF) inhomogeneity as a function of neural network architecture and training regime, rather than treating either effect in isolation.

2. Evaluation on real materials informatics problems. The study uses high-throughput-relevant tasks: predicting formation energies versus the lowest-energy isomer of peri-condensed hydrocarbons, formation energies and band gaps of double perovskites, and zero point vibrational energies of molecules from the QM9 dataset.

3. Identification of architectural and training factors linked to robustness. The paper reports which network configurations and which training outcomes correlate with greater tolerance to noise.

4. A practical mitigation strategy for activation function inhomogeneity. The authors show that retraining a network using the practically realized shapes of the NAFs can compensate for non-uniform activation functions.

Main Findings

• Neural networks are generally noise-sensitive. Model accuracy degrades rapidly as the noise level increases, indicating that reliable operation in noisy analog circuits is not automatic.

• Shallow and oversized networks fare better. Single-hidden-layer networks, and networks that are larger than the size that would be optimal in a noise-free setting, are somewhat more noise-tolerant than other configurations.

• Less overfitting correlates with more robustness. Models that overfit less — which the authors note is not necessarily the same as having the lowest test set error — are more tolerant to noise.

• Activation function inhomogeneity is fixable. Retraining the network using the practically realized NAF shapes can palliate the effect of non-uniform activation functions, turning a hardware variability problem into a training-time problem.

• Quantitative details are not in the abstract. The abstract states these trends qualitatively; specific error values, noise levels tested, dataset sizes, and comparisons against baselines are not reported in it.

Methodology in Plain English

The authors ran a computational simulation study rather than fabricating circuits. They took several materials property prediction problems, trained neural networks on them under different architectures and training setups, and then simulated what happens when those networks are effectively run under analog circuit conditions — injecting noise and varying the shapes of the neuron activation functions to mimic device-to-device dispersion. By comparing accuracy across configurations, they could identify which architectures and training outcomes hold up best. Finally, they tested a corrective approach: instead of assuming ideal activation functions, they retrained the networks using the activation shapes the hardware would actually produce, and observed whether that recovered accuracy.

Why This Matters

Impact on research: The work connects two communities that often operate separately — analog/hardware neuromorphic computing and machine learning for materials. It argues that robustness to hardware imperfection is a design variable shaped by architecture and training choices, not a fixed property, and it offers a concrete, low-cost remedy (hardware-aware retraining) for activation function variability.

Real-world applications:

• Materials discovery and screening, where high-throughput property prediction (formation energies, band gaps, vibrational energies) is the bottleneck and low-power inference is valuable.
• Remote or off-grid sensing and analysis, where a digital computer is impractical and predictions must be made in place.
• Small, mobile, and autonomous devices, such as drones or field instruments with tight power and size budgets.
• Extreme-condition operation, including environments where conventional electronics are unreliable or power is scarce.

Industry relevance: For chipmakers and hardware startups pursuing analog in-memory or neuromorphic accelerators, the results suggest that device variability need not be a blocker if training accounts for it — and that intentionally choosing a slightly over-provisioned or shallow architecture may buy robustness cheaply. For materials and chemistry teams considering such hardware for screening pipelines, it sets expectations about accuracy degradation under noise.

Future Directions

• Hardware-aware training as standard practice. If retraining with realized activation shapes works, the natural next step is co-designing the training pipeline with measured device characteristics rather than simulated ones.

• Architecture search under noise constraints. The finding that shallow or larger-than-optimal networks are more tolerant raises the question of how to systematically search for architectures that trade accuracy for robustness.

• Characterizing real NAF dispersion. The study models non-uniform activation functions computationally; measuring the actual distribution of activation shapes across fabricated devices would test how well the simulation matches reality.

• Extending beyond the tested tasks and properties. Whether the same noise-tolerance trends hold for other materials informatics targets, model families, and larger-scale networks remains open.

Target Audience

Researchers and engineers working at the intersection of neuromorphic or analog computing hardware and machine learning, particularly those applying ML to materials and chemistry. It is also relevant to ML practitioners interested in robustness to hardware-induced perturbations, and to hardware designers who need to know which device imperfections matter most for downstream model accuracy. A reader without background in either neural networks or analog circuits will find the concepts approachable but the motivation more relevant than the technical detail.

Authors’ abstract

Dedicated analog neurocomputing circuits are promising for high-throughput, low power consumption applications of machine learning (ML) and for applications where implementing a digital computer is unwieldy (remote locations; small, mobile, and autonomous devices, extreme conditions, etc.). Neural networks (NN) implemented in such circuits, however, must contend with circuit noise and the non-uniform shapes of the neuron activation function (NAF) due to the dispersion of performance characteristics of circuit elements (such as transistors or diodes implementing the neurons). We present a computational study of the impact of circuit noise and NAF inhomogeneity in function of NN architecture and training regimes. We focus on one application that requires high-throughput ML: materials informatics, using as representative problem ML of formation energies vs. lowest-energy isomer of peri-condensed hydrocarbons, formation energies and band gaps of double perovskites, and zero point vibrational energies of molecules from QM9 dataset. We show that NNs generally possess low noise tolerance with the model accuracy rapidly degrading with noise level. Single-hidden layer NNs, and NNs with larger-than-optimal sizes are somewhat more noise-tolerant. Models that show less overfitting (not necessarily the lowest test set error) are more noise-tolerant. Importantly, we demonstrate that the effect of activation function inhomogeneity can be palliated by retraining the NN using practically realized shapes of NAFs.