Towards Reinforcement Learning from Neural Feedback: Mapping fNIRS Signals to Agent Performance

AI summary

Overview

Research area: Human-in-the-loop machine learning, passive brain-computer interfaces, and reinforcement learning from human feedback (RLHF).

Technical level: Intermediate. The paper assumes familiarity with RL concepts (policies, action spaces, states), classification metrics (F1), and basic neuroimaging signal processing, but the framing and contributions are accessible to readers with a machine learning background.

Scope: The paper introduces NEURO-LOOP, a proposed framework for Reinforcement Learning from Neural Feedback (RLNF), and takes the first step toward it by testing whether functional near-infrared spectroscopy (fNIRS) signals recorded from the prefrontal cortex can be decoded into labels describing how well an agent is performing.

What This Paper Is About

RLHF usually depends on explicit feedback such as preference labels, scalar ratings, or expert demonstrations, all of which demand active participation, instruction, and cognitive effort from the human evaluator. The authors ask whether an implicit signal — the blood-flow response in the prefrontal cortex, measured by fNIRS — can instead reveal whether an agent is behaving optimally, suboptimally, or in a worst-case manner while a person simply watches. The goal is not to build a full RLHF system yet, but to solve the prerequisite "neural classification problem": showing that fNIRS signals map reliably onto agent performance labels.

Key Contributions

1. A new public dataset. The authors designed a controlled experimental protocol and publicly released a dataset of synchronized fNIRS recordings and agent transition variables collected from 25 human participants across three domains: Pick-and-Place Robot, Lunar Lander, and Flappy Bird.

2. Multi-level performance decoding. They demonstrate that machine learning can distinguish multiple levels of agent performance from neural data, extending past binary classification to a three-class setting (optimal, suboptimal, worst-case), enabling finer-grained feedback while keeping cognitive workload low for users.

3. Cross-subject generalization analysis with fine-tuning. They evaluate model performance and generalization between participants, showing that cross-subject transfer remains a challenge, but that subject-specific calibration using a small amount of data is feasible and improves performance.

4. A passive-versus-active benchmark. They compare model performance and self-reported user workload across passive (observation) tasks and active (physical demonstration) tasks, establishing a benchmark for future work in RLNF.

Main Findings

• Binary classification reaches 67% average F1. Across conditions and domains, classifiers predicting optimal versus suboptimal behavior achieved an average F1 score of 67%. Random chance for binary classification was 0.50 F1.

• Multi-class classification reaches 46% average F1. Three-way classification of optimal, suboptimal, and worst-case behavior achieved an average F1 score of 46%. Random chance for multi-class was 0.33 F1.

• Single-subject models perform best. Models trained and tested on one participant's own data distinguished binary (F1 = 0.79) and multi-class (F1 = 0.75) performance levels. However, no single-subject model was able to predict agent performance from another subject's neural data.

• Cross-subject transfer is largely unsuccessful. Cross-validation studies showed little success across conditions and participants. Only the Robot Passive condition showed notable potential for zero-shot transfer within the same condition (F1 = 0.54, σ = 0.01).

• Fine-tuning with a small amount of subject data helps substantially. Fine-tuned models used about 20% of the downsampled target participant's data. Average scores increased by 16.9% for binary models and 41.3% for multi-class models, lifting performance above random chance for an unseen target participant.

• Robot conditions outperform game conditions. Binary multi-subject F1 on Robot Passive was 0.72 ± 0.005 and Robot Active 0.67 ± 0.005, compared with Lunar Passive 0.61 ± 0.01, Lunar Active 0.62 ± 0.01, Flappy Passive 0.67 ± 0.01, and Flappy Active 0.66 ± 0.06. Lunar Lander and Flappy Bird conditions showed higher variance and lower performance than the Robot conditions.

• Multi-class performance exceeded chance in all but one condition. The exception was Flappy Passive. Some multi-class models struggled to distinguish between adjacent classes of agent performance, and the Flappy Active multi-class model performed much better than its Passive counterpart (0.51 ± 0.01 versus 0.35 ± 0.06 multi-subject).

• Regression can extract continuous performance signals. Regressors predicted the degree of deviation between the agent's chosen action and a set of near-optimal policy actions. Models trained on Active conditions performed marginally better on average (R² = 0.81) than Passive conditions (R² = 0.77). Fine-tuning with 30% of the target participant's data did not raise performance enough for application in any condition other than Lunar Active (cross-subject R² = 0.01, fine-tuned R² = 0.23).

• Active tasks impose higher perceived workload. NASA-TLX self-reports showed higher cognitive load for nearly every category during Active tasks, with Lunar Lander Active particularly high across all categories. Passive tasks were least demanding; participants described watching the autonomous Lunar Lander and Flappy Bird agents as "boring," which may contribute to slightly lower Passive condition performance.

• Continuous labels are well-separated by the multi-policy agreement system. Heatmaps across three optimality classes showed each performance category to be distinct, with a point of failure identifiable within an episode.

Methodology in Plain English

Setup. Participants sat 24 inches in front of a computer screen wearing an fNIRS headband (an ISS OxiplexTS device) that shines pulsating infrared light into the prefrontal cortex to detect changes in blood flow. The researchers recruited 25 participants aged 19 to 27 (14 female, 11 male), recruited through physical and virtual flyers. Each participant completed 3-4 of a possible six conditions, with at least 10 participants per condition. Conditions lasted 2-5 minutes each, and a 20-second rest calibrated the device before each task.

Tasks. Participants interacted with agents in three OpenAI Gymnasium domains adjusted for human interfacing: Robot Fetch and Place, Lunar Lander, and Flappy Bird. In the passive condition, participants watched an autonomous agent that began with near-optimal actions and then, with some probability, switched to non-optimal actions at a "point of failure"; all subsequent actions were labeled with a "degree of failure." In the active condition, participants physically guided the agent using a keyboard (Flappy Bird, Lunar Lander) or an Xbox controller (Robot arm). Participants completed a NASA-TLX questionnaire after each task and a post-task questionnaire at the end.

Creating labels. For discrete labels, optimal behavior was 0, suboptimal 1, and worst-case 2. For continuous labels, the authors compared the agent's chosen action against K = 10 near-optimal policies, using Kullback-Leibler divergence for discrete action spaces and Euclidean distance for continuous ones, then averaged the error across all K policies.

Signal processing. Data from left and right prefrontal cortex were measured at 690 nm and 830 nm with 110 MHz modulation and converted to oxy- and deoxy-hemoglobin. Features were sampled at 5.2 Hz and band-pass filtered from 0.001 to 0.2 Hz (3rd order). For each fixed-duration sliding window (generally 5 to 7 seconds long with 1 to 2 seconds of stride, chosen to accommodate the 5 to 7 second latency of the fNIRS signal), the researchers computed slope, mean, standard deviation, intercept, skewness, and kurtosis, producing a feature vector of length F = 8·6. Each window received a single label based on its endpoint.

Models and training. They trained a Support Vector Machine, K-Nearest Neighbors, Random Forest, and Multilayer Perceptron. Three paradigms were compared: single-subject (train and test on one person), multi-subject (train on a group, test on held-out data from that group), and fine-tuned (multi-subject models further calibrated with a fraction of a target participant's data). They used a 60-20-20 train-test-validation split, randomly down-sampled the majority class to address imbalance, and evaluated cross-subject generalization with a leave-one-participant-out approach, in some cases leave-two-participants-out.

Why This Matters

Impact on research. The work establishes a quantitative baseline for a question that has been largely unexplored: whether fNIRS, a portable and motion-tolerant neuroimaging modality, carries enough information to serve as an evaluative feedback channel for reinforcement learning. It also releases a dataset of synchronized neural and agent-transition data, giving other researchers a direct way to extend or contest these results. The passive-versus-active comparison creates a benchmark for how implicit neural feedback compares against explicit demonstration.

Real-world applications:

• Adaptive user interfaces that adjust difficulty or assistance in real time based on a user's silent evaluation of system behavior.
• Human-robot interaction, particularly for robot manipulation tasks like fetch-and-place, where a supervisor's neural response could flag an undesirable action without interrupting the workflow.
• Accessibility: hands-free, low-effort feedback channels for users who cannot provide keyboard, joystick, or gesture-based evaluations.
• Game and simulation design, where player engagement and perceived agent quality could be inferred during passive observation.

Industry relevance. Any product that trains or tunes an AI system from human preferences — recommendation, content generation, robotics, autonomous driving simulation — currently depends on explicit ratings, comparisons, or demonstrations that scale poorly and exhaust users. A passive neural channel would let evaluation happen continuously and incidentally. The paper's finding that fine-tuning with a small amount of subject-specific data (roughly 20% for classification, 30% for regression) substantially improves performance also suggests a practical deployment path: brief per-user calibration rather than lengthy training sessions.

Future Directions

• Integration into real-time RLHF. The authors identify integrating these classifiers into real-time RLHF frameworks, so that adaptive agent behavior can be driven by implicit neural feedback, as the natural next step.

• More robust cross-subject generalization. Cross-validation success was limited to Robot Passive, and fine-tuning was insufficient for regression outside of Lunar Active. The authors call for domain adaptation and deep learning techniques, more personalized calibration strategies, and expanded datasets to support transfer across participants.

• Multimodal feedback. The paper suggests combining fNIRS with other signals such as EEG, EMG, and GSR to gain greater insight into internal human assessments.

• Addressing dataset and signal limitations. The dataset is partially imbalanced because some condition episodes were shorter, lower engagement in certain passive conditions may have degraded neural signal quality, and fNIRS remains subject to signal noise, motion artifacts, and inter-subject variability. The authors highlight continued development of robust preprocessing as needed.

Target Audience

Readers who will benefit most are researchers working at the intersection of reinforcement learning and human-computer interaction, particularly those interested in brain-computer interfaces for implicit feedback, human-in-the-loop machine learning, and RLHF alternatives. It is also well suited to BCI practitioners evaluating fNIRS as a complement to EEG, to HCI researchers interested in measuring user workload during agent supervision, and to graduate students or engineers who want a concrete, reproducible dataset and baseline for neural-signal-to-agent-performance classification. Readers looking for a deployed RLNF system will not find one here; the paper is explicitly scoped as the classification prerequisite for such a system.

Authors’ abstract

Reinforcement Learning from Human Feedback (RLHF) is a methodology that aligns agent behavior with human preferences by integrating user feedback into the agent's training process. This paper introduces a framework that guides agent training through implicit neural signals, with a focus on the neural classification problem. Our work presents and releases a novel dataset of functional near-infrared spectroscopy (fNIRS) recordings collected from 25 human participants across three domains: Pick-and-Place Robot, Lunar Lander, and Flappy Bird. We train multiple classifiers to predict varying levels of agent performance (optimal, suboptimal, or worst-case) from windows of preprocessed fNIRS features, achieving an average F1 score of 67% for binary and 46% for multi-class classification across conditions and domains. We also train multiple regressors to predict the degree of deviation between an agent's chosen action and a set of near-optimal policy actions, providing a continuous measure of performance. Finally, we evaluate cross-subject generalization and show that fine-tuning pre-trained models with a small sample of subject-specific data increases average F1 scores by 17% and 41% for binary and multi-class models, respectively. Our results demonstrate that mapping implicit fNIRS signals to agent performance is feasible and can be improved, laying the foundation for future Reinforcement Learning from Neural Feedback (RLNF) systems.