Leveraging Psychophysical Attentional Distribution for Gaze-Augmented Reward Modeling

AI summary

Overview

Research area: Machine learning for language model alignment — specifically reward modeling augmented with human eye-tracking (gaze) signals, informed by cognitive psychology and psychophysics of reading.

Technical level: Intermediate. Readers should be comfortable with the standard RLHF/reward-model setup (prompt, chosen response, rejected response, pairwise preference loss) and with basic Gaussian functions, but the central idea is intuitive and explained in accessible terms.

Scope: The paper proposes a single modification to the GazeReward pipeline — redistributing token-level total reading time (TRT) across neighboring tokens using a learnable asymmetric Gaussian kernel — and evaluates whether this improves preference prediction accuracy and produces a left–right asymmetry consistent with the human perceptual span.

What This Paper Is About

Reward models in RLHF learn only from the final human preference judgment, not from how a person actually read and processed the two candidate responses. Prior work (GazeReward) addressed this by attaching predicted eye-tracking features to each token, but it treats gaze as a pinpoint signal confined to the directly fixated token. This paper argues that human visual attention is spatially graded and asymmetric — extending much further to the right of fixation than to the left in left-to-right reading — and tests whether spreading gaze features over a neighborhood around each fixation improves reward modeling.

Key Contributions

1. A gaze-redistribution module for reward modeling. The authors insert an extra step into the GazeReward pipeline: token-level TRT values from a pretrained eye-tracking generator are spread over neighboring tokens via an asymmetric Gaussian kernel with separate left and right width parameters (σ_left, σ_right) that are learned jointly with the reward model.

2. Empirical gains across models and integration settings. The method improves preference prediction accuracy over the pinpoint baseline in all four OASST1 configurations (two Llama-3 backbones × two gaze integration modes, GazeConcat and GazeAdd), and on two additional datasets (HelpSteer2, UltraFeedback).

3. A learned asymmetry that matches psychophysical findings. The optimized σ_right generally exceeds σ_left, echoing the human perceptual span in left-to-right reading, which extends roughly 3–4 letter spaces left but 14–15 letter spaces right.

4. An ablation isolating where the gain comes from. Comparing pinpoint, uniform redistribution, fixed 1:4 asymmetric Gaussian, learnable symmetric Gaussian, and learnable asymmetric Gaussian shows that learning the spread parameters matters more than the asymmetry itself.

5. A quantitative comparison against a cognitive reading model (Appendix A). The learned kernels are compared to OB1-reader's simulated attentional allocation on the Provo Corpus using Spearman correlation and Jensen–Shannon divergence.

Main Findings

• GazeConcat gains on OASST1 (Llama-3-8B-Instruct): accuracy rises from 68.38 ± 3.88 (baseline) to 74.73 ± 1.55, a 9.29% relative improvement.

• GazeConcat gains on OASST1 (Llama-3-8B): accuracy rises from 69.23 ± 4.92 to 74.81 ± 1.32, an 8.06% relative improvement.

• GazeAdd gains are smaller but consistent: Llama-3-8B-Instruct improves from 70.30 ± 1.25 to 71.89 ± 1.22 (2.26% relative); Llama-3-8B improves from 69.27 ± 0.81 to 71.17 ± 1.21 (2.74% relative).

• Largest gain on UltraFeedback: with Llama-3-8B under GazeConcat, accuracy rises from 65.65 ± 2.12 to 74.20 ± 0.46 (13.02% relative), with learned σ_left = 1.48 ± 1.69 and σ_right = 3.79 ± 1.94.

• HelpSteer2 shows the smallest gain and the one reversed asymmetry: accuracy rises from 59.48 ± 3.66 to 62.64 ± 1.68 (5.31% relative), with learned σ_left = 3.18 ± 2.34 and σ_right = 1.85 ± 2.01. The authors speculate this is because HelpSteer2 supplies only 2 responses per prompt in almost all cases, forcing use of the only available pair, whereas OASST1 averages 2.82 candidate responses per prompt and UltraFeedback has 4 responses for nearly every prompt, allowing maximally separated chosen–rejected pairs.

• Learned widths tend to be rightward-asymmetric on OASST1: for Llama-3-8B under GazeConcat, σ_left = 0.80 ± 0.56 versus σ_right = 3.47 ± 2.13; for Llama-3-8B-Instruct under GazeConcat, σ_left = 1.62 ± 1.18 versus σ_right = 2.25 ± 1.71; for Llama-3-8B-Instruct under GazeAdd, σ_left = 0.58 ± 0.21 versus σ_right = 1.99 ± 1.17; for Llama-3-8B under GazeAdd the gap is smaller (0.64 ± 0.23 versus 0.80 ± 0.31).

• Ablation (Llama-3-8B, GazeConcat, OASST1): pinpoint baseline 69.23 ± 4.92; uniform 1:4 redistribution 70.18 ± 1.15 (1.37% relative); fixed 1:4 asymmetric Gaussian 71.46 ± 2.28 (3.22%); learnable symmetric Gaussian 74.19 ± 1.86 (7.16%, learned σ = 3.28 ± 0.84); learnable asymmetric Gaussian 74.81 ± 1.32 (8.06%).

• Learning the parameters matters more than the asymmetry: the learnable symmetric and asymmetric conditions clearly beat the uniform and fixed-Gaussian conditions, but the asymmetric condition's edge over the symmetric one is small relative to seed-to-seed variability. The learned symmetric σ (3.28) is close to the asymmetric condition's σ_right (3.47), suggesting most of the predictive information lies to the right of the fixated token.

• Improved stability: standard deviations across six random seeds are generally smaller than the pinpoint baseline's, which the authors attribute to smoothing sparse token-local TRT values over neighboring tokens.

• Agreement with OB1-reader (Appendix A): with six Llama-3-8B GazeConcat checkpoints averaging σ_left = 0.804 and σ_right = 3.471 (a 1:4.32 left-to-right ratio), and OB1-reader run under 1:4 asymmetry on 100 Provo Corpus trajectories, Spearman correlation is 0.4645 (pinpoint), 0.8958 (symmetric), and 0.9095 (learned asymmetric); JS divergence is 0.2756, 0.1584, and 0.1473 respectively. Pooled weight allocation: center 26.83% (asymmetric) versus 26.69% (symmetric); left 11.39% versus 26.65%; right 61.77% versus 46.66%.

• Data loss from input-length caps is modest: OASST1 capped at 5,000 characters (307 of 8,161 samples lost, 3.76%); HelpSteer2 and UltraFeedback capped at 10,000 characters (241 of 7,592 lost, 3.17%; 1,632 of 63,966 lost, 2.55%).

Methodology in Plain English

The starting point is GazeReward, which augments a reward model with predicted eye-tracking features. Because real eye-tracking data are expensive and absent from most preference datasets, GazeReward uses a pretrained gaze predictor (from Huang and Hollenstein, 2023) to generate token-level features such as total reading time (TRT), first fixation duration, go-past time, number of fixations, and fixation proportion. These features are projected into the reward model's embedding space and either concatenated with (GazeConcat) or added to (GazeAdd) the text embeddings, after remapping between the gaze model's tokenizer and the reward model's tokenizer.

This paper keeps everything else fixed and changes only how gaze features are constructed. It takes TRT — the cumulative fixation duration on a word, a proxy for total attention — and, instead of leaving each value on its own token, spreads it across neighbors using an asymmetric Gaussian kernel centered on the fixated token. Distance is measured in token indices. The kernel is normalized per source token so total TRT mass is preserved, and the redistributed value for each token is the sum of contributions from all fixated tokens. The two width parameters, σ_left (for tokens to the left) and σ_right (for tokens at or to the right), start at 1.0 and are trained alongside the reward model with a separate, larger learning rate (5×10⁻² for the σ's versus 5×10⁻⁵ for the rest of the model), discovered through empirical tuning.

Experiments use Llama-3-8B and Llama-3-8B-Instruct reward models with LoRA (rank 8, alpha 32, dropout 0.1), trained for 2 epochs with AdamW, a cosine schedule with minimum learning-rate ratio 0.7, weight decay 0.1, per-device batch size 1 with gradient accumulation over 8 steps (effective batch size 8), and six random seeds (41–46). Evaluation is standard pairwise preference accuracy: a prediction counts as correct when the chosen response receives a higher reward than the rejected one. The main evaluation is on OASST1, with Llama-3-8B and GazeConcat (the best OASST1 configuration) carried over to HelpSteer2 and UltraFeedback.

Why This Matters

Impact on research. The paper is a case study in grounding an ML design choice in cognitive psychology rather than tuning it empirically: the asymmetric Gaussian comes directly from visual-span research (Legge et al., 2001; Rayner, 2014), and the learned parameters are then checked against that literature and against OB1-reader. It also offers a cheap, architecture-agnostic improvement — the redistribution module changes only the gaze representation, leaving the pairwise objective and reward model architecture untouched — and suggests a stability benefit from smoothing sparse supervisory signals. The authors are explicit that the fitted parameters do not constitute a direct estimate of human attention.

Real-world applications.

• Reward modeling and preference tuning pipelines that already use, or could use, gaze-augmented inputs, where a small module change yields accuracy gains without altering the training objective.
• Human–computer interaction and reading research, where learned attention profiles can be compared against established perceptual-span measurements as a sanity check on simulated gaze.
• Educational technology and readability assessment, where spatially distributed attention models could inform how text is laid out or chunked for readers.
• Any NLP system built on predicted rather than collected gaze (e.g., using ScanTextGAN-style predictors), where a denser, smoothed gaze representation may reduce sensitivity to noisy predictions.

Industry relevance. The method is lightweight and compatible with LoRA-based fine-tuning on modest hardware (the authors trained on three local servers with RTX A6000 and A5000 GPUs), and it requires no new eye-tracking data collection because it operates on synthetic gaze. Teams already running GazeReward-style pipelines can adopt it with minimal engineering. Gains are not uniform, however: the smallest improvement (HelpSteer2) coincided with datasets that offer only a single forced response pair, which is a practical caveat for teams whose preference data look like that.

Future Directions

• Beyond monotonic decay. The Gaussian kernel decreases monotonically from fixation, but prior work suggests the attentional field may be Mexican-hat-like, with enhancement near the attended location and suppression in the surround (Müller et al., 2005), and may be nonmonotonic and individually variable (Lee and Pitt, 2022). Distributions with inhibitory surround, nonmonotonic profiles, or per-individual span parameters are natural extensions.

• Letter-level rather than token-level distance. The current implementation measures distance in token indices for compatibility with the token-level eye-tracking predictor; letter-level distance might better match the spatial scale of human visual attention.

• Richer gaze features. Only TRT is redistributed here. First fixation duration, fixation probability, and number of fixations may carry complementary information and could be added to the framework.

• Context dependence of the attentional span. The authors note that the real span varies with reading skill, text difficulty, typography, task demands, and linguistic context — none of which the current single learned profile captures — and leave open why HelpSteer2's learned profile reversed direction.

Target Audience

Researchers and practitioners working on LLM alignment and reward modeling, especially those interested in multimodal or cognitively grounded supervision signals. It will also interest cognitive scientists and psycholinguists who study reading and eye movements and want to see their findings operationalized in a machine learning objective, and NLP engineers who already use gaze prediction models and want a low-cost way to make those features more robust. Readers without a background in RLHF will need to pick up the pairwise preference-loss framing from the background section, but the core mechanism is accessible to anyone comfortable with Gaussian weighting.

Authors’ abstract

Integrating human feedback into language models has gained increasing attention as a means to align model outputs with human preferences, with Reinforcement Learning from Human Feedback (RLHF) serving as a prominent framework for preference-based alignment. Recent studies have incorporated eye-tracking (ET) data as an additional supervisory signal for reward model training, grounding reward learning in human reading behavior. Building on this approach, we extend prior…