A First-Order Logic-Based Alternative to Reward Models in RLHF

AI summary

Authors: Chunjin Jiang, Xinhua Zhu (School of Computer Science and Engineering, Guangxi Normal University, Guilin, China) arXiv: 2512.14100v1 [cs.LG], 16 Dec 2025

Overview

• Research area: Reinforcement Learning from Human Feedback (RLHF) and large language model alignment, with a focus on replacing learned reward models using formal logic.
• Technical level: Advanced. The paper assumes familiarity with policy-gradient reinforcement learning, KL-divergence regularization, GRPO, DPO, SFT, and first-order logic notation.
• Scope: The paper proposes S-GRPO, a supervised variant of Group Relative Policy Optimization (GRPO) that adds a label-based supervision term to the training objective and, for preference tasks, replaces the conventional reward model with a logic-similarity reward computed over first-order logic (FOL) representations. It is evaluated on FOL translation, WMT English–German translation, and the PKU-SafeRLHF preference dataset.

What This Paper Is About

RLHF pipelines typically train a separate reward model to score model outputs, and the quality and stability of that reward model largely determine how well the language model ends up aligned. The authors argue that this dependence on a separately trained scalar reward model is both a cost and a fragility, so they build a training objective that directly incorporates gold-standard labels into the GRPO loss and, for tasks without automatic evaluation metrics, scores outputs by measuring how logically similar they are to reference answers. The goal is alignment that is more interpretable, more stable, and no longer dependent on a learned reward network.

Key Contributions

1. S-GRPO, a supervised variant of GRPO that integrates a label-based alignment term into the loss function, jointly optimizing the generation term, KL-divergence regularization, and a label-based objective; it is reported to outperform standard supervised fine-tuning (SFT) baselines.
2. A logic-similarity-based reward mechanism for preference learning that replaces conventional reward models with a formal, interpretable measure of logical alignment, computed by translating both model outputs and references into first-order logic and comparing them.
3. An optimized Logical Equivalence (LE) metric, motivated by the authors' finding that the original LE implementation from Yang et al. was inefficient and produced inaccurate matches; the optimized version scores 1–2% lower than the original but matches faster and more precisely (marked with an asterisk, LE*, in the tables).
4. A set of experiments spanning FOL translation (FOLIO), bilingual translation (WMT English–German), and preference learning (PKU-SafeRLHF), all using Qwen3-1.7B as the backbone model, with reward values constrained to the range [0, 1].

Main Findings

• FOL translation (FOLIO, split test set): Qwen3-1.7B-SGRPO reached LE* 87.4 and BLEU 66.0, versus Qwen3-1.7B-SFT at LE* 85.0 and BLEU 61.2. The authors note S-GRPO outperformed all existing methods and models on this task in their comparison.
• Reference numbers on FOLIO: ChatGPT-3.5 scored LE 80.2, LE* 77.6, BLEU 37.0; DeepSeek-V3 scored LE 83.0, LE* 79.2, BLEU 37.6; ChatGPT-4o scored LE 82.6, LE* 80.9, BLEU 38.4; LogicLLaMA-7B RLHF Corre. scored LE 84.1 and BLEU 37.8 with LE* not reported; LogicLLaMA-13B RLHF Corre. scored LE 85.8 and BLEU 38.4 with LE* not reported. The authors state they could not reproduce the LogicLLaMA results under LE* because of its relatively complex RLHF pipeline.
• WMT 2022 bilingual translation: S-GRPO scored BLEU 30.18 / COMET 86.17 for De→En and BLEU 21.07 / COMET 79.13 for En→De, improving on Qwen3-1.7B-SFT at BLEU 28.55 / COMET 85.79 and BLEU 19.57 / COMET 77.70. Numbers for NLLB-54B, GPT-4, TIM-7B, ALMA-7B, BigTranslate-13B, Bayling-13B, and ALMA-13B are also tabulated, but the authors caution that comparisons may be imperfect due to differences in training data and model architectures.
• Training stability on WMT: Over five training epochs, S-GRPO remained stable throughout, while SFT began to degrade after the first epoch. The mean reward during S-GRPO training steadily increased, and although the standard deviation rose slightly, the fluctuation remained small. The base model could not produce meaningful outputs during early training, so SFT was run first to make the model generate aligned and coherent responses; the SFT baseline was trained for two full epochs and the best-performing checkpoint was reported.
• PKU-SafeRLHF preference task: DPO achieved the best results (Win 28.69, Tie 53.95, Loss 17.35), followed by S-GRPO (Win 22.82, Tie 52.09, Loss 25.09), with SFT far behind (Win 0.95, Tie 2.41, Loss 96.62). During S-GRPO training on this dataset, the mean reward was 0.17 with a standard deviation of 0.30.
• Why SFT failed on preference data: The PKU-SafeRLHF training set consists primarily of short responses, whereas the base model tends to generate long, elaborate answers even for non-expert queries; this mismatch leads to substantial penalization. DPO's advantage is attributed to avoiding the high-variance issues common to RL-based methods, since the base model naturally satisfies π_θ(chosen|x) > π_θ(rejected|x).
• Judging protocol: DeepSeek-V3 was used as the automated evaluator rather than GPT-4, with outputs presented in alternating orders. Win means better in both orders, or wins once and ties once; Tie means ties in both orders, or wins once and loses once; Loss means worse in both orders, or ties once and loses once.

Methodology in Plain English

The authors start from GRPO, a reinforcement learning method that drops PPO's separate critic network and computes advantages by comparing responses within a group. Instead of relying on an external reward model to signal quality, they append a term to the GRPO objective that directly measures how much more likely the current model is to produce the correct label than the reference model is. This makes the objective a hybrid: one part drives exploration and generation, one part keeps the model close to its reference via KL-divergence, and one part pulls it toward gold labels. The intuition is that when the reward signal is noisy or unstable, the label term keeps updates controlled.

For tasks with automatic metrics (translation), the metric itself is used as the reward. For preference tasks, where no such objective metric exists, the authors translate both the model output and the reference response into first-order logic using a pre-trained FOL translation model, then compute logical equivalence. Each FOL expression is tokenized, pairwise cosine similarities are calculated, and tokens with similarity above a threshold of 0.6 are treated as semantically related. When one unit matches several candidates, all possible alignments are enumerated and the highest-scoring match is chosen as the final correspondence.

The appendix explains that the original LE method enumerated matches by ascending Levenshtein edit distance at O(n!) complexity (n being the length of the atomic formula set), and built NLTK parse trees by complete enumeration at O(2^(n-1)) complexity. The authors' fix computes cosine similarity between semantic units and only permutes over relevant candidates in one-to-many cases, giving a best-case O(n) and worst-case O(n!) complexity; in practice candidate sets for one-to-many matches rarely exceed three units. They also introduce chunk-based decomposition of FOL parsing with chunk size m typically 3–5 sub-units, reducing the exponential term from O(2^(n-1)) to an average case O(2^(n-1/m)), degrading to O(2^(n-1)) only in extreme cases.

Why This Matters

The paper attacks a widely acknowledged bottleneck in alignment: the learned reward model that sits between human preferences and policy optimization. By showing that label supervision plus formal logical comparison can substitute for a trained reward network, it opens a route to alignment that is more interpretable and less dependent on a potentially brittle auxiliary model. It also contributes a practical efficiency fix to a logical equivalence metric that the authors found computationally prohibitive for longer expressions.

Real-world applications the approach could support:

• Reasoning systems where logical consistency, not surface fluency, is the criterion for correctness.
• Structured translation, including formal or logic-notation translation such as the FOLIO task.
• Formal text generation, where the paper states logic-equivalence rewards remain highly promising.
• Safety and preference alignment at moderate scale, as demonstrated on PKU-SafeRLHF along the helpfulness and harmlessness dimensions.

On industry relevance: the work targets the resource constraints of RLHF, noting that PPO requires an additional critic network whose computational and memory overhead limits scalability on small or resource-constrained clusters. Removing the reward model could reduce the number of auxiliary models an organization must train and maintain, although the authors are explicit that S-GRPO still requires substantial compute.

Future Directions

1. Closing the semantic mismatch gap. The authors identify semantic variation between model outputs and reference labels as the current limit on S-GRPO's effectiveness in purely preference-based settings, and suggest that optimizing logical matching strategies could further narrow it.
2. Extending the logic-equivalence reward to logic-intensive tasks such as reasoning, structured translation, and formal text generation, where logical consistency and interpretability are described as essential.
3. Reducing training cost. S-GRPO inherits the expense of sequence generation during training, making it substantially more expensive than SFT or DPO, which use teacher forcing; the authors state that unless a task specifically benefits from reinforcement-based exploration, its training cost should be considered comparable to that of GRPO.
4. Testing more unconventional reward function designs. The authors report that S-GRPO can flexibly accommodate diverse and even unconventional reward functions, which they present as an opening for further reward-design exploration.

Target Audience

Researchers and engineers working on LLM alignment, RLHF, and preference optimization who are already comfortable with policy-gradient methods and want an alternative to learned reward models. It will also interest people working on neuro-symbolic methods and formal reasoning, and practitioners who need interpretable alignment signals under compute constraints. Readers looking for a beginner-level introduction to RLHF will find the paper's reliance on GRPO, DPO, KL-divergence, and first-order logic notation demanding.

Authors’ abstract

Reinforcement Learning from Human Feedback (RLHF) plays a crucial role in aligning large language models (LLMs) with human values and preferences. However, the quality and stability of the trained reward model largely determine the final alignment performance. Existing approaches such as Proximal Policy Optimization (PPO) rely heavily on reward models to guide LLMs toward human-aligned behaviors. In this work, we propose a logic-similarity-based reward mechanism as an alternative to conventional reward modeling. Instead of relying on heuristic reward estimation, our method leverages formal logical consistency to steer model alignment with human preferences. Since real-world questions can be interpreted from multiple perspectives, to ensure that logic-based reinforcement learning does not cause model collapse, we introduce S-GRPO, a supervised variant of the GRPO framework. S-GRPO incorporates an additional supervised component and jointly optimizes the generation term, KL-divergence regularization, and label-based objective during training. Experimental results demonstrate that S-GRPO consistently outperforms standard supervised fine-tuning (SFT) in both performance and robustness. Furthermore, it extends existing preference-learning frameworks such as GRPO and DPO, offering a more flexible and task-adaptive approach to alignment training. Our code is available at https://github.com/ChunjinJiang/sgrpo.