WeFair: Fidelity-Aware Alignment from Human Feedback for Personalized Query Rewriting in Short-Video Search

AI summary

Overview

Research area: Information retrieval and natural language generation — specifically personalized query rewriting for short-video search, using LLM post-training (supervised fine-tuning, direct preference optimization, and reinforcement learning from verifiable rewards).

Technical level: Advanced. The paper assumes familiarity with SFT, DPO, GRPO, reward modeling, retrieval pipelines, and offline/online A/B evaluation.

Scope: The paper introduces WeFair, a three-stage "fidelity-aware" training framework that matches each type of human feedback (demonstrations, consumption-based preferences, exact-match verification) to the training stage where it is most reliable, producing a compact 2B-parameter query rewriter deployed in WeChat short-video search.

What This Paper Is About

In short-video search, users type short, ambiguous queries whose real meaning depends on their recent searches and viewing history. A rewriter model must expand the query to recover that intent — but it must also know when to leave the query alone, and it must run under strict latency limits, so only small models are practical. The problem this paper targets is intent drift: when a rewriter is trained mainly on engagement/consumption signals, it learns to produce queries that retrieve catchy videos rather than what the user actually wanted, so the paper's goal is to train a compact rewriter that follows evidence of real user intent while still being fast enough for production.

Key Contributions

1. An analysis of supervision "fidelity." The authors argue and empirically show that human feedback differs in how directly it reflects personalized intent, and that each training stage tolerates a different level of fidelity: broad, lower-fidelity supervision helps curriculum SFT, consumption-derived preferences help DPO, and only strict exact-match verification should drive the final on-policy GRPO stage.

2. The WeFair framework. A progressive pipeline — demonstration (curriculum SFT) → preference (DPO with consumption pairs) → verification (GRPO with human-grounded exact-match reward) — plus a data construction procedure that mines and screens real search sessions into context-to-query, reformulation, and preservation pools.

3. Evidence that auxiliary rewards hurt on-policy alignment. Under matched training data, strict verification outperforms variants augmented with semantic similarity, consumption feedback, or a learned reward model (including a 4-vote reward-model variant) in consistent recall and overall accuracy.

4. Offline and online validation in a live system. WeFair-2B/4B beat strong large-model and adaptive-rewrite baselines offline, and a 10-day online A/B test in WeChat short-video search showed relative increases of 2.951% in click-through rate and 2.593% in effective video-view rate with a 2.350% relative reduction in query reformulation rate; the 2B model is deployed serving over 100 million queries per day.

Main Findings

• Best overall offline accuracy from a compact model: WeFair-4B reaches 73.48% overall accuracy and WeFair-2B reaches 72.81%, versus 64.42% for Gemini-3.1-Pro and 62.55% for DeepSeek-V4.1-Flash under Full CoT. WeFair-2B also posts the highest Consistent Recall at 83.95.

• Much lower intent drift than comparable rewriters: WeFair-2B shows a 2.43 hallucination rate and 5.36 intent drift rate, against 13.34/16.45/15.88 intent drift for WeWrite-0.8B/2B/4B and 14.88/13.34 for AWHCP-2B/8B. The paper describes this as reducing intent drift to about one-third of comparable methods. Large models like Gemini-3.1-Pro have slightly lower drift (4.16) but much lower rewrite recall.

• Big models under-rewrite; baselines over-rewrite: Gemini-3.1-Pro keeps the highest EM and lowest hallucination/drift but has far lower rewrite recall, while WeWrite and especially AWHCP lean toward rewriting even when preservation is correct. Even WeFair-0.8B (PD Recall 49.46) surpasses AWHCP-8B (13.39) at knowing when not to rewrite.

• Diminishing returns to scale: Overall accuracy rises 5.91 percentage points from 0.8B to 2B but only 0.67 points from 2B to 4B, with consistent recall nearly flat — most of the benefit is already present at 2B.

• Direct output beats explicit reasoning for the trained models: Direct WeFair exceeds WeFair-CoT at both 2B (72.81 vs 72.30) and 4B (73.48 vs 72.25) with lower intent drift, while for a general model (Qwen3.8-27B) CoT greatly beats direct prediction. The authors read this as fidelity-aware training producing effective implicit reasoning.

• Curriculum SFT beats its ablations: Full Curriculum reaches 72.67 consistent recall / 55.48 overall accuracy, versus 66.70/47.16 for HR Only, 69.45/47.23 for C2Q→HR, and 66.76/51.26 for CoT→HR.

• Each alignment stage contributes, verification most of all: SFT Only 55.48 overall, SFT→DPO 56.54, SFT→GRPO 70.35, full WeFair 72.81 — and the full sequence beats SFT→GRPO on consistent recall (83.95 vs 78.93), suggesting preference alignment adds value beyond its immediate gains.

• Strict rewards dominate augmented rewards: Strict Verification scores 78.93 consistent recall and 70.35 overall, versus 69.99/50.84 for the best augmented variant (RM-Augmented Verification, 4 Vote), 68.20/48.56 for RM-Augmented, 65.39/46.62 for Similarity-Augmented, and 63.47/46.14 for Consumption-Augmented. Strict Verification shows advantage purity P+ of 100.00, P− of 71.50, and P_all of 83.75.

• Cheap at inference: WeFair-2B decodes 7.60 tokens at 0.316 s mean time, versus 47.06 tokens/1.687 s for AWHCP-8B and 1023.38 tokens/61.380 s for Qwen3.8-27B with CoT.

• Online gains scale with retrieval integration: Cache Retrieval alone gives +1.954% CTR, +2.281% VV>10s and −1.628% QRR; adding vector retrieval gives +2.951% CTR, +2.593% VV>10s and −2.350% QRR (all P<0.01).

• Qualitative online cases: Three of four selected cases improved (anggwe→angew surfacing research content instead of Engwe e-bike videos; adding 意义 to a civics query; adding 招聘 to a school query), while one degraded (教育号→教育局视频号 missing the target account in the top 6).

• Judge reliability: On 500 sampled Gemini-3.1-Pro responses, the LLM judge agreed with human review 96.6% of the time (Cohen's Kappa = 0.919).

Methodology in Plain English

Step 1 — Harvest real user behaviour, then filter it hard. The team mined search sessions for three kinds of records: (a) pairs of a context and a query that led to a video watched for more than ten seconds, (b) genuine reformulations where a user rewrote their own query within a session (no click after the first query, an effective view after the second), and (c) cases where the query worked fine and should be preserved. Suggestion-triggered queries were excluded and system suggestions scrubbed from context.

Step 2 — Clean the reformulations cheaply, then with LLMs. Rule-based filters remove candidates with no new characters, only added function words, character-set Jaccard similarity of 0.06 or below, or no contextual support — about half of the initial candidates are dropped here. Survivors go to LLM screening requiring intent continuity and information gain (three models, majority vote), then a second three-model majority vote on whether the query-time context actually supports the added detail. About one third pass both checks. Rejected-but-otherwise-plausible cases become "hard" preservation examples, which teach the model not to invent unsupported personalization.

Step 3 — Teach the model in a curriculum. Stage one training has three phases: predict the query from prior context (broad, plentiful, lower fidelity), then produce a teacher-written rationale plus the target, then predict the target directly. Rationales come from Gemini-3.1-Pro-Preview, and only the final phase's direct-prediction behaviour is used at serving time.

Step 4 — Learn preferences from what people actually consumed. For each input the model samples 64 candidate rewrites and builds up to two preference pairs: human reformulation versus a candidate that retrieves none of the user's consumed videos in its top 100, and a consumption-supported candidate (a consumed video in its top 10) versus a low-engagement candidate. Preservation examples pair the preservation marker against a generated rewrite.

Step 5 — Anchor the final updates to the truth. The last stage samples 32 responses per input and rewards only exact agreement with the screened human reformulation or the preservation marker; everything else gets zero. The authors deliberately add no similarity, retrieval, or learned-judge shaping, and they guard against the model "reward hacking" by over-preserving (preservation is easier to output than a correct rewrite) with a controlled data ratio and reward weights.

Step 6 — Serve it inside the real search stack. Rewrites are used both through a cached "fake recall" path and a lightweight vector retrieval path. The original query stays a ranking anchor: the system looks at the probability of the preservation token at the first decoding step and, above a threshold, ranks using only the original query; otherwise it combines relevance to both queries.

Step 7 — Evaluate offline and live. Training used four weeks of search logs and testing the following three days, with a DeepSeek-V4.1-Flash judge backed by the human agreement check; the online test was a 10-day A/B comparison reporting CTR, effective video-view rate (VV>10s), and query reformulation rate as relative changes.

(Note: the provided text is truncated partway through Section 4.4, so the later latency analysis in Section 4.5, appendices, and any conclusion are not covered here.)

Why This Matters

The paper's core claim — that the fidelity of a feedback signal, not just its volume, should determine which training stage consumes it — is a practical recipe for any team aligning a small model on noisy behavioural logs. It also gives a concrete counterexample to the common practice of stacking extra reward terms onto RL post-training: here, more supervision made the policy worse.

Real-world applications:

• Short-video and feed search (the paper's own setting), where ambiguous queries must be resolved from recent viewing under tight latency budgets.
• E-commerce and local search, where a user's browsing or purchase history disambiguates a two-word query and mis-rewriting sends them to the wrong product category.
• Voice assistants and conversational search, where a single utterance must be resolved against dialogue context without over-personalizing.
• In-app content discovery and recommendation entry points, where deciding not to rewrite prevents the "engaging but irrelevant" failure mode that drives users to give up and re-type their query.

Industry relevance: The result is deliberately production-shaped — a 0.8B–4B model family, ~7.6 decoded tokens and ~0.32 s per sample at 2B, a deployed 2B model handling over 100 million queries per day, and a 10-day online A/B test with statistically significant (P<0.01) gains in CTR and effective video views plus a drop in query reformulation rate (a direct proxy for users' dissatisfaction with results).

Future Directions

• Better handling of the preservation-versus-rewrite trade-off. The authors identify reward hacking toward preservation as a real risk and address it with data ratios and reward weights; learned or calibrated control of that balance is left open. The single degraded online case (a missing target account) shows the cost of a wrong preservation-or-rewrite call.

• Understanding why strict rewards win. The paper shows augmented rewards underperform but does not explain the mechanism beyond advantage-purity statistics; a systematic study of what makes a reward "pure enough" for on-policy use would generalize beyond search.

• Closing the intent-drift gap to large CoT models. Gemini-3.1-Pro still has a lower drift rate (4.16 vs 5.36 for WeFair-2B) but much weaker rewrite recall; combining large-model fidelity with compact-model decisiveness is an obvious next target.

• Extending fidelity-aware staging to other signals and domains. The framework is demonstrated on consumption behaviour and reformulations in one product; whether the demonstration/preference/verification ordering transfers to purchase history, dwell time, or conversational context is untested here.

Target Audience

Applied ML and IR engineers building production query understanding or search-ranking systems; researchers working on RLHF/RLVR post-training who want an industrial case study on reward design and data curation; and technical leads evaluating whether a compact fine-tuned model can replace large-LLM rewriting under latency constraints. Readers need working knowledge of SFT/DPO/GRPO and retrieval evaluation to get the most from the results tables.

Authors’ abstract

Short-video search needs compact query rewriters that recover users' personalized intent from ambiguous queries under strict latency constraints. Existing LLM-based methods have improved rewriting by using synthetic reasoning and subsequent consumption-behavioral feedback. However, overemphasizing subsequent consumption can favor engaging content over the user's immediate need, causing intent drift even when rewritten queries are grounded in behavioral history. We propose WeFair, Fidelity-Aware Alignment from Human Feedback for Personalized Query Rewriting in WeChat short-video search. WeFair uses human feedback according to how directly it reflects personalized intent, assigning feedback fidelity to a progression from demonstration to preference to verification. We first mine actual queries and user reformulations from search sessions, and then screen them for contextual support. In the demonstration stage, a supervised fine-tuning curriculum progresses from large-scale context-to-query learning to reasoning and direct prediction on human reformulation and preservation. In the preference stage, direct preference optimization learns relative preferences from subsequent consumption feedback. In the verification stage, group relative policy optimization rewards exact agreement with reformulation or preservation targets to constrain intent drift. Further analysis shows that broad supervision benefits the full curriculum supervised training, while strict verification achieves better performance than auxiliary rewards. In offline experiments, WeFair-2B outperforms strong rewriting baselines in overall accuracy and intent preservation. An online A/B test shows relative increases of 2.593% in effective video-view rate and a 2.350% relative reduction in query reformulation rate. WeFair-2B has been deployed in the WeChat short-video search system, which serves over 100 million queries per day.