Toward Skill-Native LLMs: Skill Entropy for Benchmarking and Training Long-Horizon Reasoning

AI summary

Overview

Research area: Natural Language Processing / large language model reasoning — specifically long-horizon reasoning, benchmark design, and reinforcement-learning post-training of LLMs.

Technical level: Advanced. The paper combines a formal skill-switching metric, a synthesized 558-skill benchmark, and a GRPO-based RL training pipeline with a custom reward, so familiarity with RL post-training, reward shaping, and benchmark evaluation is assumed (though the paper walks through both the evaluation and training sides).

Scope: This paper introduces "skill entropy," a directed pairwise measure of how hard it is to switch between reasoning skills inside a task, and uses it both to build a benchmark (Skill²-Bench) that exposes a skill-switching gap across 12 models and as a reward signal (Skill-Entropy RL) that substantially improves two Qwen3 base models.

What This Paper Is About

Long-horizon reasoning tasks in the real world rarely stay in one domain — a model may need to do a math derivation, then use that result to build a schedule, then use the schedule to guide information extraction — requiring it to keep one chain of reasoning while fluidly switching skills. Existing benchmarks mostly score single skills in isolation, so they cannot measure how hard a particular skill switch is or explain why models that handle each skill well still fail when the skills are chained. The paper's goal is to define a principled measure of skill-switching difficulty, use it to benchmark models, and then convert that same measure into a training signal that teaches models to switch skills cleanly.

Key Contributions

1. Formalization of cross-skill long-horizon tasks and skill entropy. A cross-skill task is a length-L sequence of question-answer steps (L in [2, 10]) where each step uses a different skill and depends on earlier answers; skill entropy SkE(s_a, s_b) is a directed pairwise quantity measuring how hard it is to switch from skill s_a to skill s_b.

2. Skill²-Bench, a skill-entropy-calibrated benchmark. A benchmark built over 558 skills across 9 verifiable and open-ended domains, where each task receives a task-level skill-entropy score and is grouped into low, medium, and high difficulty levels.

3. Skill-Entropy RL, an RL framework with a skill-entropy reward. The model predicts a skill label alongside each step's answer, and the reward combines step-level correctness with a skill-entropy reward that grades the predicted skill sequence against the gold skill sequence.

4. Evidence that skill entropy is a reusable training signal. The same pipeline is shown to plug into off-the-shelf data (OpenR1-Math), improving performance there as well.

Main Findings

• Skill-switching gap on Skill²-Bench: Evaluating 8 frontier models (Claude-haiku-4.5, Claude-sonnet-4.5, Claude-opus-4.7, Gemini-3.1-flash, Gemini-3.1-pro, GPT-5.4-mini, O4-mini, GPT-5.5) and 4 open-source models (Qwen3-4B, Qwen3-8B, Qwen3-32B, Olmo-3-7B-Think) shows accuracy decreasing nearly monotonically as task-level skill entropy rises from low to high, for nearly all models.

• Cross-skill framing costs accuracy: Section 3.3 reports the per-model drop when the same skill is exercised inside a cross-skill task rather than a single-skill question ranges from −4% to −10% (the introduction summarizes this as −4% to −13%). The largest drops are observed on Planning skills, and drops persist even on skills the model nearly saturates in single-skill mode, such as Logic.

• Identified failure mode: In later steps, models tend to carry over the previous step's skill and answer modality instead of switching to the skill the current step requires. A representative case shows a base model reusing a math skill with a short numeric answer where "Theme Creation" with a long passage was required. Across the strongest frontier models, picking a wrong skill at a step roughly halves its accuracy.

• Skill entropy is decoupled from domain difficulty: In the skill-entropy landscape, Science has high domain accuracy (an easy domain) yet the highest skill entropies (hard skill-switching), and planning to information extraction is the hardest skill switch.

• Skill-Entropy RL improves both model sizes substantially: On Qwen3-4B-Instruct, the Skill²-Bench score rises from 34.4% to 68.4%; on Qwen3-1.7B, from 14.6% to 40.1%. This beats GRPO by +9.6% and +7.9%, and beats the strongest skill-aware baseline (STAT) by +7.0% and +7.1%.

• Gains span domains and transfer: Skill-Entropy RL attains the best per-domain accuracy on seven of nine Skill²-Bench domains at both model sizes, with the largest gain on Creative Writing — even though RL training is restricted to the six verifiable domains. It also attains the highest average score on five external benchmarks outside Skill²-Bench (MuSR, LongBench-MuSiQue, GPQA-Diamond, MMLU, IFEval).

• Works on off-the-shelf data: When applied to OpenR1-Math, vanilla GRPO plateaus while adding the skill-entropy reward keeps improving the reward without saturating; Skill-Entropy RL attains the best score on every one of six math benchmarks, improving over GRPO by +1.9% on average and over the base model by +7.7%.

Methodology in Plain English

The work proceeds in two halves: an evaluation half and a training half.

Building the benchmark. The authors first assemble a "skill bank" by prompting an LLM to label seed questions from existing datasets with 3–5 fine-grained skills, clustering those labels by embedding, and manually reviewing the clusters so skill granularity is comparable across domains. This yields 558 skills across 9 domains, sourced from datasets including OpenR1-Math, MMLU-Pro, LiveCodeBench, ZebraLogicBench, Guru-RL-92k, WikiTable, and WebSRC for the verifiable domains, plus LLM-generated skills and rubrics for the three open-ended domains (Creative Writing, Context Retrieval, Instruction Following).

Measuring switch difficulty. With Claude-opus-4.7 fixed as a reference model, they measure its accuracy on single-skill questions and on two-step cross-skill pairs. Skill entropy is the smoothed ratio of the average single-skill accuracy to the cross-skill accuracy, using Laplace smoothing α = 0.1; a value above 1 means the switch is hard, and around 1 means chaining adds little difficulty. Averaging the directed pairwise scores along a task's skill sequence gives the task-level skill entropy, and thresholding that scalar at two empirical boundaries produces low, medium, and high difficulty levels.

Generating tasks. Each cross-skill task samples a length between 2 and 10 and a skill sequence at a target entropy level. For verifiable skills, one seed question-answer pair is drawn per skill and an LLM "proposer" fuses them into a single coherent scenario in which each step depends on the previous one, preserving the underlying logic and ground-truth answers; for open-ended skills the LLM generates question-rubric pairs directly. A separate "verifier" filters out tasks that fail its checklist.

Evaluating. The held-out test set is 300 cross-skill tasks balanced across the three skill-entropy levels and nine domains. Models are queried in a single-skill mode (each step isolated) and a cross-skill mode (the full task, answered in order), sampled 4 times at temperature 0.7. Verifiable steps are graded by per-step correctness; open-ended steps are scored in [0, 1] by an LLM judge (Claude-opus-4.7) against a rubric.

Training. In the training setup, the model commits to a skill label before answering each step, using structured tags for reasoning trace, skill, and answer. Training uses GRPO with a reward summing an answer reward (mean per-step accuracy) and a skill-entropy reward, weighted λ_ans = 0.7 and λ_ent = 0.3. The skill-entropy reward is 1 minus the absolute difference between the predicted and gold skill-entropy ranks, where predicted skills are mapped to the skill bank by embedding similarity so semantically similar skills count. Training data is 9K cross-skill tasks synthesized from the six verifiable domains — 3K used for SFT warm-up with skill-annotated traces from a Qwen3-8B teacher and 6K for RL. SFT runs 4 epochs at learning rate 1e-5; RL uses group size 8, prompt batch size 256, learning rate 1e-6, KL coefficient 1e-3, and clip ratio 0.2 on 8 H100s.

Why This Matters

The paper argues that handling skill switches is an orthogonal capability to domain-specific competence — models can be strong on each skill in isolation and still degrade when the skills are chained. That reframes a class of failures that single-domain benchmarks cannot see, and it offers one object, skill entropy, that serves double duty as a benchmark scale and as a dense reward signal for training.

Real-world applications:

• Multi-domain trip planning and scheduling, the paper's own illustrative example, where a model must derive constraints, plan a schedule, and then extract supporting information in one chain.
• Research report writing and "deep research" workflows, which span literature lookup, quantitative analysis, and narrative composition.
• Agentic coding and multi-step tool use, where symbolic/coding skills interleave with planning and instruction-following steps.
• Data-analysis pipelines, such as going from table extraction to statistical modeling to written conclusions, each of which is a different skill.

Industry relevance: Reward design is a bottleneck for post-training reasoning models, and this work shows a skill-structure reward can be added to an existing RL pipeline without rebuilding the data pipeline — demonstrated directly on OpenR1-Math. It also gives practitioners a calibrated benchmark (300 held-out tasks, low/medium/high entropy levels) for measuring whether a model will hold up when tasks cross domain boundaries, plus a released codebase at the linked repository.

Future Directions

• A future-work section is not reported in the provided content, but the work raises clear open questions. One is whether the training-side gains extend to a broader set of base models beyond the two Qwen3 models reported in the main tables (the paper notes results on Llama-3.2-3B-Instruct and Olmo3-7B-Instruct follow the same trend in Section D.9).
• Dependence on a fixed reference model. Skill entropy is defined relative to Claude-opus-4.7, which the paper chooses so the scale stays constant across evaluated models; whether the ranking of skill switches holds under different reference models is a natural question.
• Training on open-ended domains. RL training is restricted to the six verifiable domains, with open-ended gains arising through transfer; whether directly training on open-ended skill switches yields further gains is unaddressed.
• Scaling and generalizing the off-the-shelf pipeline. The OpenR1-Math experiment uses a 6K labeled subset with trace segmentation and skill labeling by a Qwen3-8B annotator, leaving open how well this transfers to other off-the-shelf corpora and to agentic, tool-using settings.

Target Audience

Researchers and engineers working on LLM reasoning evaluation and post-training: benchmark designers who need measures for compositional and long-horizon tasks, RL practitioners designing reward signals beyond binary correctness, and teams building multi-step agents that must cross domain boundaries (research, planning, coding, data analysis). The paper is written for readers comfortable with reinforcement learning terminology and formal notation, though the conceptual framing of skill switching is accessible to a broader machine-learning audience.

Authors’ abstract

Long-horizon reasoning in recent LLMs demands that the model switch between distinct skills inside a reasoning chain, such as first doing a math derivation, then using the result to plan a schedule. We call such problems cross-skill long-horizon tasks: multi-step tasks whose steps require different reasoning skills and depend on earlier outputs. Existing benchmarks often evaluate individual skills, lacking a principled way to measure how well a model switches between skills. We address this gap from both the evaluation and training sides. We introduce Skill Entropy, a measure of the difficulty of switching from one skill to another. We then propose Skill^2-Bench, a benchmark of cross-skill long-horizon tasks built over 558 skills across 9 verifiable and open-ended domains. Each task is assigned a task-level skill-entropy score and grouped into three difficulty levels. Evaluating 8 frontier and 4 open-source models on Skill^2-Bench reveals a skill-switching gap: accuracy decreases on higher-entropy tasks. We then turn skill entropy from a benchmark scale into a training signal. We propose Skill-Entropy RL, an RL framework where the model predicts not only the answer at each step but also the skill used to produce it. The reward combines step-level correctness with a skill-entropy reward that measures the alignment between the model-predicted skill sequence and the gold skill sequence. On Qwen3-4B-Instruct and Qwen3-1.7B, Skill-Entropy RL improves the Skill^2-Bench score from 34.4% to 68.4% and from 14.6% to 40.1%, respectively, outperforming competitive baselines. The same pipeline can be applied to off-the-shelf training data such as OpenR1-Math, indicating that skill entropy is a reusable training signal. Code available at: https://github.com/Gen-Verse/Skill-Entropy-RL