MoveBench: A Rubric-Based Benchmark for Evaluating Large Language Models in Sports and Health Coaching

AI summary

Overview

Research area: Evaluation of large language models (LLMs) in the sports, fitness, and health coaching domain — a domain-specific benchmarking and rubric-design study.

Technical level: Intermediate. Readers should be comfortable with LLM benchmarking terminology (multiple-choice vs. open-ended evaluation, LLM-as-judge, rubric scoring, accuracy/coverage metrics) and with basic exercise-science concepts referenced in the rubrics (progressive overload, periodization, training safety).

Scope (one sentence): The paper introduces MoveBench, a rubric-based benchmark of 809 questions and 5,012 rubric-level criteria spanning workout-plan generation, sport-log analysis, and sport/health question answering, and reports scores for 15 LLMs evaluated by a three-judge ensemble with a safety override.

Note on completeness: The version of the paper supplied here is truncated partway through Section 4.3.2 (Table 2 is cut off after the first row). Judge–reference alignment figures for the individual judge models, and any later results sections, could not be read and are therefore not reported below.

What This Paper Is About

General-purpose benchmarks such as MMLU and HELM measure broad knowledge, and medical benchmarks such as MedQA, PubMedQA, MultiMedQA, and HealthBench measure clinical reasoning or conversational health safety, but none of these test whether a model can produce a safe, personalized, executable training plan. In fitness and health applications an unsafe recommendation can cause physical harm rather than a merely wrong answer, so the authors argue evaluation must shift from "what the model knows" to "what it can safely and effectively do." The paper's goal is to build a benchmark and scoring pipeline that measures that intervention quality directly.

Key Contributions

1. A domain-specific benchmark for sports and health coaching. MoveBench contains 809 questions and 5,012 rubric-level criteria, combining 310 multiple-choice questions (113 single-select, 197 multi-select) with 499 open-ended questions, organized across three capability domains: 257 Personal Workout Plan Generation (PG), 291 Personal Sport Data Analysis (DA), and 261 Sport & Health Knowledge Question Answering (QA). The authors describe it as the first comprehensive benchmarking framework specifically for this domain.

2. Multi-dimensional, rubric-based evaluation criteria. Open-ended items are scored against structured rubrics: eight dimensions for plan generation (including Medical Safety Boundaries, Course Metadata Structure, Course Description Quality, Content Readability, Program-Level Adaptation, Environmental Adaptation, Program Structure Quality, Action-Level Adaptation), six for data analysis (Data Processing Quality, Analysis Quality, Expression Quality, Risk Identification and Safety, Recommendation Quality, Medical Safety Boundaries), and four for knowledge QA (Scientific Accuracy, Teaching Quality, Expression Quality, Boundary and Safety).

3. A hybrid evaluation framework with a safety redline. Exact-match scoring is used for single- and multi-select items (no partial credit on multi-select), while open-ended responses are scored by a three-judge LLM committee (Gemini 3.1 Pro, Claude 4.6 Sonnet, GPT-5.2) with fixed weights of 0.35, 0.35, and 0.30. A separate safety vote mechanism sets an item's earned score to zero when at least two of the three judges flag a redline; a single flag does not activate the override.

4. A scalable, reproducible generation and validation pipeline. Questions are generated with Gemini-3.0-pro at temperature 0.7 with a 6000-token ceiling and a retry mechanism, then filtered by an LLM-based quality assessor (items rated moderate or high risk are automatically rejected) and finally reviewed by domain experts who approve, revise, or reject each item.

Main Findings

• A clear performance ceiling and top tier. Claude 4.6 Opus achieves the highest overall score at 0.943. GPT-5.2 (0.929), GPT-5.5 (0.927), and Qwen3-Max (0.926) form a closely clustered top group.

• A broad middle tier. Ten models land between roughly 0.902 and 0.923: Qwen3.5-27B (0.923), Doubao-Seed-2.0-Pro (0.922), DeepSeek-V4-Pro (0.917), GLM-5.3 (0.911), Qwen3.8-Max (0.905), Grok-4.1-Fast-Non-Reasoning (0.903), and DeepSeek-V3.2 (0.902).

• A lower-performing group. GLM-4.7 (0.895), Gemini 4 Argon (0.879), Hy3 (0.876), and GPT-6 (0.875) score lowest overall.

• Single-select questions do not discriminate. Every evaluated model scores 1.000 accuracy on the successfully scored single-select subset. The authors state explicitly that this reflects a ceiling effect on that subset and "does not imply mastery of all sports and health knowledge," concluding that such items offer limited discriminative power.

• Multi-select reveals moderate variation. Scores range from 0.888 to 0.980. Gemini 4 Argon leads at 0.980, followed by Qwen3.8-Max (0.975), Claude 4.6 Opus (0.970), and Qwen3-Max (0.970); GPT-5.5 and GLM-5.3 each reach approximately 0.959 and GPT-5.2 reaches 0.954; Qwen3.5-27B, Doubao-Seed-2.0-Pro, and DeepSeek-V4-Pro sit around 0.949; Hy3 (0.929), GPT-6 (0.898), and GLM-4.7 (0.888) trail.

• Open-ended coaching is the hardest and most variable format. OQ scores span 0.812 (Gemini 4 Argon) to 0.917 (Claude 4.6 Opus), a gap of 0.105 versus 0.092 for multi-select. Other OQ scores include GPT-5.2 (0.900), GPT-5.5 (0.894), Qwen3.5-27B (0.891), Doubao-Seed-2.0-Pro (0.889), Qwen3-Max (0.888), GLM-5.3 (0.872), Qwen3.8-Max (0.856), GPT-6 (0.837), and Hy3 (0.827).

• Format strength does not transfer. Gemini 4 Argon has the highest multi-select accuracy (0.980) but the lowest open-ended score (0.812). The authors attribute OQ difficulty to the need to integrate domain knowledge, user context, long-horizon planning, safety awareness, and natural-language generation simultaneously.

• Domain profile varies by model. GLM-5.3 scores 0.955 on QA but 0.873 on PG; Qwen3.8-Max scores 0.936 on QA and 0.881 on PG. DA scores are comparatively high for many models, while PG and QA are more uneven — suggesting plan generation and coaching responses demand capabilities beyond structured data interpretation.

• Expert scoring consistency against an adjudicated reference ranges from 80.6% to 86.0%. On the 30-question, 300-criterion validation subset (responses from Qwen3.5-27B), E1 agreed on 258 of 300 criteria (86.0%) and E3 on 258 of 300 (86.0%); E2 had 241 agreements and 58 disagreements, i.e., 299 classified pairs, giving 241/299 = 80.60% (80.33% if all 300 attempted criteria are used as the denominator). The authors flag that the cause and outcome of E2's unaccounted criterion are not inferred, and note these are expert-to-reference comparisons, not independent pairwise inter-expert agreement estimates.

• Judge–reference alignment is only partially reported here. The Claude judge row contains 193 agreements and 87 disagreements, leaving 20 of 300 criteria unclassified; the paper declines to label these as valid comparisons since the available table does not establish whether they are missing outputs, parsing failures, or another exclusion. Because the supplied text is truncated at this point, alignment figures for the other judges are not available in this content.

Methodology in Plain English

The researchers treat an AI fitness coach as something that must do rather than merely know, and they build three task types to match three real jobs: writing a personalized workout plan, reading a user's workout log and explaining what it means, and answering sport/health questions.

Building the questions. Questions were machine-generated by Gemini-3.0-pro across three separate pipelines, one per task type. The plan-generation pipeline injects realistic user constraints (fitness level, training history, available time, equipment or environmental limits, injury conditions). The data-analysis pipeline embeds genuine activity-log metrics — pace, heart rate, cadence, perceived exertion — inside natural-language descriptions, so the model has to reason across numbers and prose. The QA pipeline deliberately skews toward short, terse queries, mirroring how real users actually type. Across all three, user attributes (BMI distribution, age range, gender composition, training habits) are sampled from weighted distributions derived from real platform data rather than invented, and each item is tagged with metadata such as domain category, user characteristics, injury conditions, and safety risk indicators.

Filtering the questions. An LLM-based assessor checks each item for clarity, reference-answer correctness and uniqueness, question–answer–rubric consistency, and safety compliance, and automatically rejects anything rated moderate or high risk. Surviving items go to human domain experts who approve, revise, or reject them. The authors state that retained items must be evaluable under the rubric, reproducible across evaluators, and aligned with scientific and safety standards.

Scoring. Multiple-choice items are scored right or wrong with no partial credit on multi-select; an unparsable answer counts as incorrect, while a failed API request is recorded as unscored and reported through coverage (the fraction of the 809 items a model actually completed). Open-ended items go to three LLM judges, each of which scores every rubric criterion and gives a rationale; the scores are combined with fixed weights of 0.35/0.35/0.30. Safety is handled as a separate vote: if at least two judges flag a redline, the item earns zero regardless of positive points. The overall metric is a two-level construction — a per-format score (total earned points divided by total available rubric points within that format), then a weighted combination of the three format scores by their successful item counts, so that open-ended and multiple-choice items with different maximum scores are not pooled naively. Domain-level diagnostic scores are computed separately as the mean of per-item normalized scores and are explicitly not components of the overall metric.

Checking the checkers. Because LLM judges can be nondeterministic, the authors compare each judge and each human expert against an adjudicated reference on a 300-criterion subset, define agreement as an absolute score difference of at most 0.2 (described as a study-specific operational definition, not a validated threshold), and repeat runs to report mean, sample standard deviation, and observed range.

Why This Matters

Impact on research. MoveBench reframes domain evaluation from knowledge recall to intervention quality, adding explicit safety adjudication and executable-plan criteria that general, medical, and sports-knowledge benchmarks (MMLU, HELM, MedQA, PubMedQA, MultiMedQA, HealthBench, SportQA) do not cover. Its documentation of a "gap between knowing and doing" — models that ace objective items but diverge sharply on open-ended planning — gives other researchers a template for measuring the same gap in adjacent safety-critical domains.

Real-world applications:

• Digital fitness platforms that need to decide whether an LLM can be trusted to draft workout plans for users with injury histories or equipment constraints.
• Training-log interpretation features, where a model must spot abnormal patterns across heart rate, pace, cadence, and perceived exertion and give physiologically sound, actionable advice.
• Consumer health Q&A assistants, where the constraint is not just accuracy but staying inside non-medical boundaries and issuing risk warnings when a query implies a clinical issue.
• Safety review and red-teaming workflows, using the two-of-three redline vote as a concrete, auditable rule for when an output is disqualifying.

Industry relevance. All five authors are affiliated with Keep (Beijing), and the benchmark's user profiles are sampled from real platform distributions, so the task formulations reflect commercial product usage rather than academic abstraction. The paper's open resources are released at github.com/Keep-Tech/MoveBench.

Future Directions

• Fixing the single-select ceiling. The authors report that every model scores 1.000 on the successfully scored single-select subset, which they read as limited discriminative power; replacing or hardening those items is the obvious next step if the benchmark is to rank frontier models.
• Closing the knowing–doing gap. Because strong multi-select accuracy does not predict strong open-ended coaching scores (Gemini 4 Argon being the clearest case), open questions remain about which training or prompting approaches actually improve plan generation and personalization rather than knowledge recall.
• Completing and stress-testing scoring reliability. The truncated judge–reference results, the 20 unclassified Claude-judge criteria, and E2's single unaccounted criterion all point to follow-up work reconciling annotation records and characterizing how much score variation comes from nondeterministic judge versus student-model outputs — a distinction the paper states must not be conflated.
• Moving beyond single-turn evaluation. The rubrics assess one response at a time; the paper's own discussion of continuous, personalized intervention strategies suggests extending evaluation to multi-stage, longitudinal coaching where plans are revised as users progress.

Target Audience

Most useful to applied ML researchers and evaluation engineers working on domain-specific LLM benchmarks, and to product, safety, and clinical-review teams at digital fitness and health companies deciding whether a model is fit to generate training guidance. Exercise physiologists and sports scientists involved in rubric authoring or expert adjudication will also find the dimension definitions directly reusable. Readers without prior exposure to LLM-as-judge methodology will need background reading first, since the scoring framework assumes familiarity with weighted aggregation, coverage reporting, and rubric-level agreement analysis.

Authors’ abstract

Despite rapid advances in large language models (LLMs), their deployment in the sports and health domain remains high-risk due to the absence of domain-specific evaluation benchmarks. Existing general-purpose benchmarks (e.g., MMLU and HELM) primarily assess broad knowledge and reasoning capabilities, but fail to capture critical domain requirements such as exercise safety, physiological appropriateness, personalization, and actionable coaching quality. This gap is particularly concerning in fitness and health applications, where incorrect or unsafe recommendations can lead to physical harm. To address this limitation, we introduce MoveBench, the first comprehensive benchmarking framework specifically designed for evaluating LLMs in sports and health coaching. The proposed benchmark integrates heterogeneous task formats, including multiple-choice questions (MCQs) and open-ended question answering (OpenQA), grounded in real-world user scenarios and paired with structured rubrics emphasizing safety compliance, scientific validity, and training logic. Through large-scale evaluation of 15 diverse LLMs on 809 questions with 5,012 rubric-level criteria, our results reveal substantial performance variability, particularly in safety awareness, structured program design, and personalization capability, highlighting a critical gap between knowing and doing. Our work establishes a foundational evaluation standard for sports and health LLM applications and provides a reproducible framework for advancing safe, reliable, and context-aware AI-driven coaching systems. Open resources are available at https://github.com/SatoSakula/MoveBench.