The Pulse
AWS adds model leaderboard to SageMaker JumpStart
A new SageMaker Studio view compares at least 70 models on benchmark scores, speed and estimated instance cost. AWS says it runs the quality tests using a common evaluation setup, with qualifications for comparisons between reasoning and non-reasoning models.

SageMaker Studio users can now compare at least 70 models on quality, speed and estimated instance cost in a new leaderboard, with models from providers including Meta, Google, Qwen, IBM and Arcee. Amazon Web Services says it runs the quality tests itself using the same evaluation setup across the models.
The JumpStart Model Leaderboard is a new view on the SageMaker JumpStart base-models page, alongside the existing Categories and All models views. It opens to large language model rankings without requiring users to answer setup questions.
What the rankings measure
The view keeps the existing search bar and filters, including filters for license, provider and input and output modality. Users can sort benchmark columns, hide metrics they do not need and share a direct link to the leaderboard.
A blue heatmap marks relative results in each column, with a deeper shade indicating a stronger result. Model names stay visible when users scroll horizontally through the available metrics.
AWS runs three public reasoning benchmarks: BIG-Bench Hard, GPQA Diamond and MMLU-Pro. BIG-Bench Hard tests tasks requiring multi-step reasoning, including logical deduction and arithmetic; GPQA Diamond consists of graduate-level science questions; and MMLU-Pro tests knowledge and reasoning across subjects including science, law and engineering.
The leaderboard displays each benchmark separately rather than combining the results into one score. AWS cautions that scores from different benchmarks are not directly comparable because the tests have different levels of difficulty.
Speed, cost and deployment choices
AWS says it measures performance and cost by sending a standardized workload to an endpoint deployed with each JumpStart configuration. The performance columns include time to first token, throughput per request in tokens per second and estimated hourly instance cost.
The estimated cost covers only SageMaker instance cost, based on the on-demand hourly rate for the configuration's default instance type. It excludes data transfer, storage, inference component costs and discounts such as Reserved Instances or AWS Savings Plans; actual bills will vary.
Two menus determine which performance and cost figures appear. The use-case profile defaults to Max Context and changes those figures but not the quality scores. The optimization goal defaults to Balanced and selects a deployment configuration for the chosen profile.
The Balanced goal selects the configuration with the lowest estimated cost per token. Cost optimized selects the lowest hourly instance price, while Latency optimized selects the lowest time to first token. Throughput optimized selects the highest-throughput configuration, which can use speculative decoding.
Limits of the comparison
Reasoning and non-reasoning models are evaluated with different generation parameters, so comparisons between the two types are approximate. GPQA Diamond scores for reasoning models are averaged over 10 passes.
AWS recommends using the leaderboard to make a shortlist and evaluating the leading candidates on an application's own data and prompts before production. It also notes that each metric applies to the configuration on which it was measured; a different deployment configuration can change quality, latency, throughput and cost.
A user can open a model from the leaderboard to deploy it or, if it supports serverless customization, customize it. Access to SageMaker Studio and permission to deploy JumpStart models are required for the workflow.
Source
Companies in this story
Related stories
- AWS adds AI data preparation tools to SageMaker AI
- AWS details OpenFold3 research preview deployment on SageMaker
- AWS reports full fine-tuning results for SageMaker serverless
- MatrAIx opens proposals for long-horizon personal agent benchmark
- LiveGeo tests find AI agents struggle as geographic conditions change