Skip to content
AI.info

The Pulse

AWS reports full fine-tuning results for SageMaker serverless

AWS reports a 90.0% semantic correctness score for full fine-tuning in a code-generation experiment as training data scaled to more than 100,000 examples. SageMaker AI has supported the technique for more than 25 open-source models since August.

AWS reports full fine-tuning results for SageMaker serverless

Full fine-tuning reached 90.0% semantic correctness on code-generation tasks as training data scaled to more than 100,000 examples, according to an Amazon Web Services experiment using SageMaker AI serverless model customization. The company published the result on November 18 alongside a comparison with a method that updates fewer model parameters.

The capability was first announced on August 3, rather than introduced in the new experiment. At the time, AWS said serverless full fine-tuning supported more than 25 open-source models, including models in the gpt-oss, Gemma, Llama, Nemotron and Qwen families.

Two ways to customize a model

Full fine-tuning updates all model parameters, while low-rank adaptation, or LoRA, freezes the pretrained weights and adds smaller matrices to model layers. Both methods can be used to adapt a pretrained model using an organization's data.

In serverless customization, users specify the base model, datasets and hyperparameters, and the service handles compute allocation and checkpoint management. The August announcement stated that customers pay only for what they use.

The coding experiment

The November post describes supervised fine-tuning experiments on the Qwen3-4B base model using both full fine-tuning and LoRA. Both training methods received the same prompts, targets and evaluation code within each experiment.

The experiments used Magicoder-Evol-Instruct-110K, a dataset of 110,000 coding instruction-response pairs covering algorithm design, debugging, data structures and systems programming. AWS split that dataset into 99,000 training records, 5,500 validation records and a held-out test set of 5,500 records.

The semantic-correctness evaluation used Amazon Nova Pro as an automated judge: it examined generated code alongside the original prompt and assessed whether the solution was logically correct. The reported 90.0% result is for that model-judged measure across code-generation tasks, not a general accuracy score across all uses.

Other checks and job setup

The team also evaluated generated code with PRIMECODE, a custom scorer that compares output with reference solutions using text-similarity measures including BLEU, ROUGE and F1. It used the MMLU benchmark, which covers 57 academic subjects, to check general-knowledge retention before and after domain-specific fine-tuning.

AWS says full fine-tuning extended further than LoRA on every dimension of its radar-chart comparison. Elsewhere in the post it says full fine-tuning “isn’t universally superior to LoRA” and reports LoRA slightly ahead on the model-judged metric in one result, 81.5% to 79.5%. The post describes tens of customization experiments across multiple supported base models, varying settings such as learning rate, batch size, warmup ratio, epoch count and dataset scale.

Each experiment, from submitting a job to obtaining evaluation results, took a few hours, according to the company. The resulting models were registered as versioned packages in SageMaker Model Registry for review or deployment.

Using the service

A sample SageMaker Python SDK job in the post selects full training, supplies training and validation datasets, sets hyperparameters and starts the job. Switching the sample to LoRA requires changing the training type, while the rest of the illustrated code stays the same.

As of the August announcement, serverless full fine-tuning was available in US East (N. Virginia), US West (Oregon), Asia Pacific (Tokyo) and Europe (Ireland). Customers can start a customization job through the JumpStart and Models page in SageMaker Studio or use the Python SDK.

Sources

  1. Amazon Web Services
  2. Amazon Web Services

Companies in this story

Related stories

Explore

More articles