# Exa says AWS chip optimizations lifted embedding throughput about 50%

Published 2026-11-19 · AI.info Team · AI.info

> Exa reports roughly 1.5 times its initial embedding throughput after optimizing workloads on AWS Inferentia2 chips. The work came as the search company re-embedded 30 billion high-priority documents for a new model.

Exa says it increased the throughput of its document-embedding workload to roughly 1.5 times its initial level by optimizing it for AWS Inferentia2 chips. In a November 18 post on the AWS machine learning blog, the search company also reported a 47% improvement in performance and cost per 1 million documents embedded against its initial baseline.

Source [aws.amazon.com](https://aws.amazon.com/blogs/machine-learning/exa-boosts-embedding-throughput-by-50-to-index-billions-of-documents-on-aws-ai-chips/)

The work came as Exa selected 30 billion high-priority documents for re-embedding with a new document model. Running the live and new embedding pipelines in parallel doubled the inference load during that migration.

## Why embedding capacity matters

Exa builds an index using web data as well as private and synthetic sources. The company uses machine learning models to place web documents and queries in the same vector space, supporting natural-language queries alongside keyword search.

Documents that pass the company’s quality threshold go through an asynchronous pipeline before they can be served from its in-house vector database. The pipeline crawls, parses, moderates and filters data before embedding it.

The reported Inferentia2 figures refer to an inf2.xlarge instance with one chip and two NeuronCores. The workload runs on a single NeuronCore, with Kubernetes pods packed onto larger instances; the same process also runs on a Trainium2 NeuronCore and scales through data parallelism.

## Changes to the workload

The starting point was a data-parallel FP16 version of the model. Exa then changed the layout of model weights to avoid unnecessary transposes and enabled autocast, allowing the compiler to use lower precision for matrix multiplication while preserving dynamic range.

The lower-precision outputs met the company’s correctness threshold of 0.999 cosine similarity against the reference, according to the post. A pre-built attention kernel from the Neuron Kernel Interface library provided another performance gain.

Profiling showed that data was repeatedly spilling to and reloading from high-bandwidth memory, leaving the Tensor Engine underused during the forward pass. The team wrote a custom fused SwiGLU MLP kernel, which it says eliminated those memory stalls.

Exa also rewrote the attention kernel around the shapes in its model to reduce computation spent on padding. It combined attention operations into single matrix multiplications where the generic kernel used multiple instruction steps, reducing inner-loop operations and memory round trips.

## Agents and compute scheduling

Agents developed the optimizations using a system for checking correctness, measuring performance and profiling the workload. Each agent runs a workflow using Flyte that compiles a Neuron executable and checks a forward pass against a baseline dataset, with a required 0.999 cosine similarity to the canonical model.

The workflow also profiles a forward pass and runs a 30-minute saturation benchmark on hardware. Lighter models search compiler parameters, reasoning agents test hypotheses and find bottlenecks, and code-generation agents write custom kernels.

The company has optimized Inferentia2 in the US West (Oregon) Region, while carrying the work over to Trainium2. Exa schedules computing across multiple Regions from one Kubernetes control plane, comparing performance per dollar across instance types and placing pods on the cheapest nodes as embedding demand changes.

## Sources

1. Original source · Blog post [Amazon Web Services](https://aws.amazon.com/blogs/machine-learning/exa-boosts-embedding-throughput-by-50-to-index-billions-of-documents-on-aws-ai-chips/) aws.amazon.com

## Companies in this story

- [Amazon](https://ai.info/companies/amazon)
- [Amazon Web Services](https://ai.info/companies/amazon-web-services)

## Related stories

- [Daisytuner raises €6 million to expand cross-chip AI software](https://ai.info/news/daisytuner-six-million-seed-cross-chip-ai-inference)
- [U.S. sets chip licensing rules targeting remote AI access](https://ai.info/news/us-chip-licensing-remote-ai-access)
- [AMD closes Taalas deal to add inference silicon to its accelerator roadmap](https://ai.info/news/amd-completes-taalas-ai-inference-acquisition)
- [Google reports Ironwood TPU results in MLPerf training tests](https://ai.info/news/google-ironwood-tpu-mlperf-training-results)
- [AntiFox-linked infrastructure tied to AWS secret access, researchers say](https://ai.info/news/antifox-linked-infrastructure-aws-secret-access)

## Explore

[More articles](https://ai.info/news)

---

Canonical: https://ai.info/news/exa-embedding-throughput-aws-ai-chips
