The Pulse
Deepgram Reports Up to 70% More Audio per Dollar on Trainium2
Deepgram and AWS say a tuned version of Deepgram’s Nova-3 speech-to-text model processed 11% to 70% more audio per dollar on Trainium2. The result applies to offline batch transcription on one EC2 instance, not to every Deepgram workload.
AI.info Team · Written with AI · fact-checked ·
A 70% ceiling, with a narrow scope
AWS and Deepgram report that a tuned version of Deepgram’s Nova-3 speech-to-text model processed 11% to 70% more audio per dollar on Trainium2 than the compute options Deepgram had been using. The upper figure is not a general result for all speech recognition: the comparison covers offline, throughput-oriented batch workloads, including contact-center analytics and medical transcription. AWS says it calculated the metric using instance prices at the time of evaluation.
That scope matters. The post describes a cost-efficiency result for batch inference, not a claim that Trainium2 improves latency or cost across Deepgram’s real-time services, text-to-speech models or voice agents. It also does not identify the comparison instances or provide absolute audio-throughput figures, leaving readers without enough detail to reproduce the full price-performance comparison.
Eight Nova-3 replicas on one instance
The final system runs eight copies of Nova-3 on a single Amazon EC2 trn2.3xlarge instance. Deepgram and AWS say they spent several months porting the model from standard PyTorch code to Trainium2 and tuning it for the chip. The goal was to make more efficient use of accelerator time on workloads that process large batches of recordings.
The team used TorchNeuron, AWS’s PyTorch integration for its Neuron compiler, and compiled the encoder and decoder as separate static-shape graphs. AWS says the compiled model ran every operation on Trainium2 without detected CPU fallbacks. The team also checked layer-level outputs against CPU results and monitored word error rate against Deepgram’s reference values.
Kernel tuning and a Rust serving path
After establishing a working baseline, the engineers replaced selected operations with fused kernels from the Neuron Kernel Interface Library, then used profiling to identify further bottlenecks. They wrote custom kernels for feed-forward layers, depthwise one-dimensional convolution and two-dimensional convolution. AWS reports that the custom feed-forward kernel cut latency in half versus its reference kernel at batch size 96, while the depthwise-convolution kernel was 1.5 times faster at that batch size.
A separate engineering task was fitting the model into Deepgram’s production serving stack, which is written in Rust. The team built a runtime in which Rust coordinates model replicas through a shared work-stealing queue, while a C layer loads compiled Neuron executable files through AWS’s Neuron Runtime API. AWS says the runtime reproduced the reference token sequences at every tested batch size.
Against a multiprocessing Python baseline, the Rust runtime produced a 7.5% throughput gain at the batch size that maximized throughput. The post says the advantage is larger at low batch sizes, where repeated decoder calls make host-side dispatch more significant. That runtime result is distinct from the headline cost-per-dollar range: it measures throughput against a Python serving baseline, not the 11% to 70% comparison against Deepgram’s prior compute options.
Self-hosting is the next stated use
AWS says the efficiency gains could also benefit customers running Deepgram models in their own virtual private clouds through Amazon SageMaker AI. Deepgram and AWS say they are continuing work on setup, reliability and observability for that deployment path, but the post does not publish a separate benchmark for customer-hosted systems.
For now, the concrete result is a tuned Nova-3 batch-inference configuration on one trn2.3xlarge instance, with eight model replicas and a reported 11% to 70% increase in audio processed per dollar against Deepgram’s previous compute options. The measured range depends on the batch workload and the instance pricing used during the evaluation.