The Pulse
OpenAI Reworks Habitat Storage for Its Next Billion Users
OpenAI says its Habitat storage platform serves more than 1 billion users across 40 regions and holds over 500 petabytes of data. The company describes changes to its Azure Cosmos DB setup, caching and Rust service that cut global median re
AI.info Team · Written with AI · fact-checked ·
“Together, these changes have reduced our overall global read p50 from roughly 120 ms to roughly 10 ms, a 12x improvement since we began focusing on performance earlier this year.”
Jon Lee, Hrachya Hakobyan, and Chaomin Yu, Members of Technical Staff, OpenAI
OpenAI’s storage platform, Habitat, now serves more than 1 billion users across 40 geographic regions and manages over 500 petabytes of data, the company said in a post published October 23, 2026. The engineering account focuses on the database layer beneath Habitat and the operational changes OpenAI made as it scaled. Its central lesson: neither a single shared database container nor a separate one for every workload worked well on its own.
Habitat supports products including ChatGPT, the API and Codex. Beyond database access, it handles routing, data residency, encryption, authorization, request limits and caching. OpenAI says its current approach uses shared Azure Cosmos DB containers by default, alongside tools for isolating workloads and moving data where needed.
Why OpenAI moved back to shared containers
OpenAI began with a shared Azure Cosmos DB container, then added dedicated containers for critical or fast-growing services. Separate containers reduced the risk that one workload would disrupt others, but shifted capacity planning onto individual product teams. Teams had to anticipate traffic before launches; OpenAI says some features jumped from 1,000 requests per second during load tests to 1 million at launch.
That separation also removed a useful feature of shared systems: aggregate growth across many workloads can provide headroom for a new service that is small in testing but large in production. With dedicated containers, teams had to provision capacity for each workload independently. OpenAI says the overhead and uneven planning eventually pushed the company back toward shared tenancy.
The shared setup brought its own problems. A heavily used partition could consume capacity and affect other workloads, while storage growth could split physical partitions and reduce the throughput available to each one. OpenAI says it worked with Microsoft’s Azure team on changes to address those limits rather than continuing to add containers and manage their capacity separately.
Azure changes target hot keys and capacity planning
The updated arrangement combines shared containers with controls for workload isolation. OpenAI describes throttling the hottest database keys first to limit their effect on other traffic, and assigning lower priority to offline work such as bulk inserts and data extracts so interactive requests take precedence.
A revised serverless mode, built by the Azure Cosmos DB team for OpenAI, lets each physical partition scale to 10,000 request units per second and can split partitions based on request-unit usage as well as storage. OpenAI says an earlier serverless option was limited to 5,000 request units per second per partition and scaled based on storage alone. The company says it has since used one shared container by default and had no provisioning or noisy-neighbor incidents in the months before the post.
Caching and replication cut global read latency
OpenAI says it replicates only selected, latency-sensitive data broadly. Authentication metadata, which services need for each request, may be replicated across all 40 Azure regions where OpenAI serves traffic; other accounts replicate to at least one region per continent. The company says capacity constraints prevent it from using that level of replication for every dataset.
For other frequently read data, Habitat uses a distributed cache built on Valkey servers near the infrastructure handling user requests. To keep cached data current, a service consumes Azure Cosmos DB’s change feed and propagates invalidations globally within seconds. When an object changes, the affected cache entry is cleared and fetched again on its next read.
OpenAI attributes the reported drop in global median read latency—from about 120 milliseconds to about 10—to several changes together: replicating selected data nearer users, adding in-region caching and rewriting the Habitat service in Rust. The post does not assign the full improvement to any one of those steps.
The next system is still in development
OpenAI says it is building an in-house, exabyte-scale online storage system for bulky Habitat data and future conversation storage. The design combines Rockset’s RocksDB Cloud core with Raft-based replication and keeps the document model used by Azure Cosmos DB. OpenAI says disaggregated storage and physical replication should reduce flash capacity needs, with lower peak queries per second as the tradeoff.
That system is still under development. For now, the published account describes a platform built around shared database containers, selective replication, cache invalidation and service-level changes—and reports that the combined work brought global median reads to roughly 10 milliseconds.