The Pulse
GitHub Says New Git Storage Architecture Supported 5,000 Concurrent Agents in Internal Testing
GitHub says its new Git storage architecture handled 5,000 concurrent agents and 7,200 pushes per second to one repository in internal testing. GitHub says the design lets storing data, serving requests and maintaining repositories scale in
AI.info Team · Written with AI · fact-checked ·
GitHub says its new Git storage architecture supported 5,000 concurrent agents working on one repository, sending 7,200 pushes per second while reads stayed consistently low in latency during internal testing. The company published its account of the design on November 5, after announcing the architecture at GitHub Universe the previous week. GitHub calls the system the foundation for supporting agentic development, and says it built the architecture underneath GitHub while the service continued running.
GitHub says most of its requirements come from scale within a single repository
GitHub divides platform scale into two axes: horizontal scale across many repositories and vertical scale within a single repository. The company says many of its current approaches to horizontal scale have been battle-tested and carry forward, including partitioning tenants across infrastructure, isolation or fairness enforcement for tenant traffic, and load-aware routing. It says most of its requirements came from the vertical axis, and that it expects individual repositories to get larger and busier as agents write more code.
GitHub says its busiest repositories currently receive at most dozens of writes in a second, typically in short bursts. It expects parallel agents working around the clock to turn those bursts into sustained write rates within the next year. Each push also prompts reads from continuous integration, deployment systems and other agents fetching the new commit. In one repository, daily requests rose from a March average of 44 million to a May average of 181 million in 2026, about four times as many.
For a single repository, GitHub set a target of at least two orders of magnitude more scale. Its requirements include accepting thousands of writes per second, supporting multiple writer nodes, minimizing commit latency and keeping reads cheap when nothing has changed. The design also needs to record changes in order for each reference and let each component scale independently.
GitHub says Rubicon separates replicas from durable data
GitHub began work on the design, called Project Rubicon, in May 2026. It says requests from Git clients, the API and the web interface still pass through Spokes, which handles routing, load balancing and tenant placement. Under Spokes, GitHub says, virtual replicas provide disposable compute for reads, writes or both; each replica's local disk acts as a cache that can be rebuilt.
GitHub says the architecture puts Git objects, including packs and read-acceleration data, in Azure Blob Storage. A separate replicated log service, Ref-CAS, stores references and the repository's pack manifest. For a write, a virtual replica indexes an incoming pack and uploads it to Blob Storage, then commits a Ref-CAS transaction that lists the pack and updates references. For a read, the replica checks the reference version in Ref-CAS and fetches needed packs from Blob Storage into its cache.
GitHub says Ref-CAS handles reference updates, not Git objects
GitHub says Git objects are immutable and named by their contents, so writers can store objects without coordinating with one another. Reference updates are the part of a push that needs agreement. Ref-CAS commits each reference update through a quorum, locking only the references involved in that transaction.
Each Ref-CAS commit receives a log sequence number, and the repository version is the number attached to its latest commit. A virtual replica can check whether its reference state is current by sending its version; if nothing has changed, Ref-CAS returns no references. GitHub says the unchanged-repository check takes a single comparison, while reference-state transactions can run at tens of thousands of operations per second.
GitHub tested two alternatives using Blob Storage for reference updates. A single blob for the repository's reference log limited writes to the mid-hundreds per second in practice, even with batching. Splitting references across multiple blobs raised the write ceiling, but made each read check every segment and wait for the slowest one.
GitHub says background compaction handles pack growth
Every push adds a pack, and GitHub says lookups become slightly more expensive as the number of packs grows. GitHub says that in the new architecture compaction runs in the background on separate nodes operating directly on durable storage, and not on the replicas serving traffic. Those nodes take leases on repository shards so that only one node compacts a shard at a time.
A compaction node gathers packs into a segment, seals it when it reaches a limit based on pack count, size or age, and combines the packs. Before publishing the result, it checks that the new packs contain exactly the same objects as the inputs. GitHub says it has consistently demonstrated a flat latency curve as writes climb; garbage collection also uses dedicated workers to remove objects that are no longer referenced.
GitHub lists further planned work including fetching a single object without downloading its whole pack, partial replicas, and replicas in any region. Its stated plans for agent operations include working without a clone, an atomic commit API for multi-file changes and event streams for reacting to repository changes.