The Pulse
Databricks says it raised default cluster-creation rate limits 7.5×
Databricks says its default cluster-creation request-rate limits are 7.5 times higher. The change pairs adaptive limits on active provisioning workflows with queues and a fixed API guardrail.

Default limits on cluster-creation requests are 7.5 times higher, Databricks says, after it changed how it manages provisioning work. The company also says it removed more than a hundred customer-specific rate-limit overrides in the reported scope.
The change applies to cluster creation for Databricks classic compute, the company’s Compute Platform team wrote in a November 18 engineering post. The team provisions virtual machines and coordinates cluster creation across AWS, Azure and Google Cloud.
From fixed rates to active work
Previously, Databricks used fixed request-rate limits to protect the system. It says conservative limits rejected customer requests even when capacity was available, while still accepting requests when services or cloud providers slowed down. During incidents, engineers manually lowered limits and raised them again after recovery.
A cluster-creation API handler validates a request, records a work item and returns. A background workflow then coordinates services and provisions resources, including cloud virtual machines; the API call may take milliseconds, while the workflow may take minutes.
One request can provision a different number of virtual machines from another, and cloud-provider health affects how long provisioning takes. When a dependency slows or fails, retries from many active workflows can add pressure to it, according to the post. Those internal retries come from accepted work and are not visible to the API request-rate limiter.
Provisioning slots
Under the new approach, an accepted request waits for a provisioning slot before its workflow starts. An active workflow holds that slot while it provisions virtual machines and waits for dependencies, then releases it when the workflow finishes.
The slot covers provisioning work after the API returns, rather than the lifetime of the resulting cluster. A launched cluster can continue running customer workloads without occupying a provisioning slot.
The controller uses signals such as provisioning duration and saturation in critical thread pools. When those signals indicate congestion, it reduces the number of concurrent provisioning workflows; as conditions recover, it cautiously allows more. A lower limit stops new operations from starting until active work falls below the limit, while existing operations still need time to finish.
Queues and the remaining guardrail
Databricks says bounded queues and fair scheduling help absorb bursts and share capacity. Customers can face longer launch waits when accepted requests queue during a slowdown.
The company says concurrency limits cannot repair an unavailable dependency, though they can limit added pressure while it recovers. A fixed API request-rate ceiling remains as a guardrail against exceptionally high incoming traffic.
Sources
- Original source · Blog post Databricks databricks.com
Companies in this story
Related stories
- Databricks puts Genie ZeroOps pipeline agent into public beta
- Databricks plans always-on Genie agent for enterprise work
- Databricks previews publishing metric views to Power BI
- AWS makes snapshot-based AgentCore V2 the default for new runtimes in select regions
- CoreWeave cites study showing up to 65% lower GB200 NVL72 cloud costs