The Pulse
Meta Puts Idle Inference Capacity to Work on Ad Ranking
Meta says it uses temporarily idle AI inference capacity to evaluate more ad candidates and route selected requests to more powerful models. The company describes safeguards for sudden capacity losses but publishes no numerical ad-performan
AI.info Team · Written with AI ·
Meta’s fleet can be constrained and idle at once
Meta says its AI inference fleet can be short of capacity overall even as pockets of computing power sit unused in particular regions, at certain times or on hardware that cannot serve the workload at hand. On October 12, the company described a system that pools some of that temporary slack for ad recommendations, where it can be reclaimed with little notice.
“Inference fleets face a capacity paradox: They can be globally constrained while meaningful pockets of capacity remain locally idle.” Jinfu Leng, Rohith Menon, Nicolas Stier et al., Meta
That tension shapes the system’s design: extra ranking work can improve which ads the system considers, but it cannot be allowed to crowd out the capacity serving ordinary traffic. Meta says its approach puts otherwise idle resources to work without ordering additional capacity, though its post does not give a measured performance lift or a figure for how much capacity the system uses.
Two methods use different kinds of spare capacity
Meta calls its first method Rank More Ads, or RMA. Its ad system narrows millions of candidates through a series of ranking stages, with increasingly powerful machine-learning models assessing smaller groups. When spare capacity is available, RMA sends more candidates through those stages, increasing the chance of finding ads relevant to a person.
That extra work needs both CPU capacity for upstream tasks such as retrieval and feature processing, and accelerator capacity for model inference. Meta’s second method, Serve Better Models, or SBM, keeps the number of candidates steady but routes a larger share of requests to a more powerful model. Because it does not substantially increase upstream CPU work, SBM can use spare accelerator capacity that RMA cannot.
A request-value signal helps SBM decide which requests to route to the stronger model. Meta says an explore-and-exploit process measures the additional outcome for different request segments, then adjusts the allocation; it also preserves some exploratory traffic so the system can keep testing whether its choices are still effective.
Baseline ad serving gets a separate safety margin
The two methods protect ordinary traffic differently. SBM uses separate routing targets: dedicated replicas serve the baseline model, while replicas on opportunistic capacity handle requests selected for the stronger model. If that spare capacity contracts, the system lowers the share sent to the stronger model and sends those requests through the baseline path.
RMA uses a logical limit instead. Meta calculates how much extra candidate-ranking work the replicas on opportunistic capacity can support, then sets a global budget for that work. When capacity drops, RMA lowers its candidate-volume target; in both systems, the company says optional work is removed rather than shifted onto capacity reserved for baseline serving.
Central feedback loops adjust the targets as conditions change. RMA’s controller uses observed tail latency to raise or lower the number of candidates sent downstream, while SBM’s controller sets coverage for the stronger model and client-side routers apply it. Meta says those central components operate on the order of minutes, leaving a gap when capacity disappears suddenly.
Local shedding handles sudden capacity losses
Meta’s faster fallback operates at both servers and clients. Servers can immediately remove lower-value opportunistic work if the available pool shrinks before clients receive updated targets; clients also monitor server status and shed extra work when they detect overload. The company says both kinds of shedding react within a few milliseconds.
Meta says it is working to expand the range of candidate volumes and stronger models the system can use, and to make models portable across more regions and accelerator types. It is also developing precomputation, which moves some work ahead of a live request, and an optimization-based system to allocate capacity among the available options. The October 12 post describes the mechanisms and intended benefits, but offers no numerical result for ad relevance, advertiser outcomes or capacity savings.