The Pulse
Rafay Lets Token Factory Operators Cap Model Spending by Organization, Project and User
Rafay Systems says Token Router and Token Budgets let Token Factory operators route inference across in-cluster deployments and external providers. The company says administrators can set spending limits for organizations, projects and user
AI.info Team · Written with AI · fact-checked ·
Rafay Systems announced new Token Router and Token Budgets capabilities in Rafay Token Factory on Nov. 5. Rafay says the additions let operators direct inference requests to in-cluster models or external providers including OpenAI and Anthropic. According to Rafay, Token Router chooses where a request goes based on priority, provider health and cost, and Token Budgets let administrators set daily or monthly spending limits for an organization, project or individual user and enforce them before a request is admitted.
One endpoint, several inference destinations
Rafay says Token Router sits between applications and inference providers. Operators manage it through one API, issue consumers one set of credentials and view token spending in one place, while changing model sources without requiring application code changes. Token Factory can serve external models from OpenAI, Anthropic and any OpenAI-compatible endpoint alongside in-cluster deployments, using the same API surface, cost tracking and failover controls. Each backend can arrange providers into ordered priority groups: traffic goes to the highest-priority group first and moves to the next if the primary is unavailable.
Rafay lists self-hosted first with burst to cloud, frontier first with an in-cluster fallback, and multi-vendor resilience across providers as common routing patterns. Requests are spread evenly across healthy providers in a group, which can mix in-cluster deployments and external APIs. Providers that fail health checks leave the rotation automatically and return when they recover. Operators can set separate rates for input, output, cached input and cache-write tokens for each group, so each request is metered at the rate of the group that served it.
Rafay says budgets can steer traffic, not only stop it
Rafay says request-rate limits do not make AI spending predictable, because a call to a small model can cost a fraction of a cent, while a long reasoning request to a premium model costs far more. Administrators can set daily or monthly budgets at the organization, project and user levels, in the currency supported by the service provider. A user’s budget follows them across projects, and each request is checked against all applicable budgets before admission. When a budget is exhausted, operator policy can deny requests, continue serving them or send them to a lower-cost model.
New projects and users inherit default budgets, which administrators can raise for heavier workloads without changing applications or endpoints. Threshold alerts notify administrators before a limit is reached; developers can see their own spending and remaining budget in the Rafay Developer Hub. Budgets reset automatically each day or month. When a tenant, team or API key nears or reaches its budget, Token Router can move its traffic to a lower-cost group, such as a lower-priced or self-hosted model, instead of rejecting requests.
Rafay targets neoclouds and enterprises running internal token factories
Rafay says neoclouds delivering open-source models as a service can add frontier models to their catalogs through the same API and credentials their customers already use. They can offer customers a daily frontier-model allowance, then send the rest of the day’s traffic to open-source models hosted on their own GPUs. Tenant administrators can manage budgets for their own projects and users. Rafay says every token is billed at the rate of the model that served it.
For enterprises, a platform team can set a daily frontier quota for each team or application. After that quota is spent, Token Router can move requests to an open-source model on the company’s GPUs for the rest of the day, while applications continue calling the same endpoint. The quota resets the next day, and a token usage dashboard with per-user filtering and CSV export supports internal showback and chargeback.
Guardrails, cache rates and key rotation also added
Rafay says guardrail services that implement the LiteLLM Generic Guardrail API can inspect prompts and responses on routed traffic, with the same policies applied to in-cluster and external models. Operators can price tokens served from the prefix cache separately, passing prompt-caching savings on to agentic AI and retrieval workloads. Operations teams can also enforce API key expiration, automatic rotation with an overlap window, and alerts before keys expire.
“Token Router and Token Budgets turn that tradeoff into policy. Operators decide which model serves each request and what it costs, and applications never have to change. Neoclouds get a way to sell frontier access while keeping their own capacity busy, and enterprises get AI spending they can predict.”
Mohan Atreya, chief product officer at Rafay Systems
Token Router, Token Budgets and the other new capabilities are available to Rafay Token Factory customers as of Nov. 5, according to Rafay.