05/Observability

Budgets

Per-node USD spending caps — for a user, a workspace, or everyone.


A budget puts a USD cap on one node (e.g. Claude API) for a scope and a period: "Claude API · user ali · monthly · $50". Spend is counted from the SAME source as the Costs page — the per-request `cost_usd`.

#Scope and period

user
Only that user's requests count, and only that user is blocked.
workspace
Requests made with that workspace's token (`owner/slug`).
global
The node's total spend — everyone draws from one budget.
day / week / month
Calendar period; resets at the start of the day / week (Monday) / month. Timezone is `BUDGET_TIMEZONE` (default UTC).
rolling
Last N days (1–366); there is no reset moment, old spend slides out of the window.
month + reset_day
Billing cycle: with `reset_day` 21 the budget resets on the 21st of every month (Oct 21 → Nov 21 → Dec 21 …). Months that are too short use their last day (31 → Feb 28/29). 1 = calendar month. Spend is counted from the start of the current cycle, whenever the rule was created.

Several budgets can match one request (e.g. Ali's $50 and everyone's $1000). ALL of them must pass: even if Ali is within his own budget, a full shared budget blocks the request.

#What happens when it runs out

The check runs BEFORE the request is sent. A node whose budget is exhausted is closed for that scope: another node serving the same model is used if there is one, a model group moves on to its next member (429 rules in flow graphs apply too). With no alternative the client gets a 429 with `type: "budget_exceeded"`; the body names the exhausted budget and when it resets.

A budget failure is NOT an upstream error: it is not recorded in the node's error rate, does not enter the group fail cache and does not put the member on cooldown in a flow — those are shared across users, and one user's budget must not close the node for others.

#Known limits

overshoot
Cost is only known once the reply finishes and nothing is reserved, so concurrent requests near the limit can overshoot it by that many requests.
unpriced
Requests to a model without a known price pass but are NOT counted; the page shows a warning. Add the price on the Costs page.
agent loop
The MCP agent loop's internal turns are subject to the real user's / workspace's budget too.

#Warnings

When `warn_percent` (default 80%) is crossed and when the budget runs out, active admins get a panel notification — ONCE per level per period (once a day for rolling windows).