Autonomy Budgets: The Missing Control Surface for AI Agents

The most damaging AI agent failures are rarely dramatic. They are quiet. A permission change that seems harmless. A configuration tweak that spreads further than expected. A well meant automation that becomes the next incident.

That is because the unit of risk is changing. For the last few years, most AI risk debates focused on what a model says. Agents are AI systems that can take actions, not just generate text. When an agent can act, the risk becomes an unintended state change, meaning a change to a running system, data, or configuration.

This shift is arriving just as cybersecurity guidance is moving from principles to operational controls. On December 16, 2025, NIST released a preliminary draft Cybersecurity Framework Profile for Artificial Intelligence and opened it for public comment through January 30, 2026. If you care about what “good” looks like in real operations, this is the moment to weigh in. NIST IR 8596 initial preliminary draft PDF.

If we take this moment seriously, we need a missing control surface for AI agents: limited autonomy that is measurable, enforceable, and able to fall back to a safer mode when risk rises. That control surface is what I call an Autonomy Budget.

The Contrarian Claim

Agent safety is no longer exclusively a model problem. It is an operational controls problem.

Better prompts help. Better evaluations help. Better guardrails help. But none of them answer the question teams actually live with: which actions are allowed, in which environment, with what permissions, under what approvals, and what happens when the system becomes uncertain.

An Autonomy Budget makes autonomy a contract that can be monitored and enforced continuously.

What an Autonomy Budget Is

An Autonomy Budget is a set of explicit limits that define when an agent may do three kinds of things: write, execute, and commit. These are the three control points where you can attach checks, limits, and approvals. See Figure 1.

Here is the distinction that prevents confusion. Permissions in identity and access management and role based access control define what an agent can ever do, meaning who can do what. Budgets define what it is allowed to do right now, given current risk and context.

When the budget is exhausted, or when risk signals spike, the agent falls back to recommend only mode. Some teams call this human in the loop, meaning actions require approval. Others call it shadow mode when the system runs in parallel without effect. In practice, recommend only is the useful output in both cases: propose and explain, but do not change state.

Define the Currency and Reset Rules

If you call it a budget, you need a currency.

A simple model is action spend inside a time window, weighted by risk level and multiplied by scope of impact. The goal is not perfect math. The goal is a measurable limit that creates a hard stop, and a predictable fallback, before a runaway loop becomes damage.

Figure 2 shows what this looks like in practice: a cap, a spend curve, a hard stop at zero that blocks execute and commit, and reset triggers. It also includes the fail safe rule that matters most: if classification is uncertain, default to the more restrictive level.

Enforce Budgets Where Actions Happen

Enforcement must work across many agents and many tool calls. Do not track budgets only inside each agent process. Enforce them at a shared action gateway, meaning a single enforcement point between agents and sensitive tools. This is where policies and budgets stay consistent, and where every action can be recorded.

Figure 3 shows a practical architecture: the agent proposes actions, a gateway applies policy and budget checks, commits route through an approval queue, and an action record is written for investigation and learning.

A Scenario Every Team Will Recognize

Imagine an agent that helps during incident mitigation.

It detects rising error rates and increased latency. It proposes two actions:

1. Increase capacity for a single service and restart a small subset of instances
2. Rotate a set of credentials that might be compromised

Both actions can be reasonable. Only one should ever be autonomous.

With an Autonomy Budget, the first action can be treated as reversible and limited impact, allowed in production under constraints and with a clear rollback path.

The second action is higher impact and often hard to undo cleanly. It should require explicit approval. If approval is not available, the agent must remain in recommend only mode, providing the steps, the expected side effects, the rollback plan, and the checks a human should perform before proceeding.

Predefined autonomy limits reduce decision friction during incidents because teams stop debating what the agent is allowed to do in the middle of an outage.

What Triggers the Fallback to Recommend Only

You do not need an all knowing risk score. Start with signals that are measurable and obvious to practitioners:

1. Environment: Production versus test, freeze windows, elevated incident
2. Action category :Does it touch access control, credentialing, data deletion, or broad
3. Drift from a known good baseline: If the proposed state change deviates significantly from the last known safe configuration for that environment, treat it as elevated risk, increase the action cost, or force recommend only mode.
4. Uncertainty: Conflicting tool outputs, missing required context, or an inability to justify the action with evidence.
5. Abnormal behavior: Unusual volume of actions, repeated failures, or repeated near

To make uncertainty measurable without hand waving, use consistency checks. If two separate runs produce materially different action plans, or if tool outputs conflict, treat that as uncertainty and fall back.

Recommend only mode should reduce operator load, not add to it. The fallback output should be a pre staged plan that includes the proposed change in a reviewable form, the risk category and level, the rollback plan, and the evidence referenced by pointer.

Make the contract explicit in the agent’s own words: “I have reached my autonomy limit for this environment. I can propose a plan, but I cannot apply it. Here is the change, the risk category, the rollback, and the evidence I used.”

One more operational detail helps in the real world. If an agent repeatedly hits a limit, impose a cooldown period. Keep it in recommend only mode until a human reviews the policy, the context, or the tool permissions that are causing repeated boundary hits.

The Minimal Action Record

Budgets are meaningless without traceability. If an agent takes an action, you need to answer three questions quickly:

1. What did it do
2. Why did it do it
3. Under what permissions and constraints did it do it

A minimal but sufficient action record can include: an action summary, the inputs used as pointers, the decision context including currency spent, approvals, and the outcome including rollback steps.

Human approval must be meaningful, not ceremonial. If the system asks for dozens of approvals a day, approval fatigue will set in and people will rubber stamp. That is a design failure, not a people failure. Keep commit prompts rare, make the fallback output easy to review, and require the approver to confirm they reviewed the proposed change and acknowledged the risk category.

Why Now?

NIST has put a concrete draft on the table and is asking for public feedback through January 30, 2026. You can use Autonomy Budgets as a lens when commenting. Push for clearer expectations on action boundaries, budget currency and enforcement points that work across many agents, fallback requirements, minimum action record expectations, and approval design that avoids fatigue while preserving meaningful oversight.

Autonomy Budgets are a practical bridge between high level guidance and real engineering controls. If agents are going to act, we should decide in advance what they are allowed to do, how we measure those limits, and what happens when uncertainty rises.

That is what budgets are for.

About the Author

Kostakis Bouzoukas is a principal level technology leader and systems architect with 15 years of experience building and governing large scale software ecosystems, with a focus on reliability, security, and risk bounded automation. He holds an MSc in Information and Telecommunications Technologies and an MBA from Warwick Business School. He writes often about AI, work and governance and he’s a member of the IEEE Computer Society (including HCI-adjacent communities SSIT/TEMS/SMC). Views are his own.

Disclaimer: The authors are completely responsible for the content of this article. The opinions expressed are their own and do not represent IEEE’s position nor that of the Computer Society nor its Leadership.