Govern the action, not the agent. Score each action by the consequence it carries, let the consequence decide who pulls the trigger, write that down as a mandate with a named owner, decide in advance how each failure is handled, and make every consequential action leave a receipt a person outside engineering can read.
- A recommendation is not a control. A confidence score, a guideline or a prompt instruction governs nothing. Only a mechanism in the system does.
- Consequence sets the band, confidence moves inside it. A high probability never promotes an action to more autonomy than its heat allows.
- Human in the loop is a posture per action, not a label per agent. The same agent can run unattended on one verb and need a named approver on the next.
- Toolkits enforce; they do not decide. A runtime that intercepts tool calls still needs someone to say which calls are hot and who answers for them.
01What is AI agent governance?
AI agent governance is the set of decisions, controls and evidence that let an organization let an AI agent act on its behalf and still answer for the outcome. The word that matters is act. A model that answers questions needs quality control. An agent that updates a CRM record, sends an email to a customer, approves a refund or merges code needs something else: a decided answer to who authorized that, under which limits, and what happens when it goes wrong.
As auxfirst uses the term, governance covers five questions for each agent in production:
- Ownership. Which named person answers for the business outcome, and which for the system.
- Authority. Which actions the agent may take on its own, which need approval, and which it may never take, decided by the consequence of each action.
- Supervision. Where a person sits relative to each action: in the loop, on the loop, or out of it, and what the person can see and undo.
- Exceptions. What the agent does when a run fails, with a named recipient and a time limit wherever the answer is to ask someone.
- Evidence and lifecycle. How anyone reconstructs what the agent did, how its behavior is versioned and evaluated, and how it is retired.
AI agent governance is not model safety, not an AI ethics policy and not a compliance document. Those bind to intentions. Governance binds to actions: it is only real where a sentence in it can be checked against one specific thing the agent did last week. It is also not a product category. Several toolkits now enforce policies at runtime; none of them can decide what the policy should say.
02Why most agent governance is a recommendation, not a control
When we scored twelve enterprise agent platforms in the Agent Supervision Register, median capability came out at 11 of 12 and median supervision at 6. Not one of them documented a way to reverse work an agent had performed. Every vendor could say what its agent was capable of; fewer than half could say what the person whose work it changed would see afterwards.
That gap is not an accident of the sample. It is what happens when a category is sold on autonomy and bought on trust, and the two are specified by different teams. The engineering team ships capability; the governance document, written elsewhere, says the agent “must be supervised”. Nothing connects the sentence to a mechanism.
The rule that catches most teams is short: a recommendation is not a control. A guideline in a policy, an instruction in a prompt, a confidence score next to an answer, a dashboard someone could look at: none of these governs anything. A control is a mechanism in the system that makes the ungoverned action impossible, or that puts a named person between the agent and the consequence. The framework below exists to turn recommendations into controls, one action at a time.
03The auxfirst AI agent governance framework
The framework has five layers, and one rule that runs through all of them: every layer binds to a specific action, never to the agent in general. It is assembled from auxfirst’s published frameworks, each of which has its own page, and it is an analytical model rather than a standard.
Read top to bottom, the layers are a build order. You cannot score an action for heat until someone owns the agent that takes it; you cannot choose a control posture until the heat is known; you cannot write exception behavior until you know who the approver is; and evidence is only meaningful against a mandate it can be checked against. The rest of this guide takes the layers in that order.
04Authority: consequence sets the band
Most teams argue about agent autonomy with adjectives. “Semi-autonomous.” “Mostly supervised.” “Human in the loop.” None of those binds to anything. The Action Heat Ladder replaces them with five questions you can answer about any action an agent can take, in any function, scored from 0 (cool) to 4 (hot):
| Dimension | The question | 0 · cool | 4 · hot |
|---|---|---|---|
| Reversibility | Can we take it back? | Fully undoable | Permanent, unrecoverable |
| Blast radius | How far does it spread? | A single item | The whole estate |
| Exposure | Who sees it? | Nobody outside the loop | Public, press, regulators |
| Commitment | What does it bind us to? | Binds nothing | Legally or financially binding |
| Authority | What is it allowed to touch? | Read-only | Production and permissions |
↔ scroll table
An action is as hot as its hottest dimension. No averaging. Four cool dials never buy back one hot one. Sending a routine invoice scores 0 on blast radius and 1 on reversibility, but 3 on exposure and 3 on commitment, so it is a high-heat action however routine it feels. A team scoring by average would have called it low-medium and let the agent send it; that is the rule doing its job.
The band then maps to exactly one answer to the only question that matters: who pulls the trigger?
| Band | Hottest dimension | Who pulls the trigger | Posture |
|---|---|---|---|
| Critical | 4 | A person executes; the agent prepares | Human in the loop, executing |
| High | 3 | A named approver, per action | Human in the loop, approving |
| Medium | 2 | The agent proposes, a person approves in batch or by rule | In the loop, lighter |
| Low-medium | 1 | The agent acts; a person reviews a sample on a cadence | Human on the loop |
| Low | 0 | The agent runs; the log is the control | Human out of the loop |
↔ scroll table
Two consequences follow. First, autonomy is unlocked by design moves that cool a dimension, not by trust: draft-only output drops exposure to zero, a sandbox drops authority to zero, an undo window cools reversibility, hard caps cool commitment and blast radius, a narrow allowlist cools blast radius. The de-escalators are the real product. Second, confidence never moves the band. A calibrated probability can change how loudly an agent works inside its band, how much evidence it shows, whether it proposes rather than acts. It cannot promote a high-heat action to unattended execution because the number read 0.97. Consequence is a property of the action; confidence is the model’s opinion of itself.
05Human in the loop, on the loop, out of the loop
“Human in the loop” is the most used and least defined phrase in agent governance. It describes one of three postures, and the posture is a property of the action’s band, not of the agent. The same agent can be out of the loop on a low-heat verb and in the loop, approving, on the next verb in the same run.
The grid has a budget attached. Every cell in the first column consumes a person’s time per action; every cell in the second consumes it per sample. The number of agent outputs a qualified person can genuinely check in a day is the system’s verification throughput, and it is the real capacity of any governed agent. Teams that put every action in the loop do not get more safety; they get a queue that nobody clears, which is the ungoverned state with better paperwork. The ladder exists so that human time goes to the hot end.
Explainability in agents is often asked for in general and delivered nowhere in particular. In the grid it has a location: at the decision point of the person in the loop, where the sources and the uncertainty behind the proposed action must be visible before the approval, and in the receipt afterwards, where the person affected can read what was done and why. An explanation buried in a trace nobody opens is a recommendation, not a control.
06The six supervision primitives
Independent of posture, six capabilities decide whether the accountable human can see, approve and undo what the agent did. They are the criteria auxfirst used to score commercial platforms in the Supervision Register, restated here as things to build, with the band from which each becomes mandatory.
| Primitive | The question it answers | Required from |
|---|---|---|
| Agent identity and disclosure | Can a human tell which agent acted, on whose authority, and that it was not a person? | Low · always |
| Action receipt | Is there a per-action record a non-engineer can read afterwards, available to the person affected? | Low-medium |
| Reversal | Can the action be undone, and how far back does that reach? | Low-medium |
| Consequence-scaled approval | Can approval requirements be set by how costly the action is, rather than on or off per agent? | Medium |
| Escalation handoff | Is there a documented route to a named human when the agent is out of its depth, carrying context across? | Medium |
| Provenance at the decision point | Are sources and uncertainty shown where the human decides, rather than buried in a log? | Medium |
↔ scroll table
The build order follows the band, and it is cumulative. An agent that is allowed to act and notify needs everything an approve-each-action agent needs, plus a budget governor and an audit trail; an agent allowed to run autonomously needs all of that plus a kill switch you have actually tested. A product moving an agent up the spectrum does not need new model capability. It needs new primitives.
07Ownership and the mandate
Governance binds to actions, but someone has to sign it. The AI agent operating model auxfirst uses, the Agent Owner’s Manual, is fifteen fields that make one agent an identifiable business system with a named owner, an explicit mandate, known dependencies, defined authority, human-control rules and a shutdown procedure. The thirteen questions it answers begin with the two most teams cannot: why does this agent exist, and who owns the business outcome.
Inside the manual, the mandate is the governance object. It is written per action, and a useful one reads like this:
- Agent
- Invoice handling agent, v2.3 · owner: head of finance operations · system owner: platform lead
- Action
- Send a routine invoice to an existing client
- Heat
- High · exposure 3, commitment 3, reversibility 1, blast radius 0, authority 1
- Posture
- Human in the loop, approving · one named approver per client group, approval expires after 24 hours
- De-escalator
- Draft-only below €2,000 after 90 days of clean samples · drops exposure to 0, moves the action to medium
- Limits
- Existing clients only · amount matches the signed order · no credit notes · no bulk sends
- Credentials
- Billing read, document write, no send permission until approval token is attached
- Exceptions
- Missing PO number → stop and route to the client owner · amount mismatch → refuse · low confidence on client match → ask the approver, 4-hour limit, then stop
- Evidence
- A receipt per invoice: parameters, approver, timestamp, document ID, delivery confirmation
- Disable
- Kill switch in the finance operations console · tested quarterly
Two things to notice. The credentials line matches the mandate: an agent whose credentials allow more than its mandate permits is one prompt injection away from an incident, and the gap has to be closed in the credential, not in the instruction. And the de-escalator line is where autonomy is earned: the mandate says not only what the agent may do today but what it will be allowed to do once the evidence justifies it, and which design move makes that safe.
08Exceptions: nine ways a run fails, seven permitted responses
The happy path is not governance. Operations depend on what happens when a run does not go right, and most agent descriptions are silent about it. The exception policy decides, for each way a run can fail, which of seven responses applies.
- A required identifier is missing
- Records conflict between systems
- A connected tool is unavailable
- Permissions are insufficient
- Confidence is low
- A customer disputes the result
- A policy conflict appears
- Possible fraud is detected
- Sensitive information appears unexpectedly
- Retry
- Stop
- Ask a human
- Route the case
- Use a fallback
- Log the event
- Refuse to continue
Two defaults are worth arguing about before you accept them. Retry is the most over-used response. Retrying a failure caused by insufficient permissions or conflicting records produces the same failure more expensively and hides the signal; a timeout should trigger a status check of the target system, not a blind retry. And “ask a human” without a named recipient and a time limit is not an exception handler. It is a queue that fills up. The policy has to say what happens when nobody answers, which is why that is one of the twenty-four readiness questions.
09Evidence: proving what the agent did
Every layer above produces a decision. Evidence is what lets anyone check, afterwards, that the decisions held. The unit is the action receipt: a per-action record of what was attempted, under whose authority, with which parameters, and what actually changed in the system of record. A fluent summary written by the agent is not a receipt. A confirmed change in the target system is.
Three tests tell you whether the chain is real. Can you reconstruct any single action the agent took last month, from the receipt alone, without asking the person who built it? Does the person whose work was changed get to see the receipt, not only the operator? And has an incident ever become an evaluation case, so that the thing that went wrong is now a test the next version has to pass? The lifecycle around those questions, from evaluation and release to retirement and revoking access, is the subject of the AI agent development lifecycle.
10Where the toolkits and the regulations fit
Governance has acquired a substrate. Runtimes now intercept every tool call an agent attempts, evaluate it against a policy and write an audit record; Microsoft’s open-source Agent Governance Toolkit does this in five languages with certified mappings to the OWASP Agentic AI Top 10, the NIST AI Risk Management Framework, the EU AI Act and SOC 2, and policy engines such as Open Policy Agent have done the evaluation half for years. This is good news. Enforcement in the system, not in the prompt, is the only kind that counts, and it is getting cheaper.
What a toolkit cannot do is decide. It evaluates the policy you give it; it cannot say which of the agent’s actions are hot, who the named approver is, what the person affected should see afterwards, or what happens when nobody answers an escalation. We described this in The Two-Layer Stack: the toolkit is the lower shelf, governance decisions are the upper one, and the lower shelf getting cheaper makes the upper one more valuable, not less. A YAML policy that says deny: delete_* is a control; deciding that deletion is critical heat for this agent, and that the credential should not carry it at all, is governance.
The regulatory layer reads the same way. The EU AI Act requires effective human oversight of high-risk systems, including the ability to understand, monitor, intervene in and stop them, with most of its obligations applying from August 2026. It does not tell you how. A framework built on consequence bands, named approvers, escalation routes and action receipts is how an organization demonstrates that oversight for a specific agent, instead of asserting it. Whether a given agent is high-risk is a legal classification this page does not make; whether its oversight is real is a question the twenty-four checks below will answer. The one obligation that reaches almost every agent regardless of risk class is disclosure: Article 50 has required since 2 August 2026 that people are told, at the first interaction, that they are dealing with an AI system.
11Governance readiness: twenty-four questions worth failing
Run these against one real agent, ideally your most consequential one, and answer for that agent rather than for the estate. Anything you cannot answer in under a minute is the finding.
- Can you name the accountable human, not a team?
- Is there a written autonomy map, readable outside the code?
- Is the hottest action it can take written down?
- Are limits enforced by mechanism, not by prompt?
- Does it act under its own identity, not a shared account?
- Does its behavior live in version control?
- Is the model version pinned?
- Is there a bill of materials for the running version?
- Is there an evaluation suite, and when did it last grow?
- Does it run behavioral regression before model upgrades?
- Do users of the agent get a behavior changelog?
- Can you roll back to the last known-good version today?
- Is there a kill switch you have actually tested?
- Which of its actions are irreversible?
- How many actions can it take between human checks?
- Who reviews its work, and how long does that take per month?
- Does escalation have a named recipient and a time limit?
- What happens if nobody answers an escalation?
- Can you reconstruct any single action it took last month?
- Does every consequential action leave a receipt?
- Would you detect a drop in its escalation rate?
- Do you know its cost per unit of work, including review?
- Has an incident ever become an evaluation case?
- Is there a retirement procedure, including revoking access?
Score one point per question answered confidently.
A pilot, whatever its status says. Useful, possibly in production, and not operable independently of the people who built it.
Held together by specific people rather than by design. The most common band, and the most fragile: it survives until someone leaves.
An agent you could defend in front of a client, an auditor or a board. The only definition of production that matters.
For one workflow end to end, this check is the core of an AI agent readiness assessment; for a portfolio of agents it is the first page of the owner’s manual for each.
12Frequently asked questions
What is AI agent governance?
AI agent governance is the set of decisions, controls and evidence that let an organization let an AI agent act on its behalf and still answer for the outcome. It answers five questions for each agent: who owns it, what it may do on its own, when a person is in the loop, what happens when it fails, and how anyone can prove afterwards what it did. It is about agents that act in systems of record, not about the model's general safety.
How is an AI agent governance framework different from an AI policy?
A policy binds to intentions: it says agents must be supervised, transparent and safe. A framework binds to actions: for each thing an agent can do, it states the consequence band, the control posture, the approver, the exception behavior and the evidence left behind. The test is whether a sentence in the document can be checked against one specific action the agent took last week.
What does human in the loop mean for AI agents?
Human in the loop means a named person approves a specific action before it executes. It is one of three control postures. Human on the loop means the agent acts and a person reviews afterwards, with the power to reverse. Human out of the loop means the agent runs and the log is the control. Which posture applies should be decided by the consequence of the action, not per agent and not by the model's confidence.
Who should own an AI agent?
A named person who owns the business outcome, with a second named person for the technical system if they differ. A team is not an owner. The owner signs the mandate, decides the exception policy, receives escalations and can disable the agent. This is the first of the fifteen fields in the Agent Owner's Manual and the first of the twenty-four readiness questions.
Does a governance toolkit such as Microsoft's Agent Governance Toolkit solve this?
It solves enforcement: intercepting tool calls, evaluating them against a policy and writing a tamper-evident audit record. It cannot decide what the policy should say, which actions are hot, who approves them, what the person affected should see afterwards or what happens when nobody answers an escalation. Those are governance decisions, and they sit above the toolkit. Both layers are needed; neither replaces the other.
How does AI agent governance relate to the EU AI Act?
The Act requires effective human oversight of high-risk systems, including the ability to understand, monitor, intervene in and stop them, with most obligations applying from August 2026. A governance framework built on consequence bands, named approvers, escalation routes and action receipts is how an organization demonstrates that oversight for a specific agent, rather than asserting it in a policy. Whether a given agent is high-risk is a legal classification this page does not make.
What is the difference between governance and observability?
Observability tells you what the agent did and how it is behaving over time: traces, metrics, drift. Governance decides what it was allowed to do and who answers for it. Observability without governance produces a detailed record of actions nobody authorized; governance without observability produces a mandate nobody can verify. The action receipt is where the two meet.
Where should a team start with AI agent governance?
With one agent, the most consequential one, and the twenty-four readiness questions. Then list its actions as verbs, score each on the five heat dimensions, and write the mandate action by action. Most teams find that the hottest action is not the one they expected, and that the agent's credentials allow more than its mandate does. Closing that gap is the first governance change worth making.
13Terms used on this page
- AI agent governance
- The set of decisions, controls and evidence that let an organization let an AI agent act on its behalf and still answer for the outcome: who owns it, what it may do on its own, when a person is in the loop, how it fails, and how anyone can prove what it did.
- Mandate
- The written statement, per action, of what an agent is authorized to do, under which conditions, with which limits, and who approves the rest. The mandate is the unit governance binds to; a policy binds to intentions.
- Heat
- The consequence an action carries if it goes wrong, scored on five dimensions (reversibility, blast radius, exposure, commitment, authority) from 0 to 4. An action is as hot as its hottest dimension. Heat decides who pulls the trigger.
- Human in the loop
- A control posture in which a named person approves a specific action before it executes. On the Action Heat Ladder it is required for high-heat actions; for critical actions the person executes and the agent only prepares.
- Human on the loop
- A control posture in which the agent acts and a person reviews afterwards, on a cadence or a sample, with the power to stop and reverse. Appropriate for low and low-medium heat, and only where reversal exists.
- Supervision primitive
- One of six capabilities that decide whether the accountable human can see, approve and undo what an agent did: identity and disclosure, action receipt, reversal, consequence-scaled approval, escalation handoff, and provenance at the decision point.
- Action receipt
- A per-action record a non-engineer can read afterwards: what was attempted, under whose authority, with which parameters, and what actually changed in the target system.
- Exception policy
- The decided response to each way an agent's run can fail: for each of nine exceptions, one of seven permitted responses, with a named recipient and a time limit wherever the response is to ask a human.
14Sources and further reading
Standards, toolkits and regulation cited on this page
- Microsoft — Agent Governance Toolkit: open-source runtime enforcement of agent policies, with certified mappings to OWASP, NIST, the EU AI Act and SOC 2
- EU AI Act, Article 14: human oversight of high-risk AI systems, including the ability to interrupt them
- OWASP GenAI Security Project — LLM06:2025 Excessive Agency: minimal functionality and permissions, human approval, authorization in downstream systems
- OWASP GenAI Security Project — LLM01:2025 Prompt Injection: why credentials, not instructions, have to carry the limits
- NIST — AI Risk Management Framework: govern, map, measure, manage
- Open Policy Agent: a general-purpose policy engine of the kind a governance runtime evaluates against
The governance stack on auxfirst
- the Action Heat Ladder — five dimensions, five bands, 73 actions mapped; the authority layer
- AI agent operating model — the Agent Owner's Manual: ownership, mandate, human control and shutdown
- The Agent Supervision Register — twelve platforms scored on the six supervision primitives
- AI agent development lifecycle — evaluation, release, operations, governance and retirement, with the readiness check
- agent-first architecture — where authority, approval and evidence live in the system
- The auxfirst skill — the whole procedure as an installable skill for Claude, with the mandate and owner's manual templates
- The Two-Layer Stack — governance decisions above runtime enforcement
- A probability is not a permission — why confidence moves inside the band and never the band
Method · This guide is auxfirst's working framework, assembled from its published and dated frameworks; the five-layer model, the posture grid in Figure 2 and the mandate card are auxfirst's and not an industry standard. The supervision findings are from the Agent Supervision Register, scored in June 2026. Toolkit and regulatory references are linked to their primary sources; the page makes no legal classification of any system. The invoice example is illustrative, not a client case. External sources checked on 26 October 2026.
Have an agent that works and nobody can defend?
Start with the twenty-four questions on one agent. If the score lands in the middle band, the AI agent operating model engagement writes the owner's manual and the mandate; if the question is whether a workflow should get an agent at all, start with an AI agent readiness assessment.