AI agents are taking on real work. Who should sign off?

From assistant to operator

A conventional AI assistant waits for a prompt and returns an answer. An agent can be given a goal and access to tools such as a browser, code environment, database or business application. It can then decide which step to take next.

That shift matters because an answer can be wrong without immediately changing the outside world. An action can create a purchase, modify a record, send a message or change a system. The cost of an error can therefore be much higher than the cost of a bad suggestion.

The permission boundary matters more than the demo

A useful agent should not automatically receive every permission available to a human employee. Product teams can separate read access from write access, low-risk actions from high-risk actions, and reversible changes from irreversible ones.

A practical design is to let the agent handle routine, reversible work automatically while requiring approval for actions with financial, legal, security or customer impact. OpenAI describes similar principles in its own work on coding agents: clear technical boundaries, explicit approval for higher-risk actions and telemetry that allows teams to audit what happened.

Human approval is a product feature

Human-in-the-loop does not have to mean a person manually approving every click. The better question is where human judgment adds the most value.

For example, an agent might be allowed to prepare a refund, but a person approves the final payment. It might draft a production change, but an engineer approves deployment. It might collect information for a procurement request, but a manager approves the purchase.

The approval step should show the user what the system intends to do, why it intends to do it and what will happen if the action succeeds.

Auditability turns mistakes into learning

An agent should leave a useful trail: the goal it received, tools it called, important decisions, approvals, errors and final outcome. That record helps teams investigate incidents and improve prompts, policies, permissions and tests.

Monitoring is especially important when the environment changes. A workflow that worked last month can fail after a software update, a changed API, a new business rule or a new type of customer input.

A practical checklist for teams

Before giving an agent real permissions, ask: What can it read? What can it change? Which actions need approval? Can every action be traced? Can the system stop safely? Can a human take over? Is there a rollback path? What happens when a tool fails?

The strongest agent products will not be the ones that remove humans from every step. They will be the ones that automate predictable work while making uncertainty visible and consequential decisions deliberate.

From chatbot to delegated worker

The important shift is not that models can produce better text. It is that an agent can connect a model to tools and carry out a sequence of actions. Depending on the system, that might mean reading a document, querying a database, creating a ticket, drafting a response, changing a record or initiating a transaction. The product therefore has to manage both intelligence and authority.

That distinction matters because an incorrect answer and an incorrect action have different consequences. A mistaken summary can be corrected by a reader. A mistaken purchase, deletion, approval or customer communication may create a real operational cost. Agent design should therefore begin with a map of possible actions and consequences.

Permissions are a product feature

A useful way to design an agent is to give it the smallest set of permissions needed to complete its job. Read access can be separated from write access. Low-risk actions can be automated while higher-risk actions require confirmation. Spending limits, approved destinations and rate limits can reduce the blast radius of a mistake.

This is similar to good security engineering: trust should not automatically expand because a model is capable. An agent that can access everything may be impressive in a demonstration and difficult to govern in production.

Human oversight should be specific

“Human in the loop” is not enough as a product requirement. Teams need to define what the human actually reviews. Does a person approve every external message? Only payments? Only low-confidence cases? Can an operator stop an agent while it is running? Can an action be reversed?

Good oversight is therefore event-driven. The system should surface the cases where human judgment adds the most value instead of creating a human approval queue for every trivial action.

What a production checklist looks like

• Scope: document the exact tasks the agent is intended to perform.
• Authority: define tools, data and actions the agent can access.
• Guardrails: add spending, data, destination and action limits.
• Observability: record meaningful steps so an incident can be reconstructed.
• Evaluation: test normal cases, edge cases and adversarial cases before release.
• Recovery: provide cancellation, rollback or human escalation where feasible.

Questions product teams should ask

Should every AI agent require human approval?

No. The appropriate level of approval depends on risk, reversibility and the consequences of failure. A low-risk formatting task can usually run automatically. A financial transaction or irreversible production change deserves a different control model.

Who is accountable when an agent fails?

The answer should be established through product ownership, organizational controls and applicable law or contracts. An agent itself cannot be treated as a substitute for accountable people and organizations.

What should teams measure?

Accuracy is only one metric. Useful operational measures include successful task completion, intervention rate, rollback rate, policy violations, time to resolution and the severity of failures. These metrics reveal whether the system is genuinely useful rather than merely impressive in a demo.

The broader lesson is simple: autonomy is not a feature that can be switched on at the end of an AI project. It changes the product's permission model, monitoring needs and responsibility structure. Teams that design those controls early have a better chance of making agents useful without making them ungovernable.

Designing an agent workflow that can be trusted

A practical agent deployment starts with a workflow map, not a model demo. Teams should identify the exact task, the systems the agent can access, the decisions it may make, and the consequences of an incorrect action. A useful first version usually handles a narrow, reversible task before it is allowed to affect customers, money or production systems.

Permissions should follow the principle of least privilege. An agent that drafts a refund recommendation does not need permission to issue every refund. An agent that prepares a purchase order may be allowed to assemble the order but require a human to approve the supplier, amount and final submission. These boundaries make it easier to investigate failures and reduce the possible impact of an error.

What teams should measure before expanding access

Task completion rate is only one measure. A team should also track how often a person must intervene, how frequently the agent takes an incorrect or unauthorized action, how long review takes, and whether the workflow saves time after supervision is included. A system that completes tasks quickly but creates lengthy review work may not improve the overall process.

Evaluation should include normal cases and deliberately difficult cases: missing information, contradictory instructions, unavailable tools, unexpected outputs and attempts to persuade the agent to ignore its rules. Teams should test whether the agent stops safely when it cannot establish what to do. Clear logs should record the request, relevant tool calls, approvals and final outcome while respecting privacy and data-retention requirements.

The operating model matters as much as the model itself. Assign an accountable owner, define who handles exceptions, create a rollback path and review incidents. When the workflow changes, rerun the evaluation rather than assuming previous results still apply. This approach gives an organization evidence for expanding autonomy instead of relying on confidence from a successful demonstration.