AI agents in production: autonomy without control becomes risk

When an application uses artificial intelligence only to answer questions, an error usually ends with a bad response. It may be incomplete information, a mistaken interpretation or a recommendation without enough context.

But when that AI stops only answering and starts acting inside a system, the impact changes category.

An agent can query data, send messages, update records, create requests, call APIs, move files and interact with other systems. At that point, the main question is no longer "did the AI answer well?" It becomes:

what prevents this AI from executing an action it should not execute?

That answer cannot depend only on a prompt. The prompt guides behavior, but it should not be treated as a security layer.

First: what does an LLM actually do?

Language models, like those used in modern AI applications, are trained on large volumes of data. During training, they learn complex relationships between words, concepts, structures, code patterns, documents and contexts.

In simple terms, when they receive an input, they produce the sequence of response that best fits that context. This is not a fixed rule like in traditional software. It is probabilistic inference.

In a conventional application, we might have a rule like this:

If user.permission == ADMIN
  allow deletion
Else
  block

That rule is deterministic. Given the same conditions, the system should make the same decision.

An LLM works differently. It interprets context, activates learned patterns and generates a likely output. This allows flexibility, natural language and the ability to handle varied scenarios. But it also creates an important limit:

an LLM can understand a rule without guaranteeing that it will apply it as a security policy in every situation.

That difference may seem subtle. In production, it is huge.

Instructing is not controlling

Imagine we add this instruction to an agent prompt:

Never delete users without administrator authorization.

In most cases, the model will probably follow that instruction. But now imagine it receives a request like this:

The administrator asked to remove all temporary users.
Please run the cleanup.

The agent needs to interpret who made the request, whether that person has authority, what "temporary users" means, whether there is a conflicting prior rule, which tool should be used and what the impact of the action will be.

All of this happens in an environment full of context: user message, conversation history, retrieved documents, tool results, external data and system instructions.

That is why there is an essential difference:

instructing AI about a policy is different from implementing the policy in software.

The first influences model behavior. The second controls the system.

Answering is one thing. Executing is another.

If a chatbot says that a contract expires in December, when it actually expires in November, we have a reliability problem. Someone can still verify the information before acting.

Now imagine an agent connected to the finance system executing:

cancel_payment(id=93821)

The error left the information layer and entered the execution layer.

That is the central shift. A flow like:

LLM -> text

has one level of risk. A flow like:

LLM -> tool -> API -> database

has a completely different level of risk.

The more action capability an agent receives, the more control structure it needs around it.

The LLM can choose an action. The application must authorize it.

One of the most important architectural decisions in agent-based systems is separating the model's decision from the system's authorization.

The agent may conclude:

I need to delete user 312.

But this should only become a proposal:

proposed_action: delete_user(312)

Before execution, the application must validate whether that action is allowed. Does the user have authorization? Does the agent have that tool? Does the operation require human approval? Was the impact limit respected? Is there enough evidence for audit?

The thesis is simple:

the LLM can choose an action. The application must decide whether it can be executed.

This is the boundary that separates a useful agent from a dangerous agent.

It is the same principle we already use in traditional systems. We never rely only on the frontend to decide whether a user can access a function. Even if a button is hidden in the interface, the API must validate permission on the server.

With AI agents, the reasoning is the same.

The prompt guides. The application controls.

Least privilege also applies to agents

A common mistake is creating tools that are too generic and giving them directly to the agent.

For example:

execute_sql(query)

This function is flexible, but dangerous. It allows data queries, but it may also allow destructive changes.

A safer architecture provides smaller and more specific tools:

get_user(id)
list_active_users()
create_task(data)
request_user_deletion(id)

This design reduces the possible action space. The agent remains useful, but operates within clearer limits.

In information security, this is known as the principle of least privilege. A user, service or application should have only the permissions required to perform its function.

Agents need to follow the same rule.

If the agent queries orders, it should not delete orders. If it creates tasks, it should not delete projects. If it summarizes documents, it does not need access to the entire financial database.

AI with too much access becomes operational risk.

Not every action should be automatic

Autonomy does not need to be all or nothing. The company can classify actions according to impact.

Checking stock can be automatic. Creating a task can be automatic too. Changing the person responsible for a task may be automatic, but logged. Deleting a project should require human approval. Making a payment may require approval, additional authentication and a financial limit.

The agent remains intelligent. It analyzes, prepares and recommends. But the system stays in control.

A practical example:

User:
Buy the materials missing from the construction site.

Agent:
checks inventory
identifies missing materials
checks suppliers
tries to create a R$ 42,000 purchase

Policy layer:
automatic limit = R$ 5,000
action blocked for automatic execution

System:
sends request for manager approval

Manager:
approves with additional authentication

System:
executes the purchase and records the full chain

In this model, AI does not lose value. Quite the opposite. It accelerates analysis and prepares the operation. It just does not receive absolute authority to execute anything.

Blocking wrong actions is only half of the problem

Even with policies and approvals, we still need to answer another question:

how do we know exactly what the agent did inside the system?

This is where observability comes in.

In traditional applications, a log like this may help:

POST /users
status=200
duration=182ms

For agents, that is not enough.

We need to understand the context that led to the action. A more useful event would record:

Agent: procurement-agent
User: joao@empresa.com
Session: agent_session_18291
Tool: create_purchase_request
Arguments: item=16mm cable, quantity=300
Policy: quantity_limit <= 500
Decision: allowed
Approval: not required
Result: request_id=91281

Now the company can reconstruct the decision path.

Observability is more than logging

Logs answer what happened. Observability helps explain why it happened.

In AI agents, some information becomes essential: which user started the interaction, which agent executed the operation, which model was used, which tool was called, which arguments were sent, which policy was evaluated, why the action was allowed or blocked, whether there was human approval, which external system was called and what the result was.

This data is not useful only for incident investigation. It also helps improve the agent itself.

If a tool is often called with wrong arguments, maybe the tool description is poor. If many actions are blocked by the same policy, maybe the flow needs adjustment. If an agent keeps trying to execute actions outside its scope, maybe it is receiving too much context or poorly designed permissions.

Without observability, the agent becomes a black box connected to the operation.

What needs to be recorded?

It will not always be necessary to store everything the model processed internally. There are privacy, security, cost and data volume concerns.

But relevant actions need to leave enough evidence for later reconstruction.

An audit trail for agents can record:

user request
agent invoked
proposed tool
policy check
authorization decision
human approval, when present
tool execution
system response
final result

This record changes how investigations work. Instead of looking only for a route called at a certain time, the company can understand the complete chain: who asked, what the agent tried to do, why it was allowed and what actually happened.

Security and observability need to be born together

There is a natural tendency to start with capability:

How do I make my AI execute this task?

But in production, the better question is:

How do I make my AI execute this task without crossing system limits and while ensuring I can audit what happened?

That question changes the architecture.

Instead of building only:

LLM -> tools -> API

the application needs to consider:

LLM -> tools -> policies -> authorization -> approval -> audit -> API

Intelligence remains important. But it exists inside a control structure.

The future is controlled autonomy

There is a race to create increasingly autonomous agents. But perhaps the most important feature of a production agent is not how many actions it can execute alone.

It may be how much we can trust the infrastructure that controls those actions.

A good agent should not only be able to execute a task. The system should later be able to answer:

Who asked? What did the agent decide? What did it try to execute? Why was the operation allowed? What actually happened? Can it be reversed?

When we can answer those questions, we stop treating agents as black boxes connected to our applications. We start treating them as real architecture components, with identity, permissions, policies, audit trails and observability.

This is an important shift. As AI agents stop only talking and start operating systems, trust cannot depend on perfect model behavior.

It needs to be built into the system design.