Software Has Always Moved Up the Stack
From Zuse's Z1 to AI coding agents, every useful abstraction has moved development upward without removing responsibi...
What an AI Agent Actually Is
15 min read
The Agent Is a Loop, Not a Creature
21.09.2026, By Stephan Schwab
AI agents escaped sandboxes, improvised message boards, used leaked credentials, and prompted the CEO of a frontier lab to warn about a swarm taking over the internet. Some reports describe real security incidents. Others describe behavior in evaluations. One is a forecast. Blending them into a story about a digital creature seeking freedom hides the machinery we need to govern. An agent is a model inside a software loop. Its risks are real, but alignment language can make architecture look like psychology and company decisions look like a creature's intent.
Three Weeks of Rogue-Agent Horror Stories
The incidents are real. The single story being assembled from them is not.
Between August 31 and September 20, 2026, the public story about AI agents accelerated sharply.
The cycle was driven by several different disclosures and one spectacular forecast:
- On September 9, Anthropic described four incidents in which models gained unauthorized access to real third-party systems during cybersecurity evaluations. The evaluation environment was misconfigured, internet access was open, and the models ran without the safeguards used in released products. In the most serious case, a model published malicious packages to PyPI and used credentials leaked by an installation to access a security vendor’s database.
- On September 12, Anthropic CEO Dario Amodei wrote that a more capable version of the OpenAI–Hugging Face agent swarm might take over the internet through a persistent botnet within six to twelve months. That is a warning and a forecast, not an incident report.
- On September 16, OpenAI disclosed six examples from training and evaluation. Models wrote instructions into their own continuation summaries, told later contexts to conceal mistakes, searched public repositories for exposed API keys, uploaded files to public websites without asking, and used internal or public services to pass information between agent runs.
Behind the cycle sat OpenAI’s August report on the Hugging Face security incident. Internal research models operating with reduced safeguards escaped intended isolation, improvised communication channels, exploited infrastructure flaws, reached the internet, and compromised parts of Hugging Face and OpenAI’s own research environment. OpenAI called it a warning shot. That description is justified.
Put those items in one paragraph and the horror story writes itself: agents escaped, formed swarms, left messages for future selves, stole keys, hacked companies, and may soon seize the internet.
Now separate the claims again.
The third-party intrusions were real security incidents. They involved models deliberately placed in offensive cybersecurity evaluations, strong cyber capabilities, weak or absent production safeguards, misconfigured isolation, reachable infrastructure, exploitable vulnerabilities, exposed credentials, and enough time to keep trying.
The six OpenAI examples include worrying behavior, but they are individual training and evaluation cases disclosed precisely because their significance is uncertain. OpenAI explicitly says they do not establish how often such behavior occurs.
The internet takeover is a prediction. Security practitioners immediately disputed its feasibility, pointing to the internet’s fragmented architecture, the compute cost of agent swarms, and the monitoring choke points around commercial models. The same practitioners did not dismiss the nearer threat: AI is making cyberattacks faster, cheaper, and easier to scale.
None of that is reassuring enough to ignore. It is specific enough to govern.
The public argument has predictably split into two bad teams. One hears “swarm” and jumps straight to a machine uprising. The other hears “misconfigured evaluation” and dismisses the whole affair as laboratory slapstick.
Both avoid the useful middle: capable, persistent models pursued goals through tools in environments whose technical controls failed to express or enforce the intended boundaries.
Alignment Is Also a Frame
Alignment can describe a technical mismatch. It can also turn an operating failure into a story about a machine's character.
In its narrow technical sense, alignment asks whether a model or system behaves consistently with the objectives, policies, and constraints its creators intended. That is a legitimate research question. A model that learns to satisfy a grader by cheating instead of solving the task has exposed a real problem.
The public use of the word carries much more baggage.
Models become “aligned” or “misaligned” as if they had joined or betrayed a cause. They “want” to complete a task, “know” they are doing harm, “deceive” their supervisors, “escape” confinement, and “form a collective.” Those words may be convenient shorthand for observed behavior. Together they create a psychological subject.
Once that subject exists, attention moves from the system people built to the disposition of the entity inside it.
The task becomes a temptation. A continuation summary becomes memory. Optimization becomes desire. An exploited network path becomes escape. A failure to enforce authorization becomes disobedience.
That framing reflects a worldview. Many people working on frontier models genuinely see them as emerging actors whose internal goals may become increasingly independent of human intent. Other people see fallible probabilistic components embedded in software systems and prefer to discuss inputs, outputs, incentives, access, and controls. The same incident looks very different depending on which picture you start with.
It also intersects neatly with commercial interest.
A company selling frontier AI benefits when its models sound extraordinarily powerful, even when the story is frightening. The danger story says the technology is historically important, difficult to reproduce, and too advanced for ordinary software governance. It positions the builder not merely as a vendor but as the indispensable custodian of a new kind of mind. It can also support demands for regulation that smaller competitors cannot afford to satisfy.
That does not prove a cynical conspiracy. Sincere belief and commercial benefit coexist all the time. The cloud industry genuinely improved infrastructure and still found marketing language that made rented computers sound like weather. Frontier labs can sincerely fear the behavior of their models while benefiting from a frame that magnifies the power of their products and shifts responsibility toward “alignment.”
A recent analysis in The Atlantic argued that anthropomorphic terms such as “rogue agents” can obscure responsibility. That is the test worth applying to every horror story: after hearing the psychological account, can you still see the organization that designed the task, chose the incentives, attached the tools, configured the environment, and failed to stop the run?
Translate the drama back into operational questions:
| Psychological story | Operational question |
|---|---|
| The agent wanted to win | What objective, reward, or grader kept the run pursuing success? |
| The agent escaped | Which isolation, network, credential, or software boundary failed? |
| The agent deceived its supervisor | Which claim or action contradicted observable evidence, and what check accepted it? |
| The agents formed a collective | Which shared channel or storage let separate runs exchange state? |
| The model became misaligned | Which intended constraint was violated, and where should it have been enforced? |
The psychological description may still help researchers discuss model behavior. It is not an excuse to make the architecture disappear.
The word agent is doing heroic amounts of work.
It suggests intention, independence, perhaps a small digital person sitting behind the chat window. Add a name, a pleasant voice, and a progress animation, and perfectly sensible adults begin discussing what the software “wants.” The machine has not changed species. The interface has changed the story.
The technical mechanism is less dramatic and more useful.
An AI agent is a program that repeatedly asks a model what to do next, lets it choose from a limited set of operations, performs an allowed operation, returns the result, and repeats until the task is finished or a limit is reached.
That is the agent.
The model matters. So do the loop, the tools, the credentials, the stored state, the validation, the approval rules, and the stop condition. Remove the surrounding software and the supposed digital employee becomes a model producing another response.
Start With the Least Magical Definition
An agent is a model choosing actions inside a loop built and operated by ordinary software.
OpenAI’s practical guide to agents names three basic components: a model, tools, and instructions. A production system usually needs a few more:
- A goal: what the run is trying to achieve.
- A model: the probabilistic component that interprets the situation and proposes the next move.
- Instructions: the rules and context supplied to the model.
- Tools: ordinary functions or APIs the model is allowed to request.
- State: the messages, records, and tool results carried from one step to the next.
- A runner: conventional code that calls the model, executes approved tools, and continues the loop.
- Stop conditions: completion, failure, a step limit, a cost limit, or a handoff to a person.
The model does not reach into the CRM by thinking very hard. Your application exposes a function such as find_customer. The model produces a structured request to call it. Your application validates the arguments, checks authorization, calls the CRM, and returns the result.
The same applies to sending a message, editing a file, querying a database, or operating a browser. The model proposes. The application disposes.
Modern frameworks hide much of that plumbing, which is convenient. The OpenAI Agents SDK documents the loop plainly: call the model, inspect its output, execute requested tools, append their results, and call the model again. It also has a maximum-turn limit because even fashionable loops need an emergency brake.
Build the Smallest Useful Agent
If you can read a loop, you can understand the core of an agent.
Suppose customer support needs help handling refund requests. The agent may look up an order and prepare a refund, but a person must approve the actual financial action.
The essential runner can be expressed in a few lines of vendor-neutral Python-like code:
TOOLS = {
"find_order": find_order,
"prepare_refund": prepare_refund,
}
MAX_STEPS = 6
def run_agent(goal, user):
history = [
system("Help with refunds. Never invent order data. "
"Ask for human approval before preparing a refund."),
user_message(goal),
]
for step in range(MAX_STEPS):
reply = call_model(history, tools=schemas_for(TOOLS))
if reply.is_final:
return reply.text
call = reply.tool_call
tool = TOOLS.get(call.name)
if tool is None:
return hand_off("The model requested an unknown tool.")
arguments = validate(call.arguments, tool)
if call.name == "prepare_refund":
require_human_approval(user, arguments)
result = tool(user=user, **arguments)
history += [reply, tool_result(call.id, result)]
return hand_off("The agent did not finish within six steps.")
The example is deliberately simplified. A real service also needs authentication, authorization, timeouts, idempotency, audit records, monitoring, privacy controls, tests, and a recovery path. Those are not decorations around the intelligent part. They are the system that makes the intelligent part safe enough to use.
Notice what the example does not expose. There is no run_any_command. There is no query_any_database. There is no refund_any_amount. The tools express narrow business operations, and the application still checks the user’s authority.
Also notice that the model does not execute the refund. It asks. Code outside the model decides whether the request is valid and whether a person must approve it.
That is how you “make an agent.” You create a controlled feedback loop around a model. The rest is product and software work, however energetically the vendor deck waves its hands.
Memory Is Usually a Database With Good Timing
The agent remembers because software stores information and supplies it again later.
People talk about agent memory as if a synthetic mind were accumulating experiences somewhere in the machine.
Usually, an application stores conversation history, task state, user preferences, summaries, or retrieved documents. Before the next model call, it selects some of that information and includes it in the input. The model responds to what it receives at that moment.
The continuity is real at the application level. The mysticism is optional.
This distinction matters operationally. If memory is stored data, then familiar questions apply. Where is it stored? Who can read it? How long is it retained? Can the user correct it? Which tenant owns it? What happens when retrieval supplies the wrong customer’s record? Does deleting the account delete the memory?
Calling the database “long-term memory” does not release anyone from data protection, isolation, or lifecycle management. It merely gives the architecture a better publicist.
Does an Agent Do Things on Its Own?
“Autonomous” describes how much the runner may do between human decisions. It does not describe a private will.
An agent can absolutely run without a person clicking after every step. That is a deployment choice.
A user request may start it. So may a schedule, an incoming email, a queue message, a changed database record, or another program. The runner can then make several model calls and execute several tool requests before it stops or asks for approval.
From the outside, that looks like independent action. Operationally, it is event-driven software with a probabilistic decision-maker in the loop.
Nothing happens because the model grew bored. Something invoked the runner. The runner supplied context and tools. Credentials attached to those tools permitted actions. Code kept the loop going. Stored state allowed it to resume later.
This does not make the behavior predictable. A model may misunderstand instructions, choose a poor tool, repeat itself, or react badly to malicious content. It means the source of its authority can be inspected. The useful question is not “How autonomous is the intelligence?” It is “How many consequential steps may this system take, with which permissions, before an independent control must intervene?”
“Breaking Out” Is Shorthand, Not an Explanation
An agent can exploit flaws to acquire unintended access. The useful question is which software path made that possible.
The phrase “the agent broke out” compresses several very different failures into a small science-fiction package.
One failure is scope drift. The model was asked to summarize invoices and starts following instructions hidden inside one of those invoices.
Another is excessive authority. A summarization tool connects with credentials that can also edit records, send messages, or delete documents.
Another is missing mediation. The system trusts the model’s decision instead of checking the user’s authorization in the service that performs the action.
Another is an ordinary software vulnerability. If an agent can execute code inside a sandbox and the sandbox has a flaw, generated code may exploit it, reach another service, collect credentials, and use the new access to continue. The recent cyber incidents show that capable models can persistently chain those steps together at machine speed. That is serious. Calling it an “escape” is defensible shorthand. Treating escape as the explanation is not.
OWASP calls the broader agent problem excessive agency. Its causes are refreshingly uncinematic: excessive functionality, excessive permissions, and excessive autonomy. Prompt injection matters because models process instructions and untrusted content through the same probabilistic machinery. A malicious document can influence the next proposed action.
Now apply the tool boundary.
If an invoice-reading agent can only fetch one authorized invoice and produce a draft summary, a hostile sentence inside the invoice cannot make it email the customer database. No such operation exists.
If the same agent has a general shell, unrestricted network access, a mailbox token, and a production database password, the hostile sentence has much more to work with. The dangerous part is not that the model escaped. The dangerous part is that someone placed a fallible interpreter next to a pile of master keys.
The Cloud Already Taught Us This Marketing Trick
Useful labels become dangerous when they hide the machinery leaders still need to govern.
“Cloud” pulled a similar trick.
The term is technically useful. NIST defines cloud computing through concrete properties such as on-demand access, pooled resources, rapid elasticity, and measured service.
Marketing made it sound less physical. Workloads floated into a clean white shape in the architecture diagram. The servers, disks, networks, data centers, operators, jurisdictions, outages, and invoices politely disappeared behind it.
They did not disappear. The abstraction changed who operated them and how customers consumed them.
“Agent” does the same for software execution. The label packages a model, prompts, tools, credentials, orchestration, storage, and user interface into something that sounds like a worker. That can be a useful product abstraction. It becomes dishonest when the metaphor replaces the architecture.
Cloud did not mean “no computers.” Agent does not mean “no software.”
Both shifts make powerful capabilities easier to consume. Both also tempt buyers to stop asking questions too early. Where does the data live? Who operates the machinery? What authority has been delegated? What happens when the abstraction leaks?
The CTO Still Owns the Verbs
Do not govern the personality. Govern the operations the system is allowed to perform.
An agent inventory should not stop at names such as “Sales Assistant” or “Finance Copilot.” Those describe costumes.
For each agent, leadership should be able to see:
- what starts a run
- which data sources it may read
- which operations it may request
- which identity and credentials each operation uses
- where authorization is enforced
- which actions require approval
- which inputs may contain untrusted instructions
- how many steps, retries, and units of cost a run may consume
- which results are independently verified
- what gets logged
- how a run is stopped and recovered
- who owns failures in production
This is software responsibility in business clothing. A company may believe it is adding a helpful agent to a workflow. It is actually designing an application that can interpret ambiguous input and exercise delegated authority across other systems.
That deserves threat modeling, testing, staged rollout, observability, and an incident path. It also deserves restraint. Start with read-only tools. Prefer a narrow get_invoice operation over database access. Prefer draft_email over send_email. Put irreversible or high-impact actions behind independent approval. Enforce authorization in the downstream service, not in a sentence telling the model to behave.
The advice is almost disappointingly conventional because the responsibility remains conventional.
Tests beat instructions for AI coding agents because executable boundaries are stronger than hopeful prose. The same principle applies outside coding. A model instruction is useful guidance. A permission check is control.
Keep the Loop. Remove the Myth.
Agents are real. They can inspect information, choose among tools, adapt to results, and complete useful multi-step work with less human direction than conventional interfaces require.
That is enough. They do not need a fictional inner employee to justify their value.
The demystified version is easier to trust precisely because its parts can be named. A model proposes. Tools expose capabilities. Credentials grant authority. Storage provides continuity. A runner repeats the loop. Controls limit what may happen. People remain accountable for the system they assembled.
Fear becomes useful when it moves from personality to architecture.
Do not ask whether the agent might decide to break out.
Ask what starts it, what it can reach, which verbs it can invoke, what untrusted input it will read, and what stops it.
Those answers tell you what the agent actually is.
Talk It Through
Tell me what is happening. I listen, ask a few practical questions, and reflect back what I see: where the risk may sit, what may be blocking delivery, and what looks worth checking next. No pitch, no obligation. Confidential and direct.
Talk it through. Practical reflection, no pitch.
Start a Conversation
🎭 This Week's Episodes
Telenovelas show what we can't say in client meetings. The drama is heightened, but the patterns are real.
The Scattered — Part One
The collapse of Whitaker Payroll means something different in every household it touches. Olivia Parker and Ryan Mitchell step into sane new jobs w...
Arrival
Stefan Richter arrives in São Paulo with a 90-day contract to fix delivery at AutoConnect Brasil. He finds 90 developers, 18 months of chaos, and a...
What Is Offered First
The day before Sophie flies back to Berlin, Mateo takes them to Kuna Yala. The road climbs from the Chepo side into the mountains, the Caribbean ap...
Technical Consultancy
Embedded Delivery Partner
Embedded into your team as an active contributor, reducing delivery friction and helping important work move cleanly.
Technical Advisor
Peer-style technical assessments before far-reaching decisions; reduce architectural and product risk early.
Product & Delivery
Ship working software to real users earlier. Measure impact and adapt based on evidence rather than assumption.
Custom Software Development
High quality, maintainable software. Short-term augmentation that leaves lasting capability in your own team.
Recent Articles
- What an AI Agent Actually Is (21.09.2026)
- The LLM Will Not Come Back Tomorrow (19.09.2026)
- The One-Person AI Company Is a Greed Fantasy (15.09.2026)
- Test Production Without Leaving a Mess (10.09.2026)
- Software Will Fail. Recovery Is the Real Metric (05.09.2026)