The Real Architecture Behind Production AI Agents

Most agent diagrams follow the same clean flow: context, rules, agents, action. It looks elegant, it is easy to explain, and it makes a good pitch deck. Once you look at it as a backend engineer, it gets more complicated.

A chatbot that summarises a document is one thing. An agent that creates tickets, updates documents, reviews code, or escalates to people is another. The moment a system starts taking action, you have to be able to answer some plain questions.

• Can the agent reach only what this user is allowed to reach?
• Can it recover if a task fails halfway?
• Can a person review or override the important actions?
• Can someone audit what it did, later?
• Can we measure quality, cost, and latency?

These are not implementation details. They are the difference between a toy and something an organisation can trust.

I think about a production agent as a set of layers around the model. Each one solves a single problem, and each one keeps the system from turning into a black box with too much access and too little control.

These are responsibilities, not ten separate services. A small team can keep all of them in one application as long as the boundaries stay visible in the code and the logs. And you probably do not need several agents on day one. One capable agent inside a proper workflow is easier to test than a group of agents talking to each other.

Connectors bring in the raw material: Slack, Jira, GitHub, Outlook, SharePoint, Notion. Calling them API integrations undersells the work. A good connector handles sync jobs, webhooks, rate limits, pagination, deleted documents, and permission changes, and every source behaves differently. Syncing Slack messages is nothing like syncing SharePoint documents.

Identity is where most of the risk sits. An agent cannot safely run on one global service account. If someone asks the support agent a question, it must not pull finance or HR documents just because they happen to be in the index. Access has to be checked when content is indexed, when it is retrieved, and again when an action runs. RFC 9700 makes the same point about tokens: scope them to the resource and the action.

Context is the last piece. People reduce it to embeddings and a vector database, but one answer can need a Slack thread, a Jira ticket, a pull request, and a product spec. Good retrieval is about relevance, freshness, authority, and permission, not just similarity. I go deeper on that in the RAG 101 guide.

This is the layer simple diagrams leave out. Agents do not coordinate by magic. If a product agent drafts a spec, how does the engineering agent know to start? If QA finds a bug, how does the work get back to engineering? If the task takes twenty minutes, where is its state? If something fails, who retries?

Something has to own those answers. It can be a supervisor agent that delegates, or a plain backend with queues, workers, and a state machine. Which one depends on the work.

Whatever the shape, each step has the same division of labour. The agent proposes. Policy decides. A tool does the deterministic part. The orchestrator writes down what happened.

Real work takes minutes, hours, or days. Tests run, webhooks arrive, people approve things after lunch, rate limits reset. If all of that lives inside one request or one long conversation, a restart loses it and nobody can audit it afterwards.

So the workflow saves a checkpoint after every meaningful step and puts a timeout on every wait. It assumes an event can arrive twice, and uses a stable ID so a retry does not send the same email again. A temporary failure gets a limited number of retries. Anything else goes to a person.

A rule written into a prompt is a suggestion. The check that matters is ordinary code that runs after the model proposes something and before any tool is called: is the input well-formed, is this caller allowed, are the preconditions true, does it need approval?

The same proposal can deserve a different answer depending on who is asking, what it touches, and whether it can be undone. Reading a document is not sending an email, and sending an email is not deleting a repository. OWASP calls the failure mode excessive agency: more functionality, permissions, or autonomy than the task needs.

Human review should not be something that only happens after the system has already failed. It belongs in the workflow wherever a decision needs judgment or cannot be undone. Low-risk reads and drafts can go through on their own.

Approving is only one of the ways out of that state. The reviewer needs to see the proposed action and the evidence behind it, and be able to edit it or send it back. One approval covers one action, not everything after it. And when nobody answers, the request has to expire or move to someone else.

Ordinary monitoring tells you the system is up. For an agent you also need to know whether the work was useful, grounded in real sources, permitted, and finished.

That means one ID that follows a request through every model call, retrieval, tool call, retry, and approval. It means counting outcomes such as completed, corrected, escalated, and abandoned, with cost and latency for each step. And it means testing against real tasks, including hostile ones. Prompts and tool results often contain secrets, so log metadata by default and the content only when you have to.

OpenTelemetry’s semantic conventions give you a shared vocabulary for this. The generative-AI conventions are newer than the rest, so pin the version you build against. For the governance side, the NIST Generative AI Profile is the reference I would start from.

The framework matters less than keeping these responsibilities apart. Here is the loop with nothing vendor-specific in it.

def run_workflow(workflow_id: str, request: WorkRequest) -> Result:
    state = state_store.load_or_create(
        workflow_id=workflow_id,
        requested_by=request.principal_id,
        objective=request.objective,
    )

    principal = identity.resolve_and_revalidate(state.requested_by)
    plan = planner.create_or_resume(state, request.context)

    for step in plan.pending_steps():
        action = agent.propose(step=step, context=state.safe_context())
        decision = policy.evaluate(
            principal=principal,
            action=action,
            evidence=state.evidence,
        )

        audit.record(workflow_id, action, decision)

        if decision.requires_approval:
            state.checkpoint(status="awaiting_approval", action=action)
            return approvals.request(workflow_id, action, decision.reason)

        if not decision.allowed:
            return state.finish(status="denied", reason=decision.reason)

        result = tools.execute(
            action,
            principal=principal,
            idempotency_key=f"{workflow_id}:{step.id}",
            timeout=step.timeout,
            retry_policy=step.retry_policy,
        )

        state.checkpoint(
            status="step_completed",
            step=step,
            result=result.redacted(),
        )

    return state.finish(status="completed")

Notice what the model is not in charge of: identity, policy, retries, timeouts, the audit record, and the state. It proposes work inside those boundaries.

The short version, as the questions I would ask in a review.

1. For every action the agent can take, do we know who it affects and whether it can be undone?
2. Does every tool have typed inputs and a safe way to be called twice?
3. Have we tested with several roles, including someone whose access was just revoked?
4. What happens on a timeout, a duplicate event, or a half-finished tool call?
5. Can a reviewer see the evidence, edit the proposal, and say no?
6. Can we trace one request end to end and say what it cost?

The difference between a demo and a production system is rarely the model. It is everything around it.