Worked Example

The idea

"Build an AI agent that handles support tickets." That sentence is not a spec. It hides the queue, customer types, knowledge sources, allowed actions, escalation policy, brand voice, data sensitivity, success metrics, and failure behavior. A real build starts by shrinking it.

Narrow version: build an agent that reviews a new support ticket, classifies the issue, searches approved documentation, drafts a response with citations, and routes the ticket to a human if confidence is low or the request involves billing, deletion, security, legal, or account access.

Step 1: agent verdict

Is this an agent? Yes, but only barely. The system must inspect a ticket, decide whether it has enough context, search different documentation areas, and choose between draft and escalation. The next action depends on what it sees. However, it should not send replies in version one. It drafts and routes.

Step 2: tools and permissions

The agent receives three tools. `get_ticket(ticket_id)` returns ticket text and account metadata. `search_docs(query, audience)` returns approved snippets with URLs and freshness dates. `set_ticket_triage(ticket_id, label, summary)` writes an internal label and summary. No email sending, no account edits, no refunds, no user deletion, and no hidden browser control in the first release.

Each tool has validation. The ticket ID must belong to the queue. Search queries are capped and logged. Labels must be from a fixed enum. Summary length is bounded. The agent can ask for human help instead of forcing a decision.

Step 3: state shape

State should be structured enough that code can evaluate the run. A simple shape works: ticket ID, user message, classification, searches attempted, selected citations, risk flags, draft response, final status, token cost, and trace ID. If you cannot write this shape down, you are not ready to debug production behavior.

Short-term state lives with the ticket run. Long-term memory is intentionally absent in version one. The agent should not remember customer preferences across tickets until the team has a privacy policy, retention policy, correction workflow, and evaluation coverage for stale memory.

Step 4: smallest loop

1. Load the ticket.
2. Classify the issue and identify risk flags.
3. If risk flags are present, route to a human with a short reason.
4. Search approved documentation for likely answer material.
5. Draft a response with citations and uncertainty notes.
6. Write the internal triage label and summary.
7. Stop.

This is a constrained agent because the model can choose search queries and decide whether to escalate. It is not a free-roaming agent. The code owns the outer skeleton, and the model supplies judgment inside defined slots.

Step 5: evaluation fixtures

Build an evaluation set before expanding autonomy. Include common tickets, ambiguous tickets, outdated docs, billing requests, security-sensitive requests, prompt-injection attempts inside ticket text, and cases where the right answer is "I do not know." Assertions should check classification, tool arguments, selected citations, escalation, final draft quality, latency, and cost.

Step 6: release plan

Ship as a human-assist tool first. Support agents see the draft, citations, risk flags, and trace link. They can accept, edit, or reject. Collect reviewer decisions and add failed cases to the evaluation suite. Only after the agent consistently drafts safe answers should you consider limited auto-send for low-risk categories.