"Agent" has become the default word for any AI system that does more than chat. But there is an important difference between systems where you decide the sequence of steps and systems where the model decides them at runtime. The first kind is a workflow; the second is an agent. They have very different costs, failure modes and testing needs.
Choosing correctly is one of the most consequential architecture decisions in an LLM product. Building an agent where a workflow would do leads to unpredictable behaviour, higher costs and painful debugging. Building a rigid workflow where the task genuinely varies leads to brittle systems that break on every new case. This guide defines both clearly, describes the common patterns between them, and gives a decision framework with examples.
Definitions that actually help
A useful distinction, popularised by Anthropic's 2024 essay "Building effective agents," is:
- Workflows: systems where LLMs and tools are orchestrated through predefined code paths. The developer decides the steps; the model fills in each step.
- Agents: systems where the LLM dynamically directs its own process and tool use, deciding what to do next based on what it observes, until it judges the task complete.
Both can use tools, retrieval and multiple model calls. The difference is who controls the flow.
The spectrum from simple to autonomous
Most systems sit somewhere on a spectrum. Start at the left and move right only when you must.
| Level | Pattern | Who decides the steps | Example |
|---|---|---|---|
| 1 | Single call (with retrieval or tools) | Fixed | Answer a question from retrieved docs |
| 2 | Prompt chain | Developer | Extract → summarise → format a report |
| 3 | Router | Model chooses one of several fixed paths | Classify a ticket, send to the right handler |
| 4 | Parallel / map-reduce | Developer | Review every clause of a contract in parallel |
| 5 | Orchestrator-workers | Model splits the task; workers run fixed steps | Plan which files to change, then edit each |
| 6 | Evaluator-optimiser loop | Fixed loop, model judges when done | Draft, critique, revise until criteria pass |
| 7 | Autonomous agent | Model, at every step | Investigate and fix a bug in an unfamiliar codebase |
Levels 2 to 6 are workflows or workflow-agent hybrids. Many successful "agents" in production are actually level 3 to 6 systems. The patterns for levels 2 and 4 are covered in decomposition prompts for long, messy tasks.
What agents cost you
Autonomy is not free:
- Unpredictability. The same input can produce different sequences of actions, which makes behaviour harder to reason about and support.
- Compounding errors. Each step depends on previous ones; small mistakes cascade. See AI agent failure modes.
- Cost and latency. Agents often take many more model calls than a workflow for the same task, and the number varies by run.
- Testing difficulty. You must test outcomes over many possible paths, with repeated runs; see evaluating agents end to end.
- Safety surface. More tools and more autonomy mean more ways to take harmful actions.
What agents give you
- Flexibility on tasks where the right steps cannot be known in advance.
- Recovery: an agent can notice a failed step and try another approach.
- Coverage of the long tail: cases you never designed a path for.
- Less code for highly variable tasks, since you do not have to encode every branch.
The decision guide
Ask these questions in order.
- Can a single well-designed call do it? With good retrieval, tools and a clear prompt, many tasks need nothing more. Try it first.
- Can you write down the steps in advance? If the same sequence works for most inputs, build a workflow. Handle variations with a router.
- Is the variation in *which* path, or in *how many* steps? If inputs fall into a few categories, a router to fixed workflows is enough. If the number and kind of steps truly depend on what is discovered along the way, consider an agent.
- What does a wrong action cost? If actions are irreversible or expensive, prefer workflows, or agents with human approval at those points; see human-in-the-loop patterns.
- Can you verify success automatically? Agents work best where success is checkable: tests pass, a record exists, a file compiles. Without verification, autonomous loops are risky.
- Can you afford variable cost and latency? If not, cap the agent tightly or choose a workflow; see step budgets and stopping rules.
Examples
Customer support answers from documentation. Single call with retrieval, plus escalation. No agent needed.
Invoice processing. Extract fields, validate, match against purchase orders, flag exceptions. A fixed workflow with a human queue for exceptions.
Ticket triage across teams. A router: classify, then hand off to a team-specific workflow.
Research reports. Orchestrator-workers: a model plans the sub-questions; workers search and summarise each; a final step synthesises. Bounded autonomy.
Coding tasks in a large repository. Often a genuine agent: the steps depend on what the code reveals, and tests provide automatic verification.
Booking travel or making purchases. Agent-like flexibility in search, but workflow-style control at the point of payment: explicit confirmation, fixed checks.
Hybrid designs work best
The most robust systems combine both: a workflow skeleton with agentic steps inside it. For example, a fixed pipeline that gathers context, lets an agent attempt a bounded sub-task with limited tools, then verifies the result in code and routes failures to a human. You get flexibility where it is needed and predictability everywhere else.
Migrating between the two
Architecture decisions are not permanent, and the best path often runs through both.
From agent to workflow. Teams frequently prototype with an agent because it is quick to build: give the model tools and a goal, and see what it does. Once it is working, read a few hundred traces. If most successful runs follow the same few sequences, encode those sequences as workflows and keep the agent only for the remaining cases. Costs fall, behaviour becomes predictable, and the agent's traces have effectively designed your workflow for you.
From workflow to agent. When a workflow accumulates branches and fallbacks, look at which cases fall through. If they share a pattern, such as needing a few extra lookups or a different order of steps, add a bounded agentic step for exactly that part, with its own budget and verification. Replace the whole workflow with an agent only if the variation is everywhere.
In both directions, keep the eval set constant so you can compare quality, cost and latency before and after the change.
Signs you chose wrong
You built an agent but needed a workflow if most runs follow the same sequence of steps, you keep adding prompt rules to force a particular order, or costs vary wildly for similar tasks.
You built a workflow but needed an agent if you keep adding branches for new cases, many inputs fall through to a generic fallback, or users regularly need several attempts because the fixed path does not fit their situation.
Frequently asked questions
What is the difference between an AI agent and a workflow?
In a workflow, the developer defines the sequence of steps in code and the model performs each step. In an agent, the model decides at runtime which steps and tools to use and when the task is complete.
When should I use an AI agent?
When the steps needed genuinely depend on what is discovered during the task, success can be verified automatically, and you can tolerate variable cost and latency. Coding and open-ended research are common examples.
Are AI agents more expensive than workflows?
Usually. Agents tend to make more model calls, and the number varies by run. Workflows are more predictable in cost and latency, which is why they suit high-volume tasks.
Can I combine agents and workflows?
Yes, and it is often the best design: a workflow handles the predictable structure and safety checks, while bounded agentic steps handle the parts that need flexibility.
