LLM Agents Explained: How They Work and When to Use Them
A practical explainer on how LLM agents work, the tool-call loop, failure modes, and when agents are the right choice versus a simpler.
An LLM agent is a model calling tools in a loop until a given task is complete[^claim:cm-01]. That definition, from LangChain's agent docs as of mid-2026, draws a clean line through a term that got muddy fast. The word "agent" spent 2024 and 2025 picking up baggage — some vendors used it to mean any prompt-wrapped API call, others reserved it for fully autonomous systems that plan multistep workflows, execute them, and course-correct on failure. The LangChain definition lands closer to the second camp, and it's the one this article works from.
The practical upshot: an LLM agent isn't a smarter chatbot. It's a system where the model decides what to do next, calls external tools to get new information, evaluates what comes back, and repeats. IBM Research describes the pattern plainly — LLM agents "can typically act on feedback to refine their plan of action"[^claim:cm-12]. That feedback loop is the thing that separates an agent from a single-turn completion.
Most of the confusion comes from people using "agent" to describe three different things. A prompt that returns JSON is not an agent. A RAG pipeline that retrieves chunks and stuffs them into a context window — which we covered separately — is not an agent either, though you can bolt one onto an agent as a tool. An agent is the loop: plan, act, observe, replan. NVIDIA's technical blog calls LLM agents "intelligent systems designed to plan, reason, use tools, and interact with external systems, not just respond to prompts"[^claim:cm-13]. That last clause does the heavy lifting.
The Core Loop: Model + Harness
Agent = Model + Harness[^claim:cm-02]. The harness is everything around the loop: the model, its prompt, its tools, and any middleware that shapes its behavior. The model is the brain — LangChain calls it "the main controller or 'brain' that controls a flow of operations needed to complete a task or user request"[^claim:cm-10 source note: the prompt attributes this to the Prompt Engineering Guide, but the brief-supplied evidence #10 sources it there]. The harness is the skeleton and the nervous system.
Here is the loop reduced to code. Real systems add tracing, retries, and approval gates, but the control flow is this simple:
type AgentState = {
task: string;
observations: string[];
};
while (!model.isDone(state)) {
const next = await model.plan(state);
if (next.type === "tool_call") {
const observation = await tools[next.tool](next.input);
state.observations.push(observation);
continue;
}
return next.answer;
}The important part is not the syntax; it is who decides the next step. In a normal app, the developer hard-codes the sequence. In an agent, the model chooses the next tool call from the latest state. Here's the loop in practice. The model receives a task. It decides on an action — call a search tool, read a file, run a shell command, query an API. The harness executes that action and feeds the result back to the model. The model evaluates: did I get what I needed? If not, it picks another action. This cycles until the model decides the task is done or the harness hits a stop condition — token budget exhausted, step limit reached, human-in-the-loop rejection.
The agent execution loop
Receive task
The user or upstream system hands the agent a goal: "summarize the last three quarterly reports and flag any revenue anomalies."
Plan
The model breaks the goal into steps: locate the reports, read each one, extract revenue figures, compare quarter-over-quarter, flag deviations beyond a threshold.
Act
The harness executes tool calls: filesystem reads, a search query, maybe a Python calculation. Each call returns structured output the model can parse.
Observe
The model inspects the result. Did the file read return the right quarter? If not, it adjusts the query and tries again.
Replan or finish
Once the model has enough signal, it either loops back with a refined plan or returns the final output to the user.
The harness isn't passive plumbing. It enforces constraints — which tools the model can call, how many steps it gets, what happens when a tool call fails. The middleware layer (more on that shortly) sits inside the harness and intercepts every tool call and every model response. That interception point is where customization happens.
Acting on Feedback: How the Model Refines Its Plan
Claim 12 gets at something deceptively simple: an LLM agent can act on feedback to refine its plan of action[^claim:cm-12]. The mechanics of that feedback loop are worth unpacking, because they're what make the "plan, act, observe, replan" cycle more than a slogan.
When the model calls a tool and gets a result back, it performs an evaluation step that is invisible to the user but determines everything downstream. It asks itself, implicitly, whether the tool output satisfies the intent of the call. Did the search query return relevant documents or noise? Did the file read get the right quarter's report? Did the API return a valid response or an error code? If the answer is no, the model doesn't just try again blindly — it adjusts the parameters of the next call based on what it learned from the failure. A search that returned something too broad gets a narrower query. A file read that returned the wrong document gets a more specific path. That adjustment is the "refine" in "refine their plan of action."
The replanning step goes further. Sometimes the tool output reveals that the original plan was wrong — not just the parameters, but the approach. The model might discover that the data it needs doesn't exist in the format it assumed, or that a subtask it queued is irrelevant given what it just learned. In those cases, the model rewrites the remaining plan, dropping steps, reordering them, or inventing new ones. This replanning is bounded by the harness: the model can only replan within the tool set and step budget it has been given. If the replan requires a tool the agent doesn't have, the harness can't help — the agent fails, and the failure mode is that it runs out of steps trying approaches that can't work. Understanding that boundary is the difference between designing an agent that degrades gracefully and one that spins its wheels until the token budget runs dry.
Why an LLM Alone Isn't Enough
A raw LLM behind an API endpoint knows only what was in its training data, cutoff date baked in. It can't check today's weather, can't read your company's internal docs, can't run a SQL query, can't send an email on your behalf. Ask it "what's the status of PR #342?" and it hallucinates a plausible-sounding answer because it has no access to your GitHub.
The agent pattern solves this by giving the model hands. Tools bridge the gap between the model's internal knowledge and the outside world. A search tool lets it retrieve current information. A code executor lets it run calculations the model itself would fumble — LLMs are mediocre at arithmetic, but a harness that shells out to a Python interpreter sidesteps that cleanly. Database connectors, API clients, filesystem access: each tool expands what the agent can touch.
But tools alone aren't enough either. A model calling tools without guardrails is a bull in a china shop. It might call the same expensive API 40 times. It might write to a file it shouldn't. It might follow a reasoning path that made sense three steps ago but is now clearly wrong, and nothing in the raw loop will stop it. The harness supplies the missing discipline.
The Agent Harness: What Wraps the Model
- Harness
- The software layer that wraps an LLM, manages its tool calls, enforces step limits and stop conditions, and routes all traffic through configurable middleware. Think of it as the runtime, not the model.
The harness does five things in every loop iteration. First, it formats the prompt — injecting system instructions, tool definitions, conversation history, and any retrieved context into the model's input. Second, it parses the model's output to extract a decision: is this a tool call, and if so, which tool with which parameters? Third, it executes that tool call against the real world — hitting an API, reading a file, running a shell command. Fourth, it feeds the result back into the model for the next reasoning step. Fifth, it checks termination conditions: did the model signal completion, did we hit the max step count, did a guardrail trip?
Most harnesses expose a middleware stack. Each tool call and each model response passes through a chain of middleware functions that can inspect, modify, or block the data in flight. This is the customization primitive, and it's where production agents differ from weekend projects.
Middleware: The Customization Primitive
Middleware is the primitive for customization in agent architectures[^claim:cm-03]. It's a pipeline of functions that every piece of data passes through — think of it as Express.js middleware but for agent loops instead of HTTP requests. When you need to extend or adapt an agent's behavior, middleware is the mechanism: rather than modifying the model or rewriting the harness, you compose middleware functions that intercept, transform, or gate the data flowing through the loop. This is what makes middleware the customization primitive — it's the single extension point through which all custom behavior flows.
You'd add middleware for things the base harness doesn't handle. Logging every tool call with timestamps for debugging. Rate-limiting API calls so you don't blow through your Anthropic tier limit in one bad loop. Injecting compliance checks that scan tool outputs for PII before they reach the model. In an enterprise deployment — not unlike the considerations we flagged in the AI-assisted analytics piece — compliance middleware is the difference between a prototype and something legal can sign off on.
The execution environment middleware gives the agent a workspace: tools, filesystem access, and optionally a code execution sandbox. Without this, the agent can reason about what it wants to do but can't actually do anything. With it, the agent can write a script, run it, read the output, and decide whether the result matches expectations — all inside one loop.
Context Management and Memory
Context management handles context window limits via summarization and memory[^claim:cm-04]. This is the practical constraint that shapes every agent design decision. A Claude or GPT-4 model in mid-2026 might have a 200K-token context window, which sounds enormous until an agent loop fills it in six tool calls — each one appending a 15K-token API response and a 5K-token model reasoning trace.
The solution is active context management inside the harness. When the context window approaches its limit, a summarization middleware compresses earlier conversation turns into a dense summary, discarding the verbatim exchanges. The model sees the summary plus recent history — enough signal to continue without losing the thread, without eating the whole window budget on stale tool outputs.
Memory is the other half. A persistent memory layer stores facts, preferences, and learned patterns across sessions. The agent doesn't start from scratch each time; it retrieves relevant memories at the start of a task. In practice this is often implemented as a vector store keyed to embeddings of past interactions, queried for semantically similar situations. K2view's guide describes LLM agents as "AI systems that leverage Large Language Models, tools, and memory to perform tasks, make decisions, and interact with users or other systems"[^claim:cm-11 source note: the brief-supplied evidence #11 from K2view supports this framing]. Memory is the third pillar alongside the model and the tools.
Planning and Delegation: Sub-Agents and Task Breakdown
Planning and delegation lets the main agent break work into pieces and hand them to sub-agents[^claim:cm-05]. This is the pattern that separates toy agents from ones that do real work. A single agent trying to research a topic, write a report, and fact-check its own output in one linear loop can become difficult to reason about: the context window accumulates signals from different phases, and the model may lose track of which phase it's in — which is one of the architectural motivations that makes sub-agent isolation attractive.
The sub-agent pattern splits the work. The main agent — sometimes called the orchestrator — receives the high-level task and produces a plan. That plan is a sequence of subtasks, each with clear inputs and expected outputs. The orchestrator spawns a sub-agent for each subtask, handing it an isolated context window and a focused tool set. The sub-agent runs its own loop, returns a result, and the orchestrator stitches the outputs together.
The isolation matters. A sub-agent researching competitor pricing doesn't see the internal strategy documents the orchestrator is holding for a later subtask. Its context window stays small, its reasoning stays focused, and if it fails, the orchestrator can retry or replan without contaminating the rest of the workflow. Deep Agents, which we'll get to, bakes this pattern into its default behavior: sub-agents with isolated context windows as a first-class primitive.
Fault Tolerance and Guardrails
Two middleware categories handle safety and reliability, and they're different enough to deserve separate treatment.
Fault tolerance middleware handles rate limits, model timeouts, and transient API errors at the infrastructure level[^claim:cm-06]. When your agent calls an external weather API and gets a 429 (rate limited), this middleware catches the error, applies an exponential backoff, and retries — without the model ever knowing something went wrong. When a model provider returns a 503, the middleware retries or fails over to a backup model. These are infrastructure concerns, not reasoning concerns, and keeping them out of the model's context window keeps the agent focused on the task.
Guardrails intercept data flowing through the agent loop, applying compliance rules or content policies deterministically[^claim:cm-07]. Unlike fault tolerance, guardrails are policy-layer — they don't retry, they block or rewrite. A guardrail might scan every tool output for patterns matching Social Security numbers and redact them before the model sees them. Another might check that the agent's planned action doesn't violate a allowlist of permitted API endpoints. These checks are deterministic — no model judgment involved, just pattern matching and rule evaluation — which makes them auditable in a way that "the model decided not to" never is.
Human-in-the-Loop Steering
Human-in-the-loop steering lets humans approve, edit, or redirect agent actions before they execute[^claim:cm-08]. The harness pauses the loop at designated decision points — before a file write, before an email send, before a database mutation — and waits for a human to signal proceed, edit, or abort.
The design question is where to put the human checkpoint. Too early (every tool call) and the agent is just a slow CLI. Too late (only at task completion) and the human is rubber-stamping output they don't have time to verify. The useful middle ground puts checkpoints at irreversible actions: anything that writes, sends, deletes, or spends money. Read-only actions — search queries, API GETs, file reads — run without interruption. The agent stays fast where speed is safe and pauses where mistakes have consequences.
From Research to Production: The Stack Deepens
The agent stack runs in three layers: Deep Agents (opinionated harness) on top of LangChain's create_agent (minimal harness) on top of LangGraph's graph runtime[^claim:cm-11]. This layering is deliberate, and picking the right layer for your use case is most of the engineering decision.
At the bottom, LangGraph gives you a graph runtime — nodes for processing steps, edges for control flow, built-in streaming, persistence, and checkpointing. You define the graph yourself. You wire up every tool call, every conditional branch, every retry path. Maximum control, maximum effort.
One level up, LangChain's create_agent gives you a minimal harness: a pre-built agent loop with tool calling, basic middleware support, and a standard prompt format. You supply the tools and the system prompt; the harness handles the loop. This covers most production use cases where you need an agent but don't need to customize the control flow at the graph level.
At the top, Deep Agents gives you an opinionated harness that runs out of the box — sub-agents, filesystem tools, shell access, persistent memory, human-in-the-loop approval, skills on demand, and any MCP server as tools[^claim:cm-09]. Use Deep Agents when you want the full harness without building it yourself; use create_agent for a lighter harness; drop to LangGraph when the standard loop doesn't fit your control flow.
Comparison
Which agent layer to use
Three layers of the LangChain agent stack, from full control to full convenience.
Open-Source Harnesses: Deep Agents and Beyond
Deep Agents is an open-source agent harness with sub-agents, isolated context windows, filesystem tools, shell access, persistent memory, and human-in-the-loop approval[^claim:cm-09]. The repo, langchain-ai/deepagents on GitHub, is the reference implementation of the patterns described in the LangChain agent docs — it's not a separate product, it's the same stack, with Deep Agents as the most opinionated layer.
The sub-agent implementation is worth examining. Each sub-agent gets its own context window, its own tool set, and its own middleware stack. The orchestrator communicates with sub-agents through structured task descriptions and result objects, not by sharing a context window. This isolation keeps individual agent loops short and focused — a sub-agent writing a unit test doesn't see the orchestrator's conversation about database schema design, and vice versa.
Model-Agnostic by Design
Deep Agents is model-agnostic and works with frontier APIs, open-weight models on providers like Baseten or Fireworks, and self-hosted models via Ollama, vLLM, or llama.cpp[^claim:cm-10]. The harness cares about one thing: does the model support tool calling? If yes, it works. You can prototype against Claude or GPT-4, then swap in an open-weight model when you need to run on your own hardware or control costs — no harness rewrite required.
This matters more in mid-2026 than it did even a year ago. Open-weight models with reliable tool-calling support are now viable for production agent workloads, and self-hosting eliminates the per-token cost that makes long agent loops expensive on frontier APIs. An agent that makes 40 tool calls per task, each costing a few cents in API fees, adds up fast. Running the same loop against a self-hosted model on a fixed-cost GPU instance changes the economics entirely. The cost calculus here echoes the infrastructure decisions we walked through for RAG deployments, where the hosting model often matters more than the retrieval algorithm.
Enterprise Agents: A Different Set of Demands
Enterprise agents shift the conversation from "can it work?" to "can we prove it worked, can we stop it, and can we audit every decision?" The core loop is the same; everything around it gets heavier.
Observability becomes non-negotiable. LangGraph's built-in integration with LangSmith provides tracing, evaluation, and monitoring — every tool call, every model reasoning step, every guardrail interception captured and timestamped. When an agent makes a bad decision in production, you need the full trace to understand why, and you need it before the postmortem meeting starts.
Compliance middleware moves from nice-to-have to deployment-blocking. Guardrails that scan for PII, enforce data residency rules, and block disallowed tool calls aren't optional when the agent touches customer data. The deterministic nature of guardrails — pattern matching, not model judgment — is what makes them auditable. A compliance officer can review the guardrail configuration and know exactly what it blocks, unlike the model's reasoning which requires forensic analysis.
Human-in-the-loop checkpoints multiply. An enterprise agent might have approval gates before any customer-facing communication, any contract modification, any financial transaction above a threshold. The harness configuration becomes the policy document: which actions require approval, who approves them, what the timeout is, what happens if nobody responds.
How to Think About LLM Agents in 2026
The agent pattern is real infrastructure now, not a research demo. But thinking clearly about it means avoiding two mistakes.
The first mistake is treating agents as magic. An agent is a model calling tools in a loop[^claim:cm-01]. If your model is bad at reasoning, wrapping it in a harness won't fix that — it'll just produce bad decisions faster and at higher API cost. The harness amplifies whatever reasoning capability the model brings. Start with the simplest thing that works: a single-turn completion, a RAG pipeline, a fine-tuned model for your domain. Add the agent loop only when the task genuinely requires multistep reasoning with tool use, and when you're willing to pay the latency and cost that the loop introduces.
The second mistake is overbuilding the harness before you understand what the model actually needs. Deep Agents ships with sub-agents, persistent memory, human-in-the-loop approval, shell access, and MCP integration. You probably don't need all of that on day one. Start with create_agent and a few well-chosen tools. Add middleware as failure modes surface in production. You'll learn more about your agent's actual behavior from a week of production traces than from a month of pre-launch design.
The stack is layered for a reason. LangGraph when you need to invent new control flow. create_agent when the standard loop fits. Deep Agents when you've proven that loop in production and want the full harness without maintaining it yourself[^claim:cm-11]. The model-agnostic design means the choice of which model powers the loop is decoupled from the harness decision — pick the model that reasons best for your task, at the cost profile that makes sense, and swap later if the economics shift. That decoupling is the practical consequence of the Model + Harness framing. The model thinks. The harness runs. Both evolve independently, and that's the architecture that'll carry agents through whatever the next model generation brings.