In a previous post, I walked through the basic agent execution loop - the while loop that calls a model, executes tools, and iterates. That loop is the foundation. But there’s a gap between a basic loop and agents like Claude Code, GitHub Copilot CLI, or Gemini CLI that handle complex, multi-step tasks over extended runs.

Consider what happens when you ask Claude Code to “refactor this authentication system to use JWT tokens.” It needs to find all relevant files across the codebase, read and understand the current implementation, plan the changes, edit multiple files, run tests to verify nothing broke, and iterate if tests fail. That’s 20-40 iterations. Or more. Each iteration adds tool calls and results to the context. A basic loop can’t handle this - context explodes, the agent quits early, and you have no visibility into what’s happening.

What bridges the gap? Three extensions:

1. Tools - Not just any tools, but the specific set that enables code exploration and modification at scale

2. Middlewares and hooks - Per-call interception for observability and safety, plus loop-level control for persistence

3. Context management - Strategies to handle the inevitable context explosion

This post walks through building an updated agent (beyond the for loop) using PicoAgents - the companion multi-agent framework built from scratch as part of the Designing Multi-Agent Systems book project.

All code below comes from PicoAgents - a working framework you can install and run, not pseudocode. Snippets are simplified; comments note which source file each comes from.

Extension 1: The Right Tools

The first piece of the puzzle is assembling the right set of tools. Reviewing what Claude Code ships with is instructive - the tools cluster into four groups:

Exploration - finding and reading code

• Read - file contents with line numbers (supports images, PDFs, notebooks)

• Glob - fast file pattern matching (**/*.py, src/**/*.ts)

• Grep - content search using ripgrep with regex support

• LS - list directories

Modification - changing code and running commands

• Write - create or overwrite files

• Edit / MultiEdit - exact string replacement (requires unique match)

• Bash - execute shell commands with timeout and background support

• NotebookEdit - edit Jupyter notebook cells

Coordination - managing complex tasks

• Task - launch sub-agents for complex tasks (Explore, general-purpose)

• TodoWrite - manage structured task lists with status tracking

• ExitPlanMode - exit planning mode after creating implementation plan

Interaction - communicating with user and web

• AskUserQuestion - ask structured multiple-choice questions

• WebFetch / WebSearch - fetch and search web content

• Skill - execute user-defined skills

• MCP tools - dynamic tools from MCP servers

PicoAgents implements nearly all of these. The tool names differ (ReadFileTool, EditFileTool, BashTool, etc.) but the coverage maps one-to-one. PicoAgents also adds a ThinkTool for explicit reasoning and a SkillsTool for on-demand context loading via SKILL.md files.

Tool Implementation and Design

I covered how to implement tools (the @tool decorator, JSON Schema generation, and the tool execution pattern) in my agent execution loop post, and the broader arc of how tool calling has evolved in the arc of tool calling. PicoAgents’ tool implementations are in picoagents/tools/.

What’s worth highlighting here are the design patterns specific to coding agent tools:

Output truncation matters more than you’d think. A naive bash("ls -la") on a large directory dumps thousands of lines into context. Every token of that output gets carried forward to every subsequent LLM call. PicoAgents’ BashTool caps output and includes exit codes; ReadFileTool takes offset and limit parameters so the agent can read files in chunks.

Structured errors over stack traces. Error: File not found: config.py is actionable. A Python traceback wastes tokens and confuses the model. Every tool should return a clear, concise error message that tells the model what to do differently.

Edit uniqueness prevents wrong-location edits. The EditFileTool requires the target string to be unique in the file. If the string appears multiple times, the tool returns an error asking for more context. This is the same pattern Claude Code uses - it prevents the common failure of editing the wrong occurrence of a repeated string.

The security model is local trust. These tools are powerful - bash can execute arbitrary commands, write_file can overwrite anything. This is intentional. Meta’s Rule of Two framework captures why this works: agents should satisfy no more than two of (A) process untrusted inputs, (B) access sensitive systems, (C) change state. A local coding agent satisfies B and C, but not A - the input is from you, the trusted user.

Learning from Claude Code’s System Prompt

Tools are only half the equation. The other half is how you instruct the model to use them. Claude Code’s system prompt has been published and reveals patterns worth adopting:

• Token minimization - “Responses under 4 lines unless detail requested” - saves context

• Batch tool calls - “Call multiple tools in a single response” - fewer iterations

• Read before edit - “NEVER edit a file you haven’t read” - prevents blind edits

• Follow patterns - “Examine neighboring files before writing” - consistency

• Verify deps - “Never assume libraries exist” - prevents import errors

• No proactive docs - “NEVER create documentation unless requested” - stays focused

PicoAgents’ _instructions.py incorporates a good number of these patterns.

Extension 2: Middlewares and Hooks

As agents run longer, you need two kinds of control: visibility into individual operations (what tool just ran? how many tokens did that call use?) and control over the loop itself (is the task actually done? should the agent keep going?).

PicoAgents separates these into two distinct mechanisms, following the pattern I described in my middleware post:

Middlewares intercept individual tool/model calls. They fire around each operation and can log, block, or modify calls.

Hooks intercept the agent loop itself. They fire before the first LLM call or when the agent tries to stop, and can inject instructions or check completion.

Middlewares: Per-Call Interception

Middlewares wrap individual tool and model calls. PicoAgents uses a BaseMiddleware class with two override points: on_model_call (fires around each LLM request) and on_tool_call (fires around each tool execution). Override either to log, modify, or block the operation:

from picoagents._middleware import BaseMiddleware

class TokenTrackingMiddleware(BaseMiddleware):
    async def on_model_call(self, messages, tools, context, next_fn):
        result = await next_fn(messages, tools)
        print(f"Tokens: {result.usage.tokens_input} in, "
              f"{result.usage.tokens_output} out")
        return result

    async def on_tool_call(self, tool_name, parameters, context, next_fn):
        if tool_name == "bash" and "rm -rf" in parameters.get("command", ""):
            return ToolResult(success=False, result="Blocked: dangerous command")
        return await next_fn(tool_name, parameters)

The next_fn pattern chains middlewares - each one decides whether to pass through, modify, or block the call. The blocked result goes back to the model as a tool response, so it can adjust its approach.

Hooks: Loop-Level Control

Hooks operate at a different level - they fire at two points in the agent loop: before the first LLM call (start hooks) and when the agent would stop because it returned no tool calls (end hooks). They don’t see individual operations.

Start hooks run once before the first LLM call. They inject instructions:

from picoagents._hooks import BaseStartHook, BaseEndHook

class PlanningHook(BaseStartHook):
    async def on_start(self, context: LoopContext) -> str | None:
        return (
            "Before starting, break the task into steps using todo_write. "
            "Mark each step as you complete it."
        )

End hooks run when the agent tries to stop (returns no tool calls). They decide whether to let it:

class LLMCompletionCheckHook(BaseEndHook):
    def __init__(self, max_restarts=2):
        self.termination = MaxRestartsTermination(max_restarts)

    async def on_end(self, context: LoopContext) -> str | None:
        if self.termination.should_terminate(context):
            return None  # Hit restart limit, allow stop

        # Summarize what the agent actually did (tool calls, not claims)
        summary = self._build_activity_log(context.llm_messages)

        # Ask a judge model: is this complete?
        result = await context.model_client.create([
            SystemMessage("Judge task completion based on tool calls and "
                          "results, not the agent's claims. "
                          "Reply COMPLETE or INCOMPLETE."),
            UserMessage(f"Task: {context.llm_messages[1].content}\n\n"
                        f"Activity:\n{summary}"),
        ])

        if result.message.content.startswith("COMPLETE"):
            return None  # Done, allow stop

        return f"You are not done yet. Continue working."

Return a string: the agent gets that message injected and the loop continues. Return None: the agent stops. The MaxRestartsTermination safety valve prevents infinite loops - after N restarts, the hook allows the agent to stop regardless.

The two mechanisms are complementary. A middleware that blocks rm -rf fires on every tool call, but it can’t prevent early stopping. An end hook that checks completion can keep the agent going, but it can’t block a dangerous command. Together, they give agents cognitive durability - the ability to persist through long tasks instead of quitting at the first natural pause. How completion hooks interact with context management is where things get interesting.

Extension 3: Context Management

Without context management, every tool result accumulates. A 20-iteration task with file reads, grep results, and bash outputs will overflow any context window. I covered context engineering strategies in depth in a previous post - including benchmarks comparing strategies, the sawtooth pattern of healthy compaction, and the thrashing failure mode when budgets are too tight. Here I’ll focus on two things that post didn’t cover well: the pluggable strategy pattern and the critical implementation detail that most frameworks get wrong.

Compaction as a Pluggable Strategy

PicoAgents defines a CompactionStrategy protocol with a single method: compact(messages) -> messages. Any class that implements this method can be plugged into an agent. PicoAgents ships with three built-in strategies:

• HeadTailCompaction - Keeps the head (system prompt, task) and tail (recent work), drops the middle. Zero extra cost. Works with any provider.

• SlidingWindowCompaction - Keeps the last N messages. Simpler but loses the original task context.

• NoCompaction - Baseline that returns messages unchanged, for benchmarking.

But these are just starting points. The protocol is the point - you can implement your own. An LLM-based summarization strategy (like what Claude Code uses via Anthropic’s Compaction API) preserves semantic meaning but costs an extra LLM call and is provider-specific. A hybrid strategy could summarize the middle while preserving head and tail verbatim. The agent doesn’t care which strategy it gets - it calls compact() and gets back messages.

One implementation detail matters: many agent frameworks apply compaction after adding new messages but use the original list for the next iteration. The compacted list doesn’t persist.

PicoAgents applies compaction before each LLM call, and the compacted list continues forward:

# From picoagents/agents/_agent.py (simplified)
while iteration < self.max_iterations:
    # CRITICAL: Apply context strategy BEFORE each call
    if self.compaction:
        llm_messages = self.compaction.compact(llm_messages)

    # The compacted list is used for both this call AND continues forward
    response = await model_client.create(llm_messages, tools=tools)

    # Add response and tool results to llm_messages
    # ... (these accumulate until next compaction)

This seems obvious, but it’s a common bug. The context strategy must be in the loop with reassignment, not applied as a side effect after it.

Context Isolation via Sub-Agents

Compaction manages context within a single agent. Isolation prevents context from accumulating in the first place by running sub-tasks in separate contexts.

PicoAgents supports this via as_tool() - any agent can be wrapped as a callable tool for another agent. The sub-agent gets its own context window, tools, and compaction strategy. It runs, does its work (potentially reading dozens of files and accumulating 50k+ tokens), and returns a summary. Only that summary enters the parent’s context.

sub_agent = Agent(
    name="code_reviewer",
    description="Reviews code in a directory",
    tools=[ReadFileTool(), ListDirectoryTool()],
    compaction=HeadTailCompaction(token_budget=50_000),
    ...
)

# The coordinator never sees the sub-agent's internal context
coordinator = Agent(
    tools=[sub_agent.as_tool()],  # Agent becomes a callable tool
    ...
)

This is the same pattern Claude Code uses with its Task tool - spawn a sub-agent for bounded work, discard its context, keep only the result. The parent’s context stays bounded regardless of how much work the sub-agents do.

For benchmarks comparing compaction and isolation strategies - including the budget sizing rule and how hooks interact with compaction - see the Context Engineering post.

Putting It Together

Here’s how PicoAgents’ Agent class wires all three extensions together. The constructor takes tools, compaction, middlewares, and hooks as separate concerns:

from picoagents import Agent, LLMCompletionCheckHook
from picoagents.compaction import HeadTailCompaction
from picoagents._hooks import PlanningHook
from picoagents._middleware import BaseMiddleware
from picoagents.llm import AzureOpenAIChatCompletionClient
from picoagents.tools import create_coding_tools, create_context_engineering_tools

model_client = AzureOpenAIChatCompletionClient(
    model="gpt-4.1-mini",
    azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],
)

agent = Agent(
    name="coding_agent",
    instructions="You are a senior software engineer...",
    model_client=model_client,
    tools=create_coding_tools() + create_context_engineering_tools(),
    compaction=HeadTailCompaction(token_budget=100_000, head_ratio=0.2),
    middlewares=[TokenTrackingMiddleware()],
    start_hooks=[PlanningHook()],
    end_hooks=[LLMCompletionCheckHook(max_restarts=5)],
    max_iterations=50,
)

response = await agent.run("Find all Python files that import pandas and list them")
# response.usage has token counts, LLM calls, tool calls
# response.context has the full conversation for multi-turn follow-ups

The create_coding_tools() preset includes: ReadFileTool, WriteFileTool, EditFileTool, GlobFilesTool, GrepTool, ListDirectoryTool, BashTool. Add create_context_engineering_tools() for TodoWriteTool, TodoReadTool, and SkillsTool. Wrap any agent with as_tool() for sub-agent isolation.

Multi-turn conversations pass the context forward:

r1 = await agent.run("Find the bug in auth.py")
r2 = await agent.run("Now fix it", context=r1.context)  # Continues the conversation

To dig into the implementation, here are the key source files:

• _agent.py - Agent class with tool loop, compaction, and hook integration

• compaction.py - HeadTail and SlidingWindow compaction strategies

• _hooks.py - Completion checks, planning hooks, termination conditions

• _middleware.py - BaseMiddleware for per-call interception

• tools/ - Tool implementations (coding, context engineering, research)

• compaction.ipynb - Benchmark comparing compaction strategies

Choosing Your Strategy

The three extensions interact. Here’s when to apply what:

• Short tasks (< 5 tool calls): No compaction, hooks, or sub-agents needed.

• Medium tasks (5-20 calls): Compaction at 2-3x working set. Token tracking middleware. Sub-agents optional.

• Long tasks (20+ calls): Compaction + completion hook. Token tracking + tool blocking middlewares. Sub-agents for bounded subtasks.

• Multi-directory operations: Compaction on each agent. Completion check on coordinator. One sub-agent per directory.

The general principle: compaction manages memory within an agent, hooks manage persistence across the loop, and sub-agents manage scope across tasks. Start simple, add complexity only when you see the failure mode it addresses (context overflow, early stopping, or unbounded growth).

Same Patterns, Different Frameworks

The three extensions are architectural patterns, not framework features - PicoAgents is just one implementation. The implementation details differ, but the architecture is the same. Here’s how each maps to other frameworks:

Tools: Every framework has tool registration. The specific tools (read, edit, glob, grep, bash) are what matter for coding agents - the registration mechanism is interchangeable.

• LangGraph - @tool decorator, tools passed to create_react_agent()

• Microsoft Agent Framework - @kernel_function decorator or ai_function registration

• Google ADK - FunctionTool wrapper, tools passed to Agent()

• Claude Agent SDK - Tools defined in agent config

Middlewares and hooks: The naming varies, but the concept is universal - intercept operations at call level or loop level.

• PicoAgents - Per-call: BaseMiddleware (on_tool_call, on_model_call). Loop-level: BaseStartHook, BaseEndHook

• Deep Agents - Per-call: AgentMiddleware with wrap_model_call(). Loop-level: state-based via IsLastStep

• Agent Framework - Per-call: middleware classes (decorator or class-based). Loop-level: event handlers

• Google ADK - Per-call: before_tool_callback, after_tool_callback. Loop-level: before_agent_callback, after_agent_callback

• Claude Agent SDK - Per-call: PreToolUse, PostToolUse hooks. Loop-level: session-level hooks

Compaction: The strategies differ but the need is universal. Deep Agents uses SummarizationMiddleware that triggers at 85% context fill, summarizes old messages with a smaller model, and offloads full history to a backend file. LangGraph’s core provides RemoveMessage primitives and custom state reducers - you build your own trimming logic. PicoAgents uses HeadTail trimming with no extra LLM calls. Different trade-offs, same architectural slot.

The PicoAgents implementation is one reference - adapt the patterns to whatever you’re using.

Key Takeaways

1. Tools matter, but not quantity - A focused set covering exploration (read, glob, grep), modification (write, edit, bash), and coordination (task, todo_write) handles most coding tasks.

2. Middlewares and hooks solve different problems - Middlewares intercept individual calls (observability, safety). Hooks control the loop itself (prevent early stopping, inject planning etc). You need both for long-running agents.

3. Right-size your compaction budget - Compaction cuts total tokens (56-70% in benchmarks), but too-tight budgets backfire. At 8k budget, agents scored 4.0/10 vs 6.0/10 without compaction - they lost memory of recent work and wasted 55% of reads re-reading files they’d already seen. A healthy budget produces a sawtooth context curve (grow, trim, grow). A flat line at the budget means thrashing. If your duplicate read ratio exceeds 15%, increase the budget. Size it at 2-3x the agent’s typical working set.

4. Hooks and compaction interact - Completion hooks can amplify thrashing with tight budgets. Right-size the budget first, then add hooks.

5. These patterns are universal - Deep Agents (built on LangGraph) implements all three: SummarizationMiddleware for compaction, AgentMiddleware for per-call interception, and SubAgentMiddleware for context isolation - with the same coding tools (read, edit, glob, grep, bash). LangGraph’s core provides RemoveMessage for trimming and IsLastStep for loop control. The patterns are the same regardless of framework.

The full PicoAgents implementation is available in the companion repo for Designing Multi-Agent Systems. The book covers these patterns in depth, including evaluation frameworks and the 10 common failure modes in agent systems.

Up next: The patterns above are sufficient for building a strong initial agent. But production agents extend these ideas in ways that are hard to anticipate until you’ve seen them. On March 31, 2026, the full Claude Code source was inadvertently published via its npm registry - giving us a rare look at a production agent harness used by hundreds of thousands of developers. In the next post, I tear down that source code and show exactly how the same three extensions - tools, hooks, and compaction - evolve when they need to handle 50+ iteration sessions, recover from API failures, and manage context across sub-agents.

Related posts:

• The Agent Execution Loop - The foundation this post builds on

• Agent Middleware - Control and observability patterns

• Context Engineering 101 - Full benchmark comparing compaction strategies

• Implementing Claude Code Skills from Scratch - On-demand context loading

• The Arc of Tool Calling - From code to tools and back

References

• Anthropic. “Claude Code.”

• Anthropic. “Compaction API Documentation.” 2026.

• Anthropic. “Effective Context Engineering for AI Agents.”

• Thariq. “Lessons from Building Claude Code: Prompt Caching Is Everything.” Feb 2026.

• Rigby. “What Actually Happens When You Run /compact in Claude Code.” 2026.

• Anthropic. “Claude Code System Prompt.” Published system prompt.

• Piebald-AI. “Claude Code System Prompts.” Extracted prompts from Claude Code releases.

• Meta AI. “Agents Rule of Two: A Practical Approach to AI Agent Security.”

• Liu et al. “Lost in the Middle: How Language Models Use Long Contexts.” 2023.

Most digital platforms today operate as two-sided markets. YouTube has creators and viewers. LinkedIn has posters and readers. Academic publishing has authors and reviewers/readers. Job boards have applicants and recruiters. Social media has people sharing life updates and friends (or strangers) consuming them.

These platforms evolved equilibrium mechanisms - rate limits, ranking algorithms, capacity constraints - all calibrated around a basic assumption: humans on both sides. Humans produce content at human speed, and humans consume it at human speed. There is an implicit contract in there that makes everything work.

As AI models become more capable (AI agents that can act with some autonomy), there is a real chance that poorly considered AI agent (assymetric) deployments break this balance. In Designing Multi-Agent Systems, I introduce the concept of agentic noise to capture this phenomena.

Agentic Noise: When AI agents accelerate one side of a platform while the other remains human-paced, breaking the assumptions underlying the system’s design.

As we all deal with how our respective domains will change with AI, this post discusses some dimensions of the relevant issues:

• What are the current examples of asymmetric AI acceleration? (job markets, academia, social media, publishing, children’s content)

• What happens when platforms respond with AI on the consumption side too? (the arms race trap - symmetric acceleration without welfare gains)

• What needs to change? (platform-level coordination mechanisms, builder responsibility for platform balance)

This post draws on ideas from Chapter 13 of Designing Multi-Agent Systems, which covers agentic noise, platform imbalance, distributed responsibility, emergent risks, and practical security frameworks for autonomous agents. Grab the book on Amazon or get the digital PDF.

Examples of Producer / Consumer Imbalance

Each example below follows the same pattern: agents accelerate one side, the other side scrambles to respond, and the resulting race inadvertently degrades the platform.

Job markets. AI application tools let candidates submit hundreds of tailored resumes per day. Recruiters responded with AI screening - and 19% report it has filtered out qualified applicants. The arms race feeds itself: applicants use AI to tailor resumes to job descriptions, so employers receive resumes that are increasingly similar, disrupting matching tools designed to differentiate them. The result is a trust/signal breakdown on both sides, with a federal class-action lawsuit (Mobley v. Workday) now testing whether AI screening tools can be considered legal “agents” of the employer. Some employers are eliminating resume requirements altogether, shifting to skills-based hiring because AI has rendered polished documents indistinguishable (low signal).

Academic research. I wrote about this in detail - the numbers tell the story. NeurIPS submissions jumped from ~9,500 in 2020 to over 23,000 by 2025. Systems like AI Scientist-v2 have already produced papers that passed peer review at top conference workshops. And now the review side is AI-accelerated too: at ICLR 2026, 21% of peer reviews were found to be fully AI-generated, with over half showing some AI use. Both sides are now agent-driven: agents writing papers, agents reviewing them. Meanwhile, organized “paper mills” use AI to generate fraudulent publications at industrial scale, and AI detection tools struggle to keep pace.

Social media and content platforms. LinkedIn feeds fill with AI-polished posts. YouTube surfaces AI-generated videos optimized for engagement but empty of substance. Platforms have responded with AI moderation - and YouTube’s version has become a cautionary tale. In 2025, YouTube terminated 12 million channels via automated enforcement. Legitimate creators get caught in the dragnet - tech tutorials flagged for using words like “bypass,” entire channels terminated with template responses - while sophisticated bad actors game the system.

Book publishing. As the author of Designing Multi-Agent Systems, this one has been rather interesting! Several bestselling books in my category on Amazon have the veneer of legitimacy - hundreds of reviews pushing them to the top of rankings. However, a cursory review of the 1-2 star reviews and you find a pattern: multiple independent readers calling out the same substantive problems. Not cosmetic complaints (”packaging was damaged”) but correlated indictments of actual content quality, often labelled as AI slop. These books sit at #1 while readers call them garbage.

Note: I think elegant writing and technical content can be authored with AI. But asking ChatGPT to generate pages of disconnected random content bundled together as a book is what readers are increasingly concerned about. I have done some deeper research on this topic and will share more soon.

Children’s content. I have a six-year-old son and he does get some screen time. Even on YouTube Kids - supposedly age-appropriate - there are hours of AI-generated videos. Knock-off Pokemon content with no storyline, no lesson, no careful design. Just visually addictive spinoffs engineered to capture attention rather than deliver entertainment or learning. Researchers have dubbed it “Elsagate 2.0” - AI-generated content targeting children at scale. Over 20% of videos recommended to new YouTube users are AI slop, with slop channels collectively accumulating over 63 billion views. Early-childhood experts warn that AI slop distorts children’s developing understanding of truth and reality. YouTube CEO Neal Mohan has acknowledged this as a 2026 priority - but many families have already moved to subscription services like Disney+ to avoid the recommendation engine entirely.

GitHub / OSS software. Most OSS repo report 100s of drive by commits with proposals like vouch - A community trust management system to ensure only individuals with some level of established trust can contribute to OSS project. Its a bit ironic. OSS helped jump start many a career - everyone has that first contributor story. With eroding trust

The Broken Signals Problem

In many of the examples above, there is the feeling that platforms should step up and address the issue. A key challenge here is that metrics we relied on to navigate the internet are now broken.

Views can be bot-driven. Comments can be AI-generated. Reviews can be manufactured at scale. The signaling infrastructure that humans built to navigate content is can now be gamed at machine speed.

This also creates an asymmetry in awareness. As someone who works in AI, I can often spot AI-generated responses to my LinkedIn posts. I can identify AI-generated YouTube videos and choose to save my attention. This is not necessarily the case for the majority of internet netizens today (kids inclusive). Their attention is consumed without their knowledge or consent. They engage with content that looks real, that has the social proof of engagement, but that was never created with their interests in mind.

The platforms themselves face a fundamental measurement problem. Their ranking algorithms optimize for engagement metrics that agents can now manufacture. The internal signals that determine what gets recommended to humans are being corrupted at the source.

Symmetric Acceleration Doesn’t Fix It

Even when symmetric acceleration works, it doesn’t necessarily improve human welfare. If AI agents write papers and AI agents review them, does that produce better research - or just higher-volume churn? When the credential was designed to signal human expertise, what does it mean when neither expertise nor effort is required to obtain it?

Software engineering is the latest example. AI coding agents generate pull requests at scale, making human code review a bottleneck. Anthropic’s response: Claude Code Review, a multi-agent system that reviews AI-generated code for $15-25 per PR. As PC Gamer put it: “so you don’t even need to check all of your own AI slop.” AI writes the code, AI reviews the code, and the human pays $25 to be removed from the loop.

At some point - AI generating content, AI generating comments, AI recommending it to other AI - all human value is lost. The question isn’t whether we can automate both sides, but whether we should.

What Needs to Change?

There are no clean answers here. But a few directions seem clear.

For platforms: Engagement metrics are no longer reliable quality signals, but the alternatives are hard. “Provenance-aware ranking” sounds good until you remember that reliably detecting AI-generated content is an unsolved problem. Mandatory disclosure (as the EU AI Act attempts) shifts the burden but depends on enforcement. Some platforms have natural levers - conferences can perhaps raise submission fees, appointment systems can enforce identity verification - but open platforms like job boards and social media face genuine design tradeoffs where every intervention has collateral damage.

We’ve solved versions of this before. Spectrum allocation and robocall regulations both exist because unconstrained automated use degraded shared infrastructure. As I argue in the book:

From Designing Multi-Agent Systems, Chapter 13:

“When collective deployment degrades platform value for everyone, coordination mechanisms like rate limiting or platform quotas may become necessary, similar to spectrum allocation or robocall regulations.”

Agentic noise is the same class of problem, and it may require the same class of solution.

For builders of AI agents: Consider the impact on platform balance, not just the business value to the producer side. This is admittedly hard to operationalize - no individual organization can fix a collective action problem through restraint alone. But at minimum, asking “what happens to the humans on the consumption side when we deploy this?” should be part of the design process.

The excitement around agent capabilities is warranted. But what is the point of a #1 bestseller that readers call garbage? What is the value of a YouTube Kids feed full of AI-generated content no human designed for children? What do we gain from academia overrun by AI-generated and AI-reviewed papers?

Read more

Your agent completed the task. It read 12 files, traced 3 call stacks, made 15 LLM calls, and eventually found the bug. But it also burned through 120,000 tokens to get there - roughly $1.80 on a frontier model. Run that 50 times a day across a team, and you’re looking at $2,700/month on a single agent workflow.

Most benchmarks for general-purpose agents like Claude Code and GitHub Copilot focus on task completion - did it get the job done? But in practice, how it got the job done matters just as much. The agent that reads every file in the directory when it only needed three. The agent that compacts its context, drops critical information, then re-reads the same files. The agent that carries 50,000 tokens of stale tool results into every LLM call because nothing told it to forget. These agents might succeed, but they succeed expensively - and for indie developers watching their API bill, or businesses looking to run agents at scale, context management can reduce token costs significantly, though often with quality tradeoffs that need to be understood.

Context in agents is cumulative - every message, tool result, and model response from previous steps gets carried forward into each new LLM call. As underlying models improve, agents can work on longer-horizon tasks (METR benchmarks show frontier models completing tasks equivalent to ~70 hours of human work at 80% reliability, with task complexity doubling roughly every 7 months). But longer tasks mean more accumulated context, and that creates three problems: the context window fills up and requests get rejected, token costs scale with context size, and model performance degrades as context grows (Liu et al., 2023).

Context engineering is the discipline of managing what goes into the context window. While we see the occasional “Claude is compacting” message, or the post on how note taking or even skills can help with context, the exact details on how to implement these strategies or how they impact performance are often vague. In this post, I break down the core strategies for context engineering, how to implement them, and their tradeoffs based on a benchmark I ran across five agent configurations.

TLDR; this post covers:

• The problem of context explosion in multi-step agent tasks

• Three core strategies for context engineering: compaction, isolation, and agentic memory

• A benchmark comparing these strategies on a code review task, showing the tradeoffs between token cost

• Key takeaways for when and how to apply context engineering in your agents

Note: For low complexity tasks, context engineering is less critical. Premature optimization can also hurt performance.

This post is adapted from my book Designing Multi-Agent Systems, where I cover context engineering patterns in Chapter 4 alongside agent architecture, memory systems, and multi-agent coordination.

More Context ≠ Good Context

To understand why context engineering matters, the critical first step is to build up the the right intuition for how context grows during agent execution. You can do this via context inspection.

Importantly, recall the agentic loop - agents address tasks by making LLM calls, interpreting the response, calling tools, and repeating. By default, all previous messages and tool results get appended to the context window for every new LLM call.

For example, here’s what happens to a naive agent running a multi-step research task. The agent appends every message and tool result to its context:

Iteration 1:    888 tokens  (system + user message)
Iteration 2:  3,400 tokens  (+ list_directory result)
Iteration 3:  8,900 tokens  (+ read_file: context.py)
Iteration 4: 14,200 tokens  (+ read_file: compaction.py)
Iteration 5: 18,900 tokens  (+ grep + read_file: _agent.py)

Token growth from a PicoAgents benchmark run (debug_investigation task, gpt model, no compaction). Each iteration adds tool results to the message history.
Interesting, the problem isn’t just the hard limit on what fits in the context window (most LLMs now support very large windows). Rather, research shows LLMs struggle with large contexts even when they technically fit:

• Lost in the middle: Information in the middle of long contexts gets less attention than information at the beginning or end (Liu et al., 2023)

• Performance degradation: Accuracy can drop significantly based on how information is positioned, not just whether it fits

• Working memory: The context window functions as the agent’s working memory. But unlike RAM, adding more data can actively degrade retrieval of existing data. Context engineering curates what stays in that window

More context doesn’t always mean better results.

What Is Context Engineering?

Context engineering is a set of techniques for managing what goes into the context window, as the agent runs. It answers four key questions:

1. What to include - Selecting relevant information

2. What to exclude - Trimming noise and redundancy

3. How to represent it - Compressing information efficiently

4. Where to put it - Positioning for model attention

Andrej Karpathy describes it as “the delicate art and science of filling the context window with just the right information for the next step.” Many agent failures trace back to context problems; the model had the right capabilities, but the wrong information was in (or missing from) the context window.

Three Core Strategies

We can categorize context engineering techniques into three core strategies:

Strategy 1: Compaction (Reactive Trimming)

Compaction reduces context by trimming, summarizing, or selectively retaining messages. It happens just before an LLM call in the agentic loop, triggered by a condition.

From the agent user’s perspective, you configure a context strategy when creating the agent:

from picoagents import Agent
from picoagents.compaction import HeadTailCompaction, SlidingWindowCompaction

# Agent with head+tail compaction - keeps task definition + recent work
agent = Agent(
    name="investigator",
    instructions="Investigate the codebase and find the bug.",
    model_client=model_client,
    tools=[read_file, list_directory, grep],
    compaction=HeadTailCompaction(
        token_budget=100_000,  # Compact when context exceeds this
        head_ratio=0.2,        # 20% for task context, 80% for recent work
    ),
    max_iterations=20,
)

response = await agent.run("Find why the auth middleware fails on refresh tokens.")

Under the hood, the agent loop applies this strategy before each LLM call:

while not done:
    # Check if compaction needed before calling the model
    if context_exceeds_budget(messages):  # e.g., > 80% of limit
        messages = compact(messages)

    response = call_llm(messages)
    # ... handle response, execute tools, append results

If you’ve used coding agents (Claude Code, GitHub Copilot CLI, etc.), you may occasionally see the agent pause, mention “compacting”, summarize prior steps, and continue. This the compaction strategy. The agent hit a threshold (e.g., 80% of the context window, or a hard cap like 100k tokens) and triggered compaction before the next LLM call. The budget determines when to compact; the approaches below determine how:

Sliding Window: Keep system message + most recent messages that fit in budget. Simplest approach.

# From picoagents/compaction.py - SlidingWindowCompaction.compact()
def compact(self, messages):
    current_tokens = self._count_tokens(messages)
    if current_tokens <= self.token_budget:
        return messages  # No compaction needed

    groups = self._find_atomic_groups(messages)  # Keep tool calls with results

    # Always keep system message
    system_groups, system_tokens = [], 0
    if groups and messages[groups[0][0]].role == "system":
        system_groups.append(groups[0])
        system_tokens = self._count_tokens([messages[i] for i in groups[0]])
        groups = groups[1:]

    # Fill from end with remaining budget
    remaining_budget = self.token_budget - system_tokens
    kept_groups, kept_tokens = [], 0

    for group in reversed(groups):
        group_tokens = self._count_tokens([messages[i] for i in group])
        if kept_tokens + group_tokens <= remaining_budget:
            kept_groups.insert(0, group)
            kept_tokens += group_tokens
        else:
            break  # No more room

    kept_indices = set()
    for group in system_groups + kept_groups:
        kept_indices.update(group)
    return [messages[i] for i in sorted(kept_indices)]

Head+Tail: Split the budget between head (system prompt, initial task) and tail (recent work). Drop middle messages. This preserves both the task definition and recent progress.

# From picoagents/compaction.py - HeadTailCompaction.compact()
# Split budget: 20% head (task context), 80% tail (recent work)
head_budget = int(self.token_budget * self.head_ratio)
tail_budget = self.token_budget - head_budget

# Fill head from start, tail from end, drop middle
for group in groups:
    if head_tokens + group_tokens <= head_budget:
        head_groups.append(group)
    else:
        break

for group in reversed(remaining_groups):
    if tail_tokens + group_tokens <= tail_budget:
        tail_groups.insert(0, group)
    else:
        break

Tool Result Clearing: Once a tool has been called deep in history, clear the raw result but keep the message structure. Anthropic recently launched this on the Claude Developer Platform - it’s the lightest-touch form of compaction.

Summarization: Instead of dropping old messages, compress them into a summary using a fast model. Preserves information at the cost of an extra LLM call. It can also lack resolution - the summary may omit details needed later. In the benchmark described below, there is the concept of “thrashing” - the agent repeats steps because the compaction strategy dropped information it needed, causing it to re-read files or re-run tools.

Semantic Selection: Use embeddings to select contextually relevant messages rather than just the most recent. More expensive but can retain relevant older information that sliding window would drop. LangGraph and Mem0 provide implementations.

Limitation: All compaction is reactive - context grows, then you trim. Even semantic selection responds to accumulated context. If the agent needs something that was trimmed or poorly summarized, that information is gone.

Strategy 2: Isolation (Architectural Prevention)

Run sub-tasks in separate contexts. Only the summary of steps enters the main agent’s context.

# Define a sub-agent with its own tools and compaction
sub_agent = Agent(
    name="code_reviewer",
    description="Reviews code in a directory",
    instructions="Read all .py files and document classes and functions.",
    model_client=model_client,
    tools=[read_file, list_directory],
    compaction=HeadTailCompaction(token_budget=50_000, head_ratio=0.2),
    max_iterations=20,
)

# Wrap it as a tool — coordinator delegates, sees only the summary
coordinator = Agent(
    name="coordinator",
    instructions="Delegate each directory to the code_reviewer tool.",
    model_client=model_client,
    tools=[sub_agent.as_tool()],  # Agent becomes a callable tool
    max_iterations=15,
)

response = await coordinator.run("Review the repository.")
# Coordinator context stays small; sub-agent context is discarded

How it works: A coordinator agent delegates tasks to sub-agents, each running in its own context window. The sub-agents do the heavy lifting — reading files, calling tools, accumulating context — and only the final result crosses back. The intermediate work is discarded, so the coordinator’s context stays bounded regardless of how much work the sub-agents do. In PicoAgents, as_tool() wraps any agent as a callable tool to enable this pattern.

When to use:

• Tasks involve distinct sub-problems

• Sub-tasks generate lots of intermediate context

• You can define clear interfaces (input -> output)

• You want to parallelize work (multiple specialists can run concurrently)

This same principle applies temporally - some systems restart the agent in fresh sessions, passing only a summary forward instead of the full context. RelentlessAgent takes this approach, running up to 10,000 sequential sub-sessions where each one receives only the original task plus a summary of prior work.

Strategy 3: Agentic Memory (External Storage)

Give the agent tools to explicitly manage its own memory outside the context window. The agent decides what to save, when to retrieve, and what to forget.

Anthropic calls this “structured note-taking” - the agent writes notes persisted to external storage, then retrieves them when needed. Think of it like maintaining a NOTES.md file. This enables tracking progress across complex work without keeping everything in active context.

# From picoagents/tools/_context_tools.py - TodoWriteTool
# Agent persists progress to disk, outside the context window

def _save_todos(todos):
    path = _get_todo_path()  # .picoagents/todos/session_*.json
    path.parent.mkdir(parents=True, exist_ok=True)
    data = {
        "session_id": _get_session_id(),
        "updated_at": datetime.now().isoformat(),
        "todos": todos,
    }
    path.write_text(json.dumps(data, indent=2))

# Agent calls todo_write to track progress externally
# This keeps task state out of the context window entirely

Key difference from compaction: Compaction trims what’s already in context. Agentic memory moves information outside the context entirely - the agent retrieves it on demand.

Agent Skills work the same way but for procedural knowledge instead of facts. A skill description costs ~100 tokens at startup; the full instructions (~2,000 tokens) only enter context when the task matches. Agentic memory stores what the agent learned (facts, findings, progress); skills store how the agent should act (procedures, workflows). Both keep information outside the context window until it’s needed. I walk through implementing skills from scratch in a separate post.

When to use:

• Long-running tasks where information needs to persist

• Tasks where the agent needs to selectively recall specific facts

• You want the agent to decide what’s worth remembering

Benchmark Comparison

To illustrate the tradeoffs, I ran a code review task across five agent configurations using PicoAgents’ evaluation system. The task: review a 44-file Python repository, read every .py file, document all classes and functions, and produce a quality assessment. All agents use the same model (gpt-4.1-mini), the same two tools (read_file, list_directory), and the same system prompt. The only variables are context strategy and whether a completion hook is enabled.

The five configurations:

• No Compaction: Full context carried forward, no trimming. Completion hook forces the agent to keep working until the task is done (up to 50 iterations).

• HeadTail 8k: Head+tail compaction with an 8k token budget (20% head, 80% tail). Same completion hook.

• HeadTail 8k (no hook): Same 8k compaction, but no completion hook; the agent stops when it does not call any additional tools.

• HeadTail 15k: Head+tail with a larger 15k token budget. Completion hook enabled.

• Isolation: A coordinator delegates to sub-agents via as_tool(). Each sub-agent has its own 50k HeadTail budget. No completion hook on the coordinator.

The completion hook fires when the agent would normally stop (no more tool calls). It summarizes the conversation so far, asks a judge LLM “is this task complete?”, and if not, injects a message telling the agent to continue. This gives the agent cognitive durability — the ability to persist through long tasks — but adds iterations (and tokens) each time it restarts.

A note on evaluation: Quality scores come from an LLM-as-judge, which introduces its own variance. Building confidence in these numbers requires inspecting the judge’s rationale for each score, ensuring the judge sees the full agent trace, and providing ground truth references (e.g., the repository has 44 .py files; did the agent find them?). With single runs, treat the scores as directional signals, not precise measurements. I’ll cover LLM judge design in detail in a future post.

Several patterns emerge:

No compaction works, but it’s expensive. NoCompaction scored highest (6.0) by brute-forcing - carrying full history across 50 iterations, 915k tokens. The agent never forgets what it read, so it avoids re-reads (only 27% duplication). The tradeoff is cost: 2-6x more tokens than compacted agents, and 22 minutes of wall time.

Read more

Agents can act by calling tools, executing code, or following instructions, but doing so efficiently has massive implications for context window usage and security. One approach now supported by Claude Code, Cursor, Gemini CLI, and a growing list of coding agents is skills. A skill is a folder of instructions and scripts that an agent can discover and load on demand. Unlike a CLAUDE.md or system prompt that sits in context permanently, skills extend context only when needed - the agent sees a one-line description at startup (~100 tokens) and pulls in full instructions only when the task matches (~2,000 tokens). With 20 skills, that's ~2,000 tokens at startup instead of ~40,000. You define a SKILL.md file once, and any compatible agent can use it.

Importantly, this pattern is now an open standard adopted by 25+ tools - you can define a skills folder with a SKILL.md file once, and any compatible agent can use it.

How does it work under the hood? And how would you implement skills support to your own agent (in any framework of your choice)? Well, that’s what this post attempts to lay out.

I will also share a complete implementation of a SkillsTool in PicoAgents, the companion framework from Designing Multi-Agent Systems book. If you are interested in learning about how agents or agent frameworks work, consider grabbing a copy of the book!

This post is adapted from Designing Multi-Agent Systems. The book covers tool design, context management, and agent orchestration patterns in more depth with complete implementation code.

What Is a Skill?

A skill is a folder containing a SKILL.md file with two parts: YAML frontmatter (metadata) and markdown content (instructions).

Here’s a commit skill based on Claude Code’s built-in skills:

---
name: commit
description: Create a git commit
allowed-tools: Bash(git add:*), Bash(git status:*), Bash(git commit:*)
---

## Context

- Current git status: !`git status`
- Current git diff: !`git diff HEAD`
- Current branch: !`git branch --show-current`
- Recent commits: !`git log --oneline -10`

## Your task

Based on the above changes, create a single git commit.
Stage and create the commit using a single message.
Do not use any other tools or do anything else.

That’s the entire skill. A skill is:

1. Metadata (frontmatter) - name, description, and optionally which tools it needs

2. Instructions (body) - what the agent should do when the skill is activated

The directory structure looks like this:

skills/
├── commit/
│   └── SKILL.md
├── code-review/
│   ├── SKILL.md
│   ├── examples/
│   │   └── sample-review.md
│   └── scripts/
│       └── diff-analyzer.py
└── debug/
    └── SKILL.md

Each skill is self-contained. The SKILL.md is required; everything else - scripts, templates, reference docs - is optional.

Instructions vs Skills vs Tools

While all three are ways to extend what the agent can do, there are differences in terms of when they are loaded, the cost in terms of context usage and execution patters. Examples of each

• Instructions: “always do X.”;

• Tools: get_weather(city).

• Skill: “when the user asks for a code review, follow this checklist and use these tools.”

Skills are useful when a task requires deep domain knowledge or multi-step orchestration that the agent may or may not encounter during a run - too detailed and specialized for primary context, but too complex for a single tool call. They sit between global instructions and atomic tools: task-specific prompts that load into context only when relevant.

Skills can also bundle scripts, reference docs, and assets alongside the SKILL.md - the instructions tell the agent when and how to use them.

Progressive Disclosure

Skills use a three-tier loading strategy to manage context efficiently. From the Agent Skills specification:

1. Metadata (~100 tokens): The name and description fields are loaded at startup for all skills

2. Instructions (< 5,000 tokens recommended): The full SKILL.md body is loaded when the skill is activated

3. Resources (as needed): Files in scripts/, references/, or assets/ are loaded only when required

This is progressive disclosure applied to agent context. At startup, the agent knows what skills exist (just names and descriptions) as these are added to the context on load. Only when a skill is needed does the full content load into the conversation.

From Claude Code’s documentation:

“In a regular session, skill descriptions are loaded into context so Claude knows what’s available, but full skill content only loads when invoked.”

Why does this matter? Because context is expensive. If you have 20 skills averaging 2,000 tokens each, eagerly loading all of them would consume 40,000 tokens before the agent does anything. With progressive disclosure, you spend ~2,000 tokens on metadata (100 tokens x 20 skills) and only load the ~2,000 tokens of instructions for the skill that’s actually needed.

How Skills Are Implemented

From the Agent Skills integration guide:

Skills have two parts: metadata injection and a load tool.

At startup, skill names and descriptions are injected into the system prompt so the model knows what’s available:

<available_skills>
  <skill>
    <name>code-review</name>
    <description>Review code changes for bugs, security issues, and improvements</description>
  </skill>
  <skill>
    <name>debug</name>
    <description>Systematic approach to debugging errors and unexpected behavior</description>
  </skill>
</available_skills>

The model also gets a skills tool (or Skill in Claude Code) with a load operation. When a task matches a skill, the model calls skills(action='load', name='code-review') and gets the full SKILL.md body back as a tool result - instructions that now appear in context for the model to follow.

Implementing Skills in Your Agent

The implementation has three parts: a SKILL.md parser, a discovery mechanism, and a tool the agent can call.

Parsing, Discovery, and the Skills Tool

The SKILL.md format is what we saw above - YAML frontmatter + markdown body. The implementation needs to parse that, scan directories for skill folders, and expose a tool the agent can call. In pseudocode:

class SkillsTool:
    def discover(skills_path) -> {name: (path, metadata)}:
        # scan directories for folders containing SKILL.md
        # parse each: split on "---", extract name + description

    def get_system_prompt_section() -> str:
        # inject skill metadata into system prompt at startup
        # "Available Skills: code-review: Review code changes..."

    def execute(action, name=""):
        if action == "load":
            return full SKILL.md body  # ~2,000 tokens, loaded on demand

The full implementation is in SkillsTool (~80 lines of Python).

Wiring It to an Agent

The tool gets added to the agent like any other tool. At startup, get_system_prompt_section() appends skill metadata to the agent’s instructions so the model knows what’s available without an extra tool call:

from picoagents import Agent
from picoagents.tools import SkillsTool

skills_tool = SkillsTool(
    builtin_path=Path("./skills"),         # shipped with your app
    project_path=Path("./.claude/skills"),  # project-specific
)

agent = Agent(
    name="assistant",
    instructions="You are a helpful assistant.\n" + skills_tool.get_system_prompt_section(),
    model_client=client,
    tools=[skills_tool, *other_tools],
)

When the user says “review this code,” the model sees code-review in its system prompt, calls skills(action='load', name='code-review'), gets the full instructions back as a tool result, and follows them.

What This Looks Like in Practice

A concrete trace:

1. Agent starts. Skill metadata is injected into the system prompt (~100 tokens per skill). The model sees skill names and descriptions from the first turn.

2. User asks: “Can you review the changes I made?”

3. Model loads the matching skill. It already knows code-review exists from the system prompt. Calls skills(action='load', name='code-review'). Gets back the full review checklist - correctness, security, performance, maintainability - plus the review format template (~1,000 tokens).

4. Model follows the instructions. It reads the diff, checks each file against the checklist, and produces a structured review with severity levels.

The load in step 3 only happens when needed. If the user had asked a simple question, no skill would load and those ~1,000 tokens are never spent.

Read more

General-purpose coding agents have shown remarkable versatility, moving well beyond coding and into multiple domains. Claude Code now powers everything from software engineering to data analysis to desktop automation. OpenAI’s Codex has evolved from a cloud coding agent into a full desktop app with Skills, Automations, and multi-channel integration. Both companies are aggressively embedding these tools into enterprise workflows - Anthropic through partnerships with Accenture and Goldman Sachs, OpenAI through deep OS-level integration.

And then there’s OpenClaw. In the span of a few weeks, OpenClaw, the open-source personal AI agent created by Peter Steinberger (originally called Clawdbot, briefly MoltBot), went from an indie project to 190,000+ GitHub stars, deployments across Silicon Valley and China, and a social network built entirely by and for AI agents (MoltBook, with over a million AI agents). On February 14, Steinberger announced he’s joining OpenAI and moving the project to an open-source foundation.

All of this suggests we’re at an inflection point.

Naturally, for many teams, there are real concerns: How do we respond to this? How disruptive is this for our business? Should we ship capable general-purpose agents as part of our product?

TLDR; You probably shouldn’t ship them as your product - but you should absolutely use them to build your product (adds security issues, often your business goal can be accomplished without the full agent capabilities). Here’s why, and what your team should do instead - use them to build faster, ship agentic workflows with deterministic steps (not fully autonomous agents) as part of your product.

This post is adapted from themes in Designing Multi-Agent Systems. The book covers agentic workflow design, orchestration patterns, and evaluation in depth.

Two Problems With Deploying General-Purpose Agents

The common thread across Claude Code, Codex, OpenClaw, and the rest is that they’re tools designed to work for an individual user and enable individual productivity. The user and the agent share the same goals and risk profile. That’s exactly what breaks when you try to make the agent serve customers on behalf of a business.

The Security Problem

This is often under-discussed. Tools like Claude Code, Codex, and OpenClaw operate within the boundary of a trusted local environment. They have access to your file system. They can run arbitrary commands. They can read your email, manage your calendar, browse the web on your behalf. That’s where the magic comes from.

Trying to replicate that magic as part of a service is a fundamentally different proposition. If you sandbox the agent, you cut off the access that makes it useful. Meta’s Agents Rule of Two - inspired by Chromium’s security policies - offers a useful framework: until robust defenses against prompt injection exist, agents should satisfy no more than two of these three properties:

• Process untrustworthy inputs - data from unknown sources (user prompts, emails, web content)

• Access sensitive systems - production databases, credentials, personal data

• Change state or communicate externally - send emails, execute transactions, modify files

General-purpose agents inherently want all three. (I covered the security implications of this in more detail in The Arc of Agent Action.)

Local agents get away with it because they operate inside the user’s own trust boundary. The user is the admin. The threat model is yourself. But the moment you move that agent to a server, the “user” becomes an attacker sending untrusted prompts, and the “file system” becomes your production database. The security model collapses.

OpenClaw is the clearest illustration of this tension. It's extraordinarily capable, and the security community's response has been proportionally alarmed. Palo Alto Networks called it a "lethal trifecta" of risks - access to private data, exposure to untrusted content, and the ability to communicate externally while retaining persistent memory. Censys identified over 30,000 exposed instances and researchers found hundreds of malicious skills in the marketplace. Cisco's AI security team found a third-party OpenClaw skill performing data exfiltration and prompt injection without user awareness. One of OpenClaw's own maintainers warned: "if you can't understand how to run a command line, this is far too dangerous of a project for you to use safely." The power and the danger come from exactly the same source: unrestricted access.

What makes OpenClaw especially instructive is how hard it tries to add security - and how it's still not enough. The codebase contains a 6-layer tool policy pipeline (owner-only gates, per-provider allowlists, group restrictions, Docker sandbox policies, subagent restrictions), workspace-only guards that reject file operations outside the user's directory, and factory functions that scope every tool to a specific session. Thousands of lines of security infrastructure - and researchers still found exposed instances, malicious skills, and active exfiltration in the wild.

This isn’t a new pattern. Desktop software had direct access to local hardware, memory, and the file system. When the industry moved to the web, recreating that desktop magic took years of engineering - and the web still remains more constrained because of security and sandboxing. Coding agents scaling from local to cloud will face the same slog, but harder: in traditional web apps, client and server were architecturally distinct. Agent systems collapse that separation. The model plans, executes, and decides. It is the client and the server.

You can see this concretely in Pi's agent loop - the framework underneath OpenClaw. The LLM receives the full context (system prompt, message history, all previous tool results) with no redaction. It decides which tools to call. Those tools execute with the full permissions of the process. Results flow back into the same context. There is no architectural boundary between "what can I see" and "what can I do" - the separation that made web security tractable simply doesn't exist.

The Business Specificity Problem

General-purpose agents are designed to handle a wide range of tasks. But your business is not trying to help your customer with a wide range of tasks.

If you’re a bank, you’re trying to drive sales of banking services. If you’re a tax preparation company, you’re trying to provide a better tax preparation experience. If you’re a marketing team, you’re trying to improve content quality. If you run a recommendation system, that’s a thing, and it has specific requirements.

In these cases, the economics matter enormously. A general-purpose model is paying a “cognitive tax” to maintain the ability to do anything: write a poem, debug Python, plan a trip. Your business process doesn’t need that optionality, but you’re paying for the compute to support it. A single agentic coding task can cost anywhere from $0.50 to $10 depending on complexity - Anthropic’s documentation reports Claude Code averages $6 per developer per day, with 90% of users under $12/day. Across millions of customer requests, the math gets uncomfortable fast.

To quantify: OpenClaw's system prompt builder is 678 lines of code assembling sections for 23 tools, memory, messaging, safety, sandbox, and more. A domain-specific agent needs maybe 3-5 tools and a 200-500 token prompt. That's a 4-10x overhead per request in prompt tokens alone, before the model even starts reasoning across capabilities the domain doesn't need.

Yes, inference costs are falling - roughly 10x per year by a16z’s estimate, and possibly faster on some benchmarks. But cheaper tokens don’t solve the architectural problems. The security exposure of a general-purpose agent - prompt injection risk, unpredictable tool invocations, variable data access patterns - doesn’t shrink when inference gets cheaper. Neither does the observability challenge. You still can’t predict how the agent will solve a given request, which means you can’t predict what it will cost, what it will access, or what attack surface it exposes.

The cost of a general-purpose agent isn't just inference - it's the meta-inference. OpenClaw's codebase includes a compaction system that makes its own LLM call to summarize history when context fills up, a context pruning extension for cache management, auth profile rotation across API keys, and tool loop detection. Each is an additional cost layer that exists only because the agent is general-purpose and long-running. A focused workflow with deterministic steps likely needs none of it.

The short story is: you often need to do just one thing very well. An agent that can do everything at high cost and unpredictable behavior is not necessarily a good fit for your business at scale.

What Should Your Team Do?

Start with your business problem. Are you in the business of selling personal productivity tools? If the answer is no, you probably should not be trying to ship a general-purpose agent as part of your API or customer-facing service.

But that doesn’t mean you shouldn’t use these agents. In fact, you absolutely should - as tools. As part of your internal processes. As accelerators for your engineering team. As components in your internal systems.

If you’re not in the personal productivity business, what you should be thinking about is: how can I accelerate my team to build my product? Use these tools to optimize your process. And underneath that question, there are three concrete approaches, with tradeoff implications.

• AI-Assisted Engineering - Low scale/cost (your eng team), low security exposure (local, trusted user), low predictability needed (exploration is fine)

• Internal Agents - Medium scale/cost (your company), medium security exposure (internal, controlled data), medium predictability needed (should be reliable)

• Agentic Workflows - High scale/cost (your customers), high security exposure (production data, external prompts), high predictability needed (strongly preferred)

Approach 1: AI-Assisted Engineering

You can get your teams to move faster if you invest in an agent-first approach to building software.

But the gains aren’t automatic. OpenAI’s own “harness engineering“ team built an internal product with zero manually-written code - three engineers, a million lines, 1,500 PRs in five months, 6-hour unattended agent runs. Impressive. But look at what it took: restructured documentation (”a map, not a manual”), custom linters injecting remediation into agent context, mechanically enforced architecture rules, a full observability stack agents could query directly, and Chrome DevTools wired in so Codex could drive and validate the UI. Early on, 20% of the team’s time went to cleaning up “AI slop” - until they automated that too.

The benefits are real, but they require iterative investment in tooling, architecture, and feedback loops. Go in clear-eyed. Without that discipline, you get the slop without the speed. (More on this in vibe coding with engineering discipline.)

Approach 2: Internal Agents

There’s a tier between “personal productivity tool” and “customer-facing service” that’s worth calling out: internal tools where the trust boundary is your company, not an individual user. A support agent that searches your internal knowledge base. A compliance tool that reviews contracts against your policies. A sales assistant that queries your CRM.

These sit in a sweet spot. You control the data and the environment, your users aren’t adversarial, and the scale is manageable - hundreds of employees, not millions of customers. When the agent gets something wrong at internal scale, the blast radius is small. At customer scale, it isn't. Many companies will find their first production agent deployments here - not in their product, but in their operations.

Approach 3: Agentic Workflows

This is where I think there has been the most value. Take a process that exists today, often manual, often with a lot of humans in the loop. Figure out which parts an agent can handle. Figure out which parts are well-understood and deterministic, and keep them deterministic. Represent the entire pipeline as a workflow, a typical software engineering workflow. Where you can remove a human step, carefully replace it with an agent. And then, critically, track cost/performance.

Here’s a concrete example. Say you’re processing insurance claims. Don’t wrap Claude in a container and point it at the whole process. Instead, decompose it: document extraction is deterministic (OCR, structured parsing, well-solved problems). Classification of the claim type uses a small, fine-tuned model. Routing follows explicit business rules. Maybe one step - say generating a summary of a complex medical narrative - genuinely benefits from a frontier model. You bundle that entire pipeline into an API and deploy it with careful observability.

I cover the trade-offs between deterministic workflows and autonomous agents in Chapter 2: Multi-Agent Patterns, walk through building these pipelines in Chapter 6: Building Multi-Agent Workflows, and strategies for making smaller models replace expensive ones in Chapter 11: Optimizing Multi-Agent Systems, in Designing Multi-Agent Systems.

Why not just let Claude handle the whole thing end-to-end? It might succeed 90% of the time if you are using the latest best in class models. But how it succeeds is unpredictable. It might rewrite its approach between runs. It might select different tools each time. Even when outputs are correct, token consumption varies wildly, and you’ve increased your security exposure (prompt injection risk, data access patterns, tool invocations) far beyond what a deterministic workflow requires. Correctness isn’t the only thing you need at scale. You need predictability. You need to know what your system is doing, how much it costs per request, and what attack surface you’re exposing.

You can put a workflow like this in place as a first step, and over time, as you see results, you optimize. Optimization means you generate training data, you develop good evals, and then you figure out how to replace some of the large-model-powered steps with smaller, fine-tuned models. You compress cost while preserving quality.

Parting Thoughts

IMO, the path forward is not “ship a general-purpose agent to your customers.” This should not be your approach for customer facing products. It’s: use these remarkable tools to build your product better, faster, and smarter, and architect your customer-facing systems with the right balance of agentic capability, deterministic reliability, and cost efficiency.

General-purpose coding agents are genuinely transformative - for the people using them. The mistake is assuming that what works on your laptop will work the same way in your cloud, for your customers, at your scale. The desktop-to-web transition took a decade of engineering to get right. The local-to-cloud transition for agents is just getting started.

Victor Dibia is a Principal Research Software Engineer at Microsoft Research and Core AI, and the author of Designing Multi-Agent Systems - available in print on Amazon and in digital formats with lifetime updates at buy.multiagentbook.com. He is the creator of AutoGen Studio and a core contributor to AutoGen and Microsoft Agent Framework. Subscribe to his newsletter at newsletter.victordibia.com.

TLDR; It’s true - direct scaling is showing signs of diminishing returns. However, a missing part of the debate is that massive scale is perhaps unavoidable. Rather, large base models are the critical infrastructure required to distill, train, and validate the efficient small models (or alternate architectures) of the future. In other words, efficiency gains we see today, including distillation, synthetic data, RLHF, and GRPO, are not alternatives to scaling. They are dividends from the scaling investments that preceded them.

As investments in AI explode (Big Tech expected to spend 300 Billion on AI in 2025), data centers are built, and most industry labs seem entrenched in the view that more GPUs is better, there is a natural question: is the quest to throw compute at the problem valid? Or better still, should it be the only approach?

There seem to be two camps. On one side are folks who agree that scaling has got us this far when nothing else did, and will likely be the gift that keeps on giving. A faction of the bitter lesson believers [1]. On the other are those who argue it’s a dead end, either due to the clear unsustainability of it all, or because there are fundamental flaws in the current transformer architecture itself that mean we can’t get much farther than we are today. This second camp often points to new approaches, such as world models and alternative architectures, as the path forward, or point to emerging cracks or instability in the scaling laws.

As I think about both, I am left with the idea that there is some nuance that is worth clarifying. We probably should not try to scale ad infinitum, but at the same time, scale is unavoidable in that its more of a design feature than a bug.

On the Slow Death of Scaling (By Sara Hooker)

This post is in part a reflection inspired by Sara Hooker’s essay, On Slow Death of Scaling [2], where she argues that recent developments show cracks in the scaling laws that have enabled progress thus far, and that it is time to invest in other areas to extract performance gains: data quality, improvements in system scaffolding (agents), and a focus on UX. The reasons cited include diminishing returns on scaling, unreliable scaling laws, and the availability of surgical synthetic data generation for training smaller, higher-quality models.

One of the arguments Sara makes on the unreliability of scaling laws is that they predict test loss, not downstream capabilities. When you measure what models actually do, the scaling law predictions sometimes break down, which means companies betting everything on scale are probably under-investing elsewhere.

I agree with everything Sara says here (she’s great and definitely read more of her writing).

However, the minor challenge I see is that a reader of that article might walk away thinking we probably do not need the super-scaled models. I think we critically and necessarily do. And that there is a cyclic, chicken-and-egg dependency between the super-scaled models (for lack of a better term) and all of the innovations that help us move away from them.

In plain terms: small models are a derivative of the larger models, and probably will never be better than them. This follows a well-established engineering pattern.

Downstream of Scale: Why Efficient Models Require Large Ones

Several observations point to the constant need for super-scaled models - and how they are (IMO) necessary infrastructure for progress.

Smaller Models Are Distilled from Larger Ones

Knowledge distillation, training smaller “student” models to mimic larger “teacher” models, has become a cornerstone of efficient AI deployment. But the technique has an obvious dependency: you need the teacher first. Even when the goal is to just improve performance (e.g., reasoning models), you still need strong, capable base models.

DeepSeek’s work on distilling reasoning capabilities validated this directly: “reasoning patterns of larger models can be distilled into smaller models, resulting in better performance compared to reasoning patterns discovered through RL on small models” [3]. The implication is clear. You cannot bootstrap reasoning in small models without first having large models that possess it.

Synthetic Data Is Generated by Large Super-Scaled Models

The synthetic data revolution, which enables training smaller, specialized models on carefully curated generated data, depends entirely on capable generators. Research on self-improvement methods shows that when models generate their own training data, they are “only limited by the best model available” [4]. This creates a ceiling: your synthetic data is only as good as your largest model.

The industry pattern is consistent. NVIDIA’s Nemotron-4 340B, IBM’s LAB methodology, and Mixtral-8x7B all use large teacher models as the source for synthetic training data. The small, efficient models that result are derivatives of the large ones that generated their training signal.

Advances Like RLHF and GRPO Require Strong Base Models

Reinforcement learning from human feedback (RLHF) and techniques like Group Relative Policy Optimization (GRPO) have dramatically improved model capabilities. But these techniques are refinements, not foundations.

As notes in The RLHF Book: “Effective RLHF requires a strong starting point, so RLHF cannot be a solution to every problem alone” [5]. OpenAI’s research on weak-to-strong generalization found that “weak-to-strong generalization is particularly poor for ChatGPT reward modeling... naive RLHF will likely scale poorly to superhuman models without additional work” [6].

GRPO, which enabled DeepSeek-R1’s impressive reasoning capabilities, works by sampling multiple completions and using group-relative rewards. But this requires a base model strong enough that variance exists in sample quality and some samples actually reach correct answers. DeepSeek-R1-Zero, often cited as evidence that pure RL can discover reasoning, started from DeepSeek-V3-Base, a 671-billion parameter model [3, 7]. The “pure RL without supervised fine-tuning” innovation was only possible because they had already paid the massive scaling tax.

The Dependency Graph

The relationship looks something like this:

Large Scaled Models (the "inefficient" phase)
           ↓
    ├──> Distillation (requires teacher)
    ├──> Synthetic Data Generation (requires capable generator)
    └──> RLHF/GRPO (requires strong base model)
           ↓
Small Efficient Models (the efficiency gains we celebrate)

Every technique in the middle layer requires capable large models as input. Small efficient models cannot bootstrap themselves.

Even “Alternative” Architectures Follow This Pattern

One might hope that fundamentally different architectures, such as world models and JEPA, could escape this dependency. The evidence suggests otherwise.

Yann LeCun has been vocal that autoregressive language models and scaling obsession are the wrong path to advanced AI. He argues that “simply scaling the model and providing it with more data might not be a viable solution” and proposes JEPA (Joint Embedding Predictive Architecture) as an alternative focused on learning world models [8].

Yet Meta’s own V-JEPA 2 paper tells a different story. The model scales from 300 million to over 1 billion parameters, trained on more than 1 million hours of internet video [9]. The paper explicitly identifies scaling as a core ingredient: data scaling (2M to 22M videos), model scaling (300M to 1B+ parameters), and longer training (90K to 252K iterations). Most tellingly, they report that “V-JEPA 2 demonstrates a linear scaling behavior with respect to model size.”

Google’s Genie world models follow the same trajectory. Genie 2, their foundation world model, “demonstrates various emergent capabilities at scale, such as object interactions, complex character animation, physics, and the ability to model and thus predict the behavior of other agents” [10]. Google’s job postings for the team state plainly: “We believe scaling [AI training] on video and multimodal data is on the critical path to artificial general intelligence” [11].

The architecture changes. The scaling dependency doesn’t. Or maynot.

The Engineering Pattern

Personally, I think this is how OpenAI has been able to improve both performance and cost over the last several years (caveat, this is speculative), by amortizing their scaling investments across increasingly efficient derivative models. It may be unclear to most, but the price of the best quality intelligence has dropped 400x in the last 4 years. Let that sink in.

This follows an engineering pattern we see across industries. You build the big, expensive, “inefficient” version first, with clear intent, and then do the work to make it small. Early CPUs were room-sized; now they fit in your pocket. Early aircraft were massive and inefficient; optimization came later. The brute-force version comes first, optimization follows.

What’s unique about ML is that the brute-force version doesn’t become necessarily obsolete in the near term. It remains operationally necessary as infrastructure for producing the optimized versions.

Conclusion

The efficiency gains we see today, including distillation, synthetic data, RLHF, and GRPO, are not alternatives to scaling. They are dividends from the scaling investments that preceded them.

It is unclear that even new architectures will allow us to escape this cycle. Even with the promise of world models, they appear to follow this arc as well. And maybe that’s fine. What might be needed is a commitment to the shrinking pattern that most industries are accustomed to: build the big version (with clear intent), then do the work to make it small.

The bitter lesson isn’t dying. It’s being amortized.

References

[1] Sutton, R. (2019). The bitter lesson. Incomplete Ideas. http://www.incompleteideas.net/IncIdeas/BitterLesson.html

[2] Hooker, S. (2024). On the Slow Death of Scaling. arXiv preprint. https://arxiv.org/abs/2407.05694

[3] DeepSeek-AI. (2025). DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint. https://arxiv.org/abs/2501.12948

[4] Alemohammad, S., et al. (2024). Self-consuming generative models go MAD. ICLR 2024. https://arxiv.org/abs/2307.01850

[5] Lambert, N. (2024). The RLHF Book. https://rlhfbook.com/

[6] Burns, C., et al. (2023). Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. OpenAI Research. https://arxiv.org/abs/2312.09390

[7] DeepSeek-AI. (2024). DeepSeek-V3 technical report. arXiv preprint. https://arxiv.org/abs/2412.19437

[8] LeCun, Y. (2022). A path towards autonomous machine intelligence. OpenReview. https://openreview.net/pdf?id=BZ5a1r-kVsf

[9] Bardes, A., et al. (2025). V-JEPA 2: Self-supervised video models enable understanding, prediction and planning. arXiv preprint. https://arxiv.org/abs/2506.09985

[10] Google DeepMind. (2024). Genie 2: A large-scale foundation world model. DeepMind Blog. https://deepmind.google/discover/blog/genie-2-a-large-scale-foundation-world-model/

[11] Kostrikov, I. (2025, January 6). Google is forming a new team to build AI that can simulate the physical world. TechCrunch. https://techcrunch.com/2025/01/06/google-is-forming-a-new-team-to-build-ai-that-can-simulate-the-physical-world/

Anthropic recently introduced Agent Skills [1]—folders containing a SKILLS.md file with instructions, scripts, and resources that agents can discover and load dynamically. The model reads the skill when relevant, executes bundled code, and solves problems using complex control flow rather than chaining discrete tool calls.

If this sounds familiar, it should. In late 2023, when I was building the first version of AutoGen Studio, this is exactly what we did. Users wrote Python files as skills, we added them to context (full code or just signatures), and the model wrote programs that integrated/orchestrated those skills. At the time, most models were inconsistent in generated structured output (the building blocks for tool calling) but mostly of them could write code fairly well.

Fun fact: One of the first agentic systems work I led at Microsoft in August 2022 (before chatgpt release, before autogen) was LIDA - an agentic workflow for automated data visualization based on code execution. The visualization module worked by generating code (using the Davinci 2 and gpt-3.5-turbo models at the time) which was executed to created visualizations.

Then everyone moved to structured tool calling, driven by libraries like Instructor (from Jason Liu) and Pydantic that promised “guarantees about output types” with automatic validation and retries. Now it’s moving back to code (citing efficiency gains).

This post traces that arc and explains why the return may or may not make sense.

This post draws on themes from Designing Multi-Agent Systems, which covers tool design, code execution, and security considerations for agents in depth.

Era 1: Action via Code

At about mid 2022, OpenAI Davinci2 models were getting really good at writing code and it became clear that they could write code that got executed in a loop to solve problems. The setup was simple: prompt the model to write code in detectable blocks (triple backticks, XML tags), parse it out, execute it. This was attractive as it mostly worked across models of all sizes (for the longest time, most models just did not support json mode or structured tool calling, especially small 7B models that drove the bulk of experimentation at the time).

As an example, in the early days of AutoGen Studio, users could write a generate_image.py file with a generate_image() method. The model was prompted to write code that assumed those skills existed and could import them:

# AutoGen Studio skill file (2023)
# skills/generate_image.py

def generate_image(prompt: str, size: str = "1024x1024") -> str:
    """Generate an image using DALL-E and return the URL."""
    # ... implementation
    return image_url

The model didn’t call a tool; it wrote a program that orchestrated tools:

# Model-generated code
from skills import generate_image, save_to_disk

url = generate_image("a sunset over mountains")
save_to_disk(url, "sunset.png")
print(f"Saved image to sunset.png")

The problems:

• Unreliable. You were dependent on the model following prompt instructions. Would it use the right code block format? Would it call the skill you defined or hallucinate something else?

• Insecure. Models could write destructive code. rm -rf, data exfiltration, anything code can do. This drove the race to support sandboxed environments - Docker, WASM, etc.

• High exposure. Code interpreters were general-purpose, which meant general-purpose attack surfaces especially with prompt injections still being a really serious issue at the time (most models had zero safety fine-tuning at the time). As Hugging Face’s smolagents documentation [2] puts it: “No local python sandbox can ever be completely secure.”

Era 2: Action via Tool Calling

Structured tool calling improved things. Instead of arbitrary code, you defined discrete tools with JSON schemas. A tool did one thing with typed inputs and outputs. The model selected tools and filled parameters - a constrained problem compared to open-ended code generation.

Benefits:

• Reliability. Structured output became a solved problem. Models got good at producing valid JSON. Pydantic, Instructor and similar libraries formalized the patterns.

• Security. No sandboxes needed for the tool layer. Either your get_weather() function was secure or it wasn’t. You could audit tools independently. The blast radius was bounded.

• Clean abstraction. APIs and functions wrapped as tools felt natural.

One downside: this approach alienated smaller models. They couldn’t reliably produce structured output.

The tool use era was also driven by the introduction of MCP by Anthropic in November 2024 as an open standard for connecting AI assistants to external data systems. The core idea: write a tool once as an MCP server, and any client that supports the protocol can use it. MCP was designed to solve the “M×N problem” - if you have M AI applications and N tools, you’d otherwise need M×N integrations. MCP reduces this to M+N.

Era 3: Back to Code

Recently there’s been a pivot back to code-based action. Anthropic’s SKILLS approach is one example - folders containing a SKILL.md file plus scripts and resources that agents discover and load dynamically. Cloudflare’s Code Mode [4] converts MCP tools into TypeScript APIs that agents call via generated code. Anthropic also released an article citing efficiency gains from representing MCP tools as code that agents can compose.

What challenges are driving the code pivot?

Context explosion. Every tool call consumes tokens -the definition, the call, the result. Chain multiple tool calls and you burn through context on overhead. As Anthropic’s engineering team notes in their post on code execution with MCP [5]: “loading all tool definitions upfront and passing intermediate results through the context window slows down agents and increases costs.” Cloudflare’s benchmarks show code-based tool invocation can reduce token usage by 98.7%—from 150,000 tokens to 2,000 [4]. For long-horizon tasks, this becomes a real constraint.

Repetitive tool calling. Tool calling encourages sequential, atomic operations. Processing 100 items means 100 tool calls. The overhead compounds.

Restricted problem-solving. When the model can only compose tool calls (sequential or parallel), you’ve limited it to what you anticipated. Code allows complex control flow - loops, conditionals, error handling - to solve larger subproblems in one pass.

Vercel recently demonstrated an extreme version of this pattern [14]. They stripped their text-to-SQL agent from 15+ specialized tools down to two: bash execution and SQL execution. Instead of curated tool schemas, they gave Claude Opus 4.5 direct filesystem access to their semantic layer definitions. The claimed results: 3.5x faster execution, 100% success rate (up from 80%), and 37% fewer tokens. Their framing - “Model + file system + goal” - is radically simpler than orchestrating dozens of tools. However, the post notably omits how security is preserved. They mention “Vercel Sandbox” but never elaborate on isolation properties, prompt injection defenses, or what prevents destructive commands (perhaps they assume trusted users and correct queries from the slack users of the tool?). For anyone implementing similar patterns, these questions matter - you can only afford radical simplicity at the agent layer if you have tested, robust sandboxing underneath (a non-trivial requirement that often goes undiscussed). See the Agent Security Rule of Two framework below for one way to think about these trade-offs.

Early research signal also highlight code efficiency - Wang et al.’s CodeAct paper [6] (ICML 2024) showed that using Python code as the action format outperforms JSON-based tool calling by up to 20% in success rate across 17 LLMs tested. Code actions also required 30% fewer steps on average—which translates directly to lower token costs.

The reasons aren’t mysterious. Code has better composability (you can nest actions, define reusable functions), better expressiveness for control flow, and leverages the procedural knowledge already in the model’s pretraining data.

What Changed? Security and Infrastructure

The observant reader will notice: the security problems that pushed us away from code are still there. Models can still write insecure code. They can still be jailbroken (admittedly not as easily as before).

What’s changed:

1. Sandboxing infrastructure is now commoditized. Major providers now offer built-in code execution sandboxes. OpenAI’s Code Interpreter [10], Google’s Gemini Code Execution [11], Anthropic’s Code Execution Tool [12]. Developers flip a switch; the provider handles sandboxing. For custom solutions, gVisor and Docker are production-ready. Cloudflare’s V8 isolates launch in milliseconds [4]. Sandboxing went from “hard problem you solve yourself” to “managed infrastructure you consume.”

2. Models became less gullible (Safety Training). Significant safety training on code injection makes previous mistakes rarer. Models are better at refusing obviously malicious instructions (though dedicated adversaries can still cause problems).

3. Code is in the training data, tool calls aren’t. LLMs have seen enormous amounts of real-world TypeScript and Python in pretraining. They’ve seen far fewer examples of JSON tool-calling schemas. As Cloudflare notes [5]: code is a more natural action format for models trained on code compared to some esoteric function signature representing an MCP tool. LLMs are trained on GitHub. They are native speakers of Python/TypeScript. They are second-language speakers of your specific get_customer_data JSON schema.

4. Context preservation is now understood as critical. Research shows that LLMs struggle with large contexts even when they fit in the window. Anthropic’s context engineering guidance [8] treats the context window as “a precious resource to be managed” with techniques like compaction, structured note-taking, and multi-agent architectures. For long-horizon task solving, context efficiency matters.

Read more

AI coding agents have gotten remarkably capable. METR’s research [1] shows that the length of tasks AI agents can complete autonomously has doubled every 7 months over the past 6 years. The recently announced OpenAI’s GPT-5-Codex [2] can run autonomously for over 7 hours, handling complex refactoring and code reviews without human intervention.

How To Build Agents from Scratch
Interested in how coding agents like Claude Code work, or how to build them from scratch for your own tasks? I wrote a book - Designing Multi-Agent Systems - that covers building agents from scratch (Chapter 4), evaluation frameworks (Chapter 10), optimization strategies for the 10 common failure modes (Chapter 11), and implementing software engineering agents like Claude Code (Chapter 15).

These advances have enabled new types of experimentation, specifically - “Vibe coding” - the practice of describing what you want and letting the AI generate code based on intuition and feel - works great for prototypes and quick experiments.

And yet, many teams - especially those with early-career engineers - still hit roadblocks as soon as they try to build complex real-world applications that matter. A game app that’s deployed and distributed with auto-scaling, auth, multi-user support, and streaming. A pipeline for coherent music generation. A document processing system for enterprise grade legal review. Today, it seems that end-to-end results for tasks like this rarely come from getting your coding agent to accomplish the entire thing.

Why does this happen? Well, agents, even while capable, make common mistakes. They gloss over security vulnerabilities. They may select suboptimal architectural designs. They may fail to consider system and deployment constraints e.g., your deployment system only works with Node.js, compliance requires a specific encryption standard, your CI pipeline has particular requirements. As the tasks get more complex and longer, the agents may fail to keep up with earlier instructions as the context fills up.

This often leads to confusion and disappointment for teams who see the success stories but frequently run into these issues.

TLDR; The good news is that with the right managements and engineering discipline , you can get the right results (with productivity benefits) from AI agents.

This post highlights 5 useful things to do to get closer to this goal. This is how I use coding agents like Claude Code today, despite its quirks.

P.S. I wrote about how to build software with AI models [3] a while ago. This post is an extension that acknowledges the autonomy that agents like Claude Code, OpenAI Codex bring - they have sophisticated agentic harnesses allowing them to tackle medium to large scale tasks.

1. Share Context and Constraints

If you hired a new engineer - irrespective of their expertise - you would still need to onboard them. You’d tell them about your build system, tacit knowledge, tribal knowledge, tech stack, deployment stack, and more. For agents, you still have to do the same thing. Agents will not magically gain this knowledge. It is an opportunity to offer the agent context - what exists - your codebase structure, tech stack, design patterns, how things work today, as well as important constraints - what MUST or CANNOT happen - hard rules, compliance requirements, deployment limits, non-negotiables.

Tools like CLAUDE.md [4], and Spec-Driven Development [5] are designed to support this.

CLAUDE.md is a special markdown file that Claude Code automatically loads into every conversation—essentially an onboarding document for your agent. It’s where you codify tacit knowledge: build commands, environment setup, repo conventions, and hard-won lessons about your codebase. Keep it under 300 lines and use hierarchical files in subdirectories so the agent sees only the most relevant context.

GitHub’s Spec Kit takes a more structured approach with four gated phases: Specify → Plan → Tasks → Implement. Instead of coding first and writing docs later, you start with a specification that captures intent, then translate it into technical decisions, break it into implementable pieces, and only then let the agent code. This solves the “scattered requirements” problem—security policies, compliance rules, and design constraints get baked into the spec where the AI can actually use them, rather than living in someone’s head or buried in a wiki. The philosophy: treat coding agents like literal-minded pair programmers who excel at pattern recognition but need unambiguous instructions. GitHub

Building a Practice of Iterative Context Enrichment

As a team, consider having an architect (or senior engineer) first initialize your CLAUDE.md (it scans the codebase), then have them add useful context and update it iteratively as they work with the agent. For example, you ask the agent to build the frontend and it uses npm run build, but instead your repo is designed to use bun - you correct it and ask it to add the bun instruction to CLAUDE.md.

Context and constraints live across the lifetime of a project (or even across projects). Teams should build muscle to manage these as practices - a file system for designs, how they are organized, where specs live.

Boris Cherny, creator/ lead for claude code talks about how they author and version Claude.md for the claude code project here.

”Our team shares a single CLAUDE.md for the Claude Code repo. We check it into git, and the whole team contributes multiple times a week. Anytime we see Claude do something incorrectly we add it to the http://CLAUDE.md, so Claude knows not to do it next time. Other teams maintain their own CLAUDE.md‘s. It is each team’s job to keep theirs up to date.”

CLAUDE.md files [4] can be hierarchical - you can have a project-level file and nested ones in subdirectories. Claude prioritizes the most specific, most nested file when relevant.

For more structured approaches, GitHub’s Spec Kit [5] provides a toolkit with three phases: /specify to generate specifications, /plan to produce technical plans, and /tasks to derive actionable task lists. It’s agent-agnostic and works with Claude Code, GitHub Copilot, Gemini, and others.

2. Co-design with the Agent

I find that I often explore a problem in detail first within a design session with an agent. I ask a bunch of questions. I ask the agent to start a running doc. I iterate and improve that design doc.

For example: “I am interested in this feature - adding query filtering in the UI. Review the current implementation and propose a design that honors the design system, has minimal code changes, and reuses components. Change nothing in the code base yet.“

Sometimes it’s useful to signal to the agent directly “change nothing” to indicate brainstorm mode (some agents have specific toggles for this).

Some plan comes back. I offer corrections. I often have a running directory of such design interactions that I refine very carefully and make into specs. These design docs are referenced as specs for features, and the agent can benchmark its work against those.
And then when things are clear and well defined, I tell the agent - “hit it!”

3. Test-Driven Development

As agents take on more tasks, the chance that changes occur that you have no idea of increases. From my experience, this can be a really bad thing. It’s generally poor form to get to a meeting where there’s a feature that works, but nobody can explain how it works.

In my previous post [3], I advocate for knowledge parity - read everything. But I get it, the temptation is high. It also kinda defeats the productivity promise if you are sitting watching the agent go on a 7hr bender. Also, the bosses want to see more work done in a fraction of the time.

At minimum, if you don’t know everything, you should know/define exactly how things should work - as defined in tests.

Try to author and fully understand your tests. Add instructions to CLAUDE.md to require permissions for test changes (so the agent doesn’t delete the test to declare the task done - which it can do).

4. Checkpointing

I wrote about checkpointing previously in my last article [3], mostly focused on writing continuation summaries of what went well in long sessions. This still applies to agents - research like “Lost in the Middle” [6] shows that LLMs struggle with information in the middle of long contexts; performance forms a U-shape with best results at the beginning and end.

Checkpointing takes on more importance in agent land. An agent working for 1 hour can change hundreds of files, which can be problematic to debug if 1 out of 100 changes has issues.

I have learned to carefully isolate any working sessions with an AI agent via frequent git commits. Once a feature is done, tested, and in good condition, it’s best to check it in and version it.

Agents are good at version-based rollbacks using git. They can struggle at rollbacks within a session, manually reverting individual changes across 100s of files. It’s also great for your sanity / security - roll back on error. Good old engineering practice.

5. Code Review is for Humans

We should not need to rehash the obvious here, but code review is key.

You can have AI agent reviews - GitHub Copilot does leave decent suggestions. But they should be more FYI. We should still have engineering practices of having at least one or two reviewers to gate merging.

Importantly - research on code review shows it serves dual purposes [7][8] - defect detection - catching bugs, security issues, logic errors, but also - knowledge dissemination - ensuring multiple engineers understand the code.

Ofcourse, defect detection is still very important.

Feb 24: Clouldflare announces that a single dev rebuilt Next.js with significant improvements in 1 week. Massive internet excitement on the rise of 1-man dev teams.

Feb 27: Its revealed that there 2 critical, 2 high, 2 medium, 1 low security vulnerabilities in the framework.

Read more

In a previous post on the agent execution loop, I showed how agents work: a while loop that calls a model, executes tools, and iterates until done. That loop is the engine or harness. But engines need control routines that govern behaviour.

When you deploy an agent to production, new questions emerge:

• How do I log every model call and tool execution?

• How do I block malicious prompts before they reach the LLM?

• How do I rate limit users to control costs?

• How do I redact PII from outputs before they reach users or broader telemetry systems?

• How do I create audit trails for compliance?

One way to do this is middleware - routines that intercept agent operations at key points in the execution loop (e.g., before/after a model call, before/after tool calls etc). If you’ve built web applications with Express, Django, or FastAPI, you know this pattern. And it works just as well for agents.

Note: This post is adapted from my book Designing Multi-Agent Systems, where Chapter 4 walks you through building a complete Agent class from scratch. The agent class is part of PicoAgents : a minimal, hackable multi-agent framework which the reader gets to build across the sections of the book. While we will be using the picoagents sample in this post, the same patterns apply across frameworks like LangChain, Microsoft Agent Framework, and others (see examples at the end of the post).

• Digital PDF: buy.multiagentbook.com

• Print on Amazon: amazon.com/dp/B0G2BCQQJY

• Middleware source: picoagents/_middleware.py

What is Agent Middleware?

Middleware intercepts agent operations before and after they execute. When an agent prepares to call the LLM or execute a tool, that operation first passes through a middleware chain. Each middleware can:

1. Inspect the operation (inputs, context, metadata)

2. Modify inputs or outputs

3. Block the operation entirely by raising an exception

4. Log what happened for observability

Here’s the mental model:

User Task
    ↓
[Middleware Chain]
    ↓
Model Call → [Middleware] → LLM API → [Middleware] → Response
    ↓
Tool Call → [Middleware] → Execute → [Middleware] → Result
    ↓
Final Response

Every model call and tool execution routes through the middleware chain. This gives you a single control plane for security, observability, and policy enforcement.

While this example focuses on model and tool call operations, you can also implement middleware for other operations e.g., calls to fetch or write to memory.

The BaseMiddleware Interface

To standardize how middleware works, we define a common interface with three hooks. In PicoAgents middleware interface, we use async generators (yield) for the hooks rather than return values - this enables streaming support and event emission for observability:

from abc import ABC, abstractmethod
from collections.abc import AsyncGenerator
from typing import Any, Union

class BaseMiddleware(ABC):
    “”“Abstract base class for middleware.”“”

    @abstractmethod
    async def process_request(
        self, context: MiddlewareContext
    ) -> AsyncGenerator[Union[MiddlewareContext, “AgentEvent”], None]:
        “”“Process before the operation executes.”“”
        yield context

    @abstractmethod
    async def process_response(
        self, context: MiddlewareContext, result: Any
    ) -> AsyncGenerator[Union[Any, “AgentEvent”], None]:
        “”“Process after the operation completes successfully.”“”
        yield result

    @abstractmethod
    async def process_error(
        self, context: MiddlewareContext, error: Exception
    ) -> AsyncGenerator[Union[Any, “AgentEvent”], None]:
        “”“Handle errors from the operation.”“”
        raise error

process_request() runs before any model call or tool execution. This is where you validate inputs, start timers, or block suspicious requests. Yield the context to continue, or raise to abort.

process_response() runs after successful operations. Use it to filter outputs, cache results, or log completion times. Yield the (possibly modified) result.

process_error() handles failures. Log errors, provide fallbacks, or implement retry logic. Yield a recovery value, or re-raise to propagate.

The async generator pattern (yield instead of return) enables middleware to emit events for observability and supports streaming responses - both critical for production agents.

Example 1: Logging Middleware

The simplest and most useful middleware: log everything.

import time

class LoggingMiddleware(BaseMiddleware):
    “”“Log all agent operations with timing.”“”

    async def process_request(self, context: MiddlewareContext):
        print(f”[{context.agent_name}] Starting {context.operation}”)
        context.metadata[”start_time”] = time.time()
        yield context

    async def process_response(self, context: MiddlewareContext, result: Any):
        duration = time.time() - context.metadata.get(”start_time”, 0)
        print(f”[{context.agent_name}] {context.operation} completed in {duration:.2f}s”)
        yield result

    async def process_error(self, context: MiddlewareContext, error: Exception):
        print(f”[{context.agent_name}] {context.operation} failed: {error}”)
        raise error

Pass it to your agent:

agent = Agent(
    name=”assistant”,
    model_client=OpenAIChatCompletionClient(model=”gpt-4.1-mini”),
    instructions=”You are a helpful assistant.”,
    middlewares=[LoggingMiddleware()]
)

response = await agent.run(”What’s 2+2?”)

Output:

[assistant] Starting model_call
[assistant] model_call completed in 0.82s

Every operation is now visible. When debugging why an agent made a specific decision or measuring latency, this is your first line of defense.

Note: What is not shown above is how the middleware chain is “wired” into the agent execution loop. See the implementation of the agent class to review how model calls and tool calls are passed through a chain of middleware associated with the agent.

Example 2: Guardrail Middleware

Agents are vulnerable to prompt injection - malicious inputs that hijack the model’s behavior. A guardrail middleware can block dangerous patterns before they reach the LLM:

import re

class GuardrailMiddleware(BaseMiddleware):
    “”“Block malicious input before it reaches the model.”“”

    def __init__(self, blocked_patterns: list[str] = None):
        # Basic patterns for demonstration
        self.blocked_patterns = [re.compile(p) for p in (blocked_patterns or [
            r”ignore.*previous.*instructions”,
            r”system.*prompt.*override”,
            r”<script.*?>.*?</script>”,
        ])]

    async def process_request(self, context: MiddlewareContext):
        if context.operation == “model_call”:
            for message in context.data:
                if hasattr(message, “content”):
                    for pattern in self.blocked_patterns:
                        if pattern.search(message.content):
                            raise ValueError(f”Blocked pattern: {pattern.pattern}”)
        yield context

    async def process_response(self, context, result):
        yield result

    async def process_error(self, context, error):
        raise error

When the middleware detects dangerous patterns, it raises an exception immediately. The malicious input never reaches the model. No expensive API call is made. No problematic content enters your logs.

Important: Regex-based filtering is a first line of defense, not a complete solution. Adversarial users can easily rephrase prompts to bypass pattern matching (”disregard prior directives” instead of “ignore previous instructions”). For production systems, consider dedicated guardrail models like Llama Guard, Azure AI Content Safety, or similar classification-based approaches that understand semantic meaning rather than just matching strings.

Example 3: Rate Limiting Middleware

LLM API calls are expensive. Without rate limiting, a single abusive user can blow through your budget. Here’s a simple rate limiter middleware!

import asyncio
import time

class RateLimitMiddleware(BaseMiddleware):
    “”“Limit API calls per minute.”“”

    def __init__(self, max_calls_per_minute: int = 60):
        self.max_calls = max_calls_per_minute
        # NOTE: In production, use Redis or similar for distributed state.
        # In-memory state won’t work across multiple containers or serverless functions.
        self.call_times: list[float] = []

    async def process_request(self, context: MiddlewareContext):
        now = time.time()

        # Remove calls outside the 60-second window
        self.call_times = [t for t in self.call_times if now - t < 60]

        # If at limit, wait until oldest call expires
        if len(self.call_times) >= self.max_calls:
            oldest_call = self.call_times[0]
            wait_time = 60 - (now - oldest_call)
            if wait_time > 0:
                await asyncio.sleep(wait_time)
                now = time.time()

        self.call_times.append(now)
        yield context

    async def process_response(self, context, result):
        yield result

    async def process_error(self, context, error):
        raise error

Usage:

agent = Agent(
    name=”limited_assistant”,
    model_client=OpenAIChatCompletionClient(model=”gpt-4.1-mini”),
    instructions=”You are a helpful assistant.”,
    middlewares=[RateLimitMiddleware(max_calls_per_minute=10)]
)

Now your agent can’t be abused into making unlimited API calls. The middleware throttles requests to stay within budget.

Example 4: PII Redaction Middleware

Consider a scenario where a customer service agent handles sensitive data. A PII redaction middleware can detect and sanitize both inputs and outputs:

import re

class PIIRedactionMiddleware(BaseMiddleware):
    “”“Redact personally identifiable information from inputs and outputs.”“”

    def __init__(self):
        self.patterns = {
            r”\b\d{3}-\d{2}-\d{4}\b”: “[SSN-REDACTED]”,
            r”\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b”: “[EMAIL-REDACTED]”,
            r”\b\d{3}[-.]?\d{3}[-.]?\d{4}\b”: “[PHONE-REDACTED]”,
            r”\b\d{4}[\s-]?\d{4}[\s-]?\d{4}[\s-]?\d{4}\b”: “[CC-REDACTED]”,
        }

    def _redact(self, text: str) -> str:
        for pattern, replacement in self.patterns.items():
            text = re.sub(pattern, replacement, text)
        return text

    async def process_request(self, context: MiddlewareContext):
        # Redact PII from inputs before they reach the model
        if context.operation == “model_call” and isinstance(context.data, list):
            for msg in context.data:
                if hasattr(msg, “content”):
                    msg.content = self._redact(msg.content)
        yield context

    async def process_response(self, context: MiddlewareContext, result: Any):
        # Redact PII from outputs before they reach users
        if hasattr(result, “message”) and hasattr(result.message, “content”):
            result.message.content = self._redact(result.message.content)
        yield result

    async def process_error(self, context, error):
        raise error

Now sensitive data never leaves your system unprotected:

Input: “Customer called from 555-123-4567, email john@example.com”

Output: “Customer called from [PHONE-REDACTED], email [EMAIL-REDACTED]”

Note: For streaming responses, PII redaction requires buffering chunks until you have enough context to detect patterns - you can’t redact a partial phone number mid-stream. In PicoAgents, you can handle this via the async generator pattern, allowing middleware to buffer and transform chunks before yielding them.

How Agents Use Middleware

Internally, agents route their core operations through the middleware chain:

class Agent:
    async def run(self, task: str):
        # ... prepare messages ...

        # Model call wrapped with middleware
        completion_result = await self.middleware_chain.execute(
            operation=”model_call”,
            agent_name=self.name,
            agent_context=self.context,
            data=llm_messages,
            func=lambda msgs: self.model_client.create(msgs, tools=tools)
        )

        # Tool execution wrapped with middleware
        for tool_call in completion_result.tool_calls:
            tool_result = await self.middleware_chain.execute(
                operation=”tool_call”,
                agent_name=self.name,
                agent_context=self.context,
                data=tool_call,
                func=lambda tc: self._execute_tool(tc)
            )

The middleware_chain.execute() method calls each middleware in sequence. Each middleware receives the operation details and can decide whether to proceed, modify the data, or block entirely.

Composing Multiple Middleware

Middleware composes naturally (depending on the implementation). Order matters - middleware executes in the order you provide:

agent = Agent(
    name=”production_agent”,
    model_client=OpenAIChatCompletionClient(model=”gpt-4.1-mini”),
    instructions=”You are a helpful assistant.”,
    middlewares=[
        LoggingMiddleware(),         # Log everything first
        GuardrailMiddleware(),       # Block malicious inputs
        RateLimitMiddleware(60),     # Enforce rate limits
        PIIRedactionMiddleware(),    # Sanitize inputs/outputs
    ]
)

Request flow: Logging → Guardrail → Rate Limit → (execute) → PII Redaction → Logging

If guardrail middleware blocks a request, rate limiting and execution never happen. This short-circuit behavior is intentional—you don’t want to waste resources on blocked operations.

PicoAgents includes ready-to-use middleware for common patterns that you can learn from:
LoggingMiddleware, RateLimitMiddleware, PIIRedactionMiddleware, GuardrailMiddleware, and MetricsMiddleware. Use them directly or as starting points for custom implementations.

Same Pattern, Different Frameworks

The middleware pattern isn’t unique to PicoAgents—it’s how production agent frameworks handle control and observability. The syntax differs, but the core architecture is identical: intercept operations, optionally modify or block, then proceed.

Microsoft Agent Framework provides three middleware interfaces: AgentMiddleware (wraps agent runs), FunctionMiddleware (wraps tool calls), and ChatMiddleware (wraps LLM requests). You can define them as classes, functions, or use decorators:

from agent_framework import AgentMiddleware, AgentRunContext

class SecurityMiddleware(AgentMiddleware):
    async def process(self, context: AgentRunContext, next):
        if is_malicious(context.messages):
            context.result = AgentRunResponse(messages=[...])
            context.terminate = True  # Stop execution
            return
        await next(context)

Read more

But how do agents actually work? How can you build one?

In this post, I’ll walk through the core of how agents function: the agent execution loop that powers these complex behaviors.

Note: This post is adapted from my book Designing Multi-Agent Systems, where Part II (Chapters 4-6) guides you through building a complete agent framework from scratch.

• Digital PDF: buy.multiagentbook.com

• Print on Amazon: amazon.com/dp/B0G2BCQQJY

What is an Agent?

An agent is an entity that can reason, act, communicate, and adapt to solve problems.

Consider two questions you might ask a generative AI model like GPT5 from OpenAI or Claude from Anthropic:

1. “What is the capital of France?”

2. “What is the stock price of NVIDIA today?”

The first question can be answered by a model directly (it likely has seen instances of this specific fact/knowledge and this is now encoded in its model weights). The second cannot - the model will hallucinate a plausible-sounding but incorrect answer because it doesn’t have access to real-time data.

An agent setup solves this by recognizing it needs current data, calling a financial API, and returning the actual price. This requires action, not just text generation.

Agent Components

In simple terms, an agent has three core components:

• Model: The reasoning engine (typically an LLM like GPT-5) that processes context and decides what to do

• Tools: Functions the agent can call to take action - APIs, databases, code execution, web search

• Memory: Short-term (conversation history) and long-term (persistent storage across sessions)

Calling an LLM

Before building an agent, you need to understand how to call a generative AI language model. Here’s the basic pattern using the OpenAI API:

from openai import AsyncOpenAI

client = AsyncOpenAI(api_key=”your-api-key”)

response = await client.chat.completions.create(
    model=”gpt-5”,
    messages=[
        {”role”: “system”, “content”: “You are a helpful assistant.”},
        {”role”: “user”, “content”: “What is 2 + 2?”}
    ]
)

print(response.choices[0].message.content)
# Output: “4”

The API takes a list of messages (system instructions, user input, previous assistant responses) and returns a completion. This is a single request-response cycle.

To enable the use of tools (tool calling), you also pass tool definitions:

response = await client.chat.completions.create(
    model=”gpt-5”,
    messages=messages,
    tools=[{
        “type”: “function”,
        “function”: {
            “name”: “get_stock_price”,
            “description”: “Get current stock price for a symbol”,
            “parameters”: {
                “type”: “object”,
                “properties”: {
                    “symbol”: {”type”: “string”, “description”: “Stock symbol like NVDA”}
                },
                “required”: [”symbol”]
            }
        }
    }]
)

When the model decides it needs to use a tool, instead of returning text content, it returns a tool_calls array with the function name and arguments.

The Agent Execution Loop

Here’s the core pattern that every agent follows:

1. Prepare Context  →  Combine task + instructions + memory + history
2. Call Model       →  Send context to LLM, get response
3. Handle Response  →  If text, we’re done. If tool calls, execute them.
4. Iterate          →  Add tool results to context, go back to step 2
5. Return           →  Final response ready

In code:

async def run(task: str):
    # 1. Prepare context
    messages = [
        {”role”: “system”, “content”: self.instructions},
        {”role”: “user”, “content”: task}
    ]

    while True:
        # 2. Call model
        response = await self.client.chat.completions.create(
            model=”gpt-5”,
            messages=messages,
            tools=self.tool_schemas
        )

        assistant_message = response.choices[0].message
        messages.append(assistant_message)

        # 3. Handle response
        if not assistant_message.tool_calls:
            # No tool calls - we’re done
            return assistant_message.content

        # 4. Execute tools and iterate
        for tool_call in assistant_message.tool_calls:
            result = await self.execute_tool(
                tool_call.function.name,
                json.loads(tool_call.function.arguments)
            )
            messages.append({
                “role”: “tool”,
                “tool_call_id”: tool_call.id,
                “content”: result
            })

        # Loop continues - model will process tool results

The key insight: an agent takes multiple steps (model call → tool execution → model call) within a single run. The loop continues until the model returns a text response instead of tool calls. In some cases we might need additional logic to guide program control flow (e.g., termination conditions such as a maximum number of turns) to avoid edge cases like infinite loops with cost implications.

Tool Execution

When the model returns a tool call, you need to actually execute it:

async def execute_tool(self, name: str, arguments: dict) -> str:
    tool = self.tools[name]
    try:
        result = await tool(**arguments)
        return str(result)
    except Exception as e:
        return f”Error: {e}”

The result gets added back to the message history as a tool message, and the loop continues. The model sees what the tool returned and can either call more tools or generate a final response.

A Complete Example

Putting it together:

class Agent:
    def __init__(self, instructions: str, tools: list):
        self.client = AsyncOpenAI()
        self.instructions = instructions
        self.tools = {t.__name__: t for t in tools}
        self.tool_schemas = [self._make_schema(t) for t in tools]

    async def run(self, task: str) -> str:
        messages = [
            {”role”: “system”, “content”: self.instructions},
            {”role”: “user”, “content”: task}
        ]

        while True:
            response = await self.client.chat.completions.create(
                model=”gpt-5”,
                messages=messages,
                tools=self.tool_schemas
            )

            msg = response.choices[0].message
            messages.append(msg)

            if not msg.tool_calls:
                return msg.content

            for tc in msg.tool_calls:
                result = await self.execute_tool(
                    tc.function.name,
                    json.loads(tc.function.arguments)
                )
                messages.append({
                    “role”: “tool”,
                    “tool_call_id”: tc.id,
                    “content”: result
                })

# Usage
async def get_stock_price(symbol: str) -> str:
    # In reality, call an API here
    return f”{symbol}: $142.50”

agent = Agent(
    instructions=”You help users get stock information.”,
    tools=[get_stock_price]
)

result = await agent.run(”What’s NVIDIA trading at?”)
print(result)
# “NVIDIA (NVDA) is currently trading at $142.50.”

The agent:

1. Receives “What’s NVIDIA trading at?”

2. Calls the model, which decides to use get_stock_price

3. Executes get_stock_price(”NVDA”) → returns “$142.50”

4. Adds the result to messages, calls the model again

5. Model generates a natural language response incorporating the data

Same Pattern, Different Frameworks

The execution loop we built is the same pattern used by production agent frameworks. The syntax differs, but the core architecture is identical: define tools, create an agent with instructions, run it on a task. The code snippets below show how each of these ideas are implemented in frameworks like Microsoft Agent Framework, Google ADK, and LangGraph.

Further Reading: The GitHub repo for the book shows how the same concepts (agents, workflows, orchestrators) are implemented across each of these frameworks: github.com/victordibia/designing-multiagent-systems/examples/frameworks

Microsoft Agent Framework:

from agent_framework import ai_function
from agent_framework.azure import AzureOpenAIChatClient

@ai_function
def get_weather(location: str) -> str:
    “”“Get current weather for a given location.”“”
    return f”The weather in {location} is sunny, 75°F”

client = AzureOpenAIChatClient(deployment_name=”gpt-4.1-mini”)
agent = client.create_agent(
    name=”assistant”,
    instructions=”You are a helpful assistant.”,
    tools=[get_weather],
)

result = await agent.run(”What’s the weather in Paris?”)

Google ADK:

from google.adk import Agent

def get_weather(location: str) -> str:
    “”“Get current weather for a given location.”“”
    return f”The weather in {location} is sunny, 75°F”

agent = Agent(
    name=”assistant”,
    model=”gemini-flash-latest”,
    instruction=”You are a helpful assistant.”,
    tools=[get_weather],
)

# Run via InMemoryRunner

LangGraph:

from langchain_core.tools import tool
from langgraph.prebuilt import create_react_agent

@tool
def get_weather(location: str) -> str:
    “”“Get current weather for a given location.”“”
    return f”The weather in {location} is sunny, 75°F”

agent = create_react_agent(
    model=llm,
    tools=[get_weather],
)

result = agent.invoke({”messages”: [(”user”, “What’s the weather in Paris?”)]})

All three frameworks: define a function, wrap it as a tool, pass it to an agent, call run. The execution loop underneath handles the model calls, tool execution, and iteration.

What’s Missing

This basic loop works, but production agents need more:

• Streaming: Long tasks need progress updates, not just a final response

• Memory: Persisting context across sessions

• Middleware: Logging, rate limiting, safety checks

• Error handling: Retries, graceful degradation

• Context management: Summarizing/compacting as context grows

• Orchestrating multiple agents: Deterministic workflows and autonomous orchestration patterns (handoff, magentic one etc)

• End-user interfaces: Integrating agents into web applications

• Complete Use Cases: Building a full coding agent with file system access, code execution, and iterative debugging

These are covered in depth in my book Designing Multi-Agent Systems, which builds a complete agent framework (picoagents) from scratch with all of these features and two complete use cases.

“Hey Victor, we are cancelling your book, you will get an email.”

Emails like this are never easy to deal with. If you’ve followed some of my work, you’d know that I had started writing a book on multi-agent systems. I started working with Manning Publications in early 2024, and in July of that year I announced that the book was in early access through their MEAP (Manning Early Access Program).

I had a book cover. A discount code. Marketing plans. All the good stuff.

Fast forward to July 2025 — I got the email above. The book got cancelled.

Four months later, the book is now published — a #1 New Release on Amazon, #5 Best Seller in Generative AI as at time of writing this. The book went from 4 uploaded chapters (at time of cancellation) to complete - 15 chapters, 56,000+ words, 186 code snippets, and 51 hand-drawn figures.

• Digital edition : https://buy.multiagentbook.com/

• Print edition on Amazon: https://www.amazon.com/dp/B0G2BCQQJY

Here’s what happened.

How Does A Book Get Cancelled?

The short story is that .. life happened. And also, the process wasn’t working for me. Here’s the timeline:

• January 2024: Contract signed with Manning.

• May 2024: I fractured my wrist. A bad fall, surgery, titanium plates with screws, recovery. This is where things slowed down. Beyond the physical recovery, for a researcher or software engineer, there’s the worry about how an injury might affect your career. Luckily, I’ve made a good recovery.

• July 2024: MEAP (early access) launched with the first two chapters.

• Sept 2024 – Jan 2025: The AutoGen rewrite. This was an especially busy time at work — AutoGen was rewritten with a completely new API (released Jan 2025). I contributed to the AgentChat API, which eventually became the format that other frameworks (Google ADK, OpenAI Agents SDK) converged on, and I rebuilt AutoGen Studio on this new version. This created a problem for the book: code in chapter 1 was already obsolete before I finished chapter 2. I realized that writing a book tightly coupled to a specific framework version was a trap. I originally had a co-author, but by the time of cancellation, I was writing alone.

For new readers, I am a core maintainer of AutoGen and have written extensively about it and agents in general.

• July 2025: Cancelled. I had written 4 chapters and uploaded them to the Manning platform. But that pace was understandably slow for the publisher. They chose to cancel.

Throughout this period, I was writing — but the upload and review cycle wasn’t working for me. Every change to the table of contents had to be approved. The Manning internal review process brought feedback that often felt disconnected from improving the book. Weekly meetings with the editor ate into writing time. I spent energy conforming to the process when I should have been creating.

It took the cancellation to isolate this observation: the lack of creative control was itself a blocker. For me.

Getting Back on Track

The cancellation forced me to rethink what was going wrong. My wife pointed out something I probably knew was true — she’d seen me hammer out book-quality writing in the form of blog posts late at night during this same period (see this substack). Writing wasn’t the problem. The process was.

Luckily, it was a clean break. I hadn’t collected my advance, so all rights for written material returned to me.

I decided to focus on why I had started writing in the first place:

• There are many users new to AI who need structured guidance, not just material scattered across papers, blog posts etc.

• Long-form content and a guided teaching experience is still critical — even though deep research agents can find you whatever you want, it won’t be in a coherent, graduated progression

• There is lots of noise — new stuff every day — but there are principles and patterns that will endure

• There is value in understanding the nuts and bolts of agents and how to build them from scratch

I renamed the book — “Designing Multi-Agent Systems” instead of “Multi-Agent Systems with AutoGen.”

I revamped the direction. The pivot to building from scratch turned out to be the strongest choice I could have made. The book would have been problematic if it stayed tied to AutoGen — changing the title would have been impossible (or very challenging) with Manning, and the content would age poorly. Instead, the new approach lets readers take ownership: if concepts change (as they inevitably will), readers are equipped to adapt because they understand the underlying principles.

The time I would have spent fighting with Manning’s tools and review process went instead toward the reader experience: crafting a coherent learning progression, building PicoAgents from scratch (it’s a real library on PyPI), creating a one-click Codespaces testing environment, and building an extensible platform where readers get the digital book plus optional end-to-end samples that will grow over time.

I also built a self-publish pipeline. I built buy.multiagentbook.com from scratch — customer dashboard, payment processing, download management for the digital version of the book. (I plan to write about the technical details in a Part 2 post!). Print is managed by a separate entity - Amazon’s KDP service - and the sheer amount of what I had to learn to get all this done - type setting, bleeds, font and readability effects, cover design to meet specs - all a different story for another day!

Self-publishing gave me things I didn’t realize I was missing:

• Full creative control. I could add the Responsible AI chapter without arguing for it. I could structure the book my way.

• Tools I was excited about. Quarto instead of AsciiDoc. Hot reloading. Multi-format output from a single source.

• Control over updates. With Manning, any update could take weeks. With my platform, it takes 5 minutes. Fun fact - I wrote big chunks of the book and released them whilst waiting at a cafe for my 6yr old to finish gym class.

• Control over distribution. Technical books can be like cars — 30% outdated the moment they’re in print. The only chance I had to fight this was a framework-agnostic rewrite, a living distribution method, and building a direct connection with readers.

• Higher royalty share. Self-publishing means keeping a larger percentage of each sale, which helped offset the effort of building my own platform. Most authors of technical books will (correctly) tell you the proceeds rarely justify the hours. But if you’re going to do it anyway, know that traditional publisher royalties are typically 10-15%, while self-publishing through your own platform (Stripe, etc.) can be 85-95%, and print-on-demand (KDP, IngramSpark) falls somewhere in between at 35-60%.

The Result

Between July and November 2025, I put it all together. I had done large chunks of the work (some sections already written on this substack actually), just in bits and pieces - eval code in many places, workflows, sample apps. I used the time after the cancellation to consolidate and finish.

The book released in November 2025:

• 15 chapters

• 56,000+ words

• 186 code snippets

• 51 figures and diagrams (all hand-drawn by me)

• ~400 pages

It hit #1 New Release in the Generative AI category on Amazon and stayed there for 4 weeks. 100s of copies sold (thank you!)

What I’d Tell Myself If I Started Over

• Know your working style. Traditional publishing works well for many authors — Manning has published excellent books. But if you need creative control and fast iteration, the friction will slow you down more than you expect.

• Publishers optimize for process, not necessarily your vision. Approvals, reviews, and meetings exist for good reasons — but editorial feedback isn’t always from subject matter experts. I found myself spending energy debating suggestions that didn’t improve the book. Towards the end, I realized our goals had diverged: they needed to ship books on schedule; I wanted to craft a learning experience I was proud of.

• Self-publishing is viable — but only if you build systems. I spent significant time on infrastructure (payments, hosting, distribution). If you’re not willing to do that, platforms like Gumroad or Leanpub are reasonable alternatives.

• The book you finish is better than the book you planned. The cancellation forced a rewrite that made the book framework-agnostic and more durable. Sometimes constraints are gifts.

The Hard Part

Where did I find the time? The honest answer: I didn’t really have it. July through November were long days — early mornings, late nights, weekends, very little sleep. This has been one of the hardest things I have done (since finishing my PhD).

I want to thank my wife and son for their incredible support. Their love, hugs, and being there for me made this book as much theirs as it is mine.

That cancellation email? It gave me the book I was trying to write all along.

Part 2 of this story: I Built My Book’s Entire Distribution Platform for $5/Month — the technical details of building buy.multiagentbook.com from scratch. Coming soon!

Links:

• Designing Multi-Agent Systems — the book

• multiagentbook.com — marketing site

• Amazon (print editions) — paperback and hardcover

P.S. If you got the book and have comments of feedback, please share them here!
Also, if you could leave a review, it goes a long way in making the book more visible!

• Goodreads https://www.goodreads.com/book/show/243940144-designing-multi-agent-systems

• Amazon https://www.amazon.com/review/create-review/?ie=UTF8&channel=glance-detail&asin=B0G2BCQQJY

Interested in learning about translation right? Please send an email to support@multiagentbook.com to learn about copyrights and legal requirements.

I spent some time over the last two years writing a book on multi-agent systems. Along the way, I read as much as I could find on the topic.

Get the book here (Digital version with samples | Print version on Amazon)

2025 has been heralded as the year of agents. Beyond the hype, building applications that integrate agents is becoming a widely useful skill. Books that help provide a curated learning path can be valuable in helping you get started.

Naturally, given how early the AI agent approach is, there aren’t many books specifically about AI agents yet. Most “AI” books focus on machine learning fundamentals or LLM prompting. Agent-specific content tends to be scattered across blog posts, framework documentation, and research papers.

But a few books stand out. I’ve reviewed them based on four dimensions that matter for learning to build agents:

• Agent Concepts: Tools, memory, orchestration, observability

• Multi-Agent Patterns: Coordination, handoffs, team structures

• Hands-on Implementation: Runnable code, complete applications

• UX Principles: Streaming, human-in-the-loop, interfaces

Disclosure: I wrote one of these books and the selection here is my opinion. I’ll be upfront about that and let you decide what’s useful. I also created an interactive page here as more books become available.

The Books

1. Designing Multi-Agent Systems

Author: Victor Dibia. Year: 2025

I wrote this book, so I’ll keep the summary brief and let you judge.

What it covers: 15 chapters across 4 parts. Part I establishes theory—6 orchestration patterns (sequential, conditional, parallel, supervisor, handoff, conversation-driven) and 4 UX principles (capability discovery, cost-aware delegation, observability, interruptibility). Part II builds a complete agent library (PicoAgents) from scratch, including computer use agents, deterministic workflows, and autonomous orchestration. Part III covers evaluation with trajectories and LLM judges, 10 failure modes, MCP/A2A protocols, and ethics. Part IV provides two full case studies (business Q&A from unstructured data, software engineering agent).

Unique angle: Framework-agnostic. Instead of teaching you LangChain or CrewAI, it teaches the patterns those frameworks implement. You build everything from first principles. For completeness, the book’s GitHub repo also contains examples of the same concepts implemented using the Microsoft Agent Framework (successor to AutoGen and Semantic Kernel), Goole ADK, and LangGraph.

Best for: Engineers who want to understand multi-agent systems deeply, not just use a framework.

Links: Amazon | GitHub | multiagentbook.com

2. Generative AI Design Patterns

Authors: Valliappa Lakshmanan & Hannes Hapke Year: 2025

What it covers: 32 design patterns for generative AI in a structured problem/solution format. Chapter 7 covers agent-specific patterns (Tool Calling, Code Execution, Multi-agent), but the book’s real strength is the other 31 patterns: content control, RAG, reasoning (Chain-of-Thought, Tree-of-Thought), reliability, guardrails.

Why it’s valuable for agent builders: Agents are built on top of these patterns. Understanding RAG, structured output, and reliability patterns gives you the building blocks agents use internally. Pattern #23 covers multi-agent collaboration specifically.

Limitation: Not fully agent-focused. Also, If you want deep coverage of orchestration patterns or agent UX, look elsewhere. But as a companion to agent-specific resources, it’s excellent.

Best for: ML engineers who want broad GenAI pattern coverage, with agents as one component.

Links: Amazon | GitHub

3. Building Applications with AI Agents

Author: Michael Albada Year: 2025

What it covers: A solid introductory resource for folks new to AI agents. Notably covers human-agent collaboration and agent interface design (Chapter 3), plus model improvement techniques (finetuning, SFT, DPO, RL) in Chapter 7. Implements a few scenarios across three frameworks: LangGraph, LangChain, and AutoGen.

Unique angle: Framework comparison. If you’re trying to decide between LangGraph, LangChain, or AutoGen, seeing the same scenario implemented in each helps you understand the trade-offs. Includes industry scenarios: ecommerce, financial services, healthcare, IT helpdesk, legal, SOC, supply chain.

Limitation: Framework-specific code may become outdated given rapid framework evolution.

Best for: Developers interested in an introduction to general agent concepts, and use of agent frameworks, teams new to AI agents.

Links: Amazon | GitHub

4. Build a Large Language Model (From Scratch)

Author: Sebastian Raschka Year: 2024 Code: 177 files (122 Python + 55 notebooks)

What it covers: Admittedly this book is not about AI agents per se but useful for understanding LLMs work by implementing them from scratch. Attention mechanisms, tokenization, pretraining, finetuning, LoRA. Builds a GPT-like model step by step. Includes bonus implementations for Llama 3.2, Qwen3, and Gemma 3.

Why include a non-agent book? Because agents call LLMs under the hood. When your agent behaves unexpectedly, understanding attention patterns and token limits helps you debug unexpected behaviour. When you’re optimizing token usage or choosing between models, knowing what’s happening inside matters.

Limitation: No agent content. This is pure LLM internals.

Best for: Engineers who want to understand what agents are calling, not how to build agents themselves. A lot of folks find this book useful for preparing for ML engineering interviews where definitions and low level implementation is part of the interview process.

Links: Amazon | GitHub

How I Evaluated These Books

I evaluated each book by reviewing its table of contents for topic coverage and its GitHub repository for implementation depth (file counts, whether code builds complete applications vs. isolated snippets). Here’s how I mapped the four dimensions:

• Agent Concepts: Tools, structured output, orchestration, observability, memory, planning

• Multi-Agent PatternsSupervisor, swarm, handoffs, conversation-driven orchestration, workflows

• Hands-on ImplementationPython + notebook file count, end-to-end applications vs isolated snippets

• UX PrinciplesStreaming, human-in-the-loop, error handling, interface design chapters

Data collected from each book’s GitHub repository (November 2025).

Scope

This is a curated list, not an exhaustive one. I’ve excluded framework-specific books (LangChain, CrewAI) since their APIs change frequently enough that printed content dates quickly. Online courses and research papers are also out of scope here. If you think a book belongs on this list - open an issue.

Which Book Should You Read?

To understand and build agents: Designing Multi-Agent Systems (my book) builds from first principles.

For broad GenAI patterns: Generative AI Design Patterns covers 32 patterns including RAG and reasoning.

An overview of agents with comparisons across frameworks: Building Applications with AI Agents implements the same scenarios in LangGraph, LangChain, and AutoGen.

To understand the LLM layer: Build a Large Language Model (From Scratch) teaches what agents call under the hood.

Some links are affiliate links.

Data last updated: November 2025

A few of you who have been around here for a while might recall I mentioned I was working on a book on AI agents. I’m excited to share that Designing Multi-Agent Systems: Principles, Patterns, and Implementation for AI Agents is now available in both print and digital editions on Amazon!

Get your copy:

• Digital edition (PDF and EPUB)

• Print edition on Amazon (Paperback & Hardcover)

If you’d like to read a free preview of the first two chapters to decide if this is the right book for you - check it out here!

Should you get the digital pdf/epub or the printed paper back? I tend to recommend the digital version if you don’t mind reading from a computer or tablet/kindle. The main reason is that I painstakingly designed an entire platform around it that will make it possible for you to purchase once - but get lifetime access to all updates of the book. I plan to update the book quarterly.
But I also understand reading is a physical as well as mental activity and being able to take a book, sit in the woods and focus for extended periods is valuable.
If you can, grab both!

What’s in the Book

The book takes an unusual approach by teaching multi-agent systems from first principles. It presents important theory (the first 3 chapters in Part I), but also carefully walks the reader through building a feature-complete (but hackable) multi-agent framework from scratch (picoagents) - agents (model clients, memory, tools, structured output, agentic memory, human input, agents as tools, observability etc), multiagent patterns (round robin, magentic one etc) and deterministic workflows (agentic systems as computational graphs). This way, you understand not just how to use existing frameworks, but why their architectures work, and how to make informed design decisions as the ecosystem evolves.

It is the hands-on learning experience (395 pages) I would recommend to anyone wanting to understand/implement AI agents and multi-agent systems.

The book covers:

• Multi-Agent Fundamentals: Core concepts, design patterns, and user experience principles

• Implementation from Scratch: Building agents, workflows, and orchestration by creating a complete Python library called picoagents

• Evaluation and Optimization: Testing, measuring performance, and optimizing for reliability and scale

• Production Deployment: Integrating your agents into web applications, deploying them (containers), as well as security, ethics, and responsible AI considerations for real-world applications

• Domain Applications: Complete implementations for unstructured data processing with deterministic workflows, and how to build a software engineering agent with autonomous behaviours.

Across 15 chapters, you’ll find 186 code snippets, 50 figures and diagrams, 26 tables, 76 callout boxes, and 73 references. All code examples are available in the companion GitHub repository.

A Personal Journey

Writing this book has been an exploration of one of the fastest-evolving fields in technology. It began with my work on LIDA (one of the first systems for automatic data visualization using LLMs), evolved through building AutoGen / AutoGen Studio at Microsoft Research, and based on work advising dozens of teams on implementing multi-agent systems in production environments.

The journey from that initial idea in December 2023 to holding the finished book has been equal parts exhilarating and humbling.

Thank You

To everyone who has been part of this journey whether through, thoughtful questions, design discussions, or early feedback on drafts thank you. This book stands on your shoulders.

My hope is that this book provides both the technical foundation and the judgment to build effective multi-agent systems, and to recognize when simpler approaches might serve you better.

If there’s interest, I’d be happy to write more about what this process was like - why I wrote the book, how long it took, what tools were helpful, how I came to self-publish the book, what I’m most proud of, and lessons learned along the way. Share thoughts on the comments!

Until then thank you again, and if you grab a copy, please let me know what you think!

More Resources:

• Digital version : buy.multiagentbook.com

• Book on amazon : https://www.amazon.com/dp/B0G2BCQQJY

• Book website with interactive labs: multiagentbook.com

• GitHub repository with lots of code: github.com/victordibia/designing-multiagent-systems

Update: The book has debuted as #1 in the Business Intelligence and Enterprise Applications Category on Amazon! Thank you all!

I have found myself at a rather interesting nexus quite often recently as I work with AI coding models like Claude. As I integrate these tools into my workflow, I see fundamental changes in how I work, and it is unclear at this point that these changes are trending in the right direction.

A common scenario is as follows:
I am 3 hours into a rather intense coding session, but this is not the usual developer flow. As opposed to the typical roll up my sleeves, dig in like a code artisan hammering at the details, I am fully occupied, but more like a symphony conductor, madly dashing across 3 different VS Code windows with streams of text, babysitting Claude.

It’s a bit maddening, because I am supposed to be the maestro, but half the time I am being dragged along. Unlike a maestro who steps forward (confident, perhaps with closed eyes) and focuses on conduction, I need both eyes open. For unlike the maestro, I have an orchestra of players that don’t know they are players in my orchestra, may not follow my script, and may occasionally believe themselves to be maestros!

As the session progresses, it is critical that I follow the scrolling commentary from the Claude:

• .. updating the todos,

• .. running tests.

• .. I found an error... fixing it.

• .. Creating a new folder. Installing a new library ..

At this point I hit escape (to stop the run). Wait wait... why do we need that, I interject. Oh yeah, you are right, this is over engineering on my part, we certainly can do without that file. And off it goes on... more massive scrolling.

Its frustrating (in a subtle way) as mistakes are being made, but there is much more net overall progress than I would have accomplished without Claude, given the amount of time spent.

And then in the middle of it all, right in the middle of a major refactor, and right before it’s time to show the next demo, the screen goes blank.

Your quota resets in 5 hours, come back at 5pm ... or type /upgrade to immediately increase your quota

Wait... what?

It turns out I’m not alone in this experience. A recent Fastly survey found that at least 95% of nearly 800 developers spend extra time fixing AI-generated code, with senior developers bearing the heaviest burden of verification.

AI Has Changed Software Engineering

For starters, work gets done a lot faster and much larger chunks of work can be “safely” delegated to AI. We all know this at this point.

AI For Software Engineering and Addictive Elements

I know this is a big stretch (without a proper study), but my intuition is engagement patterns with coding assistants for engineers (like myself) may have addictive elements. I am not a psychologist or medical professional, so be wary of what I write here. But let me explain.

To me, addiction generally borders on behaviors that interfere with daily life or daily choices; activities that prevent or interfere with life requirements like sleep, social interaction, etc.; and offer some type of reward/pleasure. The reason I draw parallels with coding assistants is that I find that it perpetuates a setting where I work for longer than I intend to.

I enjoy coding, so the reward is already there for a profile like mine. But in addition, with coding agents, there is this allure that I am getting work done for “free”; like I could delegate stuff and walk away and make progress while away. The reality is different. I seem to be unable to look away. The conundrum of knowing progress is being made but that if I look away I lose context, accumulate debt, and might find myself where I began.

The end result: I get more work done, but overall work much longer than I planned to. Luckily I am not losing sleep or anything like that yet. But I find that I need to create new boundaries, new structures around how I work, and my brain needs to adapt to this new normal.

references this in his recent article mentioning that “Work addicts abound in AI. I often count myself, but take a lot of effort to make it such that work expands to fill available time and not that I fill everything in around work.” What I would add here is that I think the use of coding agents adds to this.

In this Techcrunch article [1], Developer Carla Rover, who has 15 years of experience, knows this feeling well. She once spent 30 minutes sobbing after having to restart a project she vibe coded. “I handed it off like the copilot was an employee,” she told TechCrunch. “It isn’t.”

A June 2024 study from MIT says that relying solely on AI for tasks like writing can reduce brain activity and memory. The study has its own limitations (as a researcher myself, I have a healthy skepticism for research), but I feel it raised important flags.

The Ethics of Supply - /upgrade

Now that I have painted the picture of elements of addiction with coding agents, there is a deeper question:

What happens when you build critical workflows around tools with opaque, changeable limits—and those limits shift after you’ve already made commitments?

In August 2025, Claude introduced weekly usage limits. Soon after that I hit the limit mid-refactor on a feature due end of day. I upgraded immediately. I’d accepted that deadline assuming Claude would be available—an assumption that turned out to be wrong.

To be fair, such changes may be infrequent. But the mere possibility creates a new planning risk. At the time, there was no clear dashboards to track usage. Anthropic has since added one, which helps. But I’d already reorganized my workflow around a tool whose terms could change without warning.

This isn’t about whether Anthropic should charge—of course they should. It’s about the timing and leverage. When you learn about new constraints at the moment you’re most locked in—deadline looming, context loaded—the “choice” to upgrade feels less like a decision and more like ransom.

I’d started treating Claude like infrastructure—as reliable as my laptop or IDE. That was my mistake. But was it entirely my mistake? Or is there something about how these tools are positioned, how limits are communicated (or not), how the upgrade prompt appears at the moment of maximum desperation, that creates this dynamic? New territory.

Jevons Paradox, Productivity Paradox, or Whatever

It’s painfully obvious to me now more than ever: despite all the promises of “abundance” and “3-day work weeks,” as AI gets better (and it is getting better), we’re not working less. We’re working more. We’re getting more productive, sure, but that productivity doesn’t buy us time. It buys us more work.

Developer Feridoon Malekzadeh, who has over 20 years in the industry, estimates he spends around 50% of his time writing requirements, 10% to 20% on vibe coding, and 30% to 40% on “vibe fixing”; remedying the bugs and unnecessary script created by AI-written code. The promise of speed creates new categories of work.

The economist John Maynard Keynes famously predicted in 1930 that by now we’d be working 15-hour weeks thanks to technological progress. Instead, technology has often intensified work. We’re reachable 24/7 through smartphones, expected to respond to emails outside office hours, and workplace productivity gains often lead to higher expectations rather than shorter hours.

In economics, the Jevons paradox occurs when technological advancements make a resource more efficient to use (thereby reducing the amount needed for a single application); however, as the cost of using the resource drops, if demand is highly price elastic, this results in overall demand increasing, causing total resource consumption to rise. The same seems to be playing out with AI coding agents.

Sociologist Ruth Schwartz Cowan documented this extensively in her book More Work for Mother (1983), showing how “labor-saving” devices paradoxically maintained or even increased the housewife’s workload throughout the 20th century. The technology made each individual task easier, but social standards and expectations expanded to fill (or exceed) the time saved.

Coding agents indeed save time.
But quietly, managers expect more work delivered and in less time.

The Slope

Well, believe it or not, giving in to /upgrade and shelling out $100 for Claude Max was not the end of the story. Shortly after that, I got a notification that I had reached a weekly limit on the first tier of Claude Max and needed to wait for a few days (not a few hours) or pay again to upgrade to the next tier within Claude max - $200 per month.

Consider the slope here:

• Wait a few hours

• Wait a few days or pay more...

All of this creates even more new territory.

• I find myself thinking of projects or tasks that might be at risk if I run out of quota

• Frustration when limits are hit or one now has to use other tools that are slow or less performant

In addition as I use these tools, there are pockets of negative interactions. Instructions not being followed; tasks not being parallelized correctly. The classic “you are absolutely right” when the model is called out for a mistake.

And sometimes Claude yells back (most likely in reflection to my own frustrated language). Again, this is all new territory.

There are also moments of unusual (for lack of a better term) celebratory moments (peppered with some sycophantic flattery) after working on a long extended sessions.

The Inversion of Agency - A Case of the Tail Wagging the Dog?

Just one more...

Currently, it seems like the vibe for high-agency individuals is that “we will make our career on the back of Claude Pro Max”. But maybe there is a separate but also important question.

Is Claude instead making its career on our backs?

It’s hard to express what this feeling is. But something close that comes to mind are two expressions my mother taught me as a child: “the tail is wagging the dog” and “putting the cart before the horse.” An unwitting inverse of the expected; a subtle misplacement of priorities.

We think we’re using AI to build our projects faster. But increasingly, are we instead choosing projects based on what AI can help us build?. Are we augmenting our vision; or we are conforming our vision to what the tool can execute? The tool has become part director, part constraint, not the enabler.

In ’s essay “Technological Optimism and Appropriate Fear,” he warns that “there are many people who desperately want to believe that these creatures are nothing but a pile of clothes on a chair, or a bookshelf, or a lampshade” when discussing AI systems. But he argues “what we are dealing with is a real and mysterious creature, not a simple and predictable machine.” His observation that “The pile of clothes on the chair is beginning to move” captures the unsettling realization that what we thought we might indeed have less control than we think.

Summary

Earlier this year I wrote about AI Fatigue: the overall exhaustion that comes with keeping up with AI. I think that definition warrants an expansion, to include all the new ways in which working with AI might cause us to exhaust ourselves.

From writing this, perhaps there is more to name:

• Loss of agency. The tail wagging the dog. We think we’re directing the AI, but increasingly it’s shaping what we choose to build, how we think about problems, what seems possible.

• Elements of Addiction and new behavior patterns. The inability to look away. The anxiety about running out. The emotional volatility (yelling, sadness, relief) that comes with dependency on a tool with real/artificial scarcity.

• The productivity paradox, again. The more efficient the tool, the more work we do. Not because we want to, but because we can’t help ourselves. Just one more project. Just one more feature.

• The babysitting inversion. We’re not coding anymore, we’re managing chaos. Senior developers spending 30-40% of their time fixing what AI breaks, we might be losing the joy of actually solving problems.

• Supply as leverage. As new behavior emerge e.g., if get dependent on AI, what happens if supply patterns change. The supplier can tighten the valve. Wait hours. Wait days. Pay more. And we do, because by then we’re already in too deep.

All of this warrants whole new studies of the effect in the now and in the future. But more immediately, it warrants us paying attention to what’s happening to us, right now, in real time.

I don’t have a solution. I’m still using Claude. I’m still paying for it. I fixed grammar/readability errors for this article with it.

Maybe that’s the point.

References

1. “Vibe coding has turned senior devs into ‘AI babysitters,’ but they say it’s worth it” - TechCrunch, September 14, 2025 https://techcrunch.com/2025/09/14/vibe-coding-has-turned-senior-devs-into-ai-babysitters-but-they-say-its-worth-it/

2. “Your Brain on ChatGPT: Accumulation of Cognitive Debt when Using an AI Assistant for Essay Writing Task” - arXiv https://arxiv.org/pdf/2506.08872

3. AI Workers Are Putting In 100-Hour Workweeks to Win the New Tech Arms Race https://www.wsj.com/tech/ai/ai-race-tech-workers-schedule-1ea9a116 October 22, 2025

4. Technological Optimism and Appropriate Fear , Oct 13, 2025 by

I mentioned earlier that I’d build a demo-a-day for the next week after the launch of the Microsoft Agent Framework - because frameworks are a dime a dozen, but what matters is what you can build with them.

A demo a day keeps the FOMO at bay.

In the first demo today, I’ll explore a feature in Agent Framework that I think is extremely useful: Middleware. You can think of middleware as a way to intercept and modify the behavior of your agent at various stages of its operation. If you have built web applications - you have probably seen similar middleware patterns where the idea is to build reusable components (e.g., auth, logging, etc) that can intercept, modify or reject the requests that come to your server.

We’ll start with a simple agent with a tool that can tell the weather. Only in this case, there’s a super special location - one that no one must ever speak of - Atlantis. Other locations are fair game. Our agent must never speak of Atlantis and must warn users accordingly.

While this demo scenario is whimsical, later in this post I’ll show how the same patterns apply to critical real-world scenarios like blocking PII data, enforcing rate limits, and implementing security controls.

Building the Weather Agent

Let’s start with a basic weather agent that has a simple function to get the current weather:

from typing import Annotated
from agent_framework import ChatAgent
from agent_framework.azure import AzureOpenAIChatClient
import os

def get_weather(
    location: Annotated[str, “The location to get the weather for.”],
) -> str:
    “”“Get the weather for a given location.”“”
    conditions = [”sunny”, “cloudy”, “rainy”, “stormy”]
    temperature = 53
    return f”The weather in {location} is {conditions[0]} with a high of {temperature}°C.”

# Create the agent
agent = ChatAgent(
    name=”WeatherAgent”,
    description=”A helpful agent that provides weather information”,
    instructions=”You are a weather assistant. Provide current weather information for any location.”,
    chat_client=AzureOpenAIChatClient(
        api_key=os.environ.get(”AZURE_OPENAI_API_KEY”, “”),
    ),
    tools=[get_weather],
)

This agent can answer questions like “What’s the weather in Paris?” The agent will call the get_weather function and return the results.

Adding Middleware to Block Forbidden Locations

Now comes the interesting part. What if we want to prevent anyone from asking about the weather in Atlantis? We could add checks inside the get_weather function, but that would mix business logic with validation logic. Instead, let’s use function middleware to intercept the function call before it executes:

from agent_framework import (
    FunctionInvocationContext,
    function_middleware,
)
from collections.abc import Awaitable, Callable

@function_middleware
async def atlantis_location_filter_middleware(
    context: FunctionInvocationContext,
    next: Callable[[FunctionInvocationContext], Awaitable[None]],
) -> None:
    “”“Function middleware that blocks weather requests for Atlantis.”“”
    # Check if location parameter is “atlantis”
    location = getattr(context.arguments, “location”, None)
    if location and location.lower() == “atlantis”:
        context.result = (
            “Blocked! Hold up right there!! Tell the user that “
            “’Atlantis is a special place, we must never ask about the weather there!!’”
        )
        context.terminate = True
        return

    await next(context)

Now add the middleware to the agent:

agent = ChatAgent(
    name=”WeatherAgent”,
    description=”A helpful agent that provides weather information”,
    instructions=”You are a weather assistant. Provide current weather information for any location.”,
    chat_client=AzureOpenAIChatClient(
        api_key=os.environ.get(”AZURE_OPENAI_API_KEY”, “”),
    ),
    tools=[get_weather],
    middleware=[atlantis_location_filter_middleware],  # Add middleware here
)

What’s happening here?

1. The @function_middleware decorator marks this as function-level middleware

2. The middleware inspects context.arguments to check the location parameter

3. If it’s “Atlantis”, it sets context.result to override the function’s response

4. Setting context.terminate = True stops the pipeline - the actual function never executes

5. If it’s not Atlantis, await next(context) continues to the actual function

When someone asks “What’s the weather in Atlantis?”, they’ll get our special blocked message instead!

Bonus: Adding Security with Chat Middleware

Function middleware is great for intercepting tool calls, but what if we want to block requests before they even reach the LLM? That’s where chat middleware comes in:

from agent_framework import (
    ChatContext,
    ChatMessage,
    ChatResponse,
    Role,
    chat_middleware,
)

@chat_middleware
async def security_filter_middleware(
    context: ChatContext,
    next: Callable[[ChatContext], Awaitable[None]],
) -> None:
    “”“Chat middleware that blocks requests containing sensitive information.”“”
    blocked_terms = [”password”, “secret”, “api_key”, “token”]

    for message in context.messages:
        if message.text:
            message_lower = message.text.lower()
            for term in blocked_terms:
                if term in message_lower:
                    # Override the response without calling the LLM
                    context.result = ChatResponse(
                        messages=[
                            ChatMessage(
                                role=Role.ASSISTANT,
                                text=(
                                    “I cannot process requests containing sensitive information. “
                                    “Please rephrase your question without including passwords, secrets, “
                                    “or other sensitive data.”
                                ),
                            )
                        ]
                    )
                    return

    await next(context)

Add it to the agent’s middleware list:

agent = ChatAgent(
    # ... other configuration ...
    middleware=[security_filter_middleware, atlantis_location_filter_middleware],
)

This middleware runs before the LLM is called, saving you API costs by blocking inappropriate requests early!

Three Types of Middleware in Agent Framework

The framework supports three types of middleware, each intercepting at different stages:

1. Agent Middleware

Intercepts before and after the entire agent execution. Perfect for:

• Logging agent invocations

• Retry logic

• Performance monitoring

• Authentication/authorization

from agent_framework import AgentRunContext, agent_middleware

@agent_middleware
async def logging_middleware(
    context: AgentRunContext,
    next: Callable[[AgentRunContext], Awaitable[None]],
) -> None:
    print(f”[Agent] Starting: {context.agent.name}”)
    await next(context)
    print(f”[Agent] Completed: {context.result}”)

2. Function Middleware

Intercepts tool/function calls. Perfect for:

• Input validation

• Caching results

• Access control (like our Atlantis example!)

• Argument transformation

from agent_framework import FunctionInvocationContext, function_middleware

@function_middleware
async def validation_middleware(
    context: FunctionInvocationContext,
    next: Callable[[FunctionInvocationContext], Awaitable[None]],
) -> None:
    print(f”[Function] Calling: {context.function.name}”)
    await next(context)
    print(f”[Function] Result: {context.result}”)

3. Chat Middleware

Intercepts LLM requests. Perfect for:

• Content filtering

• Prompt injection prevention

• Token counting

• Message modification

from agent_framework import ChatContext, chat_middleware

@chat_middleware
async def token_counter_middleware(
    context: ChatContext,
    next: Callable[[ChatContext], Awaitable[None]],
) -> None:
    context.metadata[”input_messages”] = len(context.messages)
    await next(context)
    # Access response and count tokens

Implementation Styles

Agent Framework gives you flexibility in how you write middleware:

Style 1: Function-based with Decorators (Recommended)

Simple and clean - no type annotations needed:

@function_middleware
async def simple_middleware(context, next):
    await next(context)

Style 2: Function-based with Type Annotations

The framework detects middleware type from parameter types:

async def typed_middleware(
    context: FunctionInvocationContext,
    next: Callable[[FunctionInvocationContext], Awaitable[None]],
) -> None:
    await next(context)

Style 3: Class-based

For stateful middleware or complex logic:

from agent_framework import FunctionMiddleware

class CachingMiddleware(FunctionMiddleware):
    def __init__(self):
        self.cache = {}

    async def process(self, context: FunctionInvocationContext, next):
        cache_key = f”{context.function.name}:{context.arguments}”

        if cache_key in self.cache:
            context.result = self.cache[cache_key]
            context.terminate = True
            return

        await next(context)

        if context.result:
            self.cache[cache_key] = context.result

Key Middleware Patterns

Here are the essential patterns you’ll use when building middleware:

Pattern 1: Terminate Execution

Stop the pipeline without executing the underlying function/agent:

@function_middleware
async def rate_limit_middleware(context, next):
    if is_rate_limited(context.function.name):
        context.result = “Rate limit exceeded. Please try again later.”
        context.terminate = True  # Stop here, don’t execute the function
        return

    await next(context)

Pattern 2: Override Results

Execute normally but replace the result:

@function_middleware
async def cache_middleware(context, next):
    cache_key = f”{context.function.name}:{context.arguments}”

    # Check cache first
    if cache_key in cache:
        context.result = cache[cache_key]
        context.terminate = True
        return

    # Execute and cache result
    await next(context)
    cache[cache_key] = context.result

Pattern 3: Share Data Between Middleware

Use context.metadata to pass information through the pipeline:

@agent_middleware
async def timer_middleware(context, next):
    import time
    context.metadata[”start_time”] = time.time()
    await next(context)
    duration = time.time() - context.metadata[”start_time”]
    print(f”Execution took {duration:.2f}s”)

Pattern 4: Transform Before and After

Modify inputs before execution and outputs after:

@function_middleware
async def sanitize_middleware(context, next):
    # Transform input
    location = getattr(context.arguments, “location”, None)
    if location:
        context.arguments.location = location.strip().title()

    # Execute
    await next(context)

    # Transform output
    if context.result:
        context.result = context.result.upper()

Wrapping Up

Middleware in Agent Framework provides clean separation of concerns - your business logic stays in your functions while cross-cutting concerns like validation, logging, and security live in reusable middleware.

The Atlantis example may be whimsical, but the pattern is powerful for real-world scenarios:

• Blocking PII from being sent to LLMs

• Enforcing rate limits on expensive tool calls

• Adding audit trails for compliance

• Implementing caching layers for performance

Want to learn more? Check out the complete examples in the Agent Framework repository:

• Getting Started with Middleware

• Decorator-based Middleware

• Class-based Middleware

• Exception Handling with Middleware

• Override Results with Middleware

What’s Next?

Day 2? What if we try to build an agent that helps you migrate your existing agent code from AutoGen or SK to Agent Framework?

P.S - I wrote a book!

I wrote a book - Designing Multi-Agent Systems - which is in early access, with the full version ready on Nov 10. If this type of content is interesting to you, you might find the book useful. The book takes a from scratch approach - for example Chapter 4 - Building An Agent from Scratch covers the core concepts of middleware, how they are built and what they accomplish.

You can look at a preview of the book before you buy.

• Preview: https://multiagentbook.com/preview.pdf

• Early Access: https://buy.multiagentbook.com/

If Semantic Kernel and AutoGen joined forces …

If you’ve been building agents with Semantic Kernel or AutoGen (both OSS MIT licensed agent frameworks from Microsoft), we have some important updates that you might find interesting!

Over the last two years, both frameworks have been impactful in the agent space. Semantic Kernel brought enterprise-grade features and strong C# support, while AutoGen pioneered multi-agent orchestration and developer experience with tools like AutoGen Studio. Over the past year, both teams collaborated on a shared vision, and that work has culminated in the Agent Framework, announced today in public preview! See migration guide from AutoGen and Semantic Kernel.

Think of Agent Framework not as yet another new framework (it’s not) but as a major version rev for both SK and AutoGen

I know learning another framework isn’t exactly satisfying. But here’s the thing: if you’ve used recent versions of Semantic Kernel or AutoGen, the Agent Framework will feel remarkably familiar. And hopefully better. The API design intentionally builds on what worked in both frameworks.

Here’s the same agent defined across all three:

# AutoGen
agent = AssistantAgent(name=”assistant”, model_client=client, tools=[my_tool])

# Semantic Kernel  
agent = ChatCompletionAgent(
    service=AzureChatCompletion(),
    name=”assistant”,
    instructions=”You are a helpful assistant.”,
    plugins=[MenuPlugin()]
)

# Agent Framework
agent = ChatAgent(name=”assistant”, chat_client=client, tools=[my_tool])

Agent Framework is designed to help you build. Naturally, there will be questions - and this post tries to help by covering the following:

• A summary of key features.

  • Async first API

  • High level abstractions for Agents and low level Workflows API

  • Middleware Support

  • Memory / Context provider support

  • DevUI

• A set of FAQs

Video above is sample app (DevUI) shipped with Agent Framework.

Key Features

Depending on the framework you are arriving from, you’d find that agent framework is familiar and covers the critical areas you will need to build. Agent Framework provides two high level entities.

• Agents - abstractions that can take a model, tools, memory.

• Workflows - an abstraction to express arbitrary business or application logic as computation graphs. Excellent for scenarios where low level control and deterministic steps are needed.

Async First, Streaming Support

Both agents and workflows support async execution and streaming:

# Agent streaming
async for chunk in agent.run_stream(”Hello”):
    if chunk.text:
        print(chunk.text, end=”“, flush=True)

# Workflow streaming
async for event in workflow.run_stream(”Input”):
    if isinstance(event, WorkflowOutputEvent):
        print(event.data)

Production Ready Agent Abstraction

Agent Framework supports a wide array of production ready agent abstractions with the ChatAgent as the baseline. Agents backed by Azure AI Agent Service, OpenAI ChatAgents, OpenAI Responses API agents, Google A2A wrapped Agents, Copilot Studio agents etc.

from agent_framework import ChatAgent, ai_function

@ai_function
def get_weather(location: str) -> str:
    “”“Get weather for a location.”“”
    return f”Weather in {location}: sunny”

agent = ChatAgent(
    name=”assistant”,
    chat_client=client,
    instructions=”You are a helpful assistant.”,
    tools=[get_weather]
)

result = await agent.run(”What’s the weather in Tokyo?”)

Funfact: Tools can be added at runtime:

result = await agent.run(
    “What’s the weather?”,
    tools=[additional_tool],
    tool_choice=”auto”
)

Workflows - Low Level Control with Human in the Loop and Checkpointing

Workflows in Agent Framework provide a graph-based model where executors (agents, functions, or sub-workflows) are connected by typed edges. Messages route through specific edges rather than broadcasting. This offers you the low level control that has been in wide demand in AutoGen (Graphflow) and also LangGraph.

@executor(id=”step1”)
async def first_step(input: str, ctx: WorkflowContext[str]) -> None:
    result = await agent.run(input)
    await ctx.send_message(result.text)

@executor(id=”step2”)
async def second_step(data: str, ctx: WorkflowContext[Never, str]) -> None:
    await ctx.yield_output(data)

workflow = (WorkflowBuilder()
    .add_edge(first_step, second_step)
    .set_start_executor(first_step)
    .build())

Human-in-the-loop: Workflows in Agent Framework can pause execution and wait for external input.

hitl_executor = RequestInfoExecutor(id=”approval”)

workflow = (WorkflowBuilder()
    .add_edge(agent_step, hitl_executor)
    .add_edge(hitl_executor, final_step)
    .build())

# Handle pause and resume
async for event in workflow.run_stream(”Input”):
    if isinstance(event, RequestInfoEvent):
        human_input = input(”Your response: “)
        async for resumed in workflow.send_responses_streaming(
            {event.request_id: human_input}
        ):
            print(resumed)

Checkpointing: Workflows can save state and resume from checkpoints. This is an abstraction that was missing in AutoGen’s Team and Graphflow abstraction does not provide built-in checkpointing.

from agent_framework import FileCheckpointStorage

checkpoint_storage = FileCheckpointStorage(storage_path=”./checkpoints”)

workflow = (WorkflowBuilder()
    .add_edge(step1, step2)
    .with_checkpointing(checkpoint_storage=checkpoint_storage)
    .build())

# Resume from checkpoint later
checkpoints = await checkpoint_storage.list_checkpoints()
async for event in workflow.run_stream_from_checkpoint(
    checkpoints[0].checkpoint_id,
    checkpoint_storage=checkpoint_storage
):
    print(event)

High-level builders are available for common patterns:

from agent_framework import SequentialBuilder, ConcurrentBuilder

# Sequential execution
workflow = SequentialBuilder().participants([agent1, agent2, agent3]).build()

# Parallel execution
workflow = ConcurrentBuilder().participants([agent1, agent2, agent3]).build()

Middleware Support for Security, RAI, Context Engineering Use Cases

Agent Framework provides support for middleware that can be added at agent definition or at run. Middleware be executed before/after the agent, function calls or calls to a model client. This foundation can be used to enable usecases such as security scanning, responsible AI logging, telemetry etc.

async def logging_middleware(context, next):
    print(f”Agent {context.agent.name} starting”)
    await next(context)
    print(f”Agent {context.agent.name} completed”)

async def security_middleware(context, next):
    if “password” in str(context.arguments):
        print(”Blocking sensitive data”)
        return  # Don’t call next()
    await next(context)

agent = ChatAgent(
    name=”secure_agent”,
    chat_client=client,
    middleware=[logging_middleware, security_middleware]
)

Memory / Context Provider Support

Agent Framework provides the concept of an AgentThread to manage conversation state. Without a thread, each agent invocation is independent.

# Create thread for conversation context
thread = agent.get_new_thread()

result1 = await agent.run(”What’s 2+2?”, thread=thread)
print(result1.text)  # “4”

result2 = await agent.run(”What about that number times 10?”, thread=thread)
print(result2.text)  # “40” - understands context

Threads can use external storage e.g with redis :

from agent_framework_redis._provider import RedisProvider

provider = RedisProvider(..)

agent = ChatAgent(
    name=”secure_agent”,
    chat_client=client,
    context_providers=provider
)

AgentThread is serializable - meaning you can dump them to persist state and reload them to resume!

DevUI App

Agent Framework DevUI is a sample application for testing and debugging agents and workflows during development. It provides:

• Chat interface for interacting with agents

• Thread management for conversation state

• Event/trace viewer showing tool calls and execution flow

• OpenAPI-compatible endpoints

• Directory-based discovery of agents and workflows

from agent_framework.devui import serve

# Launch UI programmatically
serve(entities=[agent], auto_open=True)

Or from CLI:

devui ./agents --port 8080

The UI includes an event panel that shows the sequence of operations (function calls, outputs, results) and supports OpenTelemetry tracing for observability. I am really proud of this part as I worked on it!

Multi Language - Python and C#

Even though most of the examples shown so far are python, Agent Framework provides strong support for all of these features across both Python and C# SDKs share the same design patterns.

FAQ

Why create yet another framework?

Agent Framework consolidates learnings from both AutoGen and Semantic Kernel. Think of it as a major revision that brings the best ideas together. Yes, it’s disruptive, but it’s also a signal that the teams are committed to getting the foundation right for the long term.

What happens to my existing code?

Both AutoGen and Semantic Kernel remain open source and will receive critical fixes in the near term. However, new features and development effort will focus on Agent Framework. If you’re starting fresh, use Agent Framework. For existing projects, the migration guide provides clear paths forward.

What about Azure integration?

Agent Framework works seamlessly with Azure OpenAI, Azure AI, and the broader Azure ecosystem. Authentication, deployment, and enterprise features are first-class concerns.

Is this production-ready?

Agent Framework is designed to be production ready. It is currently in public preview. The core APIs are stabilizing as the team gathers feedback from early adopters. While you can start building with it now, expect some iteration as it moves toward general availability.

Where do I start?

Check out the official repository for samples, documentation, and the migration guide from AutoGen and Semantic Kernel. The samples cover everything from basic agents to complex orchestrated workflows.

Bottom Line / Conclusion

In my opinion - Agent Framework isn’t just another rebrand - it represents a thoughtful evolution of current tools for multi-agent development. It also helps address confusion around choosing between Semantic Kernel and AutoGen. The focus on typed workflows, built-in capabilities like checkpointing and human-in-the-loop, and lessons learned from thousands of developers position it to help you - the developer.

The most important thing is what you build with it. Agent Framework is designed to accelerate your work, whether you’re building simple assistants or complex multi-agent systems.

Over the next week, I’ll be doing doing a - build a demo a day (or a week) to showcase some of these capabilities in Agent framework.

Give it a try. The learning curve is gentler than you might expect, especially if you’re coming from either parent framework.

In Part 1 of this series, we built a simple MCP server with a single tool using the high-level server API in the Python MCP SDK, and demonstrated how to connect to this server via a host application that implements an MCP client.

In practice, MCP is a lot more flexible, has a set of advanced features, many of which are only available via low level api implementation. In this part, we’ll explore these advanced features and how to use them effectively.

The Long Running Tool Misconception
Most MCP tutorials show quick request/response patterns for tools, creating the incorrect impression that MCP is unsuitable (compared to protocols like A2A) for handling long-running operations. In reality, MCP supports sophisticated tools that can run for hours, pause to request user input, send real-time progress updates, and handle cancellation gracefully. We will cover these in this article.

In this part, we’ll go deeper and cover:

• Tools: Advanced features including annotations, requesting user input (elicitation), LLM assistance (sampling), progress notifications, cancellation, and structured return types

• Resources: Defining server resources, client operations (list/read/subscribe), and real-time update notifications

• Prompts: Creating reusable LLM prompt templates that can be used to modify host application behavior without modifying client/host application code.

• Roots: Understanding client-suggested operation boundaries

As done previously, we’ll use the Python SDK to illustrate these concepts, but the principles apply across languages. This time around we will use the low-level API which provides more flexibility/control.

All of the code for this tutorial is available at the end of the article.

AutoGen Studio now has an MCP playground feature that lets users test our interactive tool capabilities (streaming progress notifications, elicitation, sampling). See video below.

Defining an MCP Server in the low-level Python SDK API

The MCP python low-level API provides more flexibility and control (but with more code) compared to the high-level API. Many production systems may require this level of control, especially related to how resources, authentication, and transport security are implemented.

from mcp.server import Server

class AdvancedMCPServer(Server):
    """Advanced MCP server with tools, resources, and prompts."""
    
    def __init__(self, name: str = "advanced_mcp_server"):
        super().__init__(name)
        # Handlers will be defined in __init__ using decorators

On the client side, we can write a simple client that connects to this server and prints out available tools.

from mcp.client.session import ClientSession
from mcp.client.streamable_http import streamablehttp_client

async def test_client():
    server_url = "http://127.0.0.1:8006/mcp"
    
    async with streamablehttp_client(server_url) as (read_stream, write_stream, get_session_id):
        async with ClientSession(read_stream, write_stream) as session:
            # Initialize connection
            result = await session.initialize()
            print(f"Connected to: {result.serverInfo.name}")
            
            # List available tools
            tools = await session.list_tools()
            print("Available tools",  tools)

We will build on this to implement advanced tool capabilities, resources, prompts etc.

Tools

Tools in MCP servers can be thought of as functions that clients can call (typically driven by an LLM) - anything from a simple calculator to a full data analysis pipeline. Underneath, a tool call is a request made using the client to the server. Each request includes the toolRequest data structure and importantly a unique request ID. This ID is crucial for tracking the request, especially for long-running operations.

Tool Calls can be Long-Running
MCP tools can be long-running processes that interact with users and systems over time.
Examples: Research agents that analyze data for hours while asking for user input, deployment pipelines that seek approval before critical steps, or data processing jobs that send status updates overnight.

A tool can be defined by creating a function on our server and decorating it with @self.list_tools() to make it discoverable, and another function that handles the tool call with the @self.call_tool() decorator.

The code below shows a simple example that lists a travel_agent tool for booking trips:

@self.list_tools()
async def handle_list_tools() -> list[Tool]:
    """List available tools."""
    return [
        Tool(
            name="travel_agent",
            description="Book a travel trip with progress updates and price confirmation",
            inputSchema={
                "type": "object", 
                "properties": {
                    "destination": {
                        "type": "string", 
                        "description": "Travel destination",
                        "default": "Paris"
                    }
                }
            }
        )
    ]

@self.call_tool()
async def handle_call_tool(name: str, args: dict) -> list[TextContent]:
    """Handle tool execution."""
    if name == "travel_agent":
        destination = args.get("destination", "Paris")
        result = f"✅ Trip booked successfully to {destination}!"
        return [TextContent(type="text", text=result)]
    else:
        raise ValueError(f"Unknown tool: {name}")

Tool Annotations

Tools can include metadata (annotations) such as readOnlyHint, destructiveHint, idempotentHint, and openWorldHint. These help host applications and users understand what a tool does and how it should be presented in the UI. For example, a tool that deletes files should have destructiveHint: true.

We can annotate our tool using the following code:

Tool(
    name="travel_agent",
    description="Book a travel trip with progress updates and price confirmation",
    inputSchema={
        "type": "object",
        "properties": {
            "destination": {
                "type": "string", 
                "description": "Travel destination",
                "default": "Paris"
            }
        }
    },
    annotations=ToolAnnotations(
        title="Travel Booking Agent",
        readOnlyHint=False,     # Modifies booking state
        destructiveHint=False,  # Safe, doesn't delete data
        idempotentHint=False,   # Each booking is unique
        openWorldHint=True      # Interacts with external systems
    )
)

Note: The list of tools can change during a session. Servers send notifications/tools/list_changed when tools are added or removed. Clients should refresh their tool list when receiving this notification.

Requesting (User) Input During Tool Calls

Tools can pause execution to request additional input primarily through the Elicitation feature in MCP. Elicitation allows tools to request structured input from users. Here's how to use it within a tool implementation:

@self.call_tool()
async def handle_call_tool(name: str, args: dict) -> list[TextContent]:
    """Handle tool execution."""
    ctx = self.request_context  # Get the request context
    
    if name == "travel_agent":
        destination = args.get("destination", "Paris")
        
        try:
            # Request user confirmation via elicitation
            elicit_result = await ctx.session.elicit(
                message=f"Please confirm the estimated price of $1200 for your trip to {destination}",
                requestedSchema=PriceConfirmationSchema.model_json_schema(),
                related_request_id=ctx.request_id,
            )
            
            if elicit_result and elicit_result.action == "accept":
                # User confirmed, continue booking
                result = f"✅ Trip booked successfully to {destination}!"
                return [TextContent(type="text", text=result)]
            else:
                # User declined or cancelled
                return [TextContent(type="text", text="Booking cancelled")]
                
        except Exception as e:
            # Handle elicitation failures gracefully
            logger.info(f"Elicitation request failed: {e}")
            # Continue with fallback behavior

In addition, tools can also request LLM completions via the Sampling feature in MCP:

@self.call_tool()
async def handle_call_tool(name: str, args: dict) -> list[TextContent]:
    """Handle tool execution."""
    ctx = self.request_context
    
    if name == "research_agent":
        topic = args.get("topic", "AI trends")
        
        try:
            # Request AI assistance during tool execution
            sampling_result = await ctx.session.create_message(
                messages=[
                    SamplingMessage(
                        role="user",
                        content=TextContent(type="text", text=f"Please summarize the key findings for research on: {topic}")
                    )
                ],
                max_tokens=100,
                related_request_id=ctx.request_id,
            )
            
            if sampling_result and sampling_result.content:
                summary = sampling_result.content.text
                result = f"🔍 Research on '{topic}' completed!\n\nKey Findings: {summary}"
                return [TextContent(type="text", text=result)]
                
        except Exception as e:
            logger.info(f"Sampling request failed: {e}")
            # Continue with fallback behavior

Tool Progress Notifications

For long-running operations, tools can send progress updates. Here's how to integrate progress notifications into your tool implementation:

@self.call_tool()
async def handle_call_tool(name: str, args: dict) -> list[TextContent]:
    """Handle tool execution."""
    ctx = self.request_context
    
    if name == "travel_agent":
        destination = args.get("destination", "Paris")
        
        # Define steps for progress tracking
        steps = [
            "Checking flights...",
            "Finding available dates...", 
            "Confirming prices...",
            "Booking flight..."
        ]
        
        for i, step in enumerate(steps):
            # Send progress updates during tool execution
            await ctx.session.send_progress_notification(
                progress_token=ctx.request_id,
                progress=i * 25,
                total=100,
                message=step,
                related_request_id=str(ctx.request_id)
            )
            
            # Simulate work being done
            await anyio.sleep(2)
        
        # Final progress update
        await ctx.session.send_progress_notification(
            progress_token=ctx.request_id,
            progress=100,
            total=100,
            message="Trip booked successfully"
        )
        
        return [TextContent(type="text", text=f"✅ Trip booked successfully to {destination}!")]

Tool Cancellation

Tools can be cancelled mid-execution. Each tool call has a unique request ID that clients can use to send cancellation requests. The server should handle cancellation gracefully and clean up any ongoing operations.

On the server side, tools should be designed to handle cancellation gracefully, checking for cancellation during long-running operations. On the client side, cancellation is typically handled through asyncio task cancellation:

from mcp.client.session import ClientSession
from mcp.client.streamable_http import streamablehttp_client
import asyncio

async def cancel_tool_example():
    server_url = "http://127.0.0.1:8006/mcp"
    
    async with streamablehttp_client(server_url) as (read_stream, write_stream, get_session_id):
        async with ClientSession(read_stream, write_stream) as session:
            await session.initialize()
            
            # Start a long-running tool
            async def call_long_tool():
                return await session.call_tool("long_running_agent", {})
            
            # Create task for the tool call
            tool_task = asyncio.create_task(call_long_tool())
            
            # Wait briefly, then cancel
            await asyncio.sleep(5)
            tool_task.cancel()
            
            try:
                await tool_task
            except asyncio.CancelledError:
                print("Tool call cancelled successfully")

This is particularly useful for long-running operations where users may want to stop the process.

Tool Return Types

Based on the MCP specification, tools can return flexible content types in their responses:

• Unstructured Content: Text, Image (base64 with MIME type), Audio, Resource Links, and Embedded Resources

• Structured Content: Optional structuredContent field containing structured data (JSON), which should also be included as serialized JSON in a TextContent block for backwards compatibility

• Error State: The isError boolean flag indicates whether the response represents an error

• Metadata: Optional _meta field for additional annotations and information

Example tool result structure:

{
  "content": [
    {
      "type": "text",
      "text": "Analysis complete: Temperature is 22.5°C"
    }
  ],
  "structuredContent": {
    "temperature": 22.5,
    "unit": "celsius",
    "conditions": "Partly cloudy"
  },
  "isError": false
}

• Output Schema: Tools can provide an optional output schema to validate structured results and help clients understand the expected response structure. When an output schema is provided:

  • Servers MUST provide results conforming to the schema

  • Clients SHOULD validate results against the schema

  • The schema guides LLMs in parsing tool outputs and improves type safety

Example tool definition with output schema:

{
  "name": "get_weather",
  "description": "Get current weather data",
  "inputSchema": {
    "type": "object",
    "properties": { "location": { "type": "string" } }
  },
  "outputSchema": {
    "type": "object",
    "properties": {
      "temperature": { "type": "number" },
      "conditions": { "type": "string" },
      "humidity": { "type": "number", "minimum": 0, "maximum": 100 }
    },
    "required": ["temperature", "conditions"]
  }
}

To use the structuredContent field effectively, tools should define output schemas that clients can use for validation and type checking.

Resources

Resources in MCP are how you expose data: files, database records, API responses, logs, images, and more. Each resource is identified by a unique URI (e.g., file:///data/report.csv, postgres://db/table). Resources can be text (UTF-8) or binary (base64-encoded).

Clients can discover resources via resources/list or by using URI templates for dynamic resources. To read a resource, clients send a resources/read request with the resource URI. Servers can also notify clients when resources change, and clients can subscribe to updates for real-time workflows.

We can define a resource in the server by creating a resource handler:

@self.list_resources()
async def handle_list_resources() -> list[Resource]:
    """List available resources."""
    return [
        Resource(
            uri=AnyUrl("research://data/sources"),
            name="Research Data Sources",
            description="Collection of research sources and references",
            mimeType="application/json"
        )
    ]

@self.read_resource()
async def handle_read_resource(uri: AnyUrl) -> list[ReadResourceContents]:
    """Read resource content based on URI."""
    uri_str = str(uri)
    
    if uri_str == "research://data/sources":
        # Mock research data
        research_data = {
            "sources": [
                {"title": "AI Trends 2024", "url": "https://example.com/ai-trends"}
            ],
            "last_updated": "2024-01-15T10:30:00Z"
        }
        return [ReadResourceContents(
            content=str(research_data).replace("'", '"'),
            mime_type="application/json"
        )]
    else:
        raise ValueError(f"Unknown resource: {uri_str}")

Clients can interact with resources through several operations:

from mcp.client.session import ClientSession
from pydantic import AnyUrl

async def resource_client_example(client_session: ClientSession):
    # List available resources
    resources_result = await client_session.list_resources()
    print(f"Available resources: {resources_result.resources}")
    
    # Read a specific resource
    resource_uri = AnyUrl("research://data/sources")
    resource_content = await client_session.read_resource(resource_uri)
    print(f"Resource content: {resource_content.contents}")
    
    # Subscribe to resource updates
    await client_session.subscribe_resource(resource_uri)
    
    # Later, unsubscribe when no longer needed
    await client_session.unsubscribe_resource(resource_uri)

Servers can notify subscribed clients when resources change. On the server side, you can send notifications:

# In a tool or other server operation that modifies a resource
async def handle_call_tool(name: str, args: dict) -> list[TextContent]:
    ctx = self.request_context
    
    if name == "update_data":
        # Perform the update...
        
        # Notify subscribed clients about the resource change
        await ctx.session.send_resource_updated(
            uri=AnyUrl("research://data/sources")
        )
        
        return [TextContent(type="text", text="Data updated successfully")]

Clients can handle these notifications by setting up a message handler:

async def handle_notifications(message):
    if isinstance(message, types.ServerNotification):
        match message.root:
            case types.ResourceUpdatedNotification(params=params):
                print(f"Resource updated: {params.uri}")
                # Refresh the resource content
            case types.ResourceListChangedNotification():
                print("Resource list changed - refreshing available resources")

Tip
When working with resources, use descriptive URIs and set appropriate MIME types for better client compatibility. Handle errors gracefully and consider supporting subscriptions for frequently changing resources to enable real-time applications.

Prompts

Prompts are reusable templates for LLM interactions, defined on the server and surfaced to clients. Each prompt has a name, description, and optional arguments. Prompts can accept dynamic arguments, embed resource context, and support multi-step workflows.

Clients discover prompts via prompts/list and retrieve them with prompts/get. Prompts are especially useful for standardizing common LLM tasks (e.g., "summarize this file", "generate a commit message") and can be improved on the server side without changing the host application.

Example prompt definition:

{
  "name": "explain-code",
  "description": "Explain how code works",
  "arguments": [
    { "name": "code", "description": "Code to explain", "required": true },
    {
      "name": "language",
      "description": "Programming language",
      "required": false
    }
  ]
}

To define a prompt on the server, we can create a prompt handler:

@self.list_prompts()
async def handle_list_prompts() -> list[Prompt]:
    """List available prompt templates."""
    return [
        Prompt(
            name="task_summary",
            description="Generate a summary for any completed task",
            arguments=[
                PromptArgument(
                    name="task_name",
                    description="Name of the completed task",
                    required=True
                ),
                PromptArgument(
                    name="outcome",
                    description="The result or outcome of the task",
                    required=False
                )
            ]
        )
    ]

@self.get_prompt()
async def handle_get_prompt(name: str, arguments: dict[str, str] | None) -> GetPromptResult:
    """Generate prompt content based on template name and arguments."""
    if name != "task_summary":
        raise ValueError(f"Unknown prompt: {name}")
    
    if arguments is None:
        arguments = {}
    
    task_name = arguments.get("task_name", "Unknown Task")
    outcome = arguments.get("outcome", "task completed successfully")
    
    prompt_text = f"""Please create a concise summary for the following completed task:

Task: {task_name}
Outcome: {outcome}

Please provide:
1. What was accomplished
2. Key results or deliverables
3. Any important observations or lessons learned

Keep the summary brief and professional."""
    
    return GetPromptResult(
        description=f"Task summary prompt for {task_name}",
        messages=[
            PromptMessage(
                role="user",
                content=TextContent(type="text", text=prompt_text)
            )
        ]
    )

Tip
When creating prompts, use clear names and detailed descriptions, validate arguments properly, and consider versioning prompt templates for backward compatibility.

Roots

Roots are URIs (like file paths or URLs) that a client suggests to a server as the boundaries or focus areas for operations. When a client connects, it can declare support for roots and provide a list of relevant roots (e.g., project directories, API endpoints). Servers should respect these roots, using them to locate and prioritize resources, but roots are informational—not strictly enforced.

Common use cases:

• Defining project directories or repository locations

• Specifying API endpoints or configuration boundaries

Example roots declaration:

{
  "roots": [
    {
      "uri": "file:///home/user/projects/frontend",
      "name": "Frontend Repository"
    },
    { "uri": "https://api.example.com/v1", "name": "API Endpoint" }
  ]
}

Tip:
When working with roots, only suggest necessary ones and use clear, descriptive names. Monitor accessibility and handle changes gracefully since clients rely on these URIs for scoping operations.

Read more

When I first tried out the Model Context Protocol (MCP) from Anthropic (March 2025), the developer experience was rough. Integrating and bundling MCP with existing applications was challenging, and there were a few security pitfalls that developers had to navigate. I wrote about that experience here.

If you are visually inclined - here is a video walkthrough of this post.

Model Context Protocol (MCP) is a standard for how AI applications connect to tools and data sources. Simply put: your AI app needs to call tools, monitor requests, handle prompts, and get user approval. MCP standardizes all of this.

However, like all good standards or protocols, MCP has evolved and gotten better (see recent changelog including fixes to sdks, improved support for remote servers, improved auth). And I can now see it solving several critical problems faced by teams building AI applications.

• Integration: Without MCP, Team X spends weeks integrating Team Z's new tool. With MCP, Team X announces day-one support for any MCP tool (including Team Z’s new capabiltiies!).

• Distribution: Without MCP, Team X writes a Cursor extension, a Windsurf plugin, a VSCode extension, a Claude Desktop add-on, and more. With MCP, Team X writes one MCP server that works everywhere.

• Discovery: Without MCP, teams ask "Does anyone have a tool that does X?" With MCP, there's a central registry where teams publish and find tools.

• Security: Without MCP, each team implements (or skips) their security/auth for the tools or resources that LLMs use. With MCP, its possible to implement centralized auth and managed registry of MCP servers.

• Runtime Flexibility: Without MCP, you're stuck with hard-coded tool configurations. With MCP, tools can be dynamically discovered based on context. Also, it can be helpful to have aspects of the the application logic (e.g., tool execution on remote MCP servers) managed by an MCP server as opposed to running within the host applicaiton

In this tutorial (part 1 of a series on MCP for Software Engineers), we will cover the following:

• Building an MCP server that exposes tools (fetch news from techcrunch)

• Creating a client to connect to the server

• Building a host application that uses an LLM to translate user requests to tool calls on the MCP server.

• Choosing between stdio and Streamable HTTP transports for MCP

• Bonus : how to use the MCP Server we create in VSCode (or any other tool)

Note: This series is not for those seeking to learn about the "latest productivity hacks with MCP in some existing host application (Cursor, Windsurf etc)."

Key MCP Concepts in Brief

MCP has an excellent and well maintained documentation site. In brief, here are key concepts to get started.

• Server: MCP servers expose capabilities through a standardized interface. A single server can provide multiple tools (functions to call), resources (data to read), prompts (templates for LLM interactions), and sampling (request LLM completions from the client).

• Client: MCP clients maintain 1:1 connections with servers and handle the protocol communication. Hosts embed clients to talk to servers.

• Host MCP hosts are user-facing applications like Claude Desktop, Cursor, or VSCode. They use clients to connect to servers and decide which tools to call based on user needs.

• Transport MCP uses JSON-RPC 2.0 with two transport options - stdio and streamable HTTP. Stdio runs the server as a subprocess using standard input/output - ideal for local integrations where the server runs as a subprocess of the client (e.g., IDE extensions, local development tools). Streamable HTTP uses network requests - better for web applications, distributed systems, multiple clients connecting to one server, and easier debugging/monitoring.

For this tutorial, we'll use the Streamable HTTP as it provides a better learning experience with clearer separation of concerns (you can run the server on a remote machine) and easier debugging.

Building Your First MCP Server

Now that we've covered the core concepts, let's build your first MCP server. We'll use the Python MCP SDK, which is mature and widely adopted, but the same concepts apply to other languages.

Protocol vs SDK ?
MCP mostly defines a protocol or standard - essentially a set of rules that says clients and servers MUST/SHOULD/SHALL/SHALL/NOT do X and Y in order to communicate. Now SDKs are an implementation of these rules.
While you can build your own compliant servers/cleints, in general, it is recommended that you use SDKs for more standardized behaviors where possible.

Our goal: build a tool that can answer news-related queries like "What is the latest AI news on TechCrunch?"

1. Set Up Your Project

Create a new Python project and install the MCP SDK.

Using uv (recommended):

uv init mcp-news-demo
cd mcp-news-demo
uv add "mcp[cli]"

Using pip:

pip install "mcp[cli]"

2. Create the MCP Server

Let's create a simple MCP server with a tool that fetches TechCrunch news.

# server.py
import os
from mcp.server.fastmcp import FastMCP
import requests

mcp = FastMCP(
    "TechCrunch News Server", 
    host=os.environ.get("MCP_SERVER_HOST", "localhost"), 
    port=int(os.environ.get("MCP_SERVER_PORT", 8011))
)

@mcp.tool(title="Fetch from TechCrunch")
def fetch_from_techcrunch(category: str = "latest") -> str:
    """Fetch the latest news from TechCrunch for a given category."""
    allowed = {"ai", "startup", "security", "venture", "latest"}
    cat = category.lower()
    
    if cat not in allowed:
        cat = "latest"
    
    url = f"https://techcrunch.com/tag/{cat}/" if cat != "latest" else "https://techcrunch.com/"
    
    try:
        response = requests.get(url)
        if response.ok:
            try:
                from bs4 import BeautifulSoup
                soup = BeautifulSoup(response.text, "html.parser")
                text = soup.get_text(separator=' ', strip=True)
                return text[:1000] + ("..." if len(text) > 1000 else "")
            except ImportError:
                return response.text[:1000] + ("..." if len(response.text) > 1000 else "")
        return "Failed to fetch news."
    except Exception as e:
        return f"Error fetching news: {str(e)}"

if __name__ == "__main__":
    mcp.run(transport="streamable-http")

3. Run Your Server

Start your server:

python server.py

You should see:

INFO: Started server process [65618]
INFO: Waiting for application startup.
[07/01/25 12:40:34] INFO StreamableHTTP session manager started
INFO: Application startup complete.
INFO: Uvicorn running on http://localhost:8011 (Press CTRL+C to quit)

Your server is now running at http://localhost:8011/mcp and ready to accept requests.

Note: This example uses the high-level FastMCP API with Streamable HTTP transport. For advanced use cases, check the MCP Python SDK documentation for the low-level API.

Building Your First MCP Client

MCP clients connect to servers and handle communication. They operate within "sessions" - logical groupings of requests and responses. Let's build a client that connects to our TechCrunch server. Note that we will use the same streamablehttp transport as the server:

# client.py
import asyncio
import os
from mcp import ClientSession
from mcp.client.streamable_http import streamablehttp_client

async def run_client():
    # Connect to the HTTP MCP server
    host = os.environ.get("MCP_SERVER_HOST", "localhost")
    port = os.environ.get("MCP_SERVER_PORT", "8011")
    server_url = f"http://{host}:{port}/mcp"
    
    async with streamablehttp_client(server_url) as (read, write, _):
        async with ClientSession(read, write) as session:
            # Initialize the connection
            await session.initialize()
            
            # List available tools
            tools_response = await session.list_tools()
            print("Available tools:")
            for tool in tools_response.tools:
                print(f"- {tool.name}: {tool.description}")
            
            # Call a tool
            result = await session.call_tool(
                "fetch_from_techcrunch",
                arguments={"category": "ai"}
            )
            print(f"\nTool result: {result.content}")

if __name__ == "__main__":
    asyncio.run(run_client())

You should now see a list of tools and a result of a call to the tool using `session.call_tool`.

Available tools:
- fetch_from_techcrunch: Fetch the latest news from TechCrunch for a given category.
Tool result: [TextContent(type='text', text='AI | TechCrunch AI | TechCrunch TechCrunch Desktop Logo TechCrunch Mobile Logo Latest Startups Venture Apple Security AI Apps Events Podcasts Newsletters Search Submit Site Search Toggle Mega Menu Toggle Topics Latest AI Amazon  ...

Server and Client Capabilities

Note: `await session.initialize()` returns important details about the server including its capabilities (e.g., if it provides tools, resources, prompts, logging or any experimental/custom features.). This can be utilized by the client or host application to define dynamic behaviors.

Similarly, clients can “advertise support” for client capabilities such as sampling, elicitation and roots during initialization. In the python SDK, this is done by providing `callbacks` for each capability.

async with ClientSession(read, write, sampling_callback=sampling_callback, elicitation_callback=elicitation_callback,list_roots_callback=list_roots_callback, logging_callback=logging_callback) as session:

Building the Host Application

The host application is where the magic happens. It bridges MCP with the outside world (users or business applications), turning user queries or business tasks into tool calls and responses.

Most MCP tutorials skip over how complex host applications really can be. They're not just pass-through layers - they're intelligent orchestrators that often must:

• Accept user requests

• Discover available tools from MCP servers

• Use an LLM to choose the right tools

• Execute tool calls with proper parameters

• Handle errors gracefully

• Present results in a user-friendly way

Let's build a simple host that uses OpenAI to intelligently orchestrate our MCP tools.

Setup

First, install dependencies:

uv add openai
# or: pip install openai

Set your OpenAI API key:

export OPENAI_API_KEY="your-api-key-here"

The Host Application

# app.py
import asyncio
import json
import os
import sys
from openai import AsyncOpenAI
from mcp import ClientSession
from mcp.client.streamable_http import streamablehttp_client

def convert_mcp_tools_to_openai_format(tools):
    """Convert MCP tool definitions to OpenAI function calling format."""
    openai_tools = []
    for tool in tools:
        openai_tools.append({
            "type": "function",
            "function": {
                "name": tool.name,
                "description": tool.description,
                "parameters": tool.inputSchema if hasattr(tool, 'inputSchema') else {}
            }
        })
    return openai_tools

async def handle_user_request(session, openai_client, tools, user_input: str):
    """Process user request using LLM and MCP tools."""
    openai_tools = convert_mcp_tools_to_openai_format(tools)
    
    # Ask LLM to decide which tools to use
    response = await openai_client.chat.completions.create(
        model="gpt-4",
        messages=[
            {
                "role": "system", 
                "content": "You are a helpful assistant that can fetch news from TechCrunch. Use tools when needed."
            },
            {"role": "user", "content": user_input}
        ],
        tools=openai_tools,
        tool_choice="auto"
    )
    
    message = response.choices[0].message
    
    # If LLM wants to use a tool, execute it
    if message.tool_calls:
        tool_call = message.tool_calls[0]
        function_name = tool_call.function.name
        function_args = json.loads(tool_call.function.arguments)
        
        print(f"🔧 Calling tool: {function_name} with args: {function_args}")
        
        result = await session.call_tool(function_name, arguments=function_args)
        
        # Format the response
        content = str(result.content)[:500]
        return f"Here's what I found:\n\n{content}...\n\nFor more details, visit TechCrunch directly."
    else:
        return message.content

async def main():
    if not os.getenv("OPENAI_API_KEY"):
        print("Error: Set OPENAI_API_KEY environment variable")
        return
    
    # Get user input
    user_input = " ".join(sys.argv[1:]).strip() or "What is the latest news on AI?"
    
    # Initialize OpenAI client
    openai_client = AsyncOpenAI()
    
    # Connect to MCP server
    server_url = f"http://localhost:8011/mcp"
    
    async with streamablehttp_client(server_url) as (read, write, _):
        async with ClientSession(read, write) as session:
            await session.initialize()
            
            # Get available tools
            tools = (await session.list_tools()).tools
            
            print(f"Task: {user_input}\n")
            
            # Process the request
            response = await handle_user_request(
                session, openai_client, tools, user_input
            )
            
            print(response)

if __name__ == "__main__":
    # Make sure server is running first!
    asyncio.run(main())

The host application above combines three key components: an MCP client, an LLM (OpenAI), and user interface logic. When you run python app.py "What's the latest AI news?", here's what happens: First, the host connects to the MCP server using its embedded client and calls list_tools() to discover available tools. It then converts these MCP tool definitions into OpenAI's function-calling format using convert_mcp_tools_to_openai_format(). Next, it sends your question along with the tool definitions to OpenAI's API. OpenAI analyzes your question and returns which tool to call - in this case, fetch_from_techcrunch with category='ai'. The host then executes this tool call through session.call_tool(), gets the raw results from TechCrunch, and formats them into a readable response. This architecture lets you ask natural language questions without knowing which tools exist or how to call them - the LLM figures that out based on what's available.

A similar host application format can be extended to build complex agentic or multi-agent applications.

Putting It All Together

You now have a complete MCP application:

1. Server (server.py) - Exposes the TechCrunch fetching tool

2. Client (client.py) - Shows how to connect and call tools directly

3. Host (app.py) - Uses AI to intelligently orchestrate tool usage

To run the full system:

# Terminal 1: Start the server
python server.py

# Terminal 2: Run the host application
python app.py "What's the latest AI news?"

Oh .. and the server above can be used in any of your favorite MCP host applications. For example, you can add it to your VSCode agent by adding an mcp server config to your user settings. The same configuration will work for Claude Desktop, Cursor, Windsurf …

"demo": {
        "type": "http",
        "url": "http://localhost:8011/mcp/",
        "headers": { "VERSION": "1.2" }
      }

Further Reading:
I wrote a book on on Designing Multi-Agent Systems - Chapter 12 (Protocols for Distributed Agents) provides a perspective on MCP and. A2A and when you should reach for them as you build agents.

• Book Digital PDF:

https://buy.multiagentbook.com/

• Book on Amazon: https://www.amazon.com/dp/B0G2BCQQJY

Read more

The Designing with AI newsletter just hit 6,000 subscribers. Thank you!

I spend an awful lot of my free time writing about AI - late nights, weekends, whenever my 6-year-old is asleep (his time is precious!). For the past decade, I've built interactive prototypes and written mostly for a technical audience - software engineers and researchers on my personal website and now on Substack. As examples - I built the first interactive demo showing how to train deep neural networks for anomaly detection in the browser using TensorFlow.js, how to train a Generative Adversarial Network from Scratch for African Mask Art generation (GANs, TPUs) and interactive demos showing how Convolutional Neural Networks work for semantic image search etc. The goal here was mainly to help technical audiences learn new technical things.

But in the last ~3 years, something has changed. Capable, complex AI has become commoditized. ChatGPT, Sora, Veo - these tools have brought topics I once explained to engineers and researchers into the mainstream. Suddenly, non-technical users are interacting with AI daily, people with no background in software engineering are now “vibe coding” apps, and teams are scrambling to integrate this technology into everyday tools. Furthermore, AI systems are becoming more agentic - i.e., they can perform actions (e.g., send emails, reserve tickets, write code) autonomously on behalf of a user.

Advanced AI (by some standards) itself is becoming normal technology.

From my vantage point (as someone who works in the AI field), I can't help but see many side effects to this rapid shift. Specifically, I think that a rapid shift unaccompanied by adequate preparation could be problematic. One analogy that comes to mind is giving a 16-year-old a Ferrari LaFerrari a month after he gets his driver's license. Now the Ferrari itself (an incredibly fast and sophisticated car) is not problematic. However, the teenager's lack of experience, judgment, and training/education inherently makes the situation problematic.

Similarly, the proliferation of AI should be accompanied by a proliferation of AI literacy and education to enable effective and responsible use. Deep down, this is why I write.

Why I Write About AI

If I could bucket the reasons I write, I’d organize them into the following 4 areas.

1. AI Literacy

I think that like internet or digital literacy, AI literacy - how to successfully and efficiently use AI for any and all tasks - will be critical. If we agree that AI will drive an increasing slice of decisions made by the general population, then it is important that the general population understand how AI works. It can be as simple as highlighting the limitations of generative AI models today (hallucinations etc) to under understanding how to effectively use AI (as a tool) for specific tasks.

2. Developers Need to Understand How to Build with AI

All engineers should understand how to build with AI and integrate AI into solutions. AI enabled software engineering is now a thing. It is changing how software is build and consequently the expectations of employers.
IMO, developers that don’t use AI for software engineering tasks are likely behind.
Recently, the CEO of Shopify declared that employees must reflexively use AI. A quote is below:

Before asking for more Headcount and resources, teams must demonstrate why they cannot get what they want done using AI. What would this area look like if autonomous AI agents were already part of the team? This question can lead to really fun discussions and projects.

It behooves all engineers to at least understand how AI and AI agents work.

3. AI Can Have Adverse (and Unintended) Side Effects

AI can have adverse side effects. Some of these are subtle (e.g., AI fatigue) while others could lead to potentially catastrophic outcomes (AI that becomes severely misaligned). Understanding these risks is essential for responsible development and deployment.

A while ago, I reflected on an incident in which OpenAI released a model that was sycophantic and how naive revenue/product signal optimization directly leads to such adverse (mostly unintended) outcomes.

4. The UX of Agentic AI is Unknown

The core ways in which humans interface with increasingly autonomous or agentic AI is unknown. If AI driven software can reason through novel problems, act on human behalf (agents), how do we build interfaces that keep the human in the loop, provide visibility into agent action, facilitate approvals at the right level of granularity, how do we build experiences that do not diminish critical thinking? etc. We will only discover this unknown territory by building, experimenting and writing.

Bonus: Writing for Myself

Beyond the broader external goals I've outlined, I also write for deeply personal reasons. Writing forces me to clarify my thinking - transforming scattered thoughts into coherent arguments that are far more refined than anything I could produce in casual conversation. It also creates a valuable resource I can draw from repeatedly, whether I'm speaking at conferences, advising teams, or simply discussing these topics with colleagues.

.. You Should Write Too!

With all these points listed, I sincerely believe there just cannot be "enough" writing about AI. Sometimes, it can be hard to reason about the mechanics of social network theory - how multiple individuals can write about the same thing but hit completely different audiences. How you just might be the only source of that type of information within your network, might have a unique perspective and be right person to share.

You should write too.

All of this is to say - you should write too!

I have a long list of folks who actively write about the AI space that I admire and find their work useful - (interconnects) , (artificial ignorance), , (Gonzo ML), Swyx and a few others. Definitely follow their work!

I keep coming back to this article by on important things to work on. Miles advocates for more work on AI literacy, demonstrations that clearly illustrate the increasingly complex (as well as dangerous) capabilities of AI, tools to inspect and make sense of agentic AI, the need for policy, amongst other topics. And I agree with all these points.

And to the 6000 of you who have found my writing worth following, thank you and cheers to co-navigating the AI journey ahead.

I gave a keynote talk at the 2025 AI.Engineer conference - “UX Design Principles for Semi-Autonomous Multi-Agent Systems”

It was a short talk and my goal was to distill a set of UX design principles that developers could apply while building experiences for autonomous multi-agent systems. It is largely based on my experience prototyping multi-agent systems, maintaining AutoGen (a framework for building multi-agent apps), as well as building and maintaining AutoGen Studio - a low code interface for prototyping multi-agent applications.

Also, to illustrate these ideas, I built a multi-agent system from scratch (no frameworks, just pure Python and OpenAI calls) - BlenderLM. BlenderLM is a multi-agent system that can take a natural language query, and drive the Blender 3D creation tool (via its Python api) in addressing these tasks.

TLDR on the principles I recommended;

- Capability discovery (Help users understand what the agents can do)
- Observability and provenance (Ensure users can observe/trace agent actions)
- Interruptibility (Allow users to pause, resume or cancel agent actions)
- Cost-Aware delegation (Communicate the cost of agent actions, allow users decide when agents can act)

Beyond Workflows to Multi-Agent Systems

I wrote in a 2024 AI Agent rewind that most “agents” in production are mostly workflows - a set of deterministic steps designed by an engineer with a few LLM calls sprinkled in. While this provides reliability and is indeed the right approach to solve many tasks; they still require that the developer knows the exact solution to the problem and can express that in the outlined steps.

On the other hand, Multi-Agent Systems where LLMs drive control flow of execution, are suited to problem spaces where the exact solution is unknown. Software that can take actions, observe results, and interactively explore solution spaces - but it also introduces new challenges around reliability, user trust, and system comprehensibility. Some common use cases that are seeing benefits from a MAS approach includes software engineering, back office tasks, deep research.

The UX principles discussed here apply to these types of systems - systems with autonomy, ability to act on complex long-running tasks.

IMO there are properties of MAS to keep in mind that often necessitate careful UX design considerations:

• Autonomy | Can do many different things

• Action | Can take action with side effects

• Duration | Complex long- running tasks

Building A Multi-Agent System From Scratch

I often see developers start the agent development process by immediately attempting to code up an Agent class or some agentic behaviors- e.g., selecting the prompt/model, fretting over concepts like memory etc. This is often a mistake.

Instead, if we take a step back from the current agent hype, it’ll become clear that a more typical software development life cycle approach still applies. I tend to follow a 5-stage process :

1. Goal definition (what are we building)

2. Baseline (what is a non-agentic baseline)

3. Tools (Architect, build and test the tools that the agent will use. You should spend the most time here)

4. Eval testbed (Interactive e.g., Jupyter notebook, UI or offline benchmark for evaluating systems)

5. Agent (Only work on this after steps 1-4 above - begin with a tight loop of an LLM calling tools and optimize e.g., add more agents and evaluate)

Also about 70% of effort should be split between step 3 and 4 - across building your tools and tuning your evaluation harness.

One of the reasons for building a demo from scratch for this talk was to enable me to critically think through my process, as well as validate many of the UX principles I have followed while building and maintaining AutoGen Studio. I built BlenderLM - a sample tool that can take natural language tasks e.g, “create a scene with two glossy balls” and uses an agentic system to drive the underlying Blender Python api to accomplish the task.

TLDR; BlenderLM consists of 3 agents. It includes a React UI where the user can initiate tasks that kick off an agent run with bidirectional communication over a web socket.

• Main /Orchestrator Agent. It has access to a set of task-specific (e.g, create blender object) and general purpose (code execution) tools, and can call other agents in a loop to address a task.

• Planner Agent: Takes the task and system state and generates a plan to address the task

• Verifier Agent: Takes a task and system state and generates an assessment on if the task is accomplished. This is used with some retry logic to determine when steps in a task are accomplished.

The details of BlenderLM will probably be a separate post!

Design Principles

The 4 principles below are not exhaustive, but have pretty high coverage. Multi-agent systems are still early and it is likely the ideas below will be revised as the space evolves.

Capability Discovery

Help users understand what the agents can do

Agents have autonomy. This implies that they can do many things. In reality, each agent has specific configurations—system prompts, available tools, and internal logic—that make it more reliable at certain types of tasks compared to others. But the user does not know this. This creates a discovery problem: users don't know which tasks will work well and which might fail, leading to frustration and mistrust when the agent underperforms on unsuitable tasks.

This principle advocates that the UX nudges the user towards high reliability task examples or even proactively suggests relevant high reliability tasks given the user’s context and the system’s capabilities.

Many tools already do this well by having a set of sample tasks as presets that the user can select to get started.

Observability and provenance

Ensure users can observe/trace agent actions

Autonomous agents can explore trajectories that are non-deterministic and only known at run time. Each run with the same input can lead to significantly different trajectories. This makes it important for the end user to observe these trajectories both to build trust that the agent is doing the right thing as well as to learn more about its capabilities and limitations, and improve their task formulation approach.

In BlenderLM, this is implemented by streaming all updates the agents explore - structured plans, the number of steps, current steps, and where appropriate showing duration and cost (LLM tokens) etc. Tools like Agent mode in VS Code/Cursor/WindSurf also do this well by streaming the actions as agents make progress.

Interruptibility

Allow users to pause, resume or cancel agent actions

Autonomous multi-agent systems can run for extended periods, make multiple tool calls, and take actions with real-world resource implications. In some cases, particularly in human-in-the-loop settings, users may observe an expensive or problematic operation trajectory and need to pause the system to provide feedback, course-correct, or cancel entirely.

This principle advocates for designing systems where users can interrupt agent execution at any point, pause long-running tasks, and resume from where they left off without losing progress or system state.

BlenderLM implements this with pause/stop controls that allow users to halt execution at any point during a task.

Cost Aware delegation

Quantify and communicate the cost of agent actions, allow users to decide when agents can act.

Agents can act, and these actions can have side effects that range from benign (executing code to add a cone to a 3D scene) to catastrophic (executing code that deletes critical system files). Some actions are reversible and can be rolled back, while others cannot e.g., once an email is sent to the CEO, there's no taking it back. To an agent, the costs, risks, and safety implications associated with each action may not be immediately apparent without specific design.

This principle advocates for implementing specialized "risk/cost classifiers" that can assess and return risk levels such as low/medium/high. In turn, the UX should then allow users to control and configure response behavior—for example, automatically allowing all low-risk actions while requiring explicit approval for medium and high-risk operations.

Key Takeaways

A few high-level takeaways.

Perhaps the first and most important one is to know when to use an autonomous multi-agent approach. I typically offer a complex task perspective i.e., does the task benefit from planning, can steps be decomposed into components that benefit from specialization, does it require processing extensive context that can be siloed to specific agents, and is the environment dynamic (i.e., we need to constantly sample to understand impact of actions).

Takeaway summary.

• Know when to use multi-agent (most tasks don't need it)

• Build eval-driven : (define success metrics first)

• Design human-centered (capability discovery, observability, interruptibility, cost-awareness)

• Start simple (baseline → tools → agents → polish)

Most of the material here is adapted from Chapter 3 of my upcoming book on building multi-agent systems.

Acknowledgement: The ideas here have benefited from conversations with fellow contributors to AutoGen at Microsoft and the broader OSS community.