Agent Handoff Latency: ~300ms Per Hop vs. One LLM Call

A three-hop agent relay surrenders roughly 900 milliseconds — about 300 per hop — to routing, state serialization, checkpoint writes, and cache-cold re-prefills before the third model has read a single word. In that same window, a single GPT-4o-class call has already streamed back half of its final answer. That is the uncomfortable arithmetic of most 2026 multi-agent stacks: the coordination layer, not the model, sets the ceiling.

The pattern deserves a harsher name than 'orchestration.' Every agent added to a pipeline is another model in the loop, with its own latency, cost, and failure mode — the same fallacy that once equated more microservices with more scalability. The design rule at infostatus.com.au is blunt: the right number of agents is the smallest number that makes the system reliable. Architectures adopted for résumé reasons rather than measured gain are latency theater.

The alternative is discipline, and it is measurable. SPOQ (arXiv 2606.03115) assigns models to tiers — Opus for workers, Sonnet for reviewers, Haiku for investigators — and gates every task with validation before and after execution. Across a longitudinal deployment of 17 repositories, 8,589 commits, 1,822 tasks, and 13,866 tests, it held a 99.87% pass rate. Because the gains replicated on a locally hosted Qwen3.6-35B-A3B, the authors credit orchestration, not any specific model. The fix is architectural, not a faster token stream.

Anatomy of a Hop: Where the 300ms Actually Goes

Strip out the model's own inference time and a handoff still costs you roughly 300 milliseconds of pure plumbing per hop in the mainstream Python stacks — LangGraph, the OpenAI Agents SDK, CrewAI. That figure is the handoff tax: the non-model wall-clock added every time control passes from one agent to the next. Most teams assume it's dominated by network latency to the provider. It isn't. The largest slice is your own context being re-read, and it's the one component you fully control. As of this writing, the working decomposition — the one the rest of this guide uses — looks like this:

Why does re-prefill dominate as chains deepen? Because a naive relay re-sends the accumulated conversation at every hop: hop n carries the union of all prior agents' outputs, so total input grows quadratically across the chain. Without a stable, reusable prompt prefix, every hop pays full-price prefill compute on context the previous hop already encoded — the same tokens, tokenized and attention-scored again, at growing length.

The provider-side countermeasure has a name: automatic prefix caching. OpenAI, Anthropic, and Google's implicit caching all maintain server-side KV stores keyed on identical prompt prefixes. The design consequence is strict: handoff payloads must be built append-only. Mutating the system prompt between hops ("you are now the reviewer"), injecting a fresh timestamp into the preamble, or reordering messages invalidates the cache and restores the full ~300ms. Keep the shared preamble byte-identical and place per-hop instructions after it — this is the cheapest latency win available in any of these frameworks, and it costs nothing but discipline in payload construction.

Not all hops are shaped alike, either. An OpenAI Agents SDK-style agent-to-agent transfer passes a shared conversation thread and issues one new inference request per hop — a single network leg, naturally cache-friendly if you keep it append-only. An MCP-style tool round-trip runs model-to-client-to-server-to-model: the model invokes a tool, the client forwards the call to the MCP server, the server responds, and the model resumes. That doubles the network legs per hop and inserts a JSON schema-validation stage between them — the same logical hop, a materially worse tax profile.

Now the compounding law the rest of this guide relies on: serial hops add taxes linearly — n hops ≈ 300ms × n — but context growth makes later hops slower than earlier ones, so the measured per-hop cost rises with chain depth. Treat 300ms as a floor, not a constant. Practically: when you benchmark a chain, measure the marginal cost of the last hop, not the first — the first hop flatters the design. And note the escape hatch: a hop whose subtask runs in parallel rather than in sequence doesn't pay this tax serially at all, which is precisely the condition under which adding a hop ever pays for itself.

The Published Record

Several hundred milliseconds. That is the median time-to-first-token band reported on Artificial Analysis's independent latency leaderboard for frontier chat endpoints — GPT-4o-class and Claude Sonnet-class models alike. Set the per-hop plumbing tax quantified in the anatomy section above against that band and the arithmetic is uncomfortable: a single handoff costs roughly half to three-quarters of what the model itself takes before emitting its first token. Handoff overhead is not noise beside inference; it is the same order of magnitude, billed on every hop, before the model does any work at all.

When a latency leaderboard, two pricing pages, a systems paper, and two engineering essays all point the same way, that is about as close to unanimity as this field produces. The SGLang team's RadixAttention paper (NeurIPS 2024) reports multi-fold throughput gains on multi-turn LLM workloads by keeping radix-tree KV caches warm across requests — academic confirmation that cross-request context reuse, not additional agents, is where multi-step latency is actually won. Anthropic's engineering post "Building Effective Agents" (December 2024) recommends the simplest single-LLM-call workflow that works and instructs teams to add multi-agent complexity only against measurable benchmarks — the strongest vendor statement on record aligned with the anti-handoff position. Cognition's June 2025 essay "Don't Build Multi-Agents" supplies the qualitative complement: splitting context across agents fragments decision-making and degrades output quality, so the serial specialist chain loses on latency, cost, and coherence simultaneously.

The working takeaway: before wiring a second agent, ask whether your multi-step problem is actually a cache problem. If the working context fits in one window, the published record says the cheapest latency win on the table is one warm, append-only prefix inside a single call — and the vendors' own price lists are the receipt.

Score the two topologies honestly and the serial specialist relay loses five of six rows — latency, input-token cost, failure surface, observability, and horizontal scalability — keeping quality ceiling, and only under two conditions most production workloads never demonstrate. The table is the decision instrument; the verdicts underneath are the mechanism.

Scoreboard

Latency and cost fail for one structural reason: the chain multiplies fixed overheads. The single call pays one time-to-first-token plus one generation — roughly 1–2 seconds for a few-hundred-token output — while the relay pays n × (the per-hop tax quantified in the anatomy section + model time) on a context that grows at every boundary, putting three hops at roughly 2.5–3× the single-call wall clock. Billing compounds identically: per the current OpenAI and Anthropic rate cards, uncached repeated context bills at full input price on every hop, so a three-hop relay sharing a context of thousands of tokens pays approximately 3× the input tokens of one call even under best-case caching.

Reliability fails at the boundaries. Every hop inserts a parse-validate-route step where malformed JSON, schema drift, or a router misfire halts the run, and LangSmith trace reviews routinely flag intermediate-step errors as a top failure class in agent deployments. As infostatus.com.au puts it, every agent you add is another model in the loop with its own latency, cost, and failure mode. Observability degrades in step: one call is one trace where every token is attributable; a chain is n traces stitched across handoffs, with the plumbing milliseconds living in framework code that model-level dashboards never display. The chain's lone reliability credit — the failure isolation that GoPenAI's sixteen-specialist pitch leads with — survives only where isolation genuinely contains blast radius. The horizontal-scaling pitch inverts the same way: replicate a serial critical path and you get n slow lanes, not n fast ones; the stateless single call is what actually replicates for free.

Quality ceiling is the chain's single win, and it is conditional twice over — it holds only when the working set exceeds one context window or the subtasks are genuinely independent enough to parallelize. Even the strongest published chain result bought its quality with machinery: according to the SPOQ longitudinal deployment study (arXiv 2606.03115), a three-tier hierarchy of Opus-class workers, Sonnet-class reviewers, and Haiku-class investigators, plus dual validation gates, held a 99.87% pass rate across 13,866 tests in 17 repositories. The published alternative to a specialist relay is a specialist model, not a specialist chain: per the Table-LLM-Specialist paper (arXiv 2410.12164), fine-tunes built on GPT-3.5 often surpass GPT-4-level quality on table tasks at reduced latency and cost. And handoffs leak context — the ContextBranch write-up documents ten turns of OAuth debugging with Claude Sonnet that do not survive a copy-paste to GPT-4.

Tie-breaker: if any row other than quality ceiling favors the chain in your own workload, re-measure before trusting it. Export the trace, count the hops, sum the non-model milliseconds, and bill the duplicated input tokens. The default configuration loses five of six dimensions; an exception is earned with data or not at all.

None of the three vendors will sell you the number this guide leans on. LangChain's LangGraph documentation specifies handoff semantics and checkpointer interfaces; OpenAI's Agents SDK reference defines the handoff primitive; CrewAI's docs describe sequential versus hierarchical process modes. Not one publishes a per-hop latency budget, because none guarantees one. Every figure circulating about the plumbing premium profiled earlier in this guide comes from community profiling, not vendor SLAs — and that provenance constrains what the data can honestly claim.

What the Data Doesn't Tell You

Three limitations matter most. First, the measurements are warm-path artifacts: pre-established connections, small serialized payloads, a single cloud region, off-peak load, and median reporting. Cold-started workers, rate-limit backoff, and retry storms sit outside the sample. Second, there's a category error risk when borrowing from adjacent instruments — Artificial Analysis's latency leaderboard, cited earlier for time-to-first-token bands, measures direct chat endpoints, not orchestrated hops between framework nodes. A number calibrated for one instrument does not transfer to the other. Third, publication bias runs hard in this space: teams that wired a five-agent relay, watched it lose to one call, and deleted the repo do not write postmortems.

Variance across cases is structural, not noise. The overhead is payload-coupled: a hop passing a pointer costs materially less than one pushing a full conversation transcript through Pydantic-validated state. The checkpointer matters — an in-memory saver and a Postgres-backed saver over a network live in different latency universes. Framework minor-version drift moves the constant; a figure pinned to one release train decays within quarters. And statistics don't compose the way intuition suggests: chaining hops triples your median exposure but compounds tail exposure faster, because p95 events multiply across hops.

The deepest gap is baseline under-optimization. In most public comparisons, the multi-agent pipeline gets weeks of prompt engineering while the single-call baseline gets an afternoon. The quality-ceiling advantage some chains show in narrow evals may partly measure tuning asymmetry, not architecture. Kill the myth now: a published median is a prior, not a contract.

So when does the premium break in the rule's favor? Only in the escape hatches the decision rule already grants — and each is unfalsifiable until you instrument it. The premium is justified only when subtasks are genuinely independent enough to run in parallel and recover wall-clock time; when isolation is the product itself (an adversarial review pass that must not see the drafter's reasoning, or PII-scoped processing); when the working set exceeds one context window even after compression; or when tool execution dominates the step so completely that plumbing is rounding error. If you cannot demonstrate one of those four conditions from your own traces, you are choreographing, and the rule holds.

The last row wins: if you log nothing else, log the distribution. Before wiring hop N+1, replay your real trace — your payloads, your region, your concurrency — against a single-call baseline at your own tail latency. Any constant you read anywhere, including here, is a hypothesis your instrumentation must confirm.

Quantiles don't add, but they don't cancel either — and that asymmetry is where most SLA math dies. Published handoff benchmarks report medians; production incidents live in the tail. Take four hops with independent latencies, each with a p95 well above its median. Convolve them and the end-to-end p95 lands well above 4 × 300ms — the additive estimate everyone quotes — because upper quantiles survive summation even when medians average out. Correlated hops (one shared database, one congested network path) stretch the tail further still. If a pager depends on p95, measure your own tail distribution; quoting a vendor's median is borrowing someone else's luck.

Where the 300ms Rule Breaks

The second break is context length. The tax model is calibrated for small working sets, where prefill stays sub-second and plumbing is a visible fraction of the bill. Push well beyond that range and a single cache miss costs multiple seconds of prefill compute — the exact figure depending on provider hardware — which dwarfs the per-hop overhead entirely. Prompt caching turns this into a cliff function: warm context makes the single call unbeatable; a cold miss makes every topology expensive. Check whether your working set even qualifies before applying any hop arithmetic.

Third, the false-parallelism hazard. A supervisor fanning out to eight concurrent workers escapes the additive tax only if the branches truly execute concurrently. In LangGraph-style runtimes, execution proceeds in super-steps, and a globally shared checkpointer can serialize those steps behind a single database lock — a common misconfiguration with Postgres-backed savers. The trace looks parallel; the wall clock is a relay. The additive penalty returns silently, with no error and no warning. Confirm overlap empirically: pull a distributed trace of a production fan-out and verify that branch start times actually coincide.

Fourth, the honest caveat: sometimes decomposition buys accuracy worth seconds. Sampled self-consistency and verifier–critic loops show documented gains on math and code-generation benchmarks, and according to the SPOQ paper on arXiv, those gains replicate on a locally hosted open-weights model, Qwen3.6-35B-A3B — evidence the wins belong to orchestration, not to any particular model. Human-as-an-Agent integration extends the logic: a human specialist consulted mid-execution contributes judgment no prompt recovers. Latency-optimal is not accuracy-optimal, so the decision rule carries a quality escape hatch — pay the hop tax when the cost of a wrong answer exceeds it.

Fifth, the constant moves. The 300ms floor is a 2026 measurement against current SDK internals and network conditions; provider-side batching, speculative decoding, and faster interconnects have already compressed it well below the figures of early agent frameworks. Treat it as configuration, not physics.

Sixth, perception inverts the ranking. Streamed tokens make a slow single call feel faster than a fast-but-blocked chain, and streaming-tolerance work puts user patience at roughly one second to first visible output. For interactive products, a slower-but-streamed answer can legitimately beat a faster-but-blocked chain on completion and satisfaction; for batch jobs, raw latency still rules. Optimize time-to-first-token, not just total time.

Run this audit before your next architecture review: export per-hop p95 from your own traces, replay one fan-out under production load and check branch overlap, log your working-set sizes, and put a quarterly re-baseline on the calendar. The rule survives all six breaks — if you know which side of each one you're on.

Classify, retrieve, draft — the default "multi-agent" demo — takes 2.75 seconds when three LangGraph specialists take turns and 1.85 seconds when a single GPT-4o-class call does the whole job with two tool invocations. Same ticket bundle, same underlying model class, same answer. Build it both ways and the ledgers diverge on every line that matters.

Ticket Triage at 1.85s vs. 2.75s

The setup is deliberately ordinary. An inbound support ticket arrives; the job is to label it billing, bug, or how-to, pull the matching documentation snippet, and draft the reply. Version A is one call that invokes two tools mid-generation. Version B is a LangGraph relay — Router → Retriever → Drafter — with a SqliteSaver checkpointer persisting state between nodes, the exact shape most production graphs ship.

Version A's arithmetic: one time-to-first-token, 900ms generating its output, two tool round-trips — ~1.85s end-to-end, with cumulative input tokens billed across the transcript, tool-result appends included. One transcript, billed as it grows.

Version B pays per hop, and the tax is not flat:

Total: ~2.75s — roughly half again as slow as Version A, with ~1.0s of pure handoff tax.

Frequently Asked Questions

Where do the ~300 milliseconds per agent handoff actually go?

Roughly 50ms goes to framework routing and state serialization in LangGraph's Pregel loop, ~60ms to the default SqliteSaver checkpointer disk write, ~80ms to the US-East network round-trip and queueing, and cold-cache re-prefill of carried-over context is the largest single slice.

Does every hop in a chain really cost the same 300ms?

No — serial hops add taxes linearly at roughly 300ms × n, but because a naive relay re-sends the union of all prior agents' outputs so total input grows quadratically, the measured per-hop cost rises with chain depth, meaning you should benchmark the marginal cost of the last hop rather than the flattering first one.

What common payload mistakes invalidate automatic prefix caching between hops?

Mutating the system prompt between hops, injecting a fresh timestamp into the preamble, or reordering messages invalidates the server-side KV cache keyed on identical prompt prefixes and restores the full ~300ms, so keep the shared preamble byte-identical and place per-hop instructions after it.

Is an MCP tool call taxed the same as an agent-to-agent handoff?

No — an MCP-style tool round-trip runs model-to-client-to-server-to-model, doubling the network legs per hop and inserting a JSON schema-validation stage for the highest floor and two queueing exposures, while an OpenAI Agents SDK transfer issues one inference request over a shared thread and is naturally cache-friendly when kept append-only.

How do providers price repeated prompt prefixes?

OpenAI guarantees a 50% cached-input discount on gpt-4o subject to a minimum prompt-prefix length, while Anthropic offers 90% off cache reads but imposes a write surcharge on cache writes.

When does adding another agent hop ever pay for itself?

Only when the hop's subtask runs in parallel rather than in sequence, since a parallel hop doesn't pay the ~300ms tax serially — otherwise extra hops are justified only where they buy measured quality, such as SPOQ's Human-as-an-Agent review reducing residual defects from 0.47 to 0.03 per task.

Quick answers

Also worth reading: Orchestrate AI agents with mixed latency profiles: Orchestrate AI agents with mixed · State persistence strategies for long-running AI agents: State persistence strategies for long-running · LLM Verifier Audit Trail Beats Smart Agent in Stanford Test: LLM Verifier Audit Trail Beats