Migrate agent workflow: migrate if 95th percentile (p95) latency exceeds 2s across 3 tools

This guide defines when to migrate an agent workflow using p95 latency across three tools.

It covers instrumentation, sampling, and migration actions without relying on averages.

Why p95, not the average, decides

Latency distributions are right-skewed: a floor of fast responses sits beneath a long tail of slow ones, so the mean gets dragged by that tail but never shows the tail itself (qaskills.sh, June 2026). The mean averages the fast floor and the slow tail together, so it cannot distinguish a uniformly fast service from one with a rare five-second wait.

The worked arithmetic from qaskills.sh shows how severe that dilution can be: 99 calls at 50 ms plus one call at 5,000 ms yield a mean of 99.5 ms, roughly 100 ms. One real user waited five seconds, yet the average reports a tenth of a second because it smears that single wait across all 100 calls.

Percentiles avoid that smearing by stating how many requests were slower than a given value. p95 means 95% of requests were faster, so one in twenty real requests was slower; p99 means one in one hundred was slower. At one million requests per day, that one-in-one-hundred p99 pain equals 10,000 affected users (qaskills.sh, June 2026).

p95 is the deciding measure because it captures the slow-but-common tail rather than the rarest edge. p99 remains useful for painful outliers, but p95 is where slowness becomes frequent enough to describe ordinary production traffic instead of an exceptional case.

When reading a tool's results, compare p95 with the mean before drawing any conclusion. A p95 far above the mean signals a tail that the average is understating; in that situation, the mean is not evidence that the workflow is fast enough.

Treat the average as context, not as the gate. If the tail is common enough to move p95, the user pain is real even when the mean still looks small.

How the 3-tool p95 gate works

Instrument each of the three tools independently, recording latency for three spans: the agent step, the tool invocation, and the response return. Keep those measurements in separate per-tool distributions and compute p50, p95, and p99 for each tool, but treat them as diagnostics rather than as the migration decision. Raw samples matter because a summary percentile cannot be reconstructed accurately from another summary percentile.

The migration decision uses one workflow-level p95 computed across complete end-to-end runs. Collect the elapsed time from the start of an end-to-end run to its final response, then calculate the p95 over that single population of runs. A workflow run includes sequencing, retries, handoffs, and waiting between tools, so per-tool p95 values do not describe the user-visible duration. Do not derive the workflow value by averaging the three per-tool p95 values, because percentile aggregation methods vary by tool and exact versus approximate percentiles cannot always be averaged (qaskills.sh).

Wire the 2-second threshold as a pass/fail gate on that workflow-level p95. The pattern is the same one k6 uses when its thresholds block turns percentiles into automated checks in pipelines: the run passes when the workflow p95 is at or below 2 seconds and fails when it exceeds 2 seconds (qaskills.sh, June 2026). The gate is evaluated on the same population used to compute the percentile, not on a subset or on a dashboard approximation.

Keep the per-tool p50, p95, and p99 values available for diagnosis after a failure, but let the single end-to-end p95 decide the gate. That separation prevents a fast tool from masking a slow tool and prevents an averaged percentile from hiding the slow-but-common tail. This keeps the measurement recipe stable even when one tool changes its percentile implementation.

If the gate fails, inspect the per-tool p95 values to identify the slowest tool, then reorchestrate, refactor, or swap that tool. The gate itself remains the workflow-level p95, so diagnosis and decision stay separate.

The evidence on percentile-based gating

The case for percentile-first SLOs in 2026 rests on convergence across independent sources rather than on a single vendor rule. qaskills.sh's June 2026 percentile guide, OneUptime's September 2025 latency guidance, and LangChain's LangGraph documentation each address a different layer of the same problem: how to measure tail latency, how to turn it into an objective, and how to observe it inside an agent runtime. Read together, they support gating migration decisions on p95 rather than on an average.

qaskills.sh, published June 15, 2026, states that every serious performance report should lead with p95 and p99, not the average. It also treats thresholds as pass/fail checks and shows how teams wire those gates into test pipelines, using tools such as k6 thresholds blocks as the mechanism. That framing matters because it moves percentiles from descriptive reporting into enforceable pipeline conditions.

OneUptime, published September 15, 2025, documents that averages lie about latency performance and that p95 and p99 are the correct basis for SLOs. Its guidance also includes practical debugging tips organized by percentile band, so a breach is not only a pass/fail signal but a diagnostic pointer to where the tail begins.

LangChain's LangGraph documentation positions agent orchestration around reliable, observable agent runtimes and a low-level orchestration framework. In that context, per-step latency is useful only when the runtime exposes dependable observations across agent steps; percentile gates depend on that observability to be meaningful rather than decorative.

The convergence is therefore methodological: qaskills.sh makes p95 and p99 the lead metrics and gate inputs, OneUptime makes them the SLO basis with band-level debugging, and LangGraph supplies the runtime context in which per-step latency is observed. None of these sources asks teams to trust an average, and together they justify treating tail latency as the gating condition for workflow migration.

Migrate, refactor, or stay: options

The decision is not binary. Treat the workflow as having four legitimate responses, and choose by which percentile band is breaching and whether that breach survives a targeted fix.

Targeted refactor is the default winner because it attacks the slow-but-common tail directly: replace a blocking API call with streaming or caching, then re-measure. It keeps the orchestration, contracts, and observability already in place, so the change stays reversible and cheaper than moving the whole workflow.

Full migration is the correct escalation, not the starting point. Swap the orchestration framework or tool only when p95 still breaches after the slowest tool has been individually optimized; that pattern points to coordination, retries, or framework overhead rather than one call.

Doing nothing is defensible only in the narrow case where p50 is stable, the p95 breach is confined to a cold-start window, and no user-facing SLO exists. Even then, the tail should remain tracked so a cold start does not quietly become the normal path.

Use the table as a gate, not a menu: classify the breach by percentile band first, apply the cheapest response that matches, and escalate only when the cheaper response fails its re-measurement.

Numbers: thresholds, windows, cost of error

The decision gate is a workflow-level p95 above 2 seconds, but only after at least 100 runs per tool in a stable window and a second consecutive breaching window. A single window is a signal; two windows are a decision. p99 is logged with p95 so the tail's shape is visible: qaskills.sh defines p95 as the 1-in-20 slow-but-common tail and p99 as the 1-in-100 painful edge, so a p95 breach with p99 close to p95 points to a broad tail, while p99 far above p95 points to a rarer, sharper edge.

The 100-run minimum is a noise filter, not a comfort number: below it, one slow run can move a percentile enough to look like a breach. Requiring two consecutive breaching windows adds a second filter against transient load, cache cold starts, or a single bad deploy. Neither filter changes the 2-second gate; they only decide when the gate has been crossed with enough evidence to act.

Migrating too early spends reorchestration work—porting prompts, state graphs, and tool adapters—on a breach that a second window might have dismissed. The cost is not just engineering time; it is also the risk of changing a working workflow before the measurement has stabilized, which can introduce regressions while chasing noise. The two-window rule is the cheaper experiment: wait one more stable window and let the percentile either confirm or clear.

Migrating too late has the opposite asymmetry: once two windows confirm the breach, every additional run keeps the excess tail latency in production. If p95 is 3 seconds against a 2-second gate, the slow-but-common tail is 1 second above the allowed level, and that gap repeats until the workflow is reorchestrated, refactored, or the slowest tool is swapped. The cost therefore scales with run volume and delay, not with the size of the engineering change.

The economic rule is simple: pay for measurement first, then pay for migration only after the second confirmed breach. A 100-run window and a second breaching window are cheaper than either a premature rewrite or an unbounded tail.

3-tool agent pipeline

In a 150-run measurement of a three-tool workflow—retrieval, code execution, and an external API—the workflow-level p50 was 640 ms, p95 was 2.4 s, and p99 was 3.8 s. Because p95 exceeded the 2-second threshold, the migration gate was breached. The p95 gate follows the percentile methodology documented by qaskills.sh (June 2026). The 150 complete runs also yield 150 invocations per tool, above the 100-run floor for a stable percentile window, and p99 at 3.8 s records the extreme edge without changing the p95 decision.

The per-tool breakdown showed the external API tool contributing 70% of runs above 2 seconds; retrieval and code execution accounted for the remaining 30%. That concentration identifies the API tool as the first intervention target and prevents a single slow dependency from justifying a full reorchestration.

Confirmation came in two windows: the first window breached the gate at p95 = 2.4 s, and a second consecutive window also breached it at p95 = 2.2 s. Under the article's own rule, each window must contain at least 100 runs per tool before the two-window result counts as a confirmed breach, so verify both windows meet that floor before acting. Two consecutive windows above 2 seconds distinguish a persistent tail from a transient spike.

After caching the API tool's responses, workflow p95 fell to 1.6 s, below the gate, and migration was cancelled. Only the API response path changed; retrieval and code execution were left unchanged, so the before-and-after comparison isolates the fix.

Use the migration check below before acting on any p95 breach. It records the evidence, the tail owner, the confirmation window, and the cheapest targeted fix; migration proceeds only if p95 remains above 2 seconds after that fix.

Decision rules and where they break

Once the workflow-level p95 gate is in place, treat the 2-second threshold as a trigger for diagnosis, not an automatic migration order. The rules below separate a slow-but-common tail from an extreme rare tail and from a broad regression, so the response matches the shape of the latency problem.

If p95 exceeds 2 seconds in two consecutive windows of at least 100 runs per tool, refactor the slowest tool first and then re-measure. Requiring two windows prevents a single bad sample from forcing a change, while the 100-run minimum keeps the percentile stable enough to act on.

If p95 exceeds 2 seconds but p99 exceeds 8 seconds, the tail is extreme and rare rather than slow-but-common. Investigate hangs and retries first: inspect your own run logs for timeout-and-retry signatures before deciding anything, since no published source in this guide specifies a retry interlock pattern. Only after ruling out hangs and retries should you consider migrating, reorchestrating, or swapping a tool.

If p50 exceeds 2 seconds, the regression is broad: it affects the typical request, not just the tail. The p95 gate has already fired, but the fix is systemic rather than a tool swap, because the median shows the whole distribution shifted.

After any refactor, demand the same two-window, 100-run evidence before calling the change successful; one improved window can hide a retry storm or a cold-cache effect.

The 2-second gate wins when the tail is common enough to refactor and re-measure; it misleads when the problem is a rare hang or a median shift, because those cases need investigation or systemic repair before any tool change.

Frequently Asked Questions

What exact p95 latency value, and across how many tools, triggers a workflow migration?

Migrate the agent workflow when p95 latency exceeds 2s across 3 tools.

Why is average latency inadequate for detecting the slow tail that p95 catches?

Latency distributions are right-skewed, so the mean is dragged by the tail but never shows it: 99 calls at 50 ms plus one call at 5,000 ms yield a mean of only 99.5 ms, roughly 100 ms, while one real user waited five seconds.

What fraction of real requests actually experience pain at the p95 threshold?

At p95, one in twenty real requests was slower than the threshold.

At one million requests per day, how many users feel p99-level pain?

At one million requests per day, the one-in-one-hundred p99 pain equals 10,000 affected users (qaskills.sh).

If p95 decides the migration, when should I still consult p99?

p99 remains useful for painful outliers at the rarest edge, whereas p95 captures the slow-but-common tail where slowness becomes frequent enough to trigger migration.

How does a percentile avoid the smearing effect that dilutes the average?

Percentiles avoid smearing by stating how many requests were slower than a given value, so the single five-second wait is not averaged away across all 100 calls.

Quick answers

Also worth reading: Managing API rate limits for multi-agent orchestration: Managing API rate limits for · Multi agent workflow retries: Exponential vs Fixed 1% vs 93.8%: Multi agent workflow retries: Exponential · How to manage agent context windows across long-running workflows: How to manage agent context