AI Agent Load Testing with Little’s Law

AI agent load testing turns traced model and tool timing into an open-arrival queue model instead of treating one prompt call as the whole workload. This tutorial uses Little’s Law and a deterministic discrete-event lab to plan concurrency, expose retry fan-out, and set release thresholds.

AI agent load testing starts with the whole run

AI agent load testing is not a faster version of sending the same prompt in a loop. One user request can call a planner model, fan out to tools, retry a failed dependency, compact context, and call a synthesis model before it returns. The unit under test is that traced run, because every branch holds work and consumes a different pool.

A useful trace records stage start and end times, retry reason, model identity, tool identity, input class, output state, and the request-level deadline. The maintained, development-status OpenTelemetry GenAI agent-spans document gives implementers a vocabulary to evaluate for the agent boundary, while child model and tool spans retain the causal shape. Because the document is still development status, pin the revision used by your instrumentation instead of presenting it as a stable convention. An AI agent load testing receipt should name that pinned revision. Start by checking that you can trace complete AI agent runs rather than averaging unrelated calls.

The planning question is specific: given a production arrival rate and a measured distribution of stage times, how much concurrent work accumulates before latency or queueing crosses a release limit? That is narrower than benchmarking model quality, provider speed, or every possible tool. It is also more actionable because the answer maps directly to worker pools, admission policy, and a test scenario.

This tutorial uses an open, synthetic fixture so every calculation can be inspected. Replace its numbers with a redacted trace slice later; do not present the fixture as a vendor benchmark. The receipt keeps offered load, completed throughput, work in progress, queue delay, retry fan-out, and capacity decisions separate.

Turn trace spans into service demand

For AI agent load testing, begin with a stage ledger, not one request-duration histogram. For each route through the agent, calculate how many times a stage is visited and the duration of each visit. A planner called once for 380 milliseconds and a tool called twice for 220 milliseconds create different service demands even if they sit inside the same total wall time.

Tool-call latency testing must preserve fan-out and join behavior. Two tools launched together contribute two occupied slots, but their contribution to request latency is closer to the slower branch plus any queue delay. A serial retry contributes both another visit and another wait. Flattening those facts into one “tool time” erases the mechanism that creates saturation.

Use percentiles to describe the input distribution, but replay actual or deliberately constructed samples in the load model. Averages can hide a small slow class that keeps slots occupied long enough to form a queue. Segment at least by request class, route, tool family, response-size band, and outcome; collapse a segment only when its service-demand shape is genuinely similar.

The trace slice also needs a truth boundary. Redact prompts and outputs, retain timing and topology, and mark synthetic replacements. An agent concurrency planning receipt should say which spans were excluded, how retries were classified, and whether cancellation released downstream work. Without that custody, a precise concurrency number can still describe the wrong system.

Use Little’s Law as a conservation check

Little’s Law relates average work in progress L, effective throughput λ, and average time in system W: L = λW. The MIT OCW Little’s Law lecture page presents the relationship and its steady-state boundary. For AI agent load testing, the equation is most useful as a ledger check, not as a promise that a finite or overloaded run is stationary.

Measure all three terms from the same observation boundary. If completed throughput is 1.8 runs per second and average end-to-end time is 2.4 seconds, the corresponding average request-level work in progress is about 4.32. Mixing offered arrivals with completed throughput, or stage time with request-level WIP, breaks the conservation boundary.

Finite deterministic simulations offer a clean check. Integrate the number of active requests over the run window, divide by that window for L, and compare it with completed requests per second multiplied by average response time. Because the same completed jobs and window supply both sides, the values should agree apart from floating-point tolerance. Little’s Law for AI agents is most persuasive when the trace boundary and simulated boundary match.

Little’s Law does not choose a safe utilization or a latency objective. Retries, deadlines, rate limits, and business priority remain product constraints. Pair the calculation with agent run budgets that stop cleanly so timed-out work does not continue consuming capacity after the user-visible request has ended.

Drive arrivals independently of worker speed

A closed-loop test that waits for each response before sending the next request reduces pressure when the system slows. That behavior is convenient for smoke tests and misleading for capacity planning. Production demand usually arrives independently, so AI agent load testing needs an open model with arrivals scheduled by a declared rate.

The k6 constant-arrival-rate executor follows this principle: iterations start at a configured rate while the runner allocates enough virtual users to sustain them. Your LLM load testing scenario should likewise report dropped or delayed starts instead of quietly lowering the offered load. Treat those misses as evidence that the generator or system could not represent the target.

Translate the demand contract into a small scenario matrix:

• normal arrival rate with the observed request mix;
• launch burst with a bounded duration and explicit recovery window;
• elevated slow-tool share without changing model time;
• retry spike with the same original arrivals;
• dependency degradation plus cancellations at the request deadline.

Keep external quotas visible beside local pools. Rate limits are workload contracts, not merely errors to retry. A test that exceeds a provider quota and then praises local worker utilization has measured throttling behavior, not the capacity of the planned route.

Simulate the queueing network before renting traffic

The downloadable AI agent load testing lab implements a deterministic discrete-event queue with four stages: planner, parallel tool calls, an optional tool retry, and synthesis. Each stage has a declared number of slots and a repeating duration fixture. Jobs arrive at fixed intervals, take the earliest available slot, and retain their ready, start, wait, and end times.

That model is intentionally smaller than a production agent. It excludes provider batching, network jitter, token-dependent decoding, cache effects, and autoscaler warm-up. Its purpose is to make the accounting falsifiable before those details are layered in. Run the same configuration twice and the receipt hash must match; change the retry cadence and both topology and queue totals must change.

For every job, the lab emits the request arrival, completion, response time, and stage visits. From those records it derives throughput, average and p95 response time, average request WIP, peak WIP, total queue delay, retry count, and per-stage utilization. An independent test recomputes the conservation equation and verifies that no visit begins before it is ready or overlaps another visit on the same slot.

Use the program as a planning notebook. Replace one fixture dimension at a time, keep the baseline receipt, and explain why the mutation is plausible. If measured spans do not fit the model, change the topology rather than massaging durations until the answer looks comfortable.

Runnable artifact — Deterministic discrete-event planning over declared synthetic queues; not production capacity, vendor performance, or an SLA guarantee.

import assert from "node:assert/strict";
import { createHash } from "node:crypto";

const sha = (value) => createHash("sha256").update(JSON.stringify(value)).digest("hex");
const round = (value, digits = 6) => Number(value.toFixed(digits));
const numberArg = (name, fallback) => { const index = process.argv.indexOf(name); if (index < 0) return fallback; const value = Number(process.argv[index + 1]); if (!Number.isFinite(value)) throw new Error("invalid-argument:" + name); return value; };
const alternate = process.argv.includes("--alternate");
const config = {
  jobs: numberArg("--jobs", 24),
  arrivalRate: numberArg("--arrival-rate", alternate ? 2.8 : 2.4),
  plannerSlots: 2,
  toolSlots: 4,
  retrySlots: 2,
  synthesisSlots: 2,
  retryEvery: numberArg("--retry-every", alternate ? 2 : 5),
};
for (const key of ["jobs", "plannerSlots", "toolSlots", "retrySlots", "synthesisSlots", "retryEvery"]) if (!Number.isInteger(config[key]) || config[key] < 1 || config[key] > 500) throw new Error("invalid-config:" + key);
if (config.jobs < 4 || config.arrivalRate <= 0 || config.arrivalRate > 100) throw new Error("invalid-config:arrival");

const durations = {
  planner: [360, 440, 520, 400],
  toolA: [220, 310, 260],
  toolB: [300, 460, 340, 390],
  retry: [180, 260],
  synthesis: [420, 560, 480],
};
const pools = Object.fromEntries(Object.entries({ planner: config.plannerSlots, tool: config.toolSlots, retry: config.retrySlots, synthesis: config.synthesisSlots }).map(([stage, slots]) => [stage, Array.from({ length: slots }, () => 0)]));
const visits = [];
function schedule(stage, ready, duration, job, branch) {
  const pool = pools[stage];
  let slot = 0;
  for (let index = 1; index < pool.length; index += 1) if (pool[index] < pool[slot]) slot = index;
  const start = Math.max(ready, pool[slot]);
  const visit = { stage, branch, job, slot, ready, start, wait: start - ready, duration, end: start + duration };
  pool[slot] = visit.end;
  visits.push(visit);
  return visit;
}
const arrivals = Array.from({ length: config.jobs }, (_, job) => ({ job, arrival: round(job * 1000 / config.arrivalRate, 9) }));
const planner = new Map();
for (const item of arrivals) planner.set(item.job, schedule("planner", item.arrival, durations.planner[item.job % durations.planner.length], item.job, "plan"));
const toolRequests = arrivals.flatMap((item) => [
  { job: item.job, branch: "A", ready: planner.get(item.job).end, duration: durations.toolA[item.job % durations.toolA.length] },
  { job: item.job, branch: "B", ready: planner.get(item.job).end, duration: durations.toolB[item.job % durations.toolB.length] },
]).sort((left, right) => left.ready - right.ready || left.job - right.job || left.branch.localeCompare(right.branch));
const tools = new Map();
for (const item of toolRequests) tools.set(item.job + item.branch, schedule("tool", item.ready, item.duration, item.job, item.branch));
const retries = new Map();
const retryRequests = arrivals.filter((item) => (item.job + 1) % config.retryEvery === 0).map((item) => ({ job: item.job, ready: tools.get(item.job + "B").end, duration: durations.retry[item.job % durations.retry.length] })).sort((left, right) => left.ready - right.ready || left.job - right.job);
for (const item of retryRequests) retries.set(item.job, schedule("retry", item.ready, item.duration, item.job, "B-retry"));
const synthesisRequests = arrivals.map((item) => ({ job: item.job, ready: Math.max(tools.get(item.job + "A").end, retries.get(item.job)?.end || tools.get(item.job + "B").end), duration: durations.synthesis[item.job % durations.synthesis.length] })).sort((left, right) => left.ready - right.ready || left.job - right.job);
const synth = new Map();
for (const item of synthesisRequests) synth.set(item.job, schedule("synthesis", item.ready, item.duration, item.job, "synthesize"));
const jobs = arrivals.map((item) => {
  const jobVisits = visits.filter((visit) => visit.job === item.job).sort((left, right) => left.start - right.start || left.stage.localeCompare(right.stage));
  const completion = synth.get(item.job).end;
  return { job: item.job, arrival: item.arrival, completion, response: completion - item.arrival, visits: jobVisits };
});
const windowEnd = Math.max(...jobs.map((job) => job.completion));
const percentile = (values, percentileValue) => [...values].sort((a, b) => a - b)[Math.max(0, Math.ceil(values.length * percentileValue) - 1)];
const eventDeltas = jobs.flatMap((job) => [{ at: job.arrival, delta: 1 }, { at: job.completion, delta: -1 }]).sort((left, right) => left.at - right.at || left.delta - right.delta);
let active = 0;
let peakWip = 0;
for (const event of eventDeltas) { active += event.delta; peakWip = Math.max(peakWip, active); }
const responseSeconds = jobs.map((job) => job.response / 1000);
const throughput = config.jobs / (windowEnd / 1000);
const averageResponse = responseSeconds.reduce((sum, value) => sum + value, 0) / responseSeconds.length;
const averageWip = jobs.reduce((sum, job) => sum + job.response, 0) / windowEnd;
const stageStats = Object.fromEntries(Object.entries(pools).map(([stage, pool]) => {
  const selected = visits.filter((visit) => visit.stage === stage);
  const busy = selected.reduce((sum, visit) => sum + visit.duration, 0);
  return [stage, { slots: pool.length, visits: selected.length, busyMs: busy, queueMs: selected.reduce((sum, visit) => sum + visit.wait, 0), utilization: round(busy / (pool.length * windowEnd)) }];
}));
const p95Queue = percentile(visits.map((visit) => visit.wait), .95);
const maxUtilization = Math.max(...Object.values(stageStats).map((stage) => stage.utilization));
const decision = maxUtilization > .9 || p95Queue > 1000 ? "reject" : maxUtilization > .75 || p95Queue > 500 ? "review" : "stable";
const core = {
  schema: "ai-agent-load-queue-receipt-v1",
  fixture: "synthetic open-arrival agent queue with planner, tool fan-out, bounded retry, and synthesis pools",
  config,
  durationFixturesMs: durations,
  jobs,
  summary: { windowMs: windowEnd, offeredArrivalRatePerSecond: config.arrivalRate, completedThroughputPerSecond: round(throughput), averageResponseSeconds: round(averageResponse), p95ResponseMs: percentile(jobs.map((job) => job.response), .95), averageWip: round(averageWip), littleLawRhs: round(throughput * averageResponse), peakWip, totalQueueMs: visits.reduce((sum, visit) => sum + visit.wait, 0), p95QueueMs: p95Queue, retries: retries.size, maxUtilization: round(maxUtilization), decision },
  stageStats,
  claimBoundary: "Deterministic discrete-event planning over declared synthetic queues; not production capacity, vendor performance, or an SLA guarantee.",
};
assert.ok(Math.abs(core.summary.averageWip - core.summary.littleLawRhs) < 1e-5);
console.log(JSON.stringify({ ...core, receiptHash: sha(core) }, null, 2));
console.log("PASS: queue topology, open arrivals, slot bounds, Little's Law, retry mutation, hostile inputs, and digest verified");

Read saturation as a shape, not one percentage

In AI agent load testing, a pool near full utilization can remain healthy when arrivals are smooth and service times are narrow, yet collapse under correlated slow calls. Watch queue delay, response-time slope, and recovery after the offered rate returns to normal. Saturation appears when additional arrivals grow waiting work faster than completions can drain it.

Plot offered arrival rate horizontally and p95 response time or average WIP vertically. The useful boundary is the bend where queues stop clearing inside the test window, not the highest rate that completed one lucky sample. Annotate the retry cadence and slow-class mix on every point so the phase plane does not imply that arrival rate is the only driver.

Concurrency planning should be derived from the stable side of that boundary and then checked against downstream quotas. More slots are not automatically safer: they can move the queue into a tool, database, or provider that has weaker backpressure. Admission control for stable LLM latency gives the resulting budget an enforcement point instead of leaving it as a dashboard annotation.

Use at least two failure views. A service view explains which pool filled; a request view explains which users waited or timed out. The same saturation event can look moderate in aggregate utilization while one high-cost route experiences unacceptable delay.

Set release thresholds before the run

Write the AI agent load testing pass contract before generating traffic. Include the target arrival rate and mix, maximum p95 response time, maximum queue wait by stage, allowed dropped-start rate, retry ceiling, deadline behavior, and the time allowed to recover after a burst. A result cannot become a release gate if the thresholds move after the graph appears.

Separate capacity evidence from observability evidence. The load generator proves what demand it attempted; traces prove which route the work took; infrastructure metrics explain pool and dependency behavior; the user-facing timer proves the experienced boundary. You can expose AI latency with Server-Timing to connect selected backend stages to browser evidence without leaking private prompts or internal hostnames.

Run at least one negative control that must fail. Reduce a stage pool, increase retry frequency, or tighten the deadline enough to cross a predeclared threshold. If the test still passes, either the oracle is not attached to the mutated mechanism or the load never reached the intended boundary.

Store the configuration, trace-slice identifier, generator version, source revision, receipts, and threshold result together. The artifact should make it possible to distinguish “the system passed” from “the harness failed to apply pressure.” That distinction is the difference between a reusable capacity check and a screenshot of green numbers.

Ship a capacity receipt with explicit limits

A useful AI agent load testing report ends with a decision and the evidence boundary behind it. State the tested rate, request mix, stage pools, duration source, retries, deadlines, run length, observed throughput, latency distribution, WIP, queue delay, and recovery behavior. Then name the dimensions that remain untested.

Do not extrapolate a short synthetic run into a universal SLA. Provider revisions, prompt length, tool fan-out, regional routing, cache state, and traffic correlation can move the boundary. Schedule a rerun when one of those inputs changes, and compare receipts rather than isolated percentile screenshots.

The practical loop is small: trace, model, attack, observe, decide. Use the deterministic queue to challenge the workload contract cheaply, then run the open-arrival scenario against a controlled environment, and finally confirm the same topology in production telemetry. If the three views disagree, preserve the disagreement and investigate it before adding capacity.

This is forward engineering rather than performance theater. The outcome is not the largest requests-per-second number; it is a concurrency and admission policy that can explain which work is accepted, where it waits, and how the system recovers when a model or tool slows down.