Capacity Is a Queue

Your model generates 100 tokens per second. Your user waits eight seconds for the first one.

Nothing in that sentence is contradictory.

The request may have waited behind a burst. A long prompt may have stalled active decodes. KV-cache memory may have blocked admission. A high-priority batch job may have consumed the available route. An agent task that looked like one user action may have expanded into twelve inference requests.

Peak model speed describes a machine. Product latency describes a queueing system.

The capacity question is therefore:

At what arrival rate can this workload keep its time-to-first-token, time-per-output-token, deadline, and verified-outcome promises?

That rate is lower than the rate at which a benchmark can force the hardware to emit tokens. The difference is where reliability and economics live.

Start with Little's Law, then stop before it lies to you

John Little's 1961 proof established one of the most useful identities in operations research. For a stable stationary system with finite averages:

L = lambda * W

Where:

• L is the average number of items in the system;
• lambda is the effective arrival rate of completed items;
• W is the average time each item spends in the system.

If an inference route completes 20 requests per second and requests spend an average 0.5 second from admission through completion, the route carries an average 10 in-flight requests.

L = 20 * 0.5 = 10

This is an identity, not a capacity simulator. It does not tell you the p99, the impact of prompt length, or which scheduler is best. It gives a consistency check. If dashboards claim 20 completed requests per second, 0.5-second mean response, and 100 mean in flight, their boundaries differ or the instrumentation is wrong.

For agent systems, apply the law at multiple boundaries:

• user tasks in the orchestration system;
• inference requests at the gateway;
• admitted sequences in the serving engine;
• active tokens or KV blocks in memory;
• tool operations waiting on external services;
• exceptions waiting for humans.

A queue can move rather than disappear.

Utilization creates a knee, not a straight line

Consider a deliberately simple M/M/1 queue: one server, Poisson arrivals, exponential service times, first-come-first-served, and no batching. Let service capacity be mu jobs per second and arrival rate be lambda.

utilization rho = lambda / mu
mean response W = 1 / (mu - lambda)

Normalize service capacity to one job per second:

Utilization	Mean response	Illustrative p95 response
50%	2.00 s	5.99 s
80%	5.00 s	14.98 s
90%	10.00 s	29.96 s
95%	20.00 s	59.91 s
99%	100.00 s	299.57 s

This curve is not an LLM forecast. Real serving has multiple accelerators, dynamic batching, non-exponential service, cancellation, memory admission, priorities, and two distinct inference phases. Use it for one intuition only:

Headroom is a latency feature.

Running at 95 percent of an averaged capacity number is not five percent away from safety. It can be beyond the knee where small demand changes produce large waits.

Average load is not offered load

A route averages 60 requests per second over an hour. Capacity is 100 requests per second. The utilization slide says 60 percent.

But traffic arrives as:

minute 00-04: 35 requests/s
minute 05:    180 requests/s
minute 06-14: 55 requests/s
minute 15:    160 requests/s

If the system can serve 100 requests per second, the first one-minute burst adds as many as 4,800 queued requests before considering cancellation or batching effects:

(180 - 100) * 60 = 4,800

When the burst ends at 55 requests per second, spare capacity is 45 requests per second. Draining the idealized backlog takes about 107 seconds:

4,800 / 45 = 106.7 seconds

The burst lasted one minute. Its queue contaminates almost two more minutes.

Minute averages can still hide sub-second bursts. Choose measurement windows that match the latency budget and keep raw enough arrival traces to replay.

Agent traffic multiplies before it reaches the GPU

A product receives two user tasks per second. Each task makes six inference calls on average. Retries and fallbacks multiply calls by 1.15.

inference arrival rate
= 2 tasks/s * 6 calls/task * 1.15
= 13.8 requests/s

Suppose the route's effective capacity for this workload and SLO is 18 requests per second. Offered utilization is 76.7 percent.

Using the toy single-server model only for intuition:

mean response = 1 / (18 - 13.8) = 0.238 s
toy p95       = -ln(0.05) / (18 - 13.8) = 0.713 s

Now an agent update increases average calls per task from six to 7.5. User traffic did not change.

arrival = 2 * 7.5 * 1.15 = 17.25 requests/s
utilization = 17.25 / 18 = 95.8%
toy mean = 1 / (18 - 17.25) = 1.333 s
toy p95 = -ln(0.05) / 0.75 = 3.994 s

A 25 percent increase in calls per task produces about a 5.6x increase in this toy mean and p95 response because the route crossed the utilization knee.

This is why a loop change requires a capacity review. Model requests are not user arrivals.

Track:

requests per user task
tokens per user task
parallel branches per task
retry multiplier by failure class
peak concurrent tasks
cancelled downstream work after task completion

An orchestrator that cancels losing branches late can burn capacity after the user already has an answer.

Request size is at least two-dimensional

Counting requests assumes every request resembles every other request. LLM service time depends strongly on input and output shape.

Represent each request with at least:

(input tokens, requested or observed output tokens, model, adapter, priority)

A 30,000-token document summary and a 30-token classification are not one unit each in any useful capacity model.

Prompt processing, or prefill, handles input tokens in parallel and tends to use compute heavily. Decode generates later tokens sequentially and often exercises memory bandwidth differently. Long prefills can interrupt the cadence experienced by active decodes when the phases share a worker.

Sarathi-Serve addresses this interference by splitting prefills into chunks and constructing schedules that combine chunks with ongoing decode work. The paper's reported gains belong to its evaluated models, hardware, and workloads. The transferable principle is that scheduling request phases can change both throughput and tail latency without changing model weights.

Memory is an admission constraint

Accelerator compute can appear idle while memory capacity prevents another sequence from entering.

The model weights occupy a largely fixed footprint for a loaded replica. Request state, especially the KV cache, grows with active sequence lengths and layers. Fragmentation, duplication, and uncertain future output lengths create reservation problems.

PagedAttention applies virtual-memory-style block management to reduce KV-cache waste. Better memory management raises capacity on evaluated workloads. It does not make memory infinite.

An admission controller needs estimates or bounds for:

• prompt tokens;
• maximum and expected generated tokens;
• cache reuse;
• adapter footprint;
• batch and scheduler overhead;
• safety reserve for preemption or migration.

When the estimate is wrong, the system can preempt, evict, recompute, reject, or violate a deadline. Each choice has a cost that belongs in the capacity policy.

Throughput is not goodput

A serving engine completes 1,000 requests per minute. Only 650 meet the product's TTFT and TPOT objectives. Throughput is 1,000. SLO goodput is 650.

goodput(SLO)
= completed requests satisfying all named latency constraints
  ----------------------------------------------------------
  unit time

DistServe uses goodput to reason about the maximum request rate satisfying both TTFT and time-per-output-token constraints. It separates prefill and decode onto different GPUs to reduce interference and tune resources for each phase.

Splitwise also studies phase splitting, including heterogeneous machine choices for throughput, cost, and power.

Disaggregation is not free. KV state must move between phases. Network bandwidth and placement matter. Separate pools can become imbalanced. An architecture that wins for long prompts and strict decode cadence may lose for short requests or constrained interconnect.

Use workload-specific goodput, not an architecture slogan.

Metric	Counts	Can hide
Tokens/s	Emitted tokens	Waiting, request failures, unfairness, unusable latency
Requests/s	Completed requests	Length mix, missed SLOs, wrong outcomes
TTFT goodput	Requests meeting first-token target	Slow decode or failed tools
TTFT + TPOT goodput	Requests meeting both inference targets	End-to-end workflow failure
Verified-outcome goodput	Accepted outcomes per unit time	Later false acceptance and severity

The product should ultimately plan the last line. The inference team supplies a constrained component goodput.

Tail latency is where mixed workloads meet

Mean latency can improve while p99 degrades. This happens when an optimization benefits ordinary requests and harms a minority of long, low-priority, cold, or preempted ones.

Always segment tails by:

• input and output length bucket;
• model, adapter, quantization, and route;
• cache hit class;
• priority and tenant;
• cold versus warm replica;
• preemption, fallback, and retry;
• time of day and arrival burst;
• success, cancellation, and deadline outcome.

Do not calculate latency percentiles only across successful requests if timeouts disappear from the dataset. A deadline violation recorded as an error is still latency evidence.

For agent workflows, distinguish:

• per-call p95;
• critical-path p95;
• sum of sequential waits;
• maximum of parallel branches;
• time to first acceptable branch;
• time until losing branches are actually cancelled.

Parallelism can reduce critical-path time while increasing offered load and tail latency for everyone else.

Autoscaling cannot save a request whose deadline is shorter than startup

Suppose overload is detected in 10 seconds, a replica becomes ready in 90 seconds, and the request deadline is five seconds. Autoscaling may repair future capacity. It cannot serve the current burst within deadline.

The first line of defense is demand control:

1. reject work that is unauthorized, duplicate, expired, or impossible;
2. use bounded per-tenant and per-priority queues;
3. reserve capacity for critical classes;
4. shed optional branches and speculative attempts;
5. shorten output or context only under an explicit degraded mode;
6. route to a fallback with known quality and cost;
7. return a truthful retry-after or asynchronous receipt;
8. scale for the next interval.

Silent queue growth is the worst option. It converts a capacity shortage into universal deadline failure while consuming work on requests users may already have abandoned.

Capacity planning is a workload experiment

Do not begin with GPU count. Begin with a trace distribution.

1. Capture demand

• arrivals at sub-SLO resolution;
• tasks to calls and calls to tokens;
• prompt/output joint distribution;
• route, cache, and adapter mix;
• priority, deadline, cancellation;
• daily, weekly, launch, and incident bursts.

2. Define objectives

• TTFT and TPOT by product class;
• complete response and verified-outcome deadlines;
• minimum acceptance and availability;
• fairness or tenant isolation;
• maximum cost per outcome.

3. Replay the workload

Use production traces with sensitive content removed, a workload-calibrated simulator, or a controlled load test. Preserve arrival timing and length correlation. Uniform synthetic prompts at a steady rate are a component benchmark, not a capacity plan.

4. Sweep offered load

Increase demand until each SLO goodput curve reaches its knee. Record:

• queue, prefill, decode, and end-to-end percentiles;
• KV occupancy, eviction, and preemption;
• batch composition and scheduler fairness;
• fallback, rejection, and cancellation;
• cost and power per SLO-satisfying request.

5. Inject bad days

• one provider or replica pool unavailable;
• cache cold after deploy;
• prompts twice normal length;
• retry storm from a downstream timeout;
• large tenant burst;
• autoscaler delayed;
• tool latency feeding back into agent concurrency.

Capacity is the promise that survives these cases, not the best point on the lab curve.

The architect's control loop

Capacity management is a feedback system:

observe arrival and service distributions
-> predict short-horizon offered load
-> admit, route, reserve, or shed
-> schedule prefill and decode work
-> measure goodput and queue age
-> scale or rebalance pools
-> update the prediction and policy

Control too slowly and queues grow. React too aggressively and replicas oscillate, caches stay cold, and cost rises. Optimize one route without the agent layer and call amplification can erase the gain.

Three numbers belong on the same operations page:

1. Offered load: work trying to enter, including rejected and deferred requests.
2. SLO goodput: work completed inside the declared inference constraints.
3. Verified-outcome goodput: accepted business outcomes completed per unit time.

The gap between one and two is serving pressure. The gap between two and three is workflow pressure.

Builder playbook

1. Replace requests per second with a distribution over input tokens, output tokens, route, and priority.
2. Multiply user arrivals by calls, branches, and retries per task.
3. Split queue, prefill, decode, tool, verification, and commit timing.
4. Define TTFT and TPOT goodput, then connect it to verified-outcome goodput.
5. Find the utilization knee by replaying bursts, not steady averages alone.
6. Measure memory admission and KV pressure as capacity constraints.
7. Establish bounded queues, deadlines, cancellation, and degraded modes before autoscaling.
8. Test colocation, chunked prefill, or phase disaggregation against the actual SLO mix.
9. Charge speculative branches and late cancellations to the originating task.
10. Keep enough headroom for the failure mode you are obligated to survive.

Cheap tokens are useful. A queue determines when you receive them.

References

1. John D. C. Little, A Proof for the Queuing Formula: L = lambda W (Operations Research, 1961). The formal basis for relating average arrival, inventory, and time in stable systems.
2. Gyeong-In Yu et al., Orca (OSDI 2022). Introduces iteration-level scheduling for autoregressive serving.
3. Woosuk Kwon et al., PagedAttention (SOSP 2023). Explains KV-cache memory management and serving capacity.
4. Amey Agrawal et al., Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (OSDI 2024). Studies chunked prefill and stall-free scheduling.
5. Yinmin Zhong et al., DistServe (OSDI 2024). Defines goodput under TTFT and TPOT constraints and evaluates prefill/decode disaggregation.
6. Pratyush Patel et al., Splitwise (ISCA 2024). Studies phase-specific serving resources across throughput, cost, and power.