HPA on Queue Depth, Not CPU

For queue-driven workers, autoscaling on CPU is scaling on the wrong signal. Kubernetes HPA on queue depth is the fix: scale on the backlog of pending work, not on CPU, because for an async worker CPU only rises after it is already processing, which is after the queue has built up. By the time CPU crosses your threshold, latency has already degraded. The queue depth is the leading indicator; CPU is the lagging one.

This is one of the most common autoscaling mistakes in production, and it is subtle because CPU autoscaling is the default everyone reaches for. It works fine for web services and quietly fails for workers.

Why the autoscaling signal matters

The Horizontal Pod Autoscaler adds and removes replicas based on a metric crossing a target. The entire usefulness of autoscaling depends on that metric being a timely measure of demand. Pick a metric that lags, and the autoscaler always reacts too late, adding capacity after the pain instead of before it.

For request-driven services, CPU is a reasonable proxy: more requests, more CPU, right now. For queue-driven workers, that relationship breaks, and the break is the whole subject of this post. This is part of the Kubernetes operations series and pairs closely with Backpressure Design for Real-Time Systems.

Why is CPU a bad autoscaling signal for queue workers?

Because a worker’s CPU usage reflects work it is currently processing, not work waiting to be processed. When a flood of messages lands in the queue, the workers keep chugging at whatever rate they can; CPU does not spike to signal the flood, because the workers were already busy. CPU only tells you about the present throughput, while the queue tells you about the unmet demand.

The result is a dangerous lag. The backlog builds, consumer lag climbs, end-to-end latency degrades, and CPU is sitting at a normal-looking level the whole time because the workers are simply saturated, not over-CPU. By the time CPU does cross the threshold (if it ever does), you are already deep in a backlog you will take a long time to drain.

	CPU as the signal	Queue depth as the signal
What it measures	Current processing	Pending, unprocessed work
Indicator type	Lagging	Leading
Reacts to a burst	After backlog forms	As backlog forms
Scale-to-zero when idle	Awkward	Natural (queue empty)
Best for	Request-driven services	Async / queue workers

What should you scale a queue worker on?

Scale on the queue: pending message count (queue depth) or consumer lag. These are direct, leading measures of how much work is waiting, so the autoscaler adds workers the moment the backlog grows, before latency degrades. The target becomes “messages per worker” rather than “CPU percent,” which maps directly to how fast you drain the queue.

The mental model is simple: if you want to keep the backlog under control, scale the number of workers to the amount of waiting work. Twice the queue depth, roughly twice the workers, and the backlog drains at a predictable rate. CPU never enters the equation, because the queue already tells you exactly how behind you are.

How do you autoscale Kubernetes on queue depth?

Feed the HPA an external metric representing the queue, most commonly via KEDA, which provides ready-made scalers for Kafka, SQS, RabbitMQ, Redis, and many others. KEDA reads the queue depth or consumer lag and drives the HPA, and it can scale to zero when the queue is empty, which native CPU-based HPA cannot do cleanly. The plain HPA can also consume external metrics if you pipe them in yourself, but KEDA removes most of the plumbing.

Scale-to-zero is an underrated benefit. A queue worker with no pending messages does not need to run at all, and KEDA can take it to zero replicas and wake it when messages arrive. For bursty async workloads, that turns idle capacity into zero cost, which a CPU-based HPA (with its minimum-one-replica floor and CPU-never-quite-zero behavior) cannot match.

When is CPU-based autoscaling fine?

CPU autoscaling is perfectly good for request-driven services where CPU rises with incoming traffic in real time, an HTTP API, a render service, anything that does CPU work synchronously as requests arrive. There, CPU is a close, timely proxy for demand, and adding replicas when CPU climbs is exactly right.

The rule is about workload shape, not a blanket preference. Synchronous, CPU-bound request handling scales fine on CPU. Asynchronous, queue-fed work scales on the queue. Many systems have both, in which case you use CPU for the web tier and queue depth for the workers, rather than forcing one signal on both.

How do you avoid autoscaling thrash on queue depth?

Tune the scaling target and stabilization so the autoscaler does not flap replicas up and down as the queue wobbles. The two levers are the target value (messages or lag per replica, set from your real drain rate) and a stabilization window that smooths brief dips so the autoscaler does not scale down the instant a burst clears. Without these, a spiky queue produces a thrashing replica count, which is its own kind of instability.

Thrash happens when the autoscaler reacts to every small fluctuation. A queue that briefly empties triggers a scale-down, then the next burst triggers a scale-up, and the churn of starting and stopping pods, each paying startup time, can be worse than running slightly more replicas steadily. The fix is a scale-down stabilization window so the autoscaler waits to confirm the queue has genuinely drained before removing capacity, while still scaling up promptly when the backlog grows.

Set the per-replica target from how fast one worker actually drains the queue. If one worker clears a known number of messages per second and you want the backlog gone within a target time, that math gives you the messages-per-replica target directly. Anchoring the target to a real drain rate, rather than a guessed number, is what makes the autoscaling both responsive and stable, scaling up fast on real demand and down calmly when the work is truly done.

Should you scale on queue depth or consumer lag?

For Kafka specifically, these are different measurements and the distinction decides whether your autoscaling works.

Consumer lag is the offset difference between the newest message in a partition and the last one your group committed — how far behind you are, in messages. Queue depth in the general sense is how much work is waiting. For Kafka, lag is the depth measurement, and it is the correct signal.

But lag has one property that CPU does not, and it changes how you configure things: lag is bounded by partition count. Kafka assigns each partition to at most one consumer in a group, so a topic with 12 partitions supports at most 12 useful consumers. Scaling to 30 pods gives you 12 workers and 18 idle processes, and the HPA — seeing lag stay high — will keep scaling, adding pods that mathematically cannot help.

This produces a genuinely confusing failure: autoscaling is working, pods are being added, and lag does not improve. Nothing in the HPA reports the problem, because the constraint lives in the topic’s partition count rather than in Kubernetes.

Two rules follow. Cap maxReplicas at the partition count for Kafka consumers, so the autoscaler cannot chase a target it is unable to reach. And choose partition count with your maximum scale in mind, since increasing partitions later is possible but changes key-to-partition mapping and therefore ordering guarantees — the reasoning in Kafka Partitions, Retention, and Compaction.

For queues without that constraint — SQS, RabbitMQ, Redis lists — any number of consumers can draw from the same queue, so replica count is bounded only by downstream capacity. That difference is worth knowing before you copy an autoscaling configuration from one system to another.

One more Kafka-specific caution: rebalancing is not free. Every time the consumer group membership changes, partitions are reassigned and consumption briefly pauses. An autoscaler that adds and removes pods aggressively triggers a rebalance on each change, and a consumer group that spends its time rebalancing processes very little. This is the strongest argument for generous stabilisation windows on scale-down: the cost of holding an extra pod for a few minutes is far lower than the cost of a rebalance storm. Scale up eagerly and scale down lazily — the asymmetry is deliberate, because the two directions have completely different risk profiles and only one of them is urgent.

That asymmetry is the single most transferable idea here. Scaling up late costs latency, breached SLOs, and sometimes an incident; scaling down late costs a few minutes of a pod you were already paying for. When the two error directions differ that much in consequence, the configuration should differ to match, and a symmetric policy is almost always the wrong default no matter which signal you scale on.

What does queue-depth autoscaling look like end to end?

A worked example makes the interaction between the numbers concrete, because the failure modes come from how they combine rather than from any one of them.

Take a notification consumer. Measured per-pod throughput is 100 messages per second. The product requirement is that a notification is delivered within 30 seconds. Baseline traffic is 500 messages per second; a marketing send produces a burst to 5,000 per second for two minutes.

Steady state needs 5 pods to keep up with 500 msg/s. The HPA target, from the earlier formula, is 30 × 100 = 3,000 messages per pod.

Now the burst. Arrivals jump to 5,000/s while capacity is 500/s, so the queue grows at 4,500 messages per second. Within about 4 seconds, depth per pod crosses 3,000 and the HPA begins scaling. To drain 5,000/s you need 50 pods, ten times the steady-state fleet.

Three things determine whether that succeeds, and none of them is the target value:

• Scale-up rate. The HPA’s default behaviour is deliberately conservative. Going 5 → 50 pods takes several scaling intervals unless you raise the scale-up policy, and every interval spent below capacity is queue that keeps growing.
• Pod startup time. Fifty pods that each take 45 seconds to become ready cannot help with a two-minute burst until it is nearly over. Startup time is frequently the binding constraint on autoscaling responsiveness, and it is invisible in the HPA configuration.
• Downstream capacity. Fifty pods hitting a database sized for five is how autoscaling turns a queue problem into an outage. The consumer’s dependencies must scale too, or the consumer needs a concurrency cap that protects them.

The lesson the arithmetic teaches: autoscaling cannot be faster than your slowest pod start, and it must never be allowed to overwhelm what sits downstream. For predictable bursts like scheduled sends, pre-scaling before the event beats reacting to it — the queue never grows, so nothing has to catch up.

What target queue depth should you set?

“Scale on queue depth” is only actionable once you pick a number, and the number should be derived rather than guessed.

The quantity that matters to users is drain time, not depth. A depth of 5,000 means nothing on its own; a depth of 5,000 against a per-pod throughput of 100 messages per second across 10 pods means five seconds to clear, and five seconds is either fine or catastrophic depending on the product.

So work from the SLO backwards:

1. State the acceptable lag. “A notification is delivered within 30 seconds of its event.”
2. Measure per-pod throughput under realistic load, in messages per second.
3. Compute the target depth per pod: acceptable_lag × per_pod_throughput. At 100 msg/s and a 30-second budget, that is 3,000 messages per pod.
4. Set the HPA target to that per-pod value. The autoscaler adds pods when average depth per pod exceeds it, which is exactly the condition under which lag would breach the SLO.

The property this gives you is the reason to prefer it over CPU: the autoscaling target is now a restatement of the SLO. When the SLO changes, one number changes. When someone asks why the target is 3,000, there is an answer that does not begin with “we tuned it until it looked stable.”

A related trap: scale on depth per pod, not total depth. A total-depth target does not adapt as the fleet grows — 10,000 messages across 2 pods and across 50 pods are wildly different situations, and only the per-pod figure distinguishes them.

How do you get queue depth into the HPA?

The Kubernetes HPA cannot read a queue by itself; it consumes metrics through an API, so something must bridge them. Three options, in decreasing order of how much work you do:

Approach	How it works	Best for
KEDA	An event-driven autoscaler with built-in scalers for Kafka, SQS, RabbitMQ, and others; manages the HPA for you	Almost everyone — it is the default answer
Prometheus Adapter	Exposes existing Prometheus metrics as Kubernetes custom metrics for the HPA	You already run Prometheus and want one metrics path
Custom metrics API	Implement the adapter yourself	An exotic queue nothing else supports

KEDA is the right default, and its most valuable feature is not the scalers — it is scale-to-zero. A CPU-based HPA cannot go below one replica, so every idle consumer costs a pod forever. KEDA can run zero pods when the queue is empty and cold-start one when a message arrives, which for spiky or low-volume workloads is a large and permanent cost reduction.

Two operational cautions. The metrics path is now on your critical scaling path: if the adapter or the queue’s metrics endpoint fails, the HPA stops receiving values and freezes at its current replica count — often precisely during the incident when scaling matters. Alert on metric staleness, not just on queue depth.

And scale-to-zero interacts badly with slow cold starts. If a pod takes 40 seconds to become ready, scaling from zero adds 40 seconds of lag to the first message after an idle period. Either keep a warm minimum for latency-sensitive consumers, or fix the startup time — the techniques are in Cutting Kubernetes Pod Startup Time.

An autoscaling-signal checklist

When configuring an HPA, decide:

• Is this workload request-driven (CPU tracks demand) or queue-driven (backlog is the truth)?
• For queue workers, the scaling metric is queue depth or consumer lag, not CPU.
• The target is expressed as work-per-replica (messages/lag per pod), tied to your drain-rate goal.
• KEDA (or an external-metrics pipeline) feeds the queue metric to the HPA.
• Scale-to-zero is enabled where idle workers cost money for no reason.
• The web tier and the worker tier use the signal appropriate to each.
• You verified the autoscaler reacts before latency degrades, by load-testing a burst.

What about scaling on latency or in-flight requests?

Queue depth is the cleanest signal for async workers, but for some workloads the right leading indicator is in-flight request count or end-to-end latency, and the same principle applies: scale on the metric that reflects demand before saturation, not after. The point was never “always use queue depth”; it was “use a leading indicator, and CPU usually isn’t one.”

For a synchronous service with bounded concurrency, the number of in-flight requests is often a better signal than CPU, because it rises the moment requests queue up at the application even if CPU has not yet maxed out. For some user-facing services, scaling on a latency SLO works too: if p95 latency starts climbing, add capacity, because rising latency is an early sign of saturation. Both are leading indicators in the way CPU is not for these shapes of work.

The unifying rule is to choose the metric that crosses its threshold while you still have time to react. CPU works when it rises with demand in real time. Queue depth works for async workers. In-flight count or latency work for concurrency-bound services. The wrong move, the one this whole post argues against, is defaulting to CPU for every workload because it is the built-in option, when the workload’s real demand shows up somewhere else first.

What I’d do differently

The mistake I have watched (and made) is reaching for CPU autoscaling everywhere because it is the default, then being confused when the worker tier falls behind under load while CPU looks fine. The autoscaler was doing its job; it was just watching the wrong number, one that could not see the backlog forming.

If I were setting up autoscaling again, I would ask one question per workload before touching the HPA: does CPU rise with demand in real time here? If yes, CPU is fine. If the work flows through a queue, I would wire KEDA to the queue depth from the start. Autoscaling is only as good as the signal it watches, and for async workers the queue is the only signal that tells the truth in time.

Sources

• Kubernetes, Horizontal Pod Autoscaler: kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale
• KEDA, Kubernetes Event-Driven Autoscaling: keda.sh/docs/latest/concepts
• Kubernetes, HPA with custom and external metrics: kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#scaling-on-custom-metrics