A load shedding strategy answers an uncomfortable product question: when demand exceeds useful capacity, which work should the system refuse, delay, or simplify so that the most valuable work can still finish? Autoscaling helps with sustained demand when more capacity exists and can arrive in time. It cannot repair a saturated database, a fixed provider quota, a regional capacity shortage, or a retry storm quickly enough to protect every request.
Without an explicit policy, overload makes the decision accidentally. Long queues increase latency for everyone, expensive requests occupy workers, retries multiply demand, and health checks may remove slow instances and concentrate traffic on fewer survivors. Deliberate admission control accepts that some work must be lost or postponed and chooses the outcome while the service can still respond predictably.
Find the resource that limits useful work
Measure where throughput stops increasing as offered load rises. CPU may be visible while the actual constraint is a connection pool, lock, downstream quota, memory, thread count, storage IOPS, or per-tenant hot partition. Track successful completions, not only accepted requests. A server that accepts more work while latency and timeouts rise is accumulating obligations, not increasing capacity.
Build a saturation envelope from representative load tests and production observations. Record maximum sustainable throughput, latency before the knee, concurrency, queue delay, resource headroom, and behavior when one failure domain disappears. Include request cost classes; a report scanning a year of data and a cached profile read should not count as equivalent requests. Revisit the envelope after schema, dependency, instance, and traffic-shape changes.
| Constraint signal | Misleading interpretation | Better admission input | Protective action |
|---|---|---|---|
| CPU near 100% | Every request costs the same | CPU per request class and completion rate | Shed expensive optional work first |
| Queue length | More queueing will absorb the burst | Oldest age versus useful deadline | Reject or expire work that cannot finish |
| Dependency latency | The client should wait longer | Remaining end-to-end deadline and in-flight calls | Fail fast and stop new dependent work |
| Quota consumption | Autoscaling will help | Remaining quota by operation and tenant | Reserve quota for critical operations |
| Error rate | All errors show overload | Overload reason codes and saturation state | Separate capacity rejection from defects |
Classify work by business value and cost
Define a small, stable set of priority classes tied to user outcomes: safety or integrity controls, interactive core actions, paid or contractual batch commitments, best-effort background work, and discretionary enrichment. Include computational cost and deadline. Priority should not be a caller-controlled string; derive it from authenticated identity, operation, product state, and policy so clients cannot promote themselves during contention.
Avoid equating priority with customer price alone. A free user’s account-recovery request may be more urgent than an enterprise export. Preserve fairness within a class using per-tenant concurrency, token buckets, weighted scheduling, or a maximum share. Reserve some capacity for low-volume critical operations so a noisy bulk client cannot consume every worker. Publish the policy to product, support, and customer teams before it is exercised.
Choose reject, queue, or degrade at admission
Reject synchronous work before expensive parsing, authentication fan-out, database access, or allocation when it cannot meet its deadline. Return a stable overload response that clients can distinguish from a permanent error. Include retry guidance only when the service can estimate it, and cap automatic client retries. Early rejection frees capacity to complete admitted work and makes latency more predictable.
Queue work only when delayed completion remains useful and acceptance can be durable. Bound queue size and item age; decide whether new or old work is discarded when full; expose status; and make consumers idempotent. Queue-based load leveling smooths bursts but does not create backend capacity. If arrival exceeds service rate for long enough, the queue still fills, so admission policy must eventually slow or refuse producers.
Degrade when a cheaper result preserves meaningful value. Omit optional fan-out, use a bounded-staleness cache, lower result detail, or switch an export to asynchronous preparation. Measure the saved constrained resource. A brownout that removes an image while database queries remain unchanged is cosmetic, not protective. Never degrade away authorization, transaction integrity, or other controls required to make the result trustworthy.
| Admission choice | Use when | Client contract | Primary risk |
|---|---|---|---|
| Immediate rejection | Deadline cannot be met or resource is saturated | No work accepted; retry is bounded or discouraged | Poor clients retry aggressively |
| Bounded queue | Work stays valuable after delay | Accepted durably with status and expiry | Backlog outlives recovery capacity |
| Degraded response | Cheaper result is truthful and useful | Reduced content, freshness, or scope is explicit | Users mistake partial result for full result |
| Reserved capacity | Small critical class must survive | Only eligible operations use the reserve | Reserve sits idle or classification is abused |
| Fair-share limit | Tenants compete for shared resources | Each tenant gets a bounded share | Small tenants may still face global saturation |
Stop retries from becoming a second traffic source
Every retry is new load generated by a failed attempt. Set end-to-end deadlines, propagate the remaining time, cancel abandoned work, use exponential backoff with jitter, and allow retries only for operations that are safe and likely to succeed. Enforce a retry budget as a fraction of original traffic so errors cannot multiply demand without bound. Choose one layer to retry where possible; retries at three layers can amplify one user operation dramatically.
Servers should reject work that cannot finish before the caller’s deadline and expose overload separately from application faults. Clients should honor Retry-After or equivalent signals but still apply local limits. Mobile and browser clients deserve special scrutiny because old versions can persist for years. Protect the service at the edge even when official clients behave well.
Implement stable and observable admission control
A useful controller reacts before collapse and relaxes gradually. Inputs may include in-flight work, queue age, recent completion latency, dependency capacity, or a concurrency limit tuned by load testing. Add hysteresis and smoothing so one noisy sample does not alternate between open and closed. Prefer local decisions for fast protection, with centrally distributed policy and bounded local defaults if the control plane fails.
Emit admitted, rejected, expired, degraded, and completed counts by reason and priority, while avoiding unbounded tenant labels. Observe useful throughput, not just rejection rate. A high rejection rate can be correct during a flash event if critical completion remains healthy. Alert on policy failures such as critical-class rejection, reserve exhaustion, excessive queue age, retry growth, or a controller stuck in an emergency state.
Test fairness, failure, and recovery
Run step, spike, and sustained overload tests with realistic mixtures of tenants and request costs. Remove a dependency replica, reduce a quota, slow one shard, and add retries. Verify that critical work completes, lower classes receive their intended shares, queues remain bounded, overload responses are cheap, and the system does not oscillate. Test configuration errors that classify all work as high priority; the controller needs a safe response to policy mistakes.
Recovery needs a ramp. When capacity returns, do not release all queued and retried work at once. Apply rate limits to backlog drain, prioritize by expiry and value, and preserve headroom for live requests. Keep shedding until latency, queue age, and resource pressure are stably below exit thresholds. Review which users were refused and whether the business ordering still makes sense.
Audit fairness after real and simulated overload. Compare admission and completion by priority class, tenant size, region, client version, and accessibility-critical operation using bounded reporting dimensions. Look for a policy that technically reserves capacity yet disadvantages small tenants through minimum batch size, slow clients through tight timeouts, or one region through a shared global limit. Product owners should approve corrections and document why unequal treatment is necessary, proportionate, and observable.
Key takeaways
- Identify the constrained resource and maximum useful throughput before choosing thresholds.
- Derive priority from authenticated business context and preserve fairness within each class.
- Reject early, queue only delay-tolerant work, and degrade only to truthful cheaper outcomes.
- Propagate deadlines and enforce retry budgets across the call path.
- Test mixed-cost overload and ramp backlog recovery instead of reopening instantly.
Frequently asked questions
Why not rely on autoscaling?
Scaling has detection and startup delay, may be limited by quotas or stateful dependencies, and can arrive after queues and retries destabilize the service. Admission control protects the system during that gap and when no additional capacity is available.
Should shedding happen only at the API gateway?
No. Edge limits are useful, but each scarce internal boundary should protect itself because request costs and local saturation differ. Coordinate signals so a caller does not continue sending work to a dependency that is already rejecting it.
Should premium customers always get highest priority?
Commercial tier can be one factor, but urgency, integrity, operation cost, and contractual purpose also matter. Use explicit weighted fairness and reserve safety-critical actions. Have product and legal teams approve any customer differentiation.
Conclusion
Overload cannot be negotiated away after saturation. A sound load shedding strategy measures the real constraint, ranks work transparently, rejects before wasting capacity, and gives delayed or reduced outcomes precise semantics. By bounding retries and recovery as carefully as admission, teams can preserve useful service instead of letting every request fail slowly together.