Most backends do not fall over because of one giant problem. They fall over because of a dozen small assumptions that were fine at a thousand requests a day and quietly stop being fine at a million. I have spent a good chunk of my career chasing those assumptions down, usually at three in the morning, and the patterns below are the ones I keep coming back to. None of them are exotic. The skill is knowing which one to apply and when to stop.
Scale up before you scale out
The first question I ask when a service is struggling is whether I can just give it a bigger machine. Vertical scaling gets a bad reputation because it has a ceiling, but that ceiling is much higher than people think. A modern instance with 64 cores and 256 gigabytes of memory will carry an enormous amount of traffic, and you get it without touching your code, your deployment, or your mental model of the system.
Horizontal scaling is where you add more machines and spread the work across them. It scales further, but it forces decisions on you. State has to live somewhere shared. Requests have to be routed. Failures multiply because now you have ten things that can break instead of one. My rule is simple. I scale up until it gets expensive or I hit the instance ceiling, and only then do I scale out. Reaching for a fleet of tiny nodes on day one is how you end up debugging distributed systems problems you did not need to have yet.
Statelessness is the thing that makes everything else possible
You cannot spread traffic across many servers if any one server is secretly special. The moment a request only works because it landed on the same box that handled the previous request, horizontal scaling is dead. This is why I treat statelessness as the foundation rather than an optimization.
In practice that means no session data in local memory, no uploaded files sitting on the local disk, no in-process counters that matter. Push session state into Redis or a signed token. Push files into object storage. Push anything durable into a database. When every application node is interchangeable, a load balancer can send a request to any of them, you can add and remove nodes freely, and a crashed node costs you nothing but the in-flight requests it was holding.
If you want the wider context for how this fits into a system, I wrote about it in modern fullstack architecture, where statelessness shows up again as a precondition for clean deploys.
Load balancing and how requests find a home
Once you have several interchangeable nodes, something has to decide where each request goes. A load balancer sits in front and distributes traffic. Round robin is the obvious starting point and it is fine for uniform workloads, but it gets dumb fast when requests vary in cost. Least connections is usually a better default because it sends new work to whichever node is least busy right now.
The part people forget is health checks. A load balancer is only as good as its ability to notice a sick node and stop sending it traffic. I always configure active health checks with a real endpoint that touches the critical dependencies, not a route that returns 200 no matter what.
upstream api_backend {
least_conn;
server 10.0.1.10:8080 max_fails=3 fail_timeout=30s;
server 10.0.1.11:8080 max_fails=3 fail_timeout=30s;
server 10.0.1.12:8080 max_fails=3 fail_timeout=30s;
}
server {
listen 443 ssl;
location / {
proxy_pass http://api_backend;
proxy_next_upstream error timeout http_502 http_503;
}
location /healthz {
access_log off;
proxy_pass http://api_backend;
}
}
Caching, the highest leverage move you have
Nothing buys you headroom faster than not doing work twice. Caching is the highest leverage tool in this whole list, and it operates at several layers. There is the CDN at the edge for static assets and cacheable responses. There is an application cache like Redis or Memcached for computed results and hot rows. There is the database query cache and the operating system page cache below that. Each layer you can serve from is a layer of work the layers beneath it never see.
The hard part is never reading from a cache. It is invalidation. A stale cache is worse than no cache because it lies confidently. I lean heavily on time based expiry because it is predictable, and I only reach for event based invalidation when staleness genuinely hurts. Cache-aside is my default pattern: check the cache, on a miss go to the source, then write the result back with a TTL.
def get_user(user_id):
key = "user:" + str(user_id)
cached = redis.get(key)
if cached is not None:
return deserialize(cached)
user = db.query("SELECT * FROM users WHERE id = %s", user_id)
redis.setex(key, 300, serialize(user))
return user
Two failure modes are worth naming. A cache stampede happens when a popular key expires and a thousand requests all miss at once and slam the database together. You fight it with jittered TTLs or a short lock so only one request rebuilds the value. The second is unbounded growth, which you handle with an eviction policy like LRU and a memory cap.
Queues, or how to stop doing slow work in the request path
A lot of what makes requests slow does not need to happen while the user waits. Sending an email, resizing an image, generating a report, syncing to a third party. If the caller does not need the result right now, get it out of the request path and onto a queue. The web tier accepts the job, hands it to a broker like RabbitMQ or SQS, returns immediately, and a pool of workers chews through the backlog at its own pace.
This does two things. It makes your latency predictable because the request no longer waits on slow downstream work. And it absorbs spikes, because a queue is a buffer. When traffic triples for an hour, the queue grows and the workers catch up afterward instead of the whole system melting. The thing you must get right is idempotency. Queues redeliver. A message will be processed more than once eventually, so every worker has to be safe to run twice on the same input.
- Make handlers idempotent with a dedupe key or an upsert so a redelivery is harmless.
- Set a dead letter queue so poison messages stop blocking the line and land somewhere you can inspect them.
- Watch queue depth as a first class metric. A growing queue is the earliest warning that workers cannot keep up.
Backpressure and graceful degradation
Scaling is not only about handling more. It is about failing well when you cannot. A system under genuine overload should shed load deliberately rather than collapse. I put rate limits on the edges, timeouts on every outbound call, and circuit breakers around flaky dependencies so a slow downstream service does not pile up threads and take the whole process with it.
Graceful degradation means deciding ahead of time what to drop first. If the recommendation service is down, show a generic list instead of an error page. If the cache is cold, serve slightly stale data rather than nothing. The users who get a degraded experience are far happier than the users who get a 500.
Where the database fits
Everything above buys time, but it eventually runs into the database, which is almost always the real bottleneck. Caching reduces reads, queues smooth writes, but past a certain point the data layer itself has to scale. That is a big enough topic that I gave it its own write up on scaling databases with replication and sharding, and if your schema is fighting you before you even get there, fix that first with solid schema design.
The order I actually apply these
If I had to compress all of this into a sequence, it would be this. Make the service stateless so it can scale at all. Scale up until it is uneconomical. Put a load balancer and more nodes in front. Add caching at the layers that hurt most. Move slow work onto queues. Add backpressure so overload degrades instead of crashes. Then, and usually only then, do the harder work of scaling the database. Measure at every step, because the bottleneck is rarely where your intuition says it is, and adding capacity to the wrong layer just moves the queue around.
