Load balancers are the traffic cops of distributed systems. Without one, adding more servers to handle load requires changing DNS, and a single backend crash takes down your entire service. With one, you distribute requests across a fleet of servers, route around failures in milliseconds, and deploy new versions without downtime.
What changed in 2026
- Envoy is the default for service-mesh proxies. Istio, Linkerd 2.x, and Consul Connect all use Envoy under the hood for sidecar load balancing with built-in observability.
- Cloud ALBs got smarter. AWS ALB and GCP Cloud Load Balancing now support gRPC, WebSockets, and HTTP/3 (QUIC) natively — no special configuration needed.
- eBPF-based load balancing. Cilium can do L4 load balancing inside the Linux kernel with near-zero overhead — relevant for high-throughput Kubernetes clusters.
- QUIC/HTTP3 changes latency profiles. Load balancers that terminate QUIC connections reduce connection establishment latency by ~100 ms on first request (no TCP + TLS handshake).
Layer 4 vs Layer 7
| Dimension |
L4 (Transport) |
L7 (Application) |
| Operates on |
TCP/UDP packets |
HTTP/gRPC/WebSocket frames |
| Routing basis |
IP + port |
URL path, headers, cookies |
| TLS handling |
Pass-through or terminate |
Terminate, inspect, re-encrypt |
| Performance |
Faster (less inspection) |
Slower, richer features |
| Sticky sessions |
IP hash |
Cookie-based |
| Examples |
AWS NLB, HAProxy TCP mode |
AWS ALB, Nginx, Envoy |
Use L4 when raw throughput matters (databases, TCP services). Use L7 for HTTP APIs where you need path-based routing, header manipulation, or mTLS.
Load balancing algorithms
| Algorithm |
How it works |
Best for |
| Round-robin |
Rotate through servers sequentially |
Uniform request durations |
| Least connections |
Send to server with fewest active connections |
Variable durations (API, DB queries) |
| Weighted round-robin |
Round-robin respecting server weights |
Mixed-capacity servers |
| IP hash |
Hash client IP → consistent server |
Stateful sessions (avoid if possible) |
| Least response time |
Track P95 latency per backend |
Latency-sensitive APIs |
| Random with two choices |
Pick 2 random servers, send to the better |
Large clusters, low overhead |
For most HTTP APIs, least connections is the safest default. It naturally routes away from slow backends without configuration.
Nginx load balancer configuration
upstream api_backends {
least_conn; # algorithm
keepalive 32; # persistent connections to backends
server 10.0.1.1:3000 weight=3;
server 10.0.1.2:3000 weight=3;
server 10.0.1.3:3000 weight=1 backup; # only used when others are down
}
server {
listen 443 ssl http2;
ssl_certificate /etc/ssl/cert.pem;
ssl_certificate_key /etc/ssl/key.pem;
location /api/ {
proxy_pass http://api_backends;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
# Health-check awareness
proxy_next_upstream error timeout http_500 http_502 http_503;
proxy_next_upstream_tries 2;
}
}
Health checks
Health checks are not optional. A load balancer must know which backends are healthy before it routes traffic:
# Nginx Plus active health check (free alternative: use HAProxy)
upstream api_backends {
server 10.0.1.1:3000;
server 10.0.1.2:3000;
# Passive health check (built-in Nginx open-source)
# Mark unhealthy after 3 failures, re-check after 30s
}
# HAProxy active health check
backend api_servers
balance leastconn
option httpchk GET /healthz HTTP/1.1\r\nHost:\ localhost
default-server inter 5s fall 3 rise 2
server s1 10.0.1.1:3000 check
server s2 10.0.1.2:3000 check
Your /healthz endpoint must check internal dependencies (DB connection, cache connectivity). A "200 OK" that just returns {"status": "ok"} without checking dependencies is a false positive.
How to pick
- HTTP/HTTPS API traffic? → L7: AWS ALB, Nginx, or Caddy.
- TCP database connections or non-HTTP protocols? → L4: AWS NLB or HAProxy in TCP mode.
- Inside a Kubernetes cluster? → Service type
ClusterIP + Envoy (via service mesh) or kube-proxy.
- gRPC services? → Must use L7 (HTTP/2-aware) load balancer — L4 LBs can't distribute individual gRPC streams.
- Zero infrastructure management? → Cloud ALB (AWS, GCP, Azure) with auto-scaling target groups.
Common mistakes
No health checks. A backend that returns 503 on every request will continue receiving traffic without active health checks. Every backend must have a /healthz endpoint and the LB must poll it.
Sticky sessions via IP hash. IP addresses are not stable (mobile users, NAT). Use cookie-based stickiness or, better, design your backends to be stateless.
Long connection timeouts. Default Nginx proxy_read_timeout is 60 s. A stuck backend will hold connections for a minute. Set aggressive timeouts and let clients retry.
Not logging the original client IP. After the LB, backends see the LB's IP. Use X-Forwarded-For or the PROXY protocol to preserve the original IP in access logs.
What to skip
- DNS-based load balancing as your only strategy — DNS TTLs mean failover takes minutes, not milliseconds.
- Session-stateful backends — state on backends makes load balancing hard. Move sessions to Redis; keep backends stateless.
- Self-managed load balancers for simple apps — cloud ALBs handle DDoS mitigation, certificate renewal, and scaling automatically at very low cost (~$20/month base).
FAQ
What is the difference between a load balancer and a reverse proxy?
A reverse proxy forwards requests to backends on behalf of clients; a load balancer is a reverse proxy that distributes requests across multiple backends. Every load balancer is a reverse proxy; not every reverse proxy is a load balancer.
Can a load balancer be a single point of failure?
Yes — so cloud LBs run multiple instances across availability zones automatically. For self-managed LBs (Nginx, HAProxy), use VRRP (keepalived) for active-passive failover.
What is connection draining / graceful shutdown?
When deregistering a backend, the LB stops sending new requests but waits for in-flight requests to complete before removing the backend. AWS ALB calls this "deregistration delay" (default 300 s).
How do load balancers handle WebSockets?
WebSockets require sticky routing (the same backend handles the whole connection lifetime) and L7 LBs that support the HTTP Upgrade mechanism. Configure proxy_http_version 1.1 and the Upgrade header in Nginx.
Where to go next