Health Checks and Graceful Shutdown

Intermediate · 36 min

Writing liveness and readiness endpoints and a SIGTERM handler that drains in-flight requests before your own process exits.

Before thisSecrets and Sensitive Data Handling

Sriniously · 36 min

Health Checks and Graceful Shutdown

The player loads from youtube-nocookie.com when you press play. Nothing reaches YouTube before that.

Taught by SriniouslyReport a problem with this video
Not started
Set this lesson's status

What you'll learn

  • A liveness check only answers 'is this process still running,' while a readiness check answers the much more useful question of whether this specific instance can actually handle a request right now, including whether its dependencies like a database connection are up.
  • SIGTERM is a request to stop that your code can catch and respond to, unlike SIGKILL, which ends the process immediately with no chance to clean up — graceful shutdown only exists because SIGTERM exists.
  • The correct shutdown order is to stop accepting new connections first, let in-flight requests finish, and only then close database pools and other outbound resources — closing them in the wrong order breaks requests that are still running.
  • Failing the readiness check the instant a shutdown begins, while keeping the liveness check passing throughout the drain, is what stops an orchestrator from routing new traffic to a dying instance or restarting it mid-drain.

A running process isn’t the same as a working service

It’s tempting to think a service is “up” the moment its process starts and begins accepting connections. That’s the assumption that gets broken constantly in production: a process can be running, scheduled, and answering TCP connections while being completely unable to do its actual job — its database connection pool never came up, a required cache is unreachable, a dependency it needs on every request is down. The process is alive. The service is not working. Any system that only checks “is the process running” (a deployment script, a load balancer, an orchestrator) will happily keep sending it traffic it can’t handle.

This is the actual problem health checks solve, and it’s worth being precise about what a health check even is before writing one: it’s an endpoint in your own service, not a feature of the infrastructure around it. A Kubernetes cluster or a load balancer calls your health check — it doesn’t invent one for you. Writing that endpoint, and writing the shutdown logic that goes with it, is squarely your code’s job, and that’s this lesson’s scope. What a load balancer or Kubernetes does with the answer — restarting pods, draining connections at the cluster level, the broader discipline of chaos engineering — is covered separately in Health Checks and Chaos Engineering. This lesson is about the handler, not the orchestrator reading it.

Two different questions: liveness and readiness

A useful health check actually answers two separate questions, and conflating them is the single most common mistake:

Liveness asks: is this process alive and functioning, or is it stuck? A deadlock, an infinite loop, a thread pool that’s exhausted and can never recover — these are liveness failures, and the correct response to one is to kill the process and start a fresh one. A liveness check should be cheap and self-contained: check that the event loop is responsive, maybe that a basic internal invariant holds. It should not check a database.

Readiness asks: can this specific instance serve a request right now? This is where the database connection actually matters — if your service can’t reach its database, it genuinely cannot do its job, and it should say so. The correct response to a readiness failure is different from a liveness failure: pull this instance out of rotation, stop sending it traffic, but don’t restart it. Restarting a process whose database is unreachable doesn’t fix anything — the database is still down after the restart — it just adds the cost of a restart on top of an outage you can’t fix locally.

Why keep them separate? Picture a shared database going down for five minutes. If every instance’s liveness check pings that database, every instance in the fleet fails liveness simultaneously, and an orchestrator restarts the entire fleet at once — a self-inflicted outage on top of the real one, and the restarts won’t even help since the database is still down. If instead only readiness depends on the database, every instance correctly drops out of traffic (there’s nothing useful to send them), liveness stays green, nothing restarts, and the moment the database recovers, readiness flips back to healthy with zero redeploys. That one design choice is the difference between a contained incident and a cascading one.

SIGTERM: the signal that makes “graceful” possible

Stopping a process isn’t a single, uniform action. The operating system gives you (at minimum) two very different ways to end one: SIGKILL, which the kernel enforces immediately with zero cooperation from your code — no cleanup, no chance to finish anything, the process is just gone — and SIGTERM, a request to stop that your process can intercept, run code in response to, and use to clean up before actually exiting. Graceful shutdown exists entirely because SIGTERM exists. Deploys, autoscaling scale-downs, and rolling restarts all send SIGTERM first, specifically so well-behaved services get a window to wind down cleanly, and only escalate to SIGKILL after a grace period if the process hasn’t exited on its own.

The naive mistake is writing no handler at all, or writing one that just exits immediately. At the instant SIGTERM arrives, your process is very likely in the middle of handling live requests. Exiting right then cuts every one of those connections off mid-response — the client sees a dropped connection, a half-written response body, or a 502 from whatever’s in front of you. That’s the exact failure graceful shutdown is for.

The shutdown sequence, in the right order

A correct handler for SIGTERM runs through a specific sequence, and the order matters:

  1. Fail readiness immediately. The instant shutdown begins, flip the readiness check to failing, before anything else happens. This tells anything watching that endpoint to stop routing new traffic here.
  2. Wait briefly for that to propagate. A load balancer or router takes a moment — often a few seconds — to notice the readiness change and stop sending new connections. If you close your listening socket before this propagation finishes, requests sent during that gap hit a socket that’s already gone, producing connection-refused errors for clients who did nothing wrong.
  3. Stop accepting new connections, by closing the listening socket, while continuing to serve anything already in progress.
  4. Drain in-flight requests. Let every request that’s already being handled run to completion and send its response normally. This is the actual “graceful” part — nothing currently being processed gets cut off.
  5. Close outbound resources last, once draining is done: the database pool, Redis clients, message-broker connections. This has to come after draining, not before — a request still being handled that needs the database will fail if the pool is already torn down underneath it.
  6. Exit.

Throughout steps 1–4, liveness should keep reporting healthy. A drain that takes a few seconds is not a stuck process, and you don’t want an orchestrator interpreting a normal, bounded drain as a hang and sending SIGKILL partway through it. It’s also worth setting an internal drain timeout — a bound shorter than whatever grace period the environment around you gives before SIGKILL — so one unusually slow request can’t hold the whole shutdown hostage indefinitely; after that timeout, exit anyway rather than wait forever.

This sequence is the same shape whether you’re writing it in Go with http.Server.Shutdown(ctx), in Node with a server.close() callback and a tracked count of in-flight requests, or in any other language’s equivalent — the mechanics differ, the order doesn’t.

Check yourself

  1. 01Why isn't a single /health endpoint that just returns 200 good enough for a real service?

    A bare 200-OK endpoint only proves the process is scheduled and answering HTTP requests — it says nothing about whether that instance can actually do its job, like reach its database. A process can be technically alive while being completely unable to serve a real request, and a single generic endpoint can't distinguish those two states.

  2. 02What's the functional difference between a liveness check and a readiness check?

    Liveness asks whether the process itself is alive and not deadlocked or stuck, and failing it should trigger a restart. Readiness asks whether this instance can currently serve traffic, and failing it should just pull the instance out of rotation without restarting anything — a database outage should fail readiness everywhere at once, not trigger a cluster-wide restart storm.

  3. 03Why can't a process just exit immediately the moment it receives SIGTERM?

    At the moment SIGTERM arrives, the process is very likely in the middle of handling one or more requests. Exiting immediately cuts those connections off mid-response, which the client sees as a dropped connection or a 502 — exactly the failure a graceful shutdown exists to prevent. The fix is to stop accepting new work but let the work already in progress finish first.

  4. 04During a graceful shutdown, why should the readiness check start failing before the server actually stops accepting connections?

    A load balancer or router takes a moment to notice that readiness has failed and to stop sending new traffic to that instance. If the server closes its listening socket at the same instant it starts shutting down, any request already in flight toward it from before that propagation delay arrives at a socket that's no longer listening. Failing readiness first, then waiting briefly before closing the listener, closes that race.

  5. 05What goes wrong if a shutdown handler closes the database connection pool before the HTTP server has finished draining in-flight requests?

    Any request still being processed that needs to run a query will try to use a pool that's already been torn down, and it fails with a connection error instead of completing normally — turning what should have been a clean, slightly-delayed response into a hard failure for a request that was otherwise going to succeed.

Go deeper

Notes

Private to you

Esc
↑↓ navigate↵ open? shortcuts

Keyboard shortcuts

Press Esc to close

Anywhere

⌘K
Search and commands
/
Search
?
This sheet
Esc
Close

Reading a lesson

J
Next lesson
K
Previous lesson
B
Bookmark