Structured Logging and Correlation IDs

Intermediate · 40 min

What to put in a log line, when to use each log level, and how a correlation ID lets you reconstruct one request across an entire call chain.

Before thisHealth Checks and Graceful Shutdown

Sriniously · 40 min

Structured Logging and Correlation IDs

The player loads from youtube-nocookie.com when you press play. Nothing reaches YouTube before that.

Taught by SriniouslyReport a problem with this video
Not started
Set this lesson's status

What you'll learn

  • A structured log line is a set of named key-value fields, not a free-text sentence — the difference is that a field can be filtered, aggregated, and queried across millions of lines, while a sentence has to be grepped and parsed by hand.
  • Log levels exist so a reader (or an alerting system) can decide how much to care without reading every line — DEBUG for development-only detail, INFO for normal operation, WARN for handled-but-unexpected situations, ERROR for failures that need attention, and FATAL for a crash in progress.
  • A correlation ID is generated once, at the edge, the moment a request enters the system, and is then threaded through every log line and every downstream call that request triggers, so one ID ties together everything that happened while serving it.
  • Propagating the ID means reading it from an incoming header if one exists, generating a new one only if it doesn't, and forwarding the exact same value on every outbound call that request makes — generating a fresh ID at each hop breaks the whole chain.

Why a log line is a record, not a sentence

Most developers start out writing logs the way they’d write a comment: console.log("User 123 created order 456"). It reads fine in a terminal during development. It falls apart the moment you have real production traffic, because a sentence like that is only searchable by string-matching — grepping for “created order” works until you need every order created by user 123 in the last hour, broken down by how long each one took, at which point you’re writing regex against prose that was never meant to be queried.

The fix is to stop writing sentences and start writing records: a structured log line emits a set of named fields — event: "order.created", userId: 123, orderId: 456, amount: 29.99, duration_ms: 42 — typically serialized as JSON, one object per line. Every mainstream backend language has a library built for exactly this: pino in Node, zap in Go, structlog in Python, Serilog in .NET. The output looks less friendly in a raw terminal, but it’s not meant to be read raw — it’s meant to be shipped to a logging backend that can filter on userId=123, aggregate duration_ms across a million requests, or alert the moment event="payment.failed" spikes. None of that is possible against free text without serious, fragile string parsing. This lesson is specifically about what you put into one of those lines and how you tag it — the dashboards, search indexes, and alerting rules that consume structured logs at scale are their own discipline, covered in Monitoring, Logging, and Alerting.

Log levels: how much should anyone care about this line

A level is a signal about urgency, separate from the content itself, and skipping this distinction is almost as common a mistake as skipping structure entirely:

  • DEBUG — verbose detail useful while actively debugging, and actively harmful in production: at real traffic volume, DEBUG-level logging can be expensive to store and impossible to read through.
  • INFO — the normal operation of the system: a request was handled, a job completed, a user logged in. This is what you’d want to see scrolling by in a healthy system.
  • WARN — something unexpected happened but the system recovered on its own: a retry that eventually succeeded, a deprecated code path that still works, a slow-but-completed query. Nobody needs to be paged for a WARN, but a human should eventually look.
  • ERROR — a failure that needs attention: a request that couldn’t be completed, an exception that propagated out.
  • FATAL — the process cannot continue and is about to crash.

The reason to bother with five levels instead of two is that levels are a filter, not just a label. A production system set to log at INFO and above stays quiet during normal operation and loud exactly when something needs a look — that’s the entire point of having the distinction in code, independent of whatever dashboard eventually reads it.

The problem correlation IDs solve

A single user-facing request routinely fans out into several internal calls: an API gateway hits an auth service, which hits a user service, which queries a database and calls a third-party API. Each of those hops produces its own log lines, in its own service, often on different machines. When something goes wrong, the question you actually need answered is “show me everything that happened while handling this one request” — and without a shared identifier, that question is nearly unanswerable. You’d be stuck correlating by timestamp across five different log streams and guessing.

A correlation ID fixes this directly: a single identifier, generated once, carried through everything that request touches. The pattern is consistent across languages and frameworks:

  1. At the edge — the first service a request hits — middleware checks for an incoming correlation header (commonly X-Correlation-ID or X-Request-ID). If one is present, reuse it exactly. If not, generate a new one, typically a UUID.
  2. Attach it to every log line produced while handling that request, as a named field — correlationId: "a1b2c3..." — the same structured-field discipline from the section above.
  3. Forward it on every outbound call that request triggers: HTTP calls to other services, messages published to a queue, anything downstream. The receiving service repeats step 1, sees the header already set, and reuses the same value rather than minting its own.
  4. Optionally echo it back in the response headers, so a client or support engineer can hand you the exact ID tied to the request they’re asking about.

Do this consistently and a single search for one correlation ID across your centralized log store reconstructs the entire path a request took — every service it touched, every log line any of them emitted, in order. It’s one of the cheapest, highest-payoff additions you can make to a system that’s otherwise hard to debug in production, and it costs almost nothing: a UUID, a header, and a field on every log call.

Where this stops being a code-level concern

Everything above — what fields go in a line, which level to use, how the ID gets generated and forwarded — is work that happens in your own application code, independent of any particular logging platform. What happens after a structured log line with a correlation ID leaves your process — ingestion, indexing, building a searchable dashboard from millions of these lines — is infrastructure, not application code, and it’s covered in Monitoring, Logging, and Alerting. And when a correlation ID’s “it’s the same string in every log line” approach isn’t enough — when you need actual per-hop timing and a visual map of where time went across a dozen services — that’s the job of Distributed Tracing, a heavier mechanism built for exactly that problem.

Check yourself

  1. 01What makes a log line 'structured,' and why does that matter more than it sounds?

    A structured log line records its information as named fields — event, userId, orderId, duration_ms — typically serialized as JSON, rather than as a sentence like 'User 123 created order 456 in 42ms.' The payoff shows up at scale: a logging backend can filter on userId=123 or aggregate duration_ms across millions of lines instantly, while a free-text sentence can only be searched with fragile string matching.

  2. 02Why have five log levels instead of just 'normal' and 'error'?

    Levels let a reader or an alerting rule decide how much attention a line deserves without reading its content. DEBUG is noise outside active debugging. INFO is routine operation. WARN flags something unexpected that was still handled, like a retry that succeeded. ERROR means something failed and needs a human. FATAL means the process is about to go down. Collapsing these means either drowning in debug noise in production or missing the 'this was weird but survived' signal that WARN exists for.

  3. 03Where does a correlation ID come from, and what happens if a request already arrives with one?

    It's generated at the edge of the system — the first service a request hits — the moment that request arrives. If the incoming request already carries one, for instance because an upstream caller or a previous hop set it, the service reuses that exact value instead of generating a new one. Only a request with no existing ID gets a freshly generated one; everything downstream just forwards what it received.

  4. 04Why does generating a new correlation ID at each service hop break the point of having one?

    The entire value of a correlation ID is that one identifier appears in every log line produced anywhere while handling one request. If service B mints its own new ID instead of forwarding the one it received from service A, then A's logs and B's logs now have no shared value to search for — you can reconstruct each service's internal behavior but not stitch the two together into one request's full story.

  5. 05What's the difference between a correlation ID and the kind of request tracing a distributed tracing system like OpenTelemetry provides?

    A correlation ID is a single opaque string you attach to log lines and forward on calls — it's a log-stitching mechanism with almost no infrastructure required. A distributed tracing system additionally captures timing, parent-child relationships between calls, and span-level detail, letting you see a visual waterfall of where time was spent across services, not just that the services were part of the same request. Correlation IDs are the cheap, immediate version; tracing is the heavier, richer one.

Go deeper

Notes

Private to you

Esc
↑↓ navigate↵ open? shortcuts

Keyboard shortcuts

Press Esc to close

Anywhere

⌘K
Search and commands
/
Search
?
This sheet
Esc
Close

Reading a lesson

J
Next lesson
K
Previous lesson
B
Bookmark