Metrics and Instrumentation in Application Code

Intermediate · 15 min

What counters, gauges, and histograms actually measure, and how adding them to your own code is what makes a service's performance measurable.

Before thisStructured Logging and Correlation IDs

DevOps Hint · 15 min

Metrics and Instrumentation in Application Code

The player loads from youtube-nocookie.com when you press play. Nothing reaches YouTube before that.

Taught by DevOps HintReport a problem with this video
Not started
Set this lesson's status

What you'll learn

  • A counter only ever goes up (or resets to zero on a restart), which makes it the right shape for anything you're counting the occurrences of — requests served, errors thrown, jobs completed.
  • A gauge is a current value that can rise and fall, which makes it the right shape for a snapshot of state right now — queue depth, open connections, memory in use.
  • A histogram records the distribution of a measurement across many observations, which is what lets you later compute a p50, p95, or p99 instead of just an average that hides outliers.
  • Logs tell you about one event; metrics tell you about the aggregate shape of many events over time — the two are complementary, not interchangeable, and a well-instrumented service needs both.

Logs tell you about one thing; metrics tell you about all of them

Structured logging answers “what happened during this one request.” That’s the right tool when you already know which request you’re investigating. It’s the wrong tool for a different, equally common question: “is the checkout endpoint, across all ten thousand requests it served in the last hour, getting slower?” Answering that by reading log lines one at a time doesn’t scale, and it was never the job logs were built for.

Metrics are the answer: numbers your code emits about its own behavior, structured specifically so they can be aggregated across every request, every job, every instance, over any time window you ask for. This lesson is about what you actually add to your own code to make that possible — the three shapes a metric can take, and why picking the right one matters more than it sounds like it should.

Counters: things that only go up

A counter is the simplest shape: a number that only increases (or resets to zero if the process restarts). requests_total, errors_total, emails_sent_total — anything you’re tallying occurrences of fits here. The reason the “only increases” constraint matters isn’t pedantry: a monitoring system reading a counter over time computes a rate from it (requests per second, errors per minute) by looking at how fast it’s climbing. That only works cleanly if the only way the number can drop is a restart — mix in a counter that sometimes decrements for business reasons, and the rate calculation can no longer tell a real drop from your own decrement, which quietly breaks every alert built on top of it.

In code, incrementing one is close to free: requests_counter.inc() (or .Add(1), or whatever your metrics library’s syntax is) at the point in your request handler where you know the event actually happened — after a request completes, inside the except block that caught an error, right after a job finishes.

Gauges: the current value of something

A gauge is the opposite shape: a number that can go up or down, representing a snapshot of current state. Queue depth, number of open database connections, memory currently in use, number of users currently connected via WebSocket — all gauges, because “how many right now” is a meaningfully different question from “how many total, ever,” and the answer can legitimately decrease.

The practical difference in code: a gauge needs to be set or adjusted in both directions — queue_depth.set(current_length), or active_connections.inc() on connect paired with active_connections.dec() on disconnect. Getting this wrong by treating a gauge like a counter (only ever incrementing it) produces a chart that climbs forever and tells you nothing true about the present.

Histograms: where percentiles actually come from

This is the one that trips people up, because it’s solving a problem an average quietly hides. Say your checkout endpoint averages 120ms. That sounds fine. It can still be true that 1% of requests take 4 seconds — a slow database connection, a cold cache, a client on a bad network — and an average, built by summing everything and dividing by the count, smooths that slice of genuinely bad experience into invisibility. The users hitting that slow tail are having a real problem, and your one average number says nothing about them.

A histogram fixes this by recording which bucket each individual observation falls into — predefined ranges like 0-10ms, 10-50ms, 50-100ms, 100-250ms, and so on — instead of only keeping a running sum and count. With the full set of bucket counts, you can compute a percentile after the fact: a p50 (median — half of requests were faster than this), a p95, a p99 (99% of requests were faster than this; the remaining 1% is your slow tail). This is literally what’s happening whenever you see a dashboard showing “p99 latency: 850ms” — someone instrumented the code with a histogram, and a query against the stored bucket counts computed that percentile. There’s no shortcut from an average to a percentile; the histogram has to exist first.

In code, this usually looks like wrapping the operation you care about: start a timer, do the work, observe the elapsed duration into the histogram — request_duration_seconds.observe(elapsed) — once per request, at the point where you know how long it actually took.

Naming and the discipline that makes this usable later

A metric nobody can find or interpret six months later isn’t worth emitting. The convention that’s become near-universal (Prometheus’s naming scheme, adopted well beyond just Prometheus itself) is snake_case, a base-unit suffix (_seconds, _bytes, not _ms or _kb), and a _total suffix specifically for counters, so http_requests_total and http_request_duration_seconds tell you their shape from the name alone, before you’ve even looked at how they’re used.

Where this stops being your code’s job

Everything above is instrumentation: what you add inside your own service so it emits these numbers at all. What collects them from every instance, stores them, turns them into dashboards, and decides when a value crossing some threshold should page someone is a separate system, covered in Monitoring, Logging, and Alerting. And once you’re using these numbers to define what “healthy” even means for a service — committing to “p99 latency under 500ms, 99.9% of the time” as a formal target — that’s the discipline covered in SLIs, SLOs, and Error Budgets. Both depend entirely on the counters, gauges, and histograms this lesson covers actually existing in the code first.

Check yourself

  1. 01Why is 'total requests served' a counter and not a gauge?

    It only ever increases while the process runs — it never makes sense for it to go down, short of the counter resetting when the process restarts. A gauge is for values that genuinely move in both directions, like current memory use; using a gauge for something monotonically increasing throws away the guarantee a counter gives you (and that monitoring tools rely on) that any decrease means a restart, not a real drop in the underlying count.

  2. 02What's the difference between 'queue depth right now' and 'total jobs ever enqueued,' and which metric type fits each?

    'Queue depth right now' rises and falls as jobs are added and processed — that's a gauge, a snapshot of current state. 'Total jobs ever enqueued' only grows, forever — that's a counter, a running tally. Mixing them up means asking a monitoring system a question its graph can't actually answer: you can't recover 'how many jobs are in the queue right now' from a counter that never resets, and you can't get a clean rate-of-change out of a gauge the way you can from a counter.

  3. 03Why can't you compute a p99 latency figure from an average response time alone?

    An average collapses every request into one number and can look perfectly healthy even when a meaningful slice of requests are slow — a few very slow requests get averaged out by a much larger number of fast ones. A histogram instead records how many requests fell into each duration bucket (0-10ms, 10-50ms, 50-100ms, and so on), and a p99 is computed from that distribution: the duration below which 99% of requests fell. You need the distribution, not just the mean, to know how bad the slow tail actually is.

  4. 04If a histogram tracking request duration uses buckets like 0-50ms, 50-100ms, 100-250ms, what's lost compared to recording every individual duration?

    You lose the exact value of any single observation — you know a request landed in the 100-250ms bucket, not that it was precisely 143ms. What you keep is the shape of the whole distribution at a fraction of the storage cost of recording every raw value, which is the trade a histogram is explicitly making: precise aggregate percentiles, approximate individual values.

Go deeper

Notes

Private to you

Esc
↑↓ navigate↵ open? shortcuts

Keyboard shortcuts

Press Esc to close

Anywhere

⌘K
Search and commands
/
Search
?
This sheet
Esc
Close

Reading a lesson

J
Next lesson
K
Previous lesson
B
Bookmark