Bayesian Thinking for Data Scientists

Intermediate · 14 min

Priors, posteriors, and why framing a problem as Bayesian versus frequentist changes what your numbers actually mean.

Before thisConfidence Intervals and Sampling

StatQuest with Josh Starmer · 14 min

Bayesian Thinking for Data Scientists

The player loads from youtube-nocookie.com when you press play. Nothing reaches YouTube before that.

Taught by StatQuest with Josh StarmerReport a problem with this video
Not started
Set this lesson's status

What you'll learn

  • Frequentist methods treat the true parameter as fixed and unknown and ask how data behaves around it; Bayesian methods treat the parameter itself as uncertain and update a probability distribution over it.
  • A prior is your belief about a parameter before seeing new data, and it gets mathematically combined with the evidence to produce a posterior, your updated belief.
  • Bayesian results let you make direct probability statements about the thing you actually care about, like "there's an 80% chance B beats A," which frequentist confidence intervals and p-values never technically allow.
  • With enough data, a reasonable prior's influence shrinks and Bayesian and frequentist estimates tend to converge, but the Bayesian framing remains more useful with small samples or sequential decisions.

Two different ways of defining “probability”

Everything in the prior four lessons — p-values, confidence intervals, significance thresholds — comes from the frequentist school of statistics, which defines probability as a long-run frequency. “The probability of heads is 50%” means that if you flipped a coin infinitely many times, half would land heads. Under this view, a parameter like “the true conversion rate of button B” is a single fixed, unknown number, and all the randomness lives in the sampling process you use to estimate it. That’s why a frequentist confidence interval is a statement about the method’s long-run success rate, not a direct probability on the parameter, as covered in the previous lesson.

The Bayesian school defines probability differently: as a degree of belief, which can apply to a fixed but unknown quantity just as naturally as to a repeatable event. Under this view, “the true conversion rate of button B” isn’t just an unknown constant. It’s something you can hold a probability distribution over, representing how confident you are in each possible value. This isn’t a minor technicality. It changes what kinds of statements you’re allowed to make. A Bayesian can say “there’s an 80% probability that button B’s true conversion rate exceeds 10%.” A frequentist, strictly, cannot say that about a single fixed parameter, only about the method used to estimate it.

Priors, likelihood, and posteriors

Bayesian inference works through a specific update rule, Bayes’ theorem: posterior ∝ prior × likelihood. The prior is what you believed about a parameter before seeing the current data. The likelihood is how probable the data you actually observed is, under each possible value of the parameter. The posterior is your updated belief after combining the two, and it becomes the prior for your next round of data if more arrives.

Worked example: you’re testing a new checkout button. From years of past redesign tests at your company, you know conversion-rate lifts from button redesigns are usually modest, centered around 0% with most falling between -3 and +5 percentage points. That historical pattern becomes your prior — you’re not starting from total ignorance. You then run the new button for a day and observe 40 conversions out of 400 visitors (10%, against a 9% baseline, a 1-point lift). That single day’s data, the likelihood, is itself pretty noisy with only 400 visitors — consistent with a lift anywhere from roughly -2 to +4 points. Combining a prior that’s skeptical of large effects with weak, noisy evidence for a modest lift produces a posterior that’s pulled toward a small-but-plausible lift, rather than swinging wildly on one day’s noisy numbers. The practical effect: your early belief doesn’t overreact to a small sample the way a bare, prior-free point estimate would.

Why this matters in practice

The biggest practical payoff is interpretability. Stakeholders ask “what’s the chance this works,” and a Bayesian analysis answers that question directly. A frequentist analysis instead has to answer an adjacent, more awkward question (“if there were truly no effect, how surprising would this data be”) and hope the audience doesn’t conflate the two — which, as covered in the hypothesis-testing lesson, people constantly do.

The second payoff is handling small samples and sequential decisions gracefully. In frequentist hypothesis testing, peeking at results early and stopping as soon as you see significance inflates your false-positive rate, so you’re expected to fix your sample size in advance. Bayesian approaches can update continuously as data trickles in without that same penalty, because the posterior always represents “current belief given everything seen so far” rather than a one-shot calculation calibrated to a single, pre-committed sample size. This is part of why many modern A/B testing platforms default to Bayesian reporting, showing “probability B beats A” rather than a raw p-value — it matches how people naturally want to reason about a decision.

The tradeoff is that your choice of prior is a real, consequential choice. A prior that’s too strong relative to the data can anchor your posterior away from what the evidence actually shows, and a bad-faith or lazy prior choice can be used to manufacture whatever conclusion you want. The discipline of good Bayesian work is justifying the prior explicitly (ideally from real historical data, as in the example above) and checking how much it’s actually driving the answer versus how much the new data is. With a large enough sample, a reasonable prior’s influence fades and Bayesian and frequentist point estimates typically converge anyway. The real divergence, and the real value of Bayesian framing, shows up precisely when data is scarce and a decision still has to be made.

Check yourself

  1. 01What is the core philosophical difference between frequentist and Bayesian statistics?

    Frequentist statistics treats the true parameter as a single fixed number and describes how sample statistics would vary across repeated sampling, while Bayesian statistics treats the parameter itself as uncertain and represents that uncertainty directly as a probability distribution that gets updated as data arrives.

  2. 02What does a prior distribution represent, and where does it come from?

    It represents your belief about a parameter's likely values before incorporating the current dataset, and it can come from historical data, domain expertise, a previous experiment, or, when you want to express genuine ignorance, a deliberately flat or weakly informative distribution.

  3. 03Why might a data scientist prefer a Bayesian approach for an early-stage A/B test with little data so far?

    Bayesian methods let you fold in a sensible prior (such as the pattern typically seen in past redesign tests) so that early, noisy results don't get over-interpreted, and they let you make a direct statement like \"a 72% chance variant B is better,\" which is often exactly the kind of answer a stakeholder is actually asking for.

Go deeper

Notes

Private to you

Esc
↑↓ navigate↵ open? shortcuts

Keyboard shortcuts

Press Esc to close

Anywhere

⌘K
Search and commands
/
Search
?
This sheet
Esc
Close

Reading a lesson

J
Next lesson
K
Previous lesson
B
Bookmark