Bayesian Inference

One line of algebra, endlessly applicable. From the definition of conditional probability:

$P(A \cap B) = P(A \mid B)\, P(B) = P(B \mid A)\, P(A).$

Rearrange the right two expressions:

$P(A \mid B) = \frac{P(B \mid A)\, P(A)}{P(B)}.$

In statistical inference we replace $A$ with a parameter $\theta$ and $B$ with observed data $D$:

$\boxed{P(\theta \mid D) = \frac{P(D \mid \theta)\, P(\theta)}{P(D)}.}$

Nothing more than the chain rule. The interpretation is where all the depth lives.

Each term in Bayes' theorem has a job:

The evidence $P(D) = \int P(D \mid \theta)\, P(\theta)\, d\theta$ is usually the hardest to compute — but for inference you often just need the shape of the posterior, so you write:

$P(\theta \mid D) \propto P(D \mid \theta)\, P(\theta).$

Posterior $\propto$ likelihood $\times$ prior. The evidence is just the normalizer that makes everything sum to 1.

A prior $P(\theta)$ is conjugate to a likelihood $P(D \mid \theta)$ when the posterior has the same functional form as the prior — only the parameters change. Conjugacy makes inference analytic.

The canonical pair: Beta prior with Binomial likelihood.

If $\theta \sim \text{Beta}(\alpha, \beta)$ and $D$ consists of $k$ successes in $n$ trials: $P(\theta \mid D) = \text{Beta}(\alpha + k,\; \beta + n - k).$

Intuition: $\alpha$ and $\beta$ act like pseudo-counts of prior successes and failures. Observing real successes and failures just adds to those counts. No integration required.

You suspect a coin is fair. You encode this as $\theta \sim \text{Beta}(10, 10)$ — a prior centered at 0.5 with moderate certainty. You flip the coin 30 times and see 22 heads.

Posterior: $\text{Beta}(10 + 22,\; 10 + 8) = \text{Beta}(32, 18)$.

Posterior mean: $\mathbb{E}[\theta \mid D] = \frac{32}{32 + 18} = \frac{32}{50} = 0.64$.

Posterior mode (MAP estimate): $\hat{\theta}_{\text{MAP}} = \frac{32 - 1}{50 - 2} = \frac{31}{48} \approx 0.646$.

MLE (no prior): $\hat{\theta}_{\text{MLE}} = 22/30 = 0.733$. The prior pulled the estimate toward 0.5.

from scipy import stats
import numpy as np

alpha_prior, beta_prior = 10, 10
k, n = 22, 30
posterior = stats.beta(alpha_prior + k, beta_prior + n - k)
print(f"Posterior mean: {posterior.mean():.3f}")  # 0.640
print(f"95% credible interval: {posterior.interval(0.95)}")
# (0.499, 0.769)

Three point estimates from the posterior, with different loss functions:

$\hat{\theta}_{\text{MLE}} = \arg\max_\theta P(D \mid \theta)$ $\hat{\theta}_{\text{MAP}} = \arg\max_\theta P(\theta \mid D) = \arg\max_\theta \left[\log P(D \mid \theta) + \log P(\theta)\right]$ $\hat{\theta}_{\text{PM}} = \mathbb{E}[\theta \mid D] = \int \theta\, P(\theta \mid D)\, d\theta$

MAP minimizes 0-1 loss (mode); posterior mean minimizes squared-error loss. MLE ignores the prior entirely.

In regularized regression: L2 regularization = MAP with a Gaussian prior; L1 regularization = MAP with a Laplace prior. What looks like a regularization trick in frequentist ML is Bayesian MAP inference in disguise.

Full Bayesian inference uses the entire posterior — not just a point — propagating uncertainty through predictions.

Once you have a posterior $P(\theta \mid D)$, predicting a new observation $\tilde{x}$ means marginalizing over $\theta$:

$P(\tilde{x} \mid D) = \int P(\tilde{x} \mid \theta)\, P(\theta \mid D)\, d\theta.$

This is the posterior predictive distribution — it accounts for both the randomness in $\tilde{x}$ given $\theta$ and the uncertainty in $\theta$ itself. It is always wider than a prediction that plugs in a single $\hat{\theta}$.

For the Beta-Binomial example: the posterior predictive for the next flip is $P(\tilde{x} = 1 \mid D) = \mathbb{E}[\theta \mid D] = \frac{\alpha + k}{\alpha + \beta + n} = \frac{32}{50} = 0.64.$

In Gaussian processes and Bayesian neural networks, entire posterior predictive distributions over function values are maintained — not just means.

A test for a disease is 99% sensitive (true-positive rate) and 99% specific (true-negative rate). The disease prevalence is 0.1%. You test positive. What's the probability you actually have the disease?

Let $D$ = disease, $T^+$ = positive test.

• $P(T^+ \mid D) = 0.99$ (sensitivity)
• $P(T^- \mid \neg D) = 0.99$, so $P(T^+ \mid \neg D) = 0.01$ (false-positive rate)
• $P(D) = 0.001$ (base rate)

By Bayes: $P(D \mid T^+) = \frac{0.99 \times 0.001}{0.99 \times 0.001 + 0.01 \times 0.999} \approx \frac{0.00099}{0.01098} \approx 9\%.$

Only 9% — not 99%. The rare base rate dominates. Most positive tests are false positives. Ignoring the prior $P(D)$ is the base-rate fallacy, and it affects medical screening, spam filters, and fraud detection equally.

The philosophical split is about what $\theta$ means:

• Frequentist: $\theta$ is a fixed (unknown) constant. Probability refers only to long-run frequencies of repeatable experiments. A 95% confidence interval does not mean "$\theta$ is in this interval with 95% probability" — $\theta$ is fixed, not random.
• Bayesian: $\theta$ is uncertain; we model uncertainty with a probability distribution. A 95% credible interval genuinely means "we assign 95% probability to $\theta$ lying here, given our prior and the data."

In practice, the two often agree asymptotically (Bernstein–von Mises theorem: with enough data, the posterior concentrates around the MLE regardless of the prior). The differences are sharpest when:

• Data is scarce.
• The prior captures real domain knowledge.
• You need full uncertainty propagation (not just point estimates).

Modern ML uses both: SGD is frequentist; Bayesian optimization, Gaussian processes, and variational inference are Bayesian.

One of the cleanest features of the Bayesian framework: today's posterior is tomorrow's prior. You don't need all data at once.

After observing $D_1$: $P(\theta \mid D_1) \propto P(D_1 \mid \theta)\, P(\theta)$. After observing $D_2$: $P(\theta \mid D_1, D_2) \propto P(D_2 \mid \theta)\, P(\theta \mid D_1)$.

This is identical to computing the posterior on all data jointly (assuming conditional independence of $D_1$ and $D_2$ given $\theta$). The order doesn't matter.

from scipy import stats

# Beta-Binomial sequential update
a, b = 1, 1  # flat prior
for heads, flips in [(3, 5), (7, 10), (2, 5)]:
    a += heads
    b += flips - heads
    posterior = stats.beta(a, b)
    print(f"After {heads}/{flips}: mean={posterior.mean():.3f}")
# After 3/5: mean=0.500
# After 10/15: mean=0.524
# After 12/20: mean=0.520

This sequential property underpins online learning, Kalman filters, and streaming ML systems.