Random Variables and Distributions

Every probability model starts with a sample space $\Omega$ — the set of all outcomes. An event is a subset $A \subseteq \Omega$. A probability measure $P$ assigns a number in $[0,1]$ to each event, with $P(\Omega) = 1$ and countable additivity: for disjoint $A_1, A_2, \ldots$,

$P\!\left(\bigcup_{i=1}^{\infty} A_i\right) = \sum_{i=1}^{\infty} P(A_i).$

From these three axioms (due to Kolmogorov) everything else follows. Most people's intuition breaks because $\Omega$ can be uncountably infinite — e.g., $\Omega = \mathbb{R}$ when modelling a real-valued measurement. That's where random variables come in: they give us a tractable handle on $\Omega$.

A random variable $X: \Omega \to \mathbb{R}$ maps outcomes to numbers. The distribution of $X$ is the induced probability law over $\mathbb{R}$.

• Discrete: $X$ takes values in a countable set (often $\{0, 1, 2, \ldots\}$). Its distribution is captured by the probability mass function $p(x) = P(X = x)$, which must satisfy $p(x) \ge 0$ and $\sum_x p(x) = 1$.
• Continuous: $X$ takes values in an interval or $\mathbb{R}$. $P(X = x) = 0$ for every single $x$; instead the distribution is characterised by a probability density function $f(x) \ge 0$ with $\int_{-\infty}^{\infty} f(x)\,dx = 1$, and $P(a \le X \le b) = \int_a^b f(x)\,dx$.

The key rule: PMF gives the probability of a point; PDF gives the probability of an interval.

The cumulative distribution function (CDF) unifies both cases:

$F(x) = P(X \le x).$

Properties (true for any $X$):

• $F$ is non-decreasing.
• $\lim_{x \to -\infty} F(x) = 0$, $\lim_{x \to +\infty} F(x) = 1$.
• $F$ is right-continuous.

For discrete $X$: $F(x) = \sum_{k \le x} p(k)$ (a staircase). For continuous $X$: $F(x) = \int_{-\infty}^{x} f(t)\,dt$, so $f(x) = F'(x)$.

The CDF is exactly what numpy and scipy.stats return via .cdf(). Knowing it lets you compute any probability as $P(a < X \le b) = F(b) - F(a)$ — no integration required.

from scipy import stats
X = stats.norm(loc=0, scale=1)  # standard Gaussian
print(X.cdf(1.96))   # 0.975  (classic 95% CI endpoint)
print(X.ppf(0.025))  # -1.960 (inverse CDF = quantile)

Bernoulli$(p)$ — a single binary trial. $P(X=1)=p$, $P(X=0)=1-p$. Mean $p$, variance $p(1-p)$. Every classifier output, coin flip, or clicked/not-clicked event is Bernoulli.

Binomial$(n,p)$ — count of successes in $n$ i.i.d. Bernoulli$(p)$ trials: $P(X=k) = \binom{n}{k}p^k(1-p)^{n-k}, \quad k=0,\ldots,n.$ Mean $np$, variance $np(1-p)$. Use when summing independent binary outcomes.

Categorical$(\boldsymbol{\pi})$ — generalises Bernoulli to $K$ classes: $P(X=k)=\pi_k$ with $\sum_k \pi_k=1$. This is the output distribution of a softmax classifier. Its multivariate cousin is Multinomial$(n,\boldsymbol{\pi})$, which counts occurrences of each class in $n$ draws.

Gaussian (Normal) $\mathcal{N}(\mu, \sigma^2)$ — the workhorse. PDF: $f(x) = \frac{1}{\sigma\sqrt{2\pi}}\exp\!\left(-\frac{(x-\mu)^2}{2\sigma^2}\right).$ Mean $\mu$, variance $\sigma^2$. Appears everywhere via the CLT; conjugate to itself.

Uniform $\mathrm{Unif}(a,b)$ — constant density $\frac{1}{b-a}$ on $[a,b]$. Mean $\frac{a+b}{2}$, variance $\frac{(b-a)^2}{12}$. The default "I know nothing" prior on a bounded interval; used in random initialization.

Exponential$(\lambda)$ — models the waiting time between events in a Poisson process: $f(x) = \lambda e^{-\lambda x}$, $x \ge 0$. Mean $1/\lambda$, variance $1/\lambda^2$. Memoryless: $P(X > s+t \mid X > s) = P(X > t)$. Models inter-arrival times, dropout survival, and hardware failure rates.

Poisson$(\lambda)$ counts events in a fixed interval when they occur independently at rate $\lambda$:

$P(X=k) = \frac{\lambda^k e^{-\lambda}}{k!}, \quad k = 0, 1, 2, \ldots$

Mean $\lambda$, variance $\lambda$ (they're equal — a signature). As $n \to \infty$, $p \to 0$ with $np = \lambda$ fixed, Binomial$(n,p) \to$ Poisson$(\lambda)$. Classic applications: number of word occurrences in a document (bag-of-words models), events per time bucket in streaming data, rare disease counts.

import numpy as np
from scipy import stats

lam = 3.5
x = np.arange(0, 15)
pmf = stats.poisson.pmf(x, mu=lam)
print(f"P(X=0) = {pmf[0]:.4f}")   # 0.0302
print(f"P(X=3) = {pmf[3]:.4f}")   # 0.2158
print(f"mean={stats.poisson.mean(mu=lam)}, var={stats.poisson.var(mu=lam)}")
# mean=3.5, var=3.5

Decision tree for picking the right distribution given the nature of the random variable:

A quick-reference table of the distributions above. $B(a,b)$ is the beta function; all others are standard notation.

Notice that Poisson has mean $=$ variance — if empirical variance $\gg$ mean, the data is overdispersed and you need a Negative Binomial instead.

Two random variables $X, Y$ are independent if their joint distribution factors:

$p(x, y) = p_X(x)\, p_Y(y) \quad (\text{discrete})$ $f(x, y) = f_X(x)\, f_Y(y) \quad (\text{continuous})$

Independence implies $\text{Cov}(X,Y)=0$ but not vice versa (uncorrelated $\neq$ independent, except for Gaussians). The marginal of $X$ is obtained by summing/integrating out $Y$: $p_X(x) = \sum_y p(x,y)$.

When $X_1, \ldots, X_n$ are i.i.d. (independent, identically distributed), many results become tractable — the CLT, the law of large numbers, maximum likelihood factorization. "i.i.d." is the backbone assumption in most supervised learning setups.

Every scipy.stats distribution follows the same API: construct with parameters, then call .pmf/.pdf, .cdf, .ppf (quantile), .rvs (random samples).

import numpy as np
from scipy import stats

# Gaussian
g = stats.norm(loc=2.0, scale=1.5)
print(g.pdf(2.0))         # peak: 0.2659
print(g.cdf(3.5))         # P(X <= 3.5): 0.8413
print(g.ppf(0.95))        # 95th percentile: 4.468
samples = g.rvs(size=10_000, random_state=0)

# Binomial
b = stats.binom(n=20, p=0.3)
print(b.pmf(6))            # P(X=6): 0.1916
print(b.mean(), b.var())   # 6.0  4.2

# Exponential (scipy uses scale=1/lambda)
e = stats.expon(scale=1/2.0)
print(e.mean())            # 0.5 = 1/lambda

Always double-check parameter conventions — scipy uses scale = 1/lambda for the Exponential, not $\lambda$ directly.