Mean-Variance Optimization: The Mathematics of Diversification

Before 1952, investment advice mostly meant finding the best individual securities. Harry Markowitz's paper "Portfolio Selection" (Journal of Finance, 1952) changed the unit of analysis: judge a holding by what it does to the whole portfolio, and judge the portfolio by two numbers together, its expected return and its variance.

The insight that follows is arithmetic, not opinion. The risk of a portfolio depends on how its holdings move together, so two risky assets can combine into something less risky than either one. A security that looks mediocre alone can be valuable if it tends to rise when the rest of the portfolio falls.

Key idea: Expected returns average linearly: a 50/50 mix of a 5% asset and a 9% asset expects 7%. Risk does not. Unless the assets move in perfect lockstep, the mix is less volatile than the weighted average of their volatilities. That gap is the only free lunch the framework offers, and the rest of this course is about how much of it survives contact with real data.

Write the portfolio as a weight vector $w$ over $n$ assets, with expected returns $\mu$ and covariance matrix $\Sigma$. Then:

For two assets the variance expands into three terms:

The last term is where diversification lives. The correlation $\rho_{12}$ runs from $-1$ to $+1$. At $+1$ the formula collapses to $(w_1\sigma_1 + w_2\sigma_2)^2$, a plain weighted average of volatilities, and there is no benefit at all. Anything below $+1$ shrinks the cross term and pulls portfolio volatility below the average.

Definition: The covariance matrix $\Sigma$ holds every asset's variance on its diagonal and every pair's covariance, $\rho_{ij}\sigma_i\sigma_j$, off it. With $n$ assets it has $n(n+1)/2$ distinct entries: 55 for 10 assets, 1,275 for 50. Every one of them is an estimate, a fact the next lesson makes central.

Take two assets that each have 20% annual volatility and hold them 50/50. The only thing that changes below is the correlation between them.

At a correlation of 1 the mix is exactly as risky as each asset, 20%. At 0.5 it falls to 17.3%, at zero to 14.1%, and at minus 0.5 to 10%. At minus 1 the two assets cancel perfectly and the portfolio has no volatility at all.

Gotcha: Correlations are not constant. Longin and Solnik (Journal of Finance, 2001) found that correlations between international equity markets rise in bear markets but not in bull markets, so diversification can weaken exactly when it was supposed to help. The formula is right; the input is the fragile part.

Now hold $N$ assets in equal weights, each with 20% volatility and a pairwise correlation of 0.3. The portfolio variance simplifies to

Most of the benefit arrives early: going from 1 to 10 assets cuts volatility from 20% to 12.2%, while going from 10 to 100 cuts it only another 1.1 points.

Markowitz posed portfolio choice as an optimisation. For each target return, find the weights with the smallest variance:

Solving this for every target $\mu^{*}$ traces the efficient frontier: the set of portfolios for which no other portfolio offers more expected return at the same risk. Two points on it get names.

• Global minimum-variance portfolio: the lowest-risk point on the frontier. Without further constraints it has a closed form, $w_{\text{mv}} = \Sigma^{-1}\mathbf{1} / (\mathbf{1}^\top \Sigma^{-1}\mathbf{1})$, and notably it uses no expected returns at all.
• Tangency portfolio: the frontier point with the highest ratio of expected excess return to volatility, $w_{\text{tan}} = \Sigma^{-1}\mu / (\mathbf{1}^\top \Sigma^{-1}\mu)$, where $\mu$ is measured in excess of the risk-free rate.

Definition: The Sharpe ratio, introduced by William Sharpe in 1966, is expected excess return divided by volatility. The tangency portfolio is the fully invested portfolio that maximises it.

Here is a small, fully specified problem you can reproduce. Expected returns are annual and in excess of the risk-free rate.

Notice what these inputs imply before any optimisation. Asset A has the lowest return but also the lowest volatility and the weakest correlations with the others, so it is the natural diversifier. Asset C pays the most but is the riskiest and is fairly correlated with B. Every asset here offers between 0.45 and 0.5 units of excess return per unit of its own volatility, so on its own no asset stands out. The optimiser's job is to exploit how they combine.

Try it: Before reading on, guess which asset gets the largest weight in the minimum-variance portfolio, and whether the equally weighted portfolio is far from the best Sharpe ratio.

Solving the closed forms from the previous steps on these inputs gives:

Three things stand out. The minimum-variance portfolio is less volatile, at 9.25%, than the least volatile asset, A at 10%: diversification doing its job. The optimiser leans heavily on A, because low volatility combined with low correlation is exactly what the variance formula rewards.

And the headline: with the inputs known exactly, the best achievable Sharpe ratio is 0.642, while simply splitting the money three ways gives 0.607. The whole value of optimising here is a gain of 0.035 in Sharpe ratio. Hold on to that number: the next lesson asks what happens to it once the inputs are estimated rather than known.

Both named portfolios are one linear solve each. Solving is preferable to forming $\Sigma^{-1}$ explicitly, which is slower and less numerically stable.

import numpy as np

mu  = np.array([0.05, 0.07, 0.09])            # expected excess returns
vol = np.array([0.10, 0.15, 0.20])
corr = np.array([[1.0, 0.3, 0.2],
                 [0.3, 1.0, 0.5],
                 [0.2, 0.5, 1.0]])
Sigma = np.outer(vol, vol) * corr              # covariance matrix

def normalise(x):
    return x / x.sum()

w_minvar  = normalise(np.linalg.solve(Sigma, np.ones(3)))
w_tangent = normalise(np.linalg.solve(Sigma, mu))

for name, w in [("min variance", w_minvar), ("tangency", w_tangent)]:
    ret, risk = w @ mu, np.sqrt(w @ Sigma @ w)
    print(f"{name:13s} {np.round(w, 3)}  ret {ret:.2%}  vol {risk:.2%}  SR {ret/risk:.3f}")
# min variance  [0.745 0.185 0.07 ]  ret 5.65%  vol 9.25%  SR 0.611
# tangency      [0.578 0.223 0.2  ]  ret 6.24%  vol 9.73%  SR 0.642

Nothing here restricts weights to be positive. With other inputs these formulas happily return large short positions, which is the first sign of the fragility the next lesson measures.

The optimiser is exact. Everything uncertain enters through its three inputs.

Mean-variance optimisation is a clean answer to a precisely stated question. The question carries assumptions worth listing before using the answer:

1. The inputs are known. The formulas treat $\mu$ and $\Sigma$ as facts. In practice both are estimated from limited, noisy history.
2. Variance is the right measure of risk. It penalises upside and downside surprises equally and says nothing about fat tails; the trading risk course on AnyLearn covers what variance hides.
3. One period. The portfolio is chosen once for a single horizon, with no rebalancing and no path.
4. No frictions. Trading is free, any position size is available, and shorting costs nothing.

Key idea: The optimiser does not know which of its inputs are reliable. It treats a noisy estimate of an expected return with the same confidence as a precise one, and it will build large positions on small differences between them. Whether that matters in practice, and by how much, is an empirical question with a well-documented answer, and it is the subject of the next lesson.