Estimation Error: Why Optimized Portfolios Disappoint

The previous lesson in this course computed mean-variance portfolios from inputs that were known exactly. Real inputs are estimated from a finite sample of history, and the optimizer cannot tell a precise estimate from a noisy one.

Richard Michaud named the consequence in "The Markowitz Optimization Enigma: Is 'Optimized' Optimal?" (Financial Analysts Journal, 1989): mean-variance optimizers behave as estimation-error maximizers. They overweight precisely the assets whose estimates look best, meaning high estimated returns, low estimated variances and negative estimated correlations, and those are disproportionately the assets whose estimates are most wrong in the flattering direction.

Key idea: An optimizer is a machine for acting on differences. Feed it inputs where the true differences are small and the estimation noise is large, and most of what it acts on is noise. The more aggressively it optimizes, the more confidently it bets on the noise.

The standard error of a sample mean shrinks only with the square root of the sample length. For annual returns with volatility $\sigma$ observed over $T$ years:

For an asset with 20% volatility, 10 years of data gives a standard error of $20\% / \sqrt{10} \approx 6.3\%$ on its expected annual return, larger than a typical equity risk premium. Pinning the mean down to within 1 percentage point would take $(20 / 1)^2 = 400$ years of stable history.

Chopra and Ziemba (Journal of Portfolio Management, 1993) measured the damage directly: for an investor with moderate risk tolerance, errors in expected returns cost roughly ten times as much as errors in variances, and roughly twenty times as much as errors in covariances.

Simulation shows the effect without any argument about real markets. Build a market whose true parameters are known: 10 assets driven by one common factor (16% volatility, 6% annual premium), betas from 0.6 to 1.4, idiosyncratic volatility from 20% to 35%, and fixed small true alphas, so the true optimal portfolio genuinely beats equal weighting. Its true Sharpe ratio is 0.382, against 0.339 for 1/N.

Now pretend not to know the parameters. Draw $T$ months of returns, estimate the means and covariance from them, compute the tangency portfolio, and score its weights with the true parameters. Repeat 1,000 times per sample length.

With 5 years of data, the estimated portfolio's true Sharpe ratio averages 0.105, less than a third of what 1/N delivers without estimating anything.

In the same simulation, count how often the estimated portfolio beats 1/N on true Sharpe ratio.

The simulation is deliberately generous to the optimizer: returns are normal, parameters never change, and the model is exactly right. Real markets add regime changes and fat tails, which make estimation harder, not easier.

DeMiguel, Garlappi and Uppal's "Optimal Versus Naive Diversification: How Inefficient Is the 1/N Portfolio Strategy?" (Review of Financial Studies, 2009) ran the comparison on real data. They evaluated 14 models, the sample mean-variance portfolio plus extensions designed specifically to reduce estimation error, across seven empirical datasets, and reported that none is consistently better than the 1/N rule in terms of Sharpe ratio, certainty-equivalent return, or turnover.

Their conclusion is that, out of sample, the gain from optimal diversification is more than offset by estimation error.

Gotcha: The result does not say optimization is useless or that 1/N is optimal. It says that with realistic amounts of data, the estimated optimum is further from the true optimum than 1/N is. 1/N wins by estimating nothing, so it cannot be hurt by estimation error. That makes it the right benchmark for any optimized strategy: an optimizer that cannot beat 1/N out of sample has not demonstrated anything.

Dropping expected returns and targeting minimum variance removes the weakest input, but the covariance matrix has its own problem: size. With 50 assets it has 1,275 distinct entries, and five years of monthly data supplies only 60 observations per asset. The sample estimate is then badly conditioned, and inverting it, which every optimizer does, amplifies its errors.

The same one-factor simulation, now with 50 assets and 60 months of data, scoring minimum-variance portfolios by their true volatility over 500 draws:

The portfolio built to have the least risk, using the sample covariance, ends up with 28.1% true volatility, far worse than 1/N at 16.5%. The same optimizer fed a shrunk covariance estimate reaches 13.8%, better than 1/N and much closer to the true minimum of 11.2%.

Ledoit and Wolf's "Honey, I Shrunk the Sample Covariance Matrix" (Journal of Portfolio Management, 2004) argues that the sample covariance matrix should not be used for portfolio optimization, because it contains exactly the kind of error that perturbs an optimizer most. Their remedy is a weighted average of the sample matrix $S$ and a simple, stable target $F$:

The target has few parameters, for example a constant correlation or a single-factor model, so it is biased but precise. The sample matrix is unbiased but noisy. The shrinkage intensity $\delta$ is estimated from the data to minimise expected error, so no tuning is required. The effect is to pull extreme coefficients toward central values, which is where the optimizer was being misled.

from sklearn.covariance import LedoitWolf
import numpy as np

lw = LedoitWolf().fit(returns)        # returns: T x n array of asset returns
Sigma_hat = lw.covariance_
print("shrinkage intensity:", round(lw.shrinkage_, 3))

w = np.linalg.solve(Sigma_hat, np.ones(Sigma_hat.shape[0]))
w_minvar = w / w.sum()

scikit-learn's implementation shrinks toward a scaled identity matrix, a simpler target than the ones in the 2004 paper.

Practitioners long imposed no-short-selling rules and position limits for business reasons, and noticed that constrained portfolios often performed better out of sample than the unconstrained optimum, even though the constraints are "wrong" in theory.

Jagannathan and Ma (Journal of Finance, 2003) explained why in "Risk Reduction in Large Portfolios: Why Imposing the Wrong Constraints Helps". A no-short-sale constraint binds precisely on the assets the optimizer wants to short, which are typically those whose estimated covariances are too high. Imposing it has the same effect as shrinking those extreme estimates. They found that with no-short-sale constraints in place, the plain sample covariance matrix performed about as well as factor models, shrinkage estimators and daily-data estimates.

Each remedy attacks a different stage of the pipeline from the previous lesson.

The estimation-error literature does not leave you with nothing. It leaves you with an ordering of how much to trust each step.

1. Trust expected-return estimates least. Historical averages over any realistic window are mostly noise. If a strategy's weights depend heavily on them, its backtest is likely to overstate what it will deliver.
2. Prefer portfolios that need fewer estimates. Minimum variance needs no means; equal weighting needs nothing at all.
3. Never feed a raw sample covariance matrix to an optimizer when the number of assets is not small relative to the number of observations. Shrink it.
4. Treat constraints as regularisation, not only as compliance rules.
5. Benchmark against 1/N out of sample. Any claimed improvement should survive that comparison after costs.

In practice: This is the portfolio version of overfitting. An unconstrained optimizer on sample inputs is a flexible model fitted to a short, noisy dataset, and it fails for the same reason. The next lesson turns to the portfolio rules that use less estimation, and what they cost.