Validation: Measuring Anything at All

Cross-validation works by hiding some data and asking whether the model generalises to it. The procedure is only meaningful if hiding the data actually hides the information in it.

With triple-barrier or any horizon-based labels, that fails. A sample at day $t$ carries a label determined by prices over $[t, t+h]$. A sample at day $t+1$ carries a label determined by $[t+1, t+h+1]$. The two windows overlap almost completely, so the two labels are near-duplicates of each other.

Shuffle those samples into folds and you place near-copies of the test labels into the training set. The model does not have to generalise; it can interpolate between neighbours that already contain the answer.

Key idea: The unit of independence in financial data is not the row. It is the time interval a label spans. Any split that does not respect those intervals is measuring memorisation and calling it generalisation.

Generate pure Gaussian noise, so no signal exists. Build trailing-window features, label by the sign of the forward return over a horizon $h$, and compare an ordinary shuffled 5-fold against a purged one. Sweep $h$.

The shape confirms the mechanism precisely. At $h = 1$ there is no overlap between labels and both methods report 50 percent, correctly. As the horizon grows the shuffled split climbs to 55.7 percent while the purged split stays flat at chance.

Key idea: The illusion is not a constant to be subtracted. It scales with how much your labels overlap, which means the longer your holding period, the more skill your validation will invent. Nothing about the data changed between the two lines; only the splitting rule did.

The fix for label overlap is direct: remove from the training set any sample whose label window overlaps the test set's time span.

$\text{drop } i \text{ from training if } [t_i,\ t_i + h_i] \cap [t_{\text{test start}},\ t_{\text{test end}}] \neq \emptyset$

Note that this uses each sample's own horizon $h_i$, which is why the labelling lesson insisted on recording the exit index. Under triple-barrier labelling every sample has a different span, because the barrier is hit at a different time, and purging by a single fixed horizon would be wrong in both directions: too aggressive for fast exits, insufficient for slow ones.

Definition: Purging removes training samples whose outcomes were determined during the test period. It is not a heuristic margin, it is the precise removal of samples that share price observations with the test set.

The cost is real. Purging discards training data, and with long horizons and many folds it can remove a substantial fraction of the dataset. That cost is the price of a number you can believe.

Purging handles labels that reach forward into the test period. A second leak runs the other way, and needs its own remedy.

Samples immediately after the test set have features built from trailing windows that extend back into it. A 20-day trailing volatility computed the day after the test block ends is largely a function of test-period prices. Training on it leaks the test period's data into the model through the features rather than through the labels.

The fix is an embargo: also drop a buffer of samples following the test set, sized to the longest trailing window in your feature set.

Gotcha: The embargo must be sized by the feature lookback, not by the label horizon. These are different numbers and are often confused. If your longest feature is a 60-day moving average and your labels span 5 days, purging removes 5 days ahead of the test block and the embargo must remove 60 days after it. Using the label horizon for both leaves the longer leak wide open.

The folds must be contiguous in time, not shuffled, so that each has a single well-defined span to purge against.

import numpy as np

def purged_kfold(n, k, label_span, embargo):
    """Yield (train_idx, test_idx). label_span[i] = exit index of sample i."""
    bounds = np.linspace(0, n, k + 1).astype(int)
    for i in range(k):
        lo, hi = bounds[i], bounds[i + 1]
        test = np.arange(lo, hi)

        keep = np.ones(n, dtype=bool)
        keep[test] = False
        # purge: any sample whose label window reaches into the test block
        keep[label_span >= lo] &= (np.arange(n)[label_span >= lo] >= hi)
        # embargo: features after the block still see it
        keep[hi:hi + embargo] = False

        yield np.where(keep)[0], test

The fold boundaries are the only place the time ordering matters; within the training set the samples can be presented in any order. And the test blocks must be contiguous because a scattered test set has no single span, so there is nothing coherent to purge against.

In practice: Run this on pure noise before you run it on real data. If it returns anything meaningfully above chance, the implementation has a bug, and finding it there is far cheaper than discovering it after a strategy is funded.

Purging fixes the train-test boundary. A related problem remains inside the training set: overlapping samples are partly redundant, so a dataset of 1,000 rows with 10-day labels contains far fewer than 1,000 independent observations.

The standard remedy is to weight each sample by its uniqueness. For each price observation, count how many labels span it; a sample's average uniqueness is the mean of the reciprocal of that count across the observations its own label spans.

$u_i = \frac{1}{|[t_i, t_i+h_i]|}\sum_{t \in [t_i,\, t_i+h_i]} \frac{1}{c_t}, \qquad c_t = \text{number of labels spanning } t$

A sample overlapping nine others gets roughly a tenth of the weight of one standing alone.

In practice: This matters most for anything that resamples, since bootstrapping from overlapping samples draws near-duplicates and produces trees that are far more correlated than the ensemble assumes. Weighting by uniqueness, or sequentially bootstrapping to favour non-overlapping draws, restores some of the diversity that makes an ensemble work at all.

Walk-forward testing, train on the past, test on the next block, roll forward, is the standard approach and it is honest about time ordering. It has a different weakness.

It produces exactly one trajectory: one sequence of decisions through one realisation of history. The Sharpe ratio you compute from it is a single draw, and you have no measure of how much of it was luck, because you have no other paths to compare against.

Combinatorial purged cross-validation, from the same source as the other methods here, holds out several blocks at a time in many combinations, purging and embargoing each. Recombining the out-of-sample predictions yields a large number of distinct backtest paths from the same data.

Key idea: The output stops being a number and becomes a distribution. "Sharpe 1.8" is not a result; "median 1.8 across paths, with 20 percent of paths below 1.0" is one, because it tells you how much of the headline figure was the strategy and how much was the particular ordering of history.

The corrections are known, and the catalogue's backtesting course covers them properly, including the deflated Sharpe ratio, which adjusts a reported Sharpe for the number of trials that produced it.

Gotcha: The trial count that matters is not the number in your final script. It includes every configuration you ever ran on this dataset, across every session, including the ones you abandoned. That number is almost never recorded, which is why it is almost always underestimated.

Even with everything in this course applied correctly, a purged cross-validation score supports a narrower claim than people usually make of it.

The right-hand column is not what this course fixes. It is the subject of the courses on backtesting, execution, capacity and risk, and reaching them with a clean dataset is the whole purpose of the four lessons here.

Key idea: Purged validation is a necessary condition, not a sufficient one. It removes the failure mode where you were never measuring anything, which is worth doing because that failure mode is both the most common and the only one that is completely invisible from the inside.

Everything in this course, as a sequence you can apply.

1. Sample on activity or events, not on the clock, and record each sample's timestamp.
2. Label with barriers scaled by entry volatility, and store each label's exit index. Nothing downstream works without it.
3. Difference fractionally to the smallest $d$ that passes a stationarity test, with a fixed-width window.
4. Build every feature causally, and fit every transform inside the training fold.
5. Split contiguously, purge by each sample's own span, and embargo by your longest feature lookback.
6. Weight samples by average uniqueness, especially under any bootstrap.
7. Report a distribution over paths, not a single number.
8. Record the total number of configurations tried, including abandoned ones, and deflate accordingly.

Try it: Before applying any of this to real data, run the whole pipeline on pure synthetic noise. A correct implementation returns chance. Any strategy that survives that test has passed the only check in this course that cannot be argued with, and it costs an afternoon.