Backtesting as Evidence

The claim, stated the way it is meant

The case for backtesting is a good one and it deserves stating properly before anything is done to it. A rule is a set of instructions. History is a record of what prices did. You can therefore run the instructions over the record and find out what the rule would have made, which is an ordinary engineering test and not a superstition. If it lost, you have learned that cheaply. If it won, you have at least established that the idea is not obviously empty. Nobody sensible claims a backtest predicts the future. The claim is narrower and it sounds unobjectionable: a backtest tells you whether the rule worked.

Here is what it actually tells you, on data this course has already printed. Take the sixty closes and one rule family: go long when a fast moving average of the closes crosses above a slow one, go flat when it crosses back below. Fast lengths from 2 to 12, slow lengths from 3 to 30, slow always longer than fast. That is 253 configurations, which is a small search by any standard, and every one of them is a rule a reasonable person might have proposed.

The execution convention matters more than it looks, so it goes on the page. A crossover is visible only once a bar has closed, so the signal is read at the close of bar i and the trade is done at the close of bar i plus one. Executing on the signal bar itself is the look-ahead that makes half the backtests on the internet work, and it is worth about a third of the apparent edge here. Costs are a penny of spread, five basis points of slippage a side on a price near 103, and half a cent a share of commission, which comes to 0.1230 a share for a round trip. Any position still open on the last bar is closed there.

Run all 253 and the best is a 2-bar fast average against a 5-bar slow one. It takes 7 trades and makes 10.70 a share gross and 9.84 after costs, against 5.68 for buying at the first close and selling at the last. Its win rate is 6 of 7, its average trade is 0.910 of one bar’s standard deviation, and the t-statistic on its own trade record is 3.65. Every one of those numbers is the kind that gets a strategy funded.

What the same search finds in a market with nothing in it

The test is cheap. Take the 59 bar-to-bar moves that make up this series and draw 59 of them at random with replacement, then lay them end to end from the same starting price. The result has the same drift and the same volatility as the original, built from literally the same moves, and no time structure whatsoever: nothing in it can be exploited by any rule that depends on the order of anything, because the order was destroyed. Do that 4,000 times from seed 20260903 and run the identical 253-cell search on each one.

Measure	Value
Best of 253, gross	10.70
Best of 253, net of costs	9.84
Buy and hold, net of costs	5.68
Edge over holding	4.16
Structureless series whose best cell beats holding	0.772
Structureless series that beat the real edge	0.525

On 77.2 per cent of series with no structure in them, the best of 253 rules beats buying and holding. On 52.5 per cent it beats holding by at least the 4.16 the real search found. The measured result is not merely unimpressive against that distribution; it sits below its median. Whatever the search found on the real closes, a search of the same size finds more than that, more than half the time, in data that contains nothing to find.

Notice what this does not say. It does not say the rule is bad, and it does not say the series has no structure. It says the number 9.84 was produced by a procedure that produces numbers like 9.84 anyway, so the number cannot be used as evidence about the market. That is a statement about the measurement, and it is the reason almost nobody you are trading against has ever checked their own: the check costs an evening and a resampling loop, and it can only take a result away.

Two columns go on the sheet this module started in lesson 62, beside the sentence. What the backtest returned, which here is 4.16 over the benchmark. And what the same search returned on data with nothing in it, which here is 4.16 or better on 52.5 per cent of tries. A row whose second column is not comfortably larger than its third has not earned anything, and the pair is worth more than either figure alone, because neither one means much without the other.

The two subtractions that come before it

Costs first, because they are the only part of this that is certain. A round trip costs 0.1230 a share here, and one bar’s standard deviation on this series is 1.5443, so a round trip is 0.0796 of an R. That sounds small until it is multiplied. The winning rule takes 7 trades and therefore pays 0.557R in costs; holding takes one and pays 0.0796R. The frequent rule starts half an R behind before either of them is right about anything, and the same cost is 0.0398R against a two-sigma stop and 0.1593R against a half-sigma one, because the cost is fixed in cents and R is not.

The benchmark second, and it is the subtraction that is almost always missing. The best cell made 9.84. Holding made 5.68. The rule’s contribution is not 9.84, it is 4.16, because the series rose and any rule that spends time long collects that rise whether or not it has understood anything. A backtest that reports the strategy’s own equity curve and never the benchmark’s has hidden the larger of the two numbers.

The row the split-sample test cannot see

The standard defence against all of this is out-of-sample testing, and it is genuinely worth doing, so do it here. Optimise on the first thirty closes only. The best cell over that window is the 2-and-5 pair again, making 5.53 net on 3 trades, over a stretch in which buying and holding lost 0.82. That looks like a rule finding something in a market that offered nothing. Freeze the parameters, touch nothing, and run them on the last thirty closes.

Window	Best rule, net	Buy and hold, net	Trades
Bars 1 to 30, optimised here	5.53	-0.82	3
Bars 31 to 60, parameters frozen	4.29	6.88	5

It passes. Performance fell from 5.53 to 4.29, a degradation of 22 per cent, and every checklist in the field treats anything under 30 per cent as a pass. The rule was not curve-fit in the sense the checklist means: it really did keep working on data it had never seen.

And over those same unseen bars, doing nothing made 6.88. The validated rule lost to the benchmark by 2.59 on the very window that was supposed to vindicate it, and the out-of-sample test could not report that, because there is no benchmark anywhere in it. The test asks whether a result repeats. It never asks whether the result was worth having.

Nothing below changes the data or the rule. The 2-and-5 crossover is the same rule at every stage, run over the same sixty closes. Only what it is measured against changes, and each stage is the one most backtests stop at.

Stop at the first stage and the rule made 10.70 a share on 7 trades with 6 winners. Stop at the second and costs take it to 9.84, which is an 8 per cent haircut and feels like diligence, because a page that subtracts costs looks careful. Stop at the third and the benchmark takes it to 4.16, which more than halves it again, and this is already further than most published backtests go.

The fourth stage is the one that changes the verdict rather than the size. Resample the moves, rerun the search, and 52.5 per cent of structureless series produce a best cell that beats holding by 4.16 or more. The rule’s own t-statistic of 3.65 is not wrong; it is answering a different question. It measures how surely this cell’s 7 trades differ from zero, and it has no way of knowing that 252 other cells were auditioned and discarded. The t-statistic is computed inside the winner. The search happened outside it.

A backtest measures the search you ran, not the market you ran it on.

What survives is more than it sounds. A backtest still kills rules, cheaply and permanently: a rule that loses in-sample, before costs, is dead, and finding that out in an afternoon is most of what the technique is for. What it cannot do alone is promote one. Promotion needs the benchmark, the costs, and the resampled null, and the null is one loop of code.

So run all four stages on your own rule tonight, in that order, and stop at the first one that takes the result away.

What this does not settle

That backtesting does not work. The page tested one rule family over one sixty-bar series with a 253-cell search, and a real programme runs thousands of bars across many instruments, where the sampling noise this page is exploiting is very much smaller. The argument about search inflation does not depend on the size of either, but the specific 52.5 per cent does, and on a longer series the same search would manufacture a proportionally smaller edge. Nothing here shows the technique is useless; it shows what a result has to survive before it means anything.

That the resampling test is the right null. Drawing moves with replacement destroys all time structure at once, which makes it a test of whether the rule exploits ordering, and nothing else. It also destroys volatility clustering, which is real and which a crossover rule can genuinely trade, so the null is in that respect too easy to beat and the true fraction is likely higher than 52.5 per cent rather than lower. A block bootstrap would preserve some of that structure and is the honest next step; this page does not run one.

That the costs are yours. A penny of spread and five basis points of slippage describe a liquid instrument at small size. On something wider they are badly optimistic, and on a deep large-cap at a hundred shares they are pessimistic. Every figure on this page moves with that assumption, and the 7-trade rule moves about seven times as fast as the benchmark does, which is the direction that flatters holding.

That buy and hold is the right benchmark. It is the right one here because the rule is long-only on a rising series, and it is the wrong one for a rule that shorts, or for a stretch that fell. On a falling series every long-only rule looks poor against holding for a reason that has nothing to do with the rule, and picking a benchmark that flatters you is the same error as picking a window that does. The benchmark has to be chosen before the test, and this page chose it after, because the series was already written.

That the split-sample result generalises. Thirty bars in each half is far too few, the in-sample window contains 3 trades and the out-of-sample 5, and with samples that small the 22 per cent degradation figure carries almost no information. The finding it supports is not that out-of-sample tests usually pass bad rules; it is the narrower and firmer one that they can pass a rule that loses to the benchmark, because they contain no benchmark. That defect is structural and does not depend on the sample size at all.

And the concession that costs most: this page tells you to subtract what the search would have produced, and gives no way to do that for the search you actually ran. The 253 cells are countable because they were coded. The rules you looked at on a chart and abandoned, the parameter you nudged twice and put back, the instrument you tried first and dropped because it was ugly — those are configurations too, they are unrecorded, and they are almost certainly more numerous than 253. The correction on this page is computable only for the part of the search that was honest enough to be written down. Lesson 64 takes that problem head on and finds that the inflation is predictable in advance from the count alone: for a search of 15,000 configurations over 156 trades, the best cell shows 0.317R of edge before the market has done anything at all.