The Price of Looking

What the count alone already tells you

Suppose every rule you are about to test is worthless. Not badly designed, not unlucky: worthless, in the exact sense that its true edge is zero. Each one still returns a number, because a finite record of trades is a sample and samples wobble. Test one and you get a wobble. Test many and you keep the largest wobble you found, because keeping the largest is what a search is. The largest of many noisy zeros is not zero, and its size is set by how many you drew rather than by what you drew them from.

That last clause is the whole lesson and it is worth being blunt about. The amount of edge a search manufactures does not depend on the instrument, the timeframe, the year, or whether the market trended. It depends on two counts: how many configurations you tried, and how many trades each one was judged over. Both are known before you run anything.

Write the t-statistic of a configuration as the usual thing: its average trade divided by the standard error of that average. Under the zero-edge assumption each configuration’s t is a draw from a standard normal, and a search of N configurations keeps the largest of N such draws. The expected value of that largest draw has a closed form accurate to better than one per cent, which is the result Bailey, Borwein, López de Prado and Zhu set out in 2014:

E[max of N] ≈ (1 − γ) Φ⁻¹(1 − 1/N) + γ Φ⁻¹(1 − 1/(N e)), γ = 0.5772156649

One substitution, done in full, so nothing here has to be taken on trust. At 15,000 configurations, 1 divided by N is 0.0000667, and the standard normal value that leaves that much in the upper tail is 3.82022. Then N times e is 40,774, 1 divided by that is 0.0000245, and its normal value is 4.06011. Weighting the two by 0.4228 and 0.5772 gives 3.95869. That is a t-statistic. To turn it into edge per trade, divide by the square root of the trade count: the square root of 156 is 12.49, and 3.95869 divided by 12.49 is 0.317. Nothing about any market entered that calculation.

The same formula answers the question a practitioner actually has, which is not what the best worthless rule scores but what score would have been surprising. Requiring that a search of N worthless configurations produces nothing above t only five times in a hundred sets the bar at the normal value of 0.95 raised to the power 1/N. The test is one-sided throughout this page, because a rule that loses is not a discovery, which is why a single configuration sits at 1.645 rather than at the two-sided 1.96. That is the third column below.

Read the first column against the third and the shape is the useful part. Going from one configuration to 253 raises the bar from 1.645 to 3.537, which is most of the damage. Going from 253 to 100,000 is 395 times as much searching and raises it by 1.350 more. The penalty grows with the logarithm of the count, so a modest search is expensive and an enormous one is barely worse than a large one. This cuts both ways, and the way it cuts against you is the one nobody mentions: you do not need a grid of ten thousand to be badly exposed. A hundred is already at 3.283.

The sheet this module started in lesson 62 gains a third column here, beside the sentence and beside the pair of figures lesson 63 added. The bar this configuration had to clear, and the count it was computed from. A row whose t does not exceed its own bar has not shown anything, and a row that does not carry the count cannot have a bar at all. Almost nobody you are trading against records the count. It costs one integer, written down before the search rather than remembered after it, and the reason it is rarely there is that it can only ever make a result look worse.

The same winner, judged five ways

Lesson 63 searched 253 moving-average configurations over sixty closes and kept the one with the highest net total: a 2-bar average against a 5-bar one, seven trades, 9.84 a share after costs, t of 3.65. Against the table above, 253 configurations put the bar at 3.537, so the winner clears it. That looks like a clean pass, and it is worth seeing why it is not one.

Count the trades each of the 253 configurations actually completed on those closes. The smallest is 1, the median is 1, the value three quarters of the way up is 2, and the largest is 16. More than half the grid never completed a second round trip. So the search was nominally 253 wide, and most of its width was configurations that barely traded, whose average is an average over one or two numbers and whose t-statistic is correspondingly wild. The formula assumes 253 comparable tests. It was handed 253 tests of wildly unequal weight.

The standard repair is a floor: a configuration is eligible to win only if it completed at least so many trades. Nobody argues with that. What nobody prints is the number. So run lesson 63’s null again — the 59 bar-to-bar moves drawn with replacement, 4,000 series, seed 20260903, the identical 253-cell search on each — and record how often the best eligible configuration reaches the winner’s 3.65, at five floors.

At a floor of seven trades, which is exactly what the winner completed, 0.68 per cent of structureless searches reach 3.65: about one in 147, and a result anybody would publish. At a floor of two, 50.5 per cent of them do. The winner is unchanged, the closes are unchanged, the 253 configurations are unchanged, and the same t-statistic is a one-in-147 finding or the toss of a coin depending on an integer that appears nowhere in any backtest report.

The correction for having searched has a dial of its own.

Two further things fall out of that table, and both are worse than the headline. The floor does not merely filter the winner, it changes which configuration wins: at floors of two and three the best t in the grid is 6.24, posted by a 3-and-11 pair on four trades, and the 2-and-5 winner is not the best cell at all. And at a floor of ten trades the best t anywhere in the 253 is 1.03. Every scrap of the apparent edge on this series lives in configurations that trade fewer than ten times, which is the regime in which a t-statistic is least entitled to be believed.

So the closed form is a pocket estimate and not a verdict. It gave 3.537 for this grid; the simulation gives a five per cent point of 3.397 at a floor of five, which is close, and 9.57 at a floor of three, which is not close at all. Use the formula to know roughly what you are up against before you start, and simulate at the floor you actually used before you believe anything.

Do this tonight: write down your configuration count and your minimum-trade floor before you look at a result, and compute the bar from the first of them.

What this does not settle

That the configurations are independent. They are emphatically not: a 2-and-5 crossover and a 2-and-6 crossover trade almost the same bars, so the 253 draws are heavily overlapping and the true bar for independent tests would be lower than 3.537. The formula therefore over-corrects on a correlated grid, which is the direction that favours caution rather than the reader. It is still a bound worth knowing, but a page that quotes it as the answer is quoting an upper limit as a measurement.

That sixty closes can carry this. Fifty-nine moves, a winner with seven trades, and floors that empty the grid at ten: every figure in the second table is computed on a series short enough that the floors and the sample length are entangled. On two thousand bars a floor of ten would leave most of the grid standing and the flip between one in 147 and a coin flip would happen, if it happened, at different floors. The mechanism does not depend on the length. The specific numbers do, entirely.

That the t-statistic is the right statistic. It assumes the trades of one configuration are independent draws of equal variance, and consecutive trades from one rule on one series are neither: they overlap in regime, they share the volatility of the same weeks, and a rule that holds longer takes larger and fewer of them. That is precisely why the trade count varies so much across the grid, so the defect the floor is patching and the assumption the statistic rests on are the same defect seen twice.

That the resampled null is the right null. Drawing the moves with replacement destroys volatility clustering along with everything else, and clustering is exactly what decides how many trades a crossover configuration completes. So the null generates trade counts that are not the ones a real series would give, and the eligible-cell counts in the second table are, for that reason, somewhat off. A block bootstrap would keep the clustering and is the honest next step; this page does not run one, for the same reason lesson 63 did not.

That choosing the floor is innocent. It is not, and this page has just demonstrated the technique it is warning about: five floors were run and all five are printed, which is honest, but a reader who runs five and reports the one where their rule clears has moved the search up a level and is now searching over floors. The count that goes on the sheet has to include every dimension you varied, and the floor is one of them the moment you try more than one.

And the concession that costs most. Lesson 63 ended unable to count the configurations that were never written down, and this page does not fix that; it makes it sharper and more uncomfortable. The bar is a function of the count, the count is knowable only for the coded grid, and the chart you looked at, the parameter you nudged and put back, the instrument you abandoned as ugly are all configurations that raise the bar and leave no trace. The formula converts an uncountable problem into a countable one only for the part of the search that was honest enough to be typed. Lesson 65 is where the count is forced down to one and the price is paid in time instead: at a single configuration the bar falls from 4.499 to 1.645, and a 15,000-cell backtest therefore needs 7.48 times as many trades as a live record to support the same claim.