Why your backtest lied to you
It worked for three years of history. It made money every month you checked. Then you traded it, and it stopped. Nothing was broken, nobody cheated, and the reason is arithmetic you were never shown.
Almost everyone who trades has had this exact experience. You find a rule. Maybe you built it, maybe you paid for it, maybe someone on Instagram posted it. You check it against history and it looks superb. You put real money behind it. Within a few months it has quietly stopped working.
The usual explanations are that the market changed, that you lacked discipline, or that you need better entries. Occasionally one of those is true. Usually the answer is simpler, stranger, and far more useful to understand: the strategy was never working in the first place. What you saw in the backtest was real in the sense that the numbers were correct. It just was not evidence of anything.
By the end of this page you will be able to work out, for any strategy you own or are being sold, the exact number it has to beat before it means anything at all. Most strategies do not beat it. Knowing how to check takes about two minutes once you have seen it done, and it will save you years.
I am not going to assume you know any statistics. Where a term is needed, I will explain it in ordinary words first.
A room full of coin flippers
Forget trading for a moment.
Put a thousand people in a room and give each one a coin. Every morning they flip it. Heads, they buy. Tails, they stay in cash. That is the entire strategy. No charts, no news, no skill: we handed them a coin.
Run it for two years. Then ask the room: who did best?
Somebody always wins. Not because they were good (we know none of them were good), but because when a thousand people do something random, the luckiest one looks remarkable. Their equity curve goes up and to the right. Their win rate is excellent. If they posted it online, you would follow them.
And here is the part that should stop you: that person cannot tell either. From the inside, a lucky streak feels exactly like skill. They are not lying to you when they say their method works. They genuinely believe it, and they have the screenshots.
You are not looking at successful traders. You are looking at survivors, and the selection happened long before you found them.
This is not a thought experiment
A thousand is too small. The real retail trading population is in the millions, and the measured outcomes look exactly like the coin room.
| Study | Who | Outcome |
|---|---|---|
| Chague, De‑Losso & Giovannetti (2020) | Brazilians who kept at it over 300 days | 97% lost money |
| Barber, Lee, Liu & Odean | Taiwan, 14 years of records | <1% reliably profitable |
| ESMA broker disclosures | EU retail accounts | 74–89% lose |
Now put those two facts side by side. Almost everybody loses. And the handful you can actually see (the ones with the courses, the signal groups, the followings) are drawn from the luckiest tail of a population of millions.
That is not an accusation of fraud. It is just what selection does. A person with a genuine five-year winning record and no skill whatsoever is the expected output of a million people trying. You would need several such people before the result was even mildly surprising.
Now the uncomfortable part: you did the same thing
You think you tested one strategy. You almost certainly did not.
You tried a 20-day average. Mediocre. So you tried 50. Then you thought it needed a stop, so you added one at 5%, then 10%. Then you decided the asset was wrong and tried a different one. Then you noticed that 2019 was ruining everything, so you started the test in 2020 instead.
Every one of those was a look. And each look is another coin flipper in the room. By the time something finally worked, you may have taken forty looks, and the best of forty random attempts looks good, for the same reason the best of a thousand coin flippers looks like a genius.
This was not cheating. You did what any sensible person does: you kept adjusting until something worked. The adjusting is the problem, because the bar for "this is real" rises every single time you look, and nobody ever told you to raise it.
Three ways people take far more looks than they think
- Settings. Trying lookbacks of 10, 20, 50, 100 and 200 is five looks, not one. Five lookbacks × three stop levels × two assets is thirty.
- Moving the dates. Starting later because the early years "aren't representative", or stopping before a bad patch, is a look. It is also almost never counted.
- Buying someone else's looks. This one catches everybody. If you take a strategy from a course, a paper or a video that tested two hundred ideas and showed you the best one, you inherit all two hundred. Somebody else did the searching on your behalf. Your "one test" is the two hundred and first.
That third one is why buying strategies rarely works, and it has nothing to do with whether the seller is honest. An honest seller who tested two hundred ideas and sold you the winner has sold you the best of two hundred coin flippers.
The notation, once
- rt the return in period t. A decimal: 0.004 is +0.4%.
- rf what cash paid over the same period. Usually tiny, never zero.
- xt = rt − rf the excess return. Everything below uses this, not rt.
- n how many periods you have.
- p periods per year. 252 for trading days, 12 for months.
- T = n / p the length of the window in years. This one matters more than you expect.
- N how many distinct things you tried before picking one.
Three words, in plain English
To check a strategy properly you need three terms. They sound technical and they are not. Read these once and the rest of this page is easy.
The only vocabulary you need
- Sharpe ratio
- How much you earned, divided by how wildly the account swung around to earn it. Two strategies that both made 20% are not equal if one was smooth and the other was terrifying. Above 1.0 is genuinely good. Below 0.5 means the returns were mostly noise. Your backtesting software already reports this.
- Median
- The middle value when you line results up smallest to largest: half above, half below. It is more honest than an average, because one spectacular result can drag an average upward while the typical case stays poor.
- Noise floor
- The score that pure luck is expected to produce, given how many times you looked and over how long. This is the number almost nobody calculates, and it is the whole point of this module. Beating it is the minimum for a result to mean anything.
The arithmetic behind that number
The Sharpe ratio is reward divided by variability:
where x̄ = (1/n) Σ xt
and s² = (1/(n−1)) Σ (xt − x̄)²
In words: average excess return, divided by its standard deviation, scaled up to a year.
The √p is the part people use without asking about. Here is why it is a square root and not p.
If daily returns are independent with mean μ and variance σ², then over p days the mean adds: μyear = pμ. But variances add, not standard deviations, so σ²year = pσ², which means σyear = √p · σ. Divide one by the other:
Return grows with time; risk grows with the square root of time. That gap is the whole reason a long track record looks better than a short one at the same skill level, and it is also why the noise floor in section 3 has a √T in the denominator.
What the √p step assumes, and when it lies
It assumes returns are independent across periods. If your strategy holds positions for weeks, consecutive daily returns are correlated, variance grows faster than pσ², and √p overstates your Sharpe. Momentum and trend strategies are the usual offenders.
It also assumes finite variance. Fat tails do not break the formula, but they make s an unstable estimate, so the whole ratio wobbles more than its decimal places suggest.
The table that settles arguments
Two things drive the noise floor, and both are intuitive once said out loud.
More looks, higher bar. Every extra attempt gives luck another go, so the best of many attempts scores higher than the best of a few.
More years, lower bar. Luck cannot keep it up. A fluke that survives six months is easy; one that survives twenty-six years is almost impossible. Time is the only thing that genuinely defeats this problem, which is why long history is worth more than clever ideas.
Find how many looks you took down the side, and how many years you measured across the top. The cell is the Sharpe ratio your strategy must beat.
| Looks you took | 2 years | 4 years | 9 years | 26 years |
|---|---|---|---|---|
| 1 (decided in advance) | 0.00 | 0.00 | 0.00 | 0.00 |
| 5 | 0.82 | 0.58 | 0.39 | 0.23 |
| 20 (a normal afternoon) | 1.32 | 0.93 | 0.62 | 0.37 |
| 100 | 1.77 | 1.25 | 0.84 | 0.49 |
| 1,000,000 (the whole room) | 3.44 | 2.43 | 1.62 | 0.95 |
Look at the highlighted row. Twenty looks over two years demands a Sharpe of 1.32. Twenty looks is one focused afternoon of tweaking settings. If your promising strategy scores 0.9, it is not a weak edge that needs refining. It is below what pure luck produces under the same amount of searching.
Now look at the top row. One look has a bar of zero. If you decide the entire rule in advance, write it down, run it once and accept the answer, there is no luckiest-of-many to beat. That is not a loophole; it is the reward for committing before you look, and it is the single most valuable habit in this course.
And compare the bottom row across: a million looks over two years needs 3.44, the same million over twenty-six years needs 0.95. Identical luck, identical searching, and the bar falls by nearly four times purely because somebody measured for longer.
Why this is good news, not bad news
This arithmetic is the reason most strategies for sale are worthless, and it is also the only tool that would let you recognise a real one. Right now you have no way to tell a genuine edge from a lucky one, which means every decision you make about strategies is a coin flip of its own. Once you can calculate the floor, you can stop paying for noise, stop grinding on dead ideas, and spend your effort where there is actually something to find.
Where that table comes from
Everything else in this course is hygiene. This is the part that changes what you believe.
First, in plain words. Go back to the coin room. Nobody there has any skill, yet somebody always ends up on top, and the more people you put in the room the better that winner looks. The table is just that sentence turned into a number: given how many times you looked, how good does the best result get by luck alone? Everything below is the arithmetic of the coin room. If you follow the idea, the symbols are optional.
The setup. You tried N variations. Assume the worst case about yourself: that not one of them has any edge, so each one's true Sharpe is zero and every score you measured is pure noise. Statisticians call that pessimistic assumption the null hypothesis: it is simply the boring explanation you have to rule out before claiming anything interesting. You then report your best variation. What score should you expect that winner to have, even though none of them is any good?
Each measured Sharpe over T years wobbles around zero, and the longer you measure the less it wobbles: its standard deviation is 1/√T, which is just the previous section with S = 0. Strip out the √T for a moment and each variation is a draw from the plain bell curve, written Zi. So the whole question reduces to one clean puzzle: if you draw N numbers from a bell curve, how big is the biggest one, on average?
Deriving E[max]
The idea behind the algebra. There is one easy question about a maximum: what are the odds that the biggest of them is below some level? That only happens when every single one is below it, and independent probabilities multiply: the odds of two coins both landing heads is a half times a half. So the odds that all N land below x is one number multiplied by itself N times. That is the whole trick; the rest is turning it into an average.
Two symbols do the work. Φ(x) is the chance that a single draw lands below x: the area under the bell curve to the left of that point. φ(x) is the height of the curve at x. Written out:
The chance that all N draws come in below x. Now turn that into the chance the maximum lands at any particular spot. In calculus that step is differentiating, and it gives:
Read it as: the winner lands at x when one of the N draws is there (N · φ(x)) and all the others fell short (Φ(x)N−1). To get the average, weight every possible x by how likely it is and add them all up, which is what the integral sign means:
For most values of N that integral has no tidy answer you can
write down, so the code adds it up by brute force instead: it slices the curve into 200,001 thin
strips between x = −12 and x = 14 and
totals their areas (_expected_max_normal(), a trapezoid rule). That is an ordinary
approximation, and the next section checks it against an answer we already know. Put the
√T back and you have the floor:
The Sharpe that pure luck is expected to reach, given that you searched
N times over T years. judge_strategy.noise_floor().
Checking it against something known
A numerical integral you cannot check is a number you cannot trust. For N = 3 there happens to be a closed form:
The integration agrees to four decimal places, and so does a 200,000-run simulation. Three independent methods, same answer. That is the standard this course asks you to hold your own numbers to.
How the floor grows
For large N the expected maximum grows like √(2 ln N). The logarithm is the important part: the bar rises, but slowly and without limit. Doubling your search does not double the bar, and no amount of searching ever makes it stop rising.
Worked: the floor at T = 3 years
| N = 1 one pre-registered test | 0.00 |
| N = 5 | 0.67 |
| N = 14 | 1.00 |
| N = 50 | 1.33 |
| N = 200 | 1.62 |
Read the N = 14 row again. After fourteen attempts over three years, a strategy with no skill whatsoever is expected to produce a Sharpe of 1.0, the number most people would call a good result and fund.
Why N = 1 gives a floor of exactly zero
With one pre-registered test there is no "best of many" to beat, so the expected maximum is the expected value of a single draw, which is zero. That is not a loophole, it is the entire reward for deciding in advance, and it is why Module 4 exists.
It does not mean any positive result passes. It means the best-of-N bar is the wrong instrument there, and you should use the standard error from section 2 instead.
A bug in this very formula, and how it was caught
An earlier version of the code used the asymptotic expression
√(2 ln N) − (ln ln N + ln 4π) / (2√(2 ln N))
That is the location parameter of the Gumbel limit: the mode of the maximum, not its mean. The two differ by roughly the Euler–Mascheroni constant over the scale parameter, and the mode is always lower. It ran 0.13 to 0.25 too low everywhere, and 29% too low at N = 3.
Every error has a direction. This one made the bar too easy, which is the direction that produces false discoveries rather than missed ones. No conclusion flipped when it was fixed, but it could have. That is what the N = 3 closed-form check above now exists to prevent.
The two bars
E[max] is the average luck result, so half of all pure-noise searches beat it. As a hurdle for real money that is weak. The stricter version stops asking what luck reaches on average and asks what it reaches on a good day. Line up the luckiest result from ten thousand imaginary searches, worst to best, and take the one 95% of the way along: that value is the 95th quantile. Only 1 run in 20 of pure luck ever gets past it:
≈ Φ−1(0.951/N) / √T
Beaten by one noise search in twenty. The closed form on the right follows straight from P(max ≤ x) = Φ(x)N: set that equal to 0.95 and invert. The Python simulates it instead, which is why the two can differ in the second decimal.
Which bar to use is a judgement, not arithmetic. The course uses E[max] for "worth collecting forward data on" and the 95th percentile for "worth real money".
What it looks like when it happens to you
Theory is easy to nod along with, so here is the same mistake with real numbers. Mine.
There is a well-known paper that publishes 101 trading signals as explicit formulas, no interpretation needed, just code them and run. Ideal test material. I implemented 63 of them.
One stood out immediately. 22.64% a year. A clean formula from a published paper, a return that would embarrass most funds. For about an hour I thought I had found something real.
Then I looked at the other 62.
The median Sharpe across all 63 was −0.13. The typical signal in that set lost money.
So the winner was not a discovery standing in a field of discoveries. It was the best of 63 attempts drawn from a pool whose middle was slightly worse than useless, which is precisely the coin-flipper problem, except the flippers were formulas. Measured against the bar for 63 looks, it did not clear it. It then died a second time on trading fees.
Had I reported only the winner, I would have published a gold mine that did not exist. And I would have believed it, because the number on my screen was genuinely what my code produced.
The rule this produced
Report the distribution, not the winner. If somebody shows you one strategy, ask how many they tested. If they tested sixty and showed you one, you are looking at a maximum, not a result. Ask for the median and the range. The answer to that single question disqualifies most things that get sold.
It is also the question almost nobody asks, which is why it works so well.
The part that is genuinely hard to hear
I tested 946 strategies: academic papers, factor libraries, Quantpedia entries, Instagram reels, and plenty of my own inventions. One survived everything.
For a long stretch, every single result I produced sat below its own noise floor. Not most of them. All of them. The best thing on my list at one point scored 0.75 against a bar of 1.42.
That is not a story about bad strategies. It is a story about a search that had eaten its own evidence: I had examined the same few years of data hundreds of times, so no result from that data could clear the bar any more, no matter how good the idea.
Two honest conclusions follow, and they point in opposite directions.
Be far more sceptical of your backtest than you are. The floor is usually above your result, and you have taken more looks than you counted.
But a floor describes your evidence, not reality. A strategy with a true Sharpe of 0.3 is serious money at scale, and it is completely invisible at a bar of 1.42. Failing to detect something is not the same as it not being there. The right response is not to give up; it is to stop interrogating the same data and go get evidence the floor cannot swallow: longer history, different markets, and results from time that had not happened yet when you picked the strategy.
That is what the remaining seven modules are. Where the long history is and how to get it free. The one cheap test that does most of the actual killing. 31 bugs from my own research, most of them ones that made my results look better than they were, with what each cost. And an honest answer to the question this forces on you: what do you do when the whole search turns up almost nothing?
Try it on your own strategy
- Count your looks honestly. Every setting you tried, every asset you swapped, every start date you moved, every version you abandoned. If the idea came from a course or a video that surveyed candidates, add theirs. Write the number down before you continue.
- Count the years you actually measured. Not the data you downloaded: the span you scored.
- Find the cell in the table above and compare it with your Sharpe ratio.
- Answer one question in writing: if my result is below the floor, what would I need in order to find out whether this strategy is real anyway?
Most people find their result sits below the bar. That is the normal, expected outcome and it is not a reason to stop. It is the reason the rest of this exists, because once you can see the floor, every method for getting evidence over it becomes worth learning.
Honest
Strategy
Testing
You get the two things that took longest to build: a map of 19 free data sources with working download scripts, and the scoring tool that calculates every number in this module for you. Plus eight modules explaining how to use them.
The strategy that passed stays mine. The 932 failures are what teach you.
€400
One payment. Lifetime access. Includes the scoring tools, the data map, and the full ledger, searchable, with the verdict and reason for every family, round and source.
Get the courseFourteen-day refund, no questions asked. If it does not change how you test, you should not have paid for it.
If that was useful
Every two weeks I send what failed and why, with the number that killed it, and the rare one that survived. No tips, no signals, and one click to stop.
Your address is used for this and nothing else. Every email has a one-click unsubscribe.