Semtum

946 strategies.
One survivor.

Honest Strategy Testing is a course in checking a trading strategy before you trust it. I built a process that rejects almost everything, then ran 946 ideas through it across 466 documented rounds, taken from academic papers, Quantpedia entries, Instagram reels and my own inventions. 932 failed, 13 are still waiting on forward data, and 1 cleared every test. This course is that process, and every one of the 932 that failed is a worked example in it.

01/06The uncomfortable arithmetic

Take a million people flipping coins to pick trades. The best of them will show a t-statistic (how unlikely a result is to be luck) around 5.3 — the conventional threshold for a discovery is 2.0. That person has zero skill and a track record that looks like genius.

This is why your backtest looks good. Not because you cheated, but because you looked more than once, and the bar rises every time you look. Almost nobody testing strategies at home knows how to calculate where that bar actually sits — so they find "edges" that are the expected best of however many things they tried.

A third of arbitrary date windows are profitable. So is your edge. The question was never "does this make money" — it is "does this beat an arbitrary version of itself?"

That single test — the placebo sweep — has ended more of my ideas than any other test. It costs one loop of code. Module 6 is entirely about it.

The test that did the killing

A real candidate of mine: take the high and low of the first five minutes of the US session and trade the breakout. It made money. Then I rebuilt the identical rule anchored to the 1st, 2nd, 3rd … 20th five-minute bar instead — twenty versions of the same idea with no story behind them at all.

12 of the 20 arbitrary anchors beat it.More than half of the versions picked for no reason did better than the real rule. The median placebo beat it and the best beat it nearly fourfold — which cannot happen if 09:30 is special. Rejected.

Round 44 in the lab. Sharpe ratio (return for each unit of risk), same data, same costs, one anchor apart.

Drag it. Watch the bar move.

This is the whole argument in one control. Slide to however many versions of a strategy you have ever tried — every parameter, every asset, every start date, including the ones you abandoned — and watch the score a completely skill-free strategy is expected to reach.

Luck alone reaches
0.00
A typical home backtest
1.00

Three years of data. Sharpe ratio on the vertical axis.

The free first module works through this arithmetic in full. No signup, about fourteen minutes.

Read it free → Get the course

02/0631 ways a backtest lies to you

Every bug below is one I actually shipped, caught, and measured. 19 of the 31 in the course made results look better — which is why they are so hard to notice. Every one of them cost me a result I believed.

The bugWhat it cost
Exits resolved from the wrong end of the bar. A dip entry whose target sits above the level price fell to, so bars that rallied then dipped scored as winners.0.87 Sharpe
PASS → FAIL
Weekend returns deleted instead of carried. Aligning a 365-day crypto series onto stock-market sessions throws away every Saturday and Sunday.31 points
of CAGR
Bootstrapped the wrong statistic. Resampled the Sharpe of the difference instead of the difference of Sharpes. The confidence interval did not contain its own point estimate.PASS → FAIL
Sharpe on raw instead of excess returns. A near-zero-volatility cash leg inflated every low-exposure benchmark — one scored 1.59 where the honest figure was 0.12.1.47 Sharpe
of illusion
A data vendor selling the wrong index. A file labelled DAX was the EURO STOXX 50 for 42 straight months. Caught only by a 271% one-minute "move".whole test
invalid
A regulator changed a form label. The filing code changed, and a parser keyed to the old string returned zero rows for recent years without raising an error.silent
zero data
Drawdown compared with the wrong operator. Drawdowns are negative; the comparison was written the intuitive way, and a 3.8-point-deeper drawdown was reported as passing.reported a
loss as a win
Timezone-aware timestamps silently converted. Taking raw values off a tz-aware column shifts everything to UTC and the labels stop matching. Twice.two broken
reindexes

What a flattering bug looks like

One of the 31: exits resolved from the wrong end of the bar. The rule never changed. The data never changed. Only the order two prices were read in changed — and the result went from fundable to worthless.

Same strategy, same data, one line of code apart.

23 more in the course, including the one where every rebalancing frequency returned an identical number and it took a week to notice that was impossible.

All 31 are in Module 7, each with the measured cost of not catching it and the test that would have.

Try the free tools → Get the course

03/06What you get

Eight modules, in the order you actually need them. Every module ends with something you can run, not just something you have read.

01

Why your backtest lied to you

Best-of-N selection, and how to calculate the noise floor for your own search. The maths of why looking twice raises the bar, and why an inflated floor is as dangerous as no floor at all.

You leave with: a function that tells you the Sharpe your result must beat, given how many things you tried and over how long.
02

The free data map

Nineteen sources I actually pulled, what each one gives you, what it costs, and the specific trap inside each. Minute bars back to 2016, FX back to 2008, fundamentals from the regulator, auction calendars, factor libraries to 1926, options greeks, volatility indices to 1990. Nearly all of it free.

You leave with: working fetch scripts and a table of which source answers which question.
03

Five ways to kill an idea in ten minutes

Where real strategies come from — papers, factor libraries, forums, social media — and how to reject most of them in ten minutes without writing a line of code.

  • Data you do not have, and how to check before you hope
  • Not specific enough to code — the two-programmer test
  • Already tested, and keeping a ledger so you know
  • Self-reported evidence, and why a screenshot is worth nothing
  • Costs, estimated in round trips per year before you build
You leave with: a one-page intake template that has killed hundreds of ideas.
04

Decide before you look

Writing down the rule, the benchmark, the number of variants and the expected result before running anything. The discipline that makes a wrong prediction valuable and a right one credible.

You leave with: the pre-registration format, and the reason one of my predictions being wrong is why I trust that result most.
05

What to compare against

Buy-and-hold is almost always the wrong comparison. How to build an exposure-matched blend — the dumbest version of the same bet — and why one of my candidates cut drawdown against buy-and-hold in 7 of 7 windows and beat the fair blend in 0 of 7.

You leave with: the construction, plus the rule for when matching exposure changes nothing at all.
06

The placebo sweep

The highest-value test in this course and the cheapest to run. Rebuild your rule at twenty arbitrary offsets and ask where the real one ranks. It kills whole categories of calendar, session and level claims in one loop.

You leave with: the sweep, and the reason it must run in both directions.
07

31 bugs from my own backtests

31 real bugs, each with the measured cost of not catching it, and a test that would have. Look-ahead, silent type coercion, survivorship, the double-subtracted risk-free rate, and the verification that verified nothing because it re-read the same file.

You leave with: a self-check harness that recomputes your results from positions and prices alone, so a wrong number cannot reach your conclusion.
08

Judging, and what to do when nothing passes

Seven binary conditions, the boundaries that stop you gaming them, and the honest answer to the question this course forces on you: what do you do when the whole search turns up almost nothing?

You leave with: the full criteria document, and a two-gate structure that separates "worth watching" from "worth funding".

Eight modules, the toolkit, the data map and the full ledger. One payment, lifetime access, free updates.

Read one free first → Get the course — €400
8written modules
11interactive figures
4scoring tools
559rows in the ledger
19data sources mapped
12download scripts
14pre-registrations
31bugs catalogued

Roughly three hours of reading if you go straight through. Most people do not, because every module ends with something to run against their own work.

Read one of the eight, free

Not an excerpt, not a teaser email, not a sample chapter with the good part removed. Module 1 in full, with its two live figures, and no signup. If it does not change how you think about your own backtest, nothing later in the course will either.

Read Module 1 free

There are no testimonials on this page

A quote is the easiest thing in the world to produce. A course whose third module teaches “self-reported evidence, and why a screenshot is worth nothing” does not get to make an exception for itself, so there is nothing here that you would have to take on trust.

What there is instead: four things you can check, in full, before you pay me anything.

  • The ledger, in fullEvery family, round, price series and screened source, with its verdict and the reason it died. Pick any row and argue with it. The ones that disagree with me are still in there.
  • Predictions I got wrong14 pre-registrations, written and numbered before each test ran. Including the ones where what I expected did not happen, which are the only ones that prove the process was real.
  • A whole module, freeOne of eight, complete, no signup. Read it and decide from the writing rather than from a claim about the writing.
  • Five tools that work nowRun them against your own strategy before you buy anything. If they tell you something uncomfortable, that is the product demonstrating itself.

04/06What it actually looks like

Screenshots of the real thing, not mockups. Everything below is a page or a tool that ships in the download, running on its own data.

Module 1 showing the coin-flip simulator: a thousand skill-free traders, the best
                of whom reaches a Sharpe ratio of 2.66.
The modules are interactiveEleven live figures across the eight modules. This one runs a thousand coin-flippers and shows you the luckiest. Press it again and a different person wins.
The searchable ledger filtered to the word placebo, showing rejected families with
                the verdict and reason for each.
The ledger is searchableEvery approach with its verdict and the reason it died. Search before you spend an afternoon on something — each line is a question you do not have to pay to ask again.
The toolkit control panel scoring a candidate as FAIL, with the boundary violation
                named underneath.
The toolkit has a screenDouble-click one file and it opens a local control panel. Score a candidate against the seven conditions as you type; fetch any of the twelve data sources by clicking Fetch. Nothing leaves your machine.
The pre-registration writer producing a dated markdown file with the noise floor
                already computed into it.
It writes your pre-registrationFill in six lines, download a dated file. It computes the bar your result has to clear and puts it in the document, before you have any idea what the answer will be.

05/06Questions people actually ask

Do I need to be able to code?

For half of it, no. Modules 1, 4, 5 and 8 are about how to decide what counts as evidence, and need nothing but reading. The five calculators on this site run in a browser with no install.

Modules 2, 3, 6 and 7 ship Python — data-fetch scripts and a scoring toolkit. You can read them and apply the thinking by hand, but you will get more from them if you can run a script. The toolkit now opens a local control panel by double-clicking a file, so you do not need to live in a terminal.

I only trade stocks. Or only crypto. Or only forex. Does this still apply?

The method is asset-agnostic; the examples are not. Most worked examples are crypto and equities, because that is what I tested. The data map covers equities, fundamentals, FX, futures positioning, factors and volatility.

Nothing in the noise floor, the placebo sweep or the benchmark construction cares what you trade. If you test ideas on price series, it applies.

How is this different from free material on YouTube?

The 932 failures. Free material shows you the thing that worked. Nobody publishes the rejections, because nobody keeps them — and the rejections are where the method lives.

The ledger here is the research record in four parts, searchable: the families I rejected and why, the price series already used and how many times each has been looked at, every round of work with its verdict, and the sources killed before any code was written. So are the 14 pre-registrations, written before each test ran, including the ones where my prediction was wrong. You can read what I expected, then what happened.

Will this make me money?

Probably not, and that is the point. Most people who trade lose money, and nothing in here changes that. This course spends far more time on why ideas fail than on how to profit.

What it can do is stop you funding something that was never real. Whether that is worth €400 depends entirely on what you would otherwise have staked on your next good-looking backtest.

Does it contain the strategy that passed?

No. It contains the method that stopped the other 945, and the criteria that one had to clear. The page says so before you pay, deliberately.

A strategy is worth very little once published and copied. A method for disproving strategies does not decay that way, which is the honest reason this is the thing for sale.

What format is it, and how long does it take?

One download: HTML pages you open in a browser, plus Python files. No video, no login, no platform, no internet connection required after download. It works offline and it will still open in ten years.

Roughly three hours of reading if you go straight through. Most people will not, because every module ends with something to run against their own work.

How does the Team licence work?

You buy once and get one download, the same files as the single licence. You may share it with up to five people in the same organisation. Who those five are is up to you, and nothing in the files checks. When someone leaves, their seat passes to whoever replaces them. Updates arrive at the same download link; pass them on the way you passed on the first copy.

An invoice for your company: it comes from the seller of record that runs the checkout, which also handles VAT. Need something specific on it? Ask before you buy.

More than five people? Write to me and we will agree a price. The licence is for use inside your organisation: using the method on your own and your employer’s strategies is what it is for, passing the files to clients or anyone outside is not.

What if it is not for me?

Fourteen days, no questions asked. Email and it is refunded — no survey, no call, no arguing. The full policy is three paragraphs and says exactly that.

If you would rather not take my word for any of this, read the first module free before deciding. It is one of the eight, in full, with no signup.

06/06What this is not

You will get

  • A method that rejects things, including things you want to believe
  • Every failure I found, with the numbers
  • Working code: noise floors, placebo sweeps, independent recomputation
  • A map of free data, with the trap inside each source

You will not get

  • My strategies. What passed stays mine; the 932 that failed are the teaching material
  • Signals, calls, or anything to copy into a broker
  • A promise that you will find an edge. Most people who apply this correctly will not
  • Anything resembling financial advice

If you want someone to tell you what to buy, this is the wrong product and I would rather you knew now. This teaches you to disprove ideas, which is a slower and considerably less exciting skill. It is also the one that keeps paying off.

Is it worth €400? Here is the only comparison I can make honestly. Module 7 documents 31 bugs I shipped, each with its measured cost. One of them turned a losing strategy into a 0.87 Sharpe on paper. Another threw away 31 points of annual return. Any one of them, undetected, is a strategy funded on a result that was never real.

I am not going to tell you what that costs you, because I do not know what you would have staked. You do. Compare this against that number rather than against other courses.

Choose a version

The Bug Catalogue

€49one payment, free updates

If you only want the part that catches you lying to yourself.

  • Module 7 in full — 31 real bugs, each with the measured cost of not catching it and the test that would have
  • The recomputation harness: rebuilds your result from positions and raw prices alone
  • Module 3, five ways to kill an idea in ten minutes
  • The method — modules 1, 4, 5, 6, 8
  • The data map and the download scripts
  • The ledger and the pre-registrations
Get the bug catalogue

Honest Strategy Testing

€400one payment, lifetime access, free updates

The whole method, and all the evidence behind it.

  • 8 written modules with eleven interactive figures
  • The criteria document — 7 binary conditions and the boundaries that stop you gaming them, the same one I am bound by
  • The scoring toolkit — noise floors, placebo sweeps, a check that the edge over the benchmark holds in every slice of time, and the recomputation harness, behind a local control panel
  • The data map — 19 sources with the trap in each, 12 of them with a working fetch script
  • 14 pre-registrations, written before results, including the ones I got wrong
  • The full ledger — every family, round, price series and screened source with its verdict and the reason, searchable
  • The N ledger — a record of how many times each dataset has already been tested, so you know how high its bar now sits
Get the course — €400

Team

€1,200five seats, one payment

For a desk, a fund or a research group where more than one person tests ideas.

  • Everything in Honest Strategy Testing, for up to five people in your organisation
  • One download you share internally. No keys, no accounts, no seat manager
  • An invoice for your company, from the seller of record that runs the checkout
  • The criteria document as a template your team can adopt as its own standard
  • Free updates for everyone, at the same download link
  • One payment. No renewal, no per-seat billing
Get five seats

Buying for a team? Write to me and pick “Team or multi-seat licence”.

Bought the bug catalogue and want the rest? Email me and you pay the difference, not the full price. That is not a launch offer with a deadline on it — it is how it works, permanently.

Still deciding?

14 daysto change your mind, on any tier

Read Module 1 free first

14-day refund, no questions asked. If it does not change how you test, you should not have paid for it. Email and it is refunded — the address, and what that promise means in full, are on the contact page.

Instant download — eight modules, the data scripts, the scoring tools and the full ledger, as one file. Updates free for life — see what has changed.

This is not financial advice, not a recommendation to buy or sell anything, and not an offer of any investment service. Trading loses money for most people who attempt it. Nothing here suggests otherwise, and the course spends more time on why ideas fail than on how to make money.

Honest Strategy Testing · €400 Get the course