From backtesting to live validation

Backtesting is one of the most useful tools in quantitative investing and one of the easiest to misread. A backtest applies a method to historical data to see how it might have performed. It is valuable for developing and comparing ideas. It is not a track record, and treating it as one is a common route to disappointment.

This article explains why, and outlines the steps that usually sit between a promising backtest and a method that can be relied on with real capital.

Why backtests flatter

A backtest is a simulation built with the benefit of hindsight. Several well-known problems can make results look better than anything achievable in practice.

  • Look-ahead bias. The test uses information that was not available at the time, such as company accounts that were published weeks after the date being simulated, or later corrections to earlier figures.

  • Survivorship bias. The test only includes companies that still exist, leaving out those that were delisted, taken over or failed. A universe of survivors tends to look healthier than the real one.

  • Data leakage. Information from the test period seeps into how the model was built or tuned, so it is effectively being tested on data it has already seen.

  • Overfitting. A model is tuned so closely to past patterns that it captures noise rather than anything persistent. The more variations that are tried, the greater the risk.

  • Unrealistic costs. Fees, bid-offer spreads, market impact and liquidity limits are understated or ignored. Strategies that trade often or hold smaller companies are especially sensitive to this.

None of these is unusual, and none implies bad faith. They are the normal hazards of working with historical data, which is why careful methodology matters so much.

Building a more honest test

Good practice is designed to reduce each of those problems.

Point-in-time data records what was known on each date, including the original version of figures that were later restated. A universe that includes delisted companies addresses survivorship bias. Keeping training and test periods strictly separate, and deciding the test design before looking at results, limits leakage and overfitting.

Walk-forward testing takes this further. The model is trained on one period, tested on the next, then retrained and moved forward again. This mirrors how a model would actually be used, updating as time passes, and shows whether performance holds across different market conditions.

Finally, realistic assumptions about trading costs, turnover and liquidity should be built in from the start rather than added at the end.

From simulation to the real world

Even a carefully constructed backtest is still a simulation. The next steps test the method against the present.

  1. Paper trading. The system runs on live data and produces real-time decisions, but no money is invested. This tests the full process, including data feeds and timing, without financial risk.

  2. Controlled live testing. A limited amount of real capital is used, within tight limits and with close supervision, under an appropriate mandate and permissions.

  3. Monitoring. Once in use, the model is watched for drift: changes in its behaviour or accuracy as market conditions shift. Thresholds are set in advance for when to investigate, pause or retrain.

  4. Incident response. There is an agreed plan for what happens when data is wrong, a system fails or results fall outside expected ranges.

Each stage is an opportunity to discover problems that a backtest could not reveal.

Reading results with care

When you see a backtested or simulated figure, a few questions help put it in context.

  • Over what dates, on what universe, and using which version of the model?

  • Was the data point-in-time, and were delisted companies included?

  • How were training and test periods separated?

  • What costs, leverage and turnover were assumed?

  • Can the result be reproduced by someone else?

  • Has the method been tested out of sample, in paper trading or live?

If those answers are not available, the figure is best treated as a hypothesis rather than evidence.

The point of validation

Validation is not about proving that a method will succeed. No amount of testing can do that, because the future will differ from the past. Its purpose is to remove avoidable errors, understand where a method is fragile and build enough evidence that a team can make an informed judgement about using it.

The honest position for any new system is to be clear about which of these steps have been completed and which have not.

This article is educational and general in nature. It is not investment advice. Simulated or back-tested results have inherent limitations and are not a reliable indicator of future results. Intelligent Wealth's software is under development and has not completed live validation.

Next
Next

What explainable AI means for investment teams