A backtest is an experiment about a frozen rule, dataset and execution model. It is not a search engine for the prettiest equity curve. Every parameter tried, chart inspected and failed idea discarded leaks information from the test set into the research process.
Freeze the research question
Before running results, write the economic hypothesis, tradable universe, observation time, entry, exit, sizing, costs and pass/fail criteria. Example: “After an unusually large one-minute move with stable market depth, liquid contracts partially mean-revert over the next five minutes.” This can be falsified. “Find RSI settings that make money” is optimization without a hypothesis.
Use only information available at the decision time
- Signal on a closed bar unless the production strategy truly observes intrabar events.
- Lag rolling features correctly; today’s high, funding settlement or universe ranking may be unknown at entry.
- Build symbol lists point in time, including delisted and failed assets where data exists.
- Model contract specification changes, listing time, precision and maintenance breaks.
- Separate mark, index, last-trade and book prices according to their real use.
Model the complete trading cost
Net PnL = price PnL − maker/taker fees − spread − market impact − funding/borrow − liquidation and operational losses. Slippage should depend on side, urgency, size and depth—not a favorable constant. Reject or delay orders that violate historical precision, minimum notional or data-freshness rules.
If only candles are available, run conservative scenarios: next-bar execution, adverse spread and several slippage levels. A strategy that survives only zero-cost same-close fills has not passed an executable test.
Choose a simulator that matches the strategy
| Model | Appropriate for | Cannot prove |
|---|---|---|
| Vectorized bars | Slow signals and broad screening | Queue position, intrabar order path or partial fills |
| Event-driven bars/trades | Order lifecycle and conditional logic | Exact fills without book/queue assumptions |
| Order-book replay | Short-horizon execution research | Your historical queue priority unless message data supports it |
Validate forward, not randomly
Time series are ordered. Use an early research window, a later validation window and walk-forward folds that repeatedly train or select on the past and evaluate on the next unseen period. Add a gap when labels or positions overlap the boundary. Keep a final untouched holdout or, better, start a selection-free paper-trading record.
Metrics that expose the distribution
- Expectancy: win rate × average win − loss rate × average loss, after all costs.
- Profit factor: gross profit ÷ absolute gross loss. Report the denominator and trade count.
- Maximum drawdown and recovery time: path risk, not only final return.
- Turnover, exposure and capacity: how much execution the result requires.
- Tail loss and losing streaks: the risk that determines survivability.
- Attribution: long/short, symbol, market regime, entry cohort and exit reason.
Add bootstrap confidence intervals by independent-enough blocks rather than treating every minute as a new sample. A positive estimate with a wide interval crossing zero is uncertainty, not a proven edge.
Account for the number of ideas tried
The best of hundreds of no-edge variants will look impressive by luck. Record every trial, keep parameter surfaces coarse, prefer stable plateaus to isolated optima and discount evidence as the search count grows. Do not use the holdout repeatedly and continue calling it out of sample.
Bailey, Borwein, López de Prado and Zhu formalize this problem in The Probability of Backtest Overfitting. Their central lesson is practical: ordinary holdouts can still fail after extensive strategy selection.
Promotion gate before live capital
- Positive fee-aware expectancy across more than one market regime.
- Acceptable drawdown under stressed costs and delayed execution.
- No single symbol, side, month or exit mechanism explains the result.
- Stable behavior across nearby parameters and walk-forward folds.
- Paper/shadow evidence uses the same code path and records rejected opportunities.
- Operational limits, monitoring and a rollback criterion are defined before activation.
Passing this gate does not prove future profitability. It earns the right to collect cleaner forward evidence at minimal risk—the next stage of research, not the end of it.