# Manual Backtesting and Evidence Build a reproducible test without hindsight selection. Use this workbook alongside the course. Write your answers before opening the solutions. Practical work is self-reviewed; scored knowledge checks are in the Academy. ## 1. Research design ### Hypothesis Start with a falsifiable question, such as whether a precisely defined setup has positive net average results under stated execution assumptions. The question should identify instrument, horizon and comparison rather than merely ask whether a chart pattern works. ### Sample definition Define the sample before reviewing outcomes. Include dates, sessions, eligible instruments and exclusion rules. Reserving a later unseen segment creates a meaningful distinction between developing a rule and evaluating the chosen rule. ### Entry and exit rules Freeze the entry, exit and sizing definitions for the evaluation. If a rule changes, assign a new version and do not pretend earlier outcomes were generated by it. Keep a record of every material variant considered. ### No-trade cases Prevent hindsight by revealing observations sequentially or recording only information available at the decision time. A complete historical chart contains future pivots, revisions and final bar values that may not have been available then. ### Worked example A learner adjusts a stop after inspecting each losing example. The resulting record evaluates many retrospectively chosen rules, not one rule that could have been followed prospectively. ### Independent exercise Write a study protocol with a fixed sample, one baseline rule, a development period and an untouched evaluation period. Include what happens if the rule changes. My inputs and assumptions: My calculation or decision: Evidence that would change my conclusion: ## 2. Execution assumptions ### Spread A fill assumption must specify quote side, timing and available price. A signal at a closing price may only permit an order after that close. Entering at a price that disappeared before the decision creates an unrealistic result. ### Fees Model spread, commission, financing and slippage where relevant. Distinguish costs reflected in fill prices from separately charged amounts. Apply the same conventions to winners, losers and the baseline. ### Slippage OHLC bars do not always reveal whether a stop or target occurred first. Mark ambiguous cases and use a documented conservative assumption or finer data. Choosing the favorable sequence every time introduces bias. ### Intrabar ambiguity Capacity and order size affect execution. A small hypothetical order's fill assumption may not scale to a large order. Where volume or queue information is unavailable, state the limitation rather than claiming a precise executable result. ### Worked example A bar's high reaches the target and its low reaches the stop, but the intrabar sequence is unknown. Recording the target first solely because it produces a win manufactures favorable information. ### Independent exercise Describe two defensible treatments of this ambiguous bar and explain why the chosen treatment must be consistent. My inputs and assumptions: My calculation or decision: Evidence that would change my conclusion: ## 3. Evaluation ### Expectancy Report net expectancy, dispersion, drawdown and the number of observations rather than only total profit. Different measures reveal different weaknesses. A profitable average can coexist with large losses or an unstable sample. ### Drawdown Compare with a simple baseline under identical dates, costs and information timing. The baseline might be a simpler rule or no activity, depending on the question. It provides context for whether complexity adds useful evidence. ### Uncertainty Sensitivity analysis changes plausible assumptions such as costs, delays or neighboring parameters. A result that disappears under small reasonable changes is fragile. Do not call the most favorable stress test a conservative case. ### Decision to retain or reject Interpret small samples cautiously. Uncertainty depends on variability and dependence, not merely trade count. Several trades from the same event may contribute less independent evidence than their number suggests. ### Worked example A study reports 30 profit over ten trades but omits a 40 intratrade drawdown and all financing. The headline is insufficient to assess either net return or the path of risk. ### Independent exercise List the additional measurements needed and explain how a higher trade count could still fail to provide independent evidence. My inputs and assumptions: My calculation or decision: Evidence that would change my conclusion: ## 4. Reproducible study submission ### Archive the rule version Archive the exact rule version, data source and transformation steps. Another learner should be able to locate the same observations and understand how the decisions were generated. A screenshot collection alone is not a reproducible study. ### Record excluded observations and reasons Record every exclusion and its reason. Distinguish missing data from a rule-based no-trade decision. Counts should reconcile from raw observations through eligible cases to executed or simulated trades. ### Produce gross and net results Publish gross and net results with the cost bridge between them. A reviewer should see how each charge or assumption changes the result. Avoid a single unexplained adjustment that cannot be traced to trades. ### Have another learner reproduce the sample Ask another learner to reproduce a bounded part of the study before relying on the whole report. Disagreements can reveal unclear rules, timestamp errors or accounting differences. Resolve them with documented changes rather than silently editing outputs. ### Worked example The report contains 100 raw candidates, 15 predefined exclusions and 85 eligible cases, of which 60 trigger entries. These stages should reconcile; the remaining 25 eligible non-entries are not missing trades. ### Independent exercise Create a flow table for those counts and state what evidence must accompany the fifteen exclusions. My inputs and assumptions: My calculation or decision: Evidence that would change my conclusion: ## Course project Submit a rulebook, chronological trade sample and unseen validation results. ### Self-review rubric - Concepts and reasoning: 25% - Calculations, data and evidence: 30% - Process and risk controls: 25% - Limitations and communication: 20% Record one correction and one next practice task. This rubric is not automatically graded. ## Worked solutions ### Exercise 1 A changed rule returns to development under a new version. The untouched period should not be repeatedly inspected to choose parameters; once used for selection, it is no longer independent evidence for that selection. ### Exercise 2 Use finer timestamped data where available, or apply a predefined conservative rule and report the count of ambiguous cases. Consistency prevents outcome-driven selection; neither approach should conceal remaining uncertainty. ### Exercise 3 Reconcile costs, record marked-to-market drawdown, show outcome distribution and identify clustering. Trades driven by the same event or overlapping positions can be dependent; their count alone does not determine precision. ### Exercise 4 Show 100 raw, 15 excluded, 85 eligible, 60 entries and 25 non-entries. Each exclusion needs its predefined reason and supporting observation. The accounting of cases prevents selective disappearance of inconvenient examples. ## Further reading - https://scikit-learn.org/stable/modules/cross_validation.html - https://www.itl.nist.gov/div898/handbook/