# Robust Validation and Model Risk Evaluate research choices without contaminating the holdout. Use this workbook alongside the course. Write your answers before opening the solutions. Practical work is self-reviewed; scored knowledge checks are in the Academy. ## 1. Bias control ### Look-ahead Lookahead occurs when a decision uses information unavailable at its timestamp. It can enter through final bar values, revised releases, future pivots or dataset-wide transformations. Audit availability, not merely the date label attached to a row. ### Survivorship Survivorship and selection arise when the sample contains only entities or strategies that remained available or attractive. Retain the inclusion process and rejected variants so the observed result is not mistaken for an unselected experiment. ### Parameter search Preprocessing can leak information when scaling, imputation or feature selection is fitted on evaluation data. Fit learned transformations on the allowed training portion and apply them to later data under the frozen procedure. ### Repeated holdout use Repeatedly checking an evaluation set to choose changes makes it part of development. Label that use honestly and obtain separate evidence for the selected final procedure rather than continuing to call the reused set untouched. ### Worked example A scaler uses the mean of the entire dataset, including future evaluation observations. Even though the prediction model is fitted only on earlier rows, its inputs have already incorporated later information. ### Independent exercise Specify a corrected transformation pipeline and explain how to apply it during each temporal evaluation fold. My inputs and assumptions: My calculation or decision: Evidence that would change my conclusion: ## 2. Validation ### Walk-forward design Temporal validation preserves the order relevant to deployment. Training on later periods to predict earlier periods may answer a different question. Use folds that reflect the intended update and evaluation sequence. ### Purging where relevant Walk-forward designs repeatedly fit on an allowed past window and evaluate later observations. Window length, update frequency and parameter selection are part of the design and should be specified before comparing results. ### Sensitivity surfaces Overlapping labels or positions can connect training and evaluation observations. Purging or gaps may be appropriate depending on the target horizon and information overlap. The size and purpose of any gap should follow the data construction, not a magic universal number. ### Cost stress Compare with simple baselines under the same folds, costs and information set. A complex model's raw score is difficult to interpret without knowing whether a simpler procedure achieves similar performance with fewer assumptions. ### Worked example A target uses the next five bars. Training labels near the split contain returns from the evaluation period. A simple row-date split does not by itself remove that overlap. ### Independent exercise Describe how to identify and remove the overlapping training labels while preserving a record of excluded observations. My inputs and assumptions: My calculation or decision: Evidence that would change my conclusion: ## 3. Governance ### Frozen rules Governance defines who can approve a candidate and what evidence is required. Separate research exploration from acceptance criteria. A successful demonstration should not automatically change operating behavior. ### Independent review Record the model's intended use, limitations and dependencies. A validation result for one instrument, horizon or cost setting does not automatically transfer to another. Changes in use can require new evaluation. ### Model limitations Monitor data quality and behavior after implementation using predefined measures. Drift detection is not a guarantee of timely failure prediction. Define what happens when monitoring itself becomes unavailable. ### Retirement criteria Retirement criteria identify when a model should stop being used or return to research. They may involve invalid assumptions, persistent operational defects or a failed evidence review. Preserve outcomes and records even when the model is retired. ### Worked example A model validated on daily data is moved to intraday trading without new execution assumptions. The original validation does not cover the new latency, spread and observation regime. ### Independent exercise Write a change-of-use review identifying which evidence can transfer and which must be regenerated. My inputs and assumptions: My calculation or decision: Evidence that would change my conclusion: ## 4. Independent validation review ### Reproduce the frozen baseline Independent validation starts by reproducing the frozen baseline and checking that data, code and configuration match the submitted record. A reviewer should not begin by optimizing the candidate to improve its results. ### Challenge fill and cost assumptions Challenge fill, fee and delay assumptions with defensible alternatives. Report which assumptions materially change the conclusion. A candidate that survives one favorable model is not thereby robust to realistic implementation uncertainty. ### Inspect selection across many trials Inspect the full search history where available. The number and diversity of tried variants affect how impressive the selected result is. Hidden experiments limit the strength of validation conclusions. ### Define rejection criteria before reviewing results Define rejection criteria before reviewing the final evaluation. A reviewer can conclude insufficient evidence rather than force a binary claim that the strategy is permanently good or permanently useless. ### Worked example A candidate's positive net result becomes negative under a modest documented delay that the operating system commonly experiences. The discrepancy directly challenges feasibility and needs resolution before acceptance. ### Independent exercise Write a validation finding with observation, impact, required correction and a bounded retest. My inputs and assumptions: My calculation or decision: Evidence that would change my conclusion: ## Course project Create a preregistered validation protocol and report rejected variants. ### Self-review rubric - Concepts and reasoning: 25% - Calculations, data and evidence: 30% - Process and risk controls: 25% - Limitations and communication: 20% Record one correction and one next practice task. This rubric is not automatically graded. ## Worked solutions ### Exercise 1 Fit the scaler and other learned preprocessing only on each fold's training portion, freeze those fitted objects and transform the corresponding later evaluation portion. Repeat the complete fitting process inside each fold without using its evaluation targets or distribution to choose the transformation. ### Exercise 2 Associate each label with its information interval, remove training examples whose target intervals overlap the evaluation information window under the chosen design, and log the exclusions. The policy depends on the target definition and should be applied consistently. ### Exercise 3 Basic code components may remain reusable, but timing, data quality, costs, labels and decision performance need review for the new use. Document the boundary instead of treating the old approval as universal. ### Exercise 4 State the tested delay and changed result, explain that the original execution assumption was not supported, require a realistic timing model and rerun the frozen comparison. Avoid searching for unrelated parameters merely to recover the old headline. ## Further reading - https://scikit-learn.org/stable/modules/cross_validation.html - https://scikit-learn.org/stable/common_pitfalls.html