# Machine Learning for Market Research Evaluate predictive models under temporal validation and limited evidence. Use this workbook alongside the course. Write your answers before opening the solutions. Practical work is self-reviewed; scored knowledge checks are in the Academy. ## 1. Problem formulation ### Target definition Define a target that matches a decision, such as a later return over a specified horizon or a probability of an event. The target's construction determines when its label becomes available and whether observations overlap. ### Decision horizon Decision horizon and execution horizon must be compatible. A model predicting a move too brief for the assumed order latency may have little practical value. Prediction quality and executable usefulness are separate questions. ### Class imbalance Class imbalance can make accuracy misleading. A model predicting the common class every time may score well while missing the cases that matter. Compare with simple baselines and metrics suited to the decision. ### Label availability Labels must not contain future information that leaks into features. Store each label's information interval. A training example's row date alone may not show that its target uses returns from a later evaluation period. ### Worked example If 95 of 100 examples are class zero, always predicting zero yields 95% accuracy but detects none of the five class-one cases. Whether this is useful depends on the decision, not the headline accuracy alone. ### Independent exercise Specify a baseline and at least two measurements that expose this model's limitations. My inputs and assumptions: My calculation or decision: Evidence that would change my conclusion: ## 2. Features ### Point-in-time construction Features should be constructed from data available at prediction time. Revised releases, finalized bars and full-period statistics can leak later information even when the feature name sounds historical. ### Scaling within training data Fit scaling and other learned preprocessing within the training portion. Apply the fitted transformation unchanged to the evaluation portion. A pipeline helps organize this procedure but still needs correct temporal splits and input definitions. ### Missingness Missingness may carry information, but its treatment must be deliberate. An imputer fitted across the full dataset can leak distribution information. Record whether missing values reflect closures, outages or unavailable releases. ### Feature leakage Feature leakage can be subtle: a post-trade tag, final holding period or outcome-dependent selection field may encode the answer. Audit the provenance and availability of every input, not only obvious future-price columns. ### Worked example A model predicts whether a trade wins using its final holding duration. That duration depends on the exit path and was not known at entry, so the apparently strong feature is unavailable for the stated prediction. ### Independent exercise Design a feature audit table for name, source, transformation, availability time and allowed prediction use. My inputs and assumptions: My calculation or decision: Evidence that would change my conclusion: ## 3. Model development ### Simple baselines Start with simple baselines such as a constant, linear model or existing rule appropriate to the task. Complexity should demonstrate incremental value under the same evaluation design, not merely improve training fit. ### Regularization Regularization constrains fitted complexity under a chosen penalty or procedure. It can reduce sensitivity but does not eliminate leakage, selection or changing market behavior. Hyperparameters remain part of the research choices. ### Hyperparameter search accounting Account for the search over models and parameters. Repeatedly trying alternatives and reporting only the best evaluation score makes that score optimistic as independent evidence. Separate development selection from final evaluation. ### Calibration Calibration concerns whether predicted probabilities align with observed frequencies under the evaluated sample. A model can rank cases well but produce poorly calibrated probabilities. Decision thresholds should use the relevant property rather than assume all scores are literal probabilities. ### Worked example A classifier assigns 0.9 to many cases, but only about half are positive in a reserved comparable sample. Its ranking may still contain information, but the numerical scores should not be interpreted as reliable 90% probabilities. ### Independent exercise Explain how you would compare ranking and calibration separately without tuning on the final evaluation set. My inputs and assumptions: My calculation or decision: Evidence that would change my conclusion: ## 4. Evaluation ### Temporal splits Temporal splits should reflect how the model would be updated and used. Avoid training on future periods for a prospective earlier prediction. Store split definitions so the evaluation can be reproduced. ### Purging overlapping labels where applicable Purging or gaps may be needed when labels overlap across split boundaries. Determine the overlap from target intervals rather than applying a universal arbitrary gap. Record removed observations and the reason. ### Economic costs Convert predictions into a specified decision policy with costs, delays and risk. A statistical score improvement may not survive trading frictions. Compare the full policy with the baseline, not only the classifier metric. ### Drift and model retirement Monitor drift and define review or retirement conditions. A historical model can become inappropriate when inputs, target relationships or implementation change. Monitoring is evidence gathering, not a guarantee of detecting failure before losses. ### Worked example A model improves classification score but generates many more trades whose costs exceed the added gross gains. The statistical improvement does not establish a better net decision policy. ### Independent exercise Write a final model report containing prediction metrics, policy results, costs, temporal design and limitations. My inputs and assumptions: My calculation or decision: Evidence that would change my conclusion: ## Course project Compare a simple baseline with one model using a frozen temporal evaluation plan. ### Self-review rubric - Concepts and reasoning: 25% - Calculations, data and evidence: 30% - Process and risk controls: 25% - Limitations and communication: 20% Record one correction and one next practice task. This rubric is not automatically graded. ## Worked solutions ### Exercise 1 Use the always-zero baseline explicitly, report the confusion table and class-specific recall or precision as appropriate, and connect errors to decision costs. A higher score must be evaluated against the same information and sample. ### Exercise 2 For each feature, show how it is produced and when it is knowable. Reject or redesign the final-duration feature for entry-time prediction. Keep the audit with the model version so later additions receive the same review. ### Exercise 3 Choose appropriate ranking and calibration measures during development, fit any calibration procedure on allowed data, then evaluate the frozen complete process on the reserved set. Report both and avoid recalibrating to the final outcomes while calling them untouched. ### Exercise 4 Report the frozen model and decision mapping, baseline comparison, net outcomes, uncertainty, split and overlap treatment, search history and monitoring assumptions. Keep unproven generalization claims out of the conclusion. ## Further reading - https://scikit-learn.org/stable/modules/cross_validation.html - https://scikit-learn.org/stable/common_pitfalls.html