What you will learn
- Simple baselines
- Regularization
- Hyperparameter search accounting
- Calibration
Simple baselines
Start with simple baselines such as a constant, linear model or existing rule appropriate to the task. Complexity should demonstrate incremental value under the same evaluation design, not merely improve training fit.
Regularization
Regularization constrains fitted complexity under a chosen penalty or procedure. It can reduce sensitivity but does not eliminate leakage, selection or changing market behavior. Hyperparameters remain part of the research choices.
Hyperparameter search accounting
Account for the search over models and parameters. Repeatedly trying alternatives and reporting only the best evaluation score makes that score optimistic as independent evidence. Separate development selection from final evaluation.
Calibration
Calibration concerns whether predicted probabilities align with observed frequencies under the evaluated sample. A model can rank cases well but produce poorly calibrated probabilities. Decision thresholds should use the relevant property rather than assume all scores are literal probabilities.
Worked example
A classifier assigns 0.9 to many cases, but only about half are positive in a reserved comparable sample. Its ranking may still contain information, but the numerical scores should not be interpreted as reliable 90% probabilities.
Try it yourself
Explain how you would compare ranking and calibration separately without tuning on the final evaluation set.
Show the worked solution
Choose appropriate ranking and calibration measures during development, fit any calibration procedure on allowed data, then evaluate the frozen complete process on the reserved set. Report both and avoid recalibrating to the final outcomes while calling them untouched.
Apply this to your course project
Compare a simple baseline with one model using a frozen temporal evaluation plan.
Keep the calculation inputs, assumptions and decisions with your work. Practical exercises are self-reviewed; the scored knowledge checks assess the questions shown, not an independent certification of practical competence.