What you will learn
- Practical significance
- Multiple testing
- Sensitivity
- Evidence thresholds
Practical significance
A decision threshold should relate to the practical question, such as whether net benefit survives costs and uncertainty. Statistical significance alone does not establish economic usefulness or executable scale.
Multiple testing
Testing many variants increases the chance of selecting an apparently strong result by chance. Preserve the search history and use a validation design that accounts for selection. A single final p-value cannot erase an undisclosed search.
Sensitivity
Effect size measures the magnitude of the difference, while uncertainty describes precision. A tiny estimated advantage may be irrelevant after implementation costs even if a large sample makes it statistically distinguishable from zero.
Evidence thresholds
A no-go decision is a legitimate research result. Define what evidence would reject the hypothesis before inspection, and record negative findings. This reduces pressure to keep changing the analysis until it produces an attractive answer.
Worked example
A rule shows a tiny gross advantage of 0.02 per trade while realistic costs are 0.10. Even a precisely estimated gross advantage would not establish positive net economics.
Try it yourself
Write a decision rule that considers both net effect and uncertainty. State how repeated variant searches should be reported.
Show the worked solution
Require the predefined net measure to meet the stated practical criterion under an appropriate uncertainty analysis, and disclose all material variants and selection stages. The exact criterion belongs to the research protocol, not a retrospective justification.
Apply this to your course project
Write a performance report with confidence intervals and dependence caveats.
Keep the calculation inputs, assumptions and decisions with your work. Practical exercises are self-reviewed; the scored knowledge checks assess the questions shown, not an independent certification of practical competence.