What you will learn
- Outlier policies
- Staleness
- Provenance
- Reproducible storage
Outlier policies
Quality checks should distinguish impossible values, suspicious values and valid extremes. Automatically deleting every large move can remove the very events needed for risk analysis. Quarantine and investigate rather than rewriting history for smoothness.
Staleness
Missingness should be characterized by source, interval and cause where known. A feed outage differs from a market closure. The downstream policy should follow the reason and study requirements.
Provenance
Cross-source comparisons require matched instruments, timestamps and price conventions. Disagreement can reveal a defect, but it can also reflect genuine venue differences. Avoid declaring one source wrong solely because another differs.
Reproducible storage
Quality reports should include counts, affected periods and impact on derived results. A pass/fail badge without details is insufficient when the remaining gaps affect a strategy's trading window.
Worked example
A dataset has complete daily closes but missing ticks during major releases. It may support a daily study while being unsuitable for measuring first-second event execution.
Try it yourself
Write a fitness-for-purpose note for both uses of the dataset.
Show the worked solution
State the available resolution and gaps. Daily-close analysis may remain possible with appropriate checks, while fine event-execution claims are unsupported. Quality depends on the question, not one universal dataset label.
Apply this to your course project
Deliver a versioned dataset and a quality report.
Keep the calculation inputs, assumptions and decisions with your work. Practical exercises are self-reviewed; the scored knowledge checks assess the questions shown, not an independent certification of practical competence.