What you will learn
- Timestamps
- Time zones
- Missing data
- Duplicate records
Timestamps
Ingestion should retain raw provider messages or files with source identifiers and receipt times. A normalized database is useful, but it should not be the only record when transformations may need investigation.
Time zones
Define a schema for symbol, price type, quantity, timestamp and source. Reject or quarantine records that do not satisfy required fields. A schema change should not silently reinterpret old values.
Missing data
Track event time and receipt time separately. Delayed messages can arrive out of order. A research pipeline needs an explicit policy for late observations rather than assuming arrival order equals market order.
Duplicate records
Make repeated ingestion idempotent using stable identifiers or documented deduplication rules. Reprocessing the same file should not double volume or P&L. Preserve evidence of rejected duplicates and genuine repeated observations.
Worked example
A provider resends yesterday's file after a reconnect. An append-only importer without identity checks doubles the rows. The correct replay should leave the normalized dataset unchanged for already stored records.
Try it yourself
Design an ingestion key and describe what happens when the same key arrives with corrected content.
Show the worked solution
Use source and stable event identity where available, with version or correction handling. A changed record should be retained as a documented revision, not silently ignored as identical or added as an unrelated event.
Apply this to your course project
Deliver a versioned dataset and a quality report.
Keep the calculation inputs, assumptions and decisions with your work. Practical exercises are self-reviewed; the scored knowledge checks assess the questions shown, not an independent certification of practical competence.