What you will learn
- Reconcile fills after a restart
- Handle partial dependency failure
- Rehearse emergency stop and recovery
- Document incident ownership and evidence
Reconcile fills after a restart
A restart rehearsal should include fills arriving while the client is offline. Verify that recovered positions and accounting match the authoritative record and that no execution is applied twice.
Handle partial dependency failure
Partial dependency failure tests situations where some services remain healthy while a critical input fails. A system should not infer overall readiness from one successful health check.
Rehearse emergency stop and recovery
An emergency stop rehearsal should verify the exact documented scope and subsequent recovery. Confirm what remains open or working, who can act and how state is communicated. Do not assume every stop button has the same semantics.
Document incident ownership and evidence
Incident ownership and evidence should be established before stress. Keep timestamps, identities, decisions and observed state, then review the rehearsal's gaps. A successful drill is evidence for that scenario, not proof against every failure.
Worked example
The data feed fails while the broker connection remains healthy. Continuing to create entries because order submission still works confuses connectivity with valid decision inputs.
Try it yourself
Write a failure-injection drill for that case, including expected withholding behavior and recovery checks.
Show the worked solution
Interrupt the required feed in a controlled demo test, verify that dependent entries stop while account monitoring remains available, restore the feed and reconcile freshness and state before resuming under the runbook.
Apply this to your course project
Run an incident drill with reconciliation and a documented shutdown.
Keep the calculation inputs, assumptions and decisions with your work. Practical exercises are self-reviewed; the scored knowledge checks assess the questions shown, not an independent certification of practical competence.