# Trading System Operations and Controls Design safe demo operation and recovery procedures. Use this workbook alongside the course. Write your answers before opening the solutions. Practical work is self-reviewed; scored knowledge checks are in the Academy. ## 1. Operations ### Monitoring Operational monitoring should expose data freshness, connectivity, account state and task health. A process being alive does not prove that its inputs are current or its outputs are being accepted. ### Data freshness Alerts need an actionable condition, severity and owner. An alert that fires continuously without a defined response becomes background noise. Test delivery and escalation rather than assuming configuration equals receipt. ### Connection state Runbooks describe concrete steps for common failures using the actual system's state and permissions. Include what evidence to collect and when to stop. A generic instruction to restart everything can destroy useful context. ### Time synchronization Service objectives should distinguish availability from correctness. A system that responds quickly with stale data is not necessarily healthy. Choose measures that reflect the decisions the service supports. ### Worked example A dashboard updates its clock while its market feed has stopped. Process liveness looks normal, but source timestamps reveal that the decisions would use stale observations. ### Independent exercise Define separate liveness and freshness checks and state which dependent activity should be withheld when freshness fails. My inputs and assumptions: My calculation or decision: Evidence that would change my conclusion: ## 2. Controls ### Order reconciliation Controls should be placed where they can actually prevent or detect the relevant failure. A visual warning does not enforce a quantity limit if another path can submit orders. Test all authorized entry paths. ### Duplicate prevention A stop or disable mechanism needs a precise scope: new entries, order submission, strategy evaluation or all activity. Disabling new entries does not necessarily cancel working orders or close positions. State these differences clearly. ### Access management Permission separation reduces accidental activation. Research tools, monitoring views and execution services can require different capabilities. A read-only audit should not obtain trading authority merely for convenience. ### Audit records Control state must survive or be reconciled after restart where required. A local toggle that resets to enabled can defeat an intended halt. Verify persisted state and startup behavior explicitly. ### Worked example An operator disables a strategy, but a previously submitted limit order remains working and later fills. The disable action stopped new signals, not existing venue instructions. ### Independent exercise Write a control description that distinguishes disabling new decisions from cancelling orders and closing positions. My inputs and assumptions: My calculation or decision: Evidence that would change my conclusion: ## 3. Recovery ### Kill switch Recovery begins by identifying what state is authoritative and what local information may be stale. Retrieve current orders, positions and recent executions before replaying pending work. ### Restart safety Use stable identities and idempotent handling for repeated messages. A restart can redeliver events or leave a request outcome uncertain. Processing the same execution twice must not double its accounting effect. ### Backup and restore Restore dependencies in an order that respects their relationships. Starting decision logic before required data or account reconciliation is ready can create invalid actions even when every process eventually becomes healthy. ### Incident review Document rollback and forward-recovery choices. Rolling back software does not roll back market events. The system must reconcile actual account state with whichever behavior version is restored. ### Worked example A restart reloads a queued order request whose earlier execution succeeded but was not acknowledged locally. Blind replay can duplicate exposure; recovery must first establish the remote outcome. ### Independent exercise Describe a startup sequence that handles this uncertainty and define the evidence required before new entries are enabled. My inputs and assumptions: My calculation or decision: Evidence that would change my conclusion: ## 4. Operational rehearsal ### Reconcile fills after a restart A restart rehearsal should include fills arriving while the client is offline. Verify that recovered positions and accounting match the authoritative record and that no execution is applied twice. ### Handle partial dependency failure Partial dependency failure tests situations where some services remain healthy while a critical input fails. A system should not infer overall readiness from one successful health check. ### Rehearse emergency stop and recovery An emergency stop rehearsal should verify the exact documented scope and subsequent recovery. Confirm what remains open or working, who can act and how state is communicated. Do not assume every stop button has the same semantics. ### Document incident ownership and evidence Incident ownership and evidence should be established before stress. Keep timestamps, identities, decisions and observed state, then review the rehearsal's gaps. A successful drill is evidence for that scenario, not proof against every failure. ### Worked example The data feed fails while the broker connection remains healthy. Continuing to create entries because order submission still works confuses connectivity with valid decision inputs. ### Independent exercise Write a failure-injection drill for that case, including expected withholding behavior and recovery checks. My inputs and assumptions: My calculation or decision: Evidence that would change my conclusion: ## Course project Run an incident drill with reconciliation and a documented shutdown. ### Self-review rubric - Concepts and reasoning: 25% - Calculations, data and evidence: 30% - Process and risk controls: 25% - Limitations and communication: 20% Record one correction and one next practice task. This rubric is not automatically graded. ## Worked solutions ### Exercise 1 Liveness checks the service response; freshness checks the age of required source observations under a documented policy. Withhold decisions that require current data while preserving monitoring and recovery access. ### Exercise 2 Define each action separately, identify its permissions and acknowledgements, and show residual exposure after completion. Avoid a single reassuring label that conceals which states remain active. ### Exercise 3 Load persisted control state, connect required services, reconcile remote orders and fills against stable identities, resolve uncertain requests and verify data freshness. Enable dependent behavior only under the documented operating decision. ### Exercise 4 Interrupt the required feed in a controlled demo test, verify that dependent entries stop while account monitoring remains available, restore the feed and reconcile freshness and state before resuming under the runbook. ## Further reading - https://owasp.org/www-project-top-ten/ - https://www.nist.gov/cyberframework