What you will learn
- Predeclare annotations
- Measure observer disagreement
- Count failed examples
- Compare to a simple entry baseline
Predeclare annotations
Write annotation rules before revealing the later chart. Use a fixed sample and preserve uncertain cases. This reduces the temptation to revise definitions around outcomes that are already visible.
Measure observer disagreement
Ask another learner to apply the same rules without seeing your labels. Disagreement identifies ambiguity in the specification. Agreement improves reproducibility but does not establish that the pattern is profitable.
Count failed examples
Count failures and non-events as well as successful examples. A collection of textbook winners is useful for recognition but not for estimating performance. Maintain a clear inclusion rule for the evaluation sample.
Compare to a simple entry baseline
Compare the proposed rule against a simple baseline using consistent costs and timing. Added complexity needs to justify itself through evidence. If both methods perform similarly within uncertainty, the more elaborate labels may not add useful information.
Worked example
Two reviewers agree on only 12 of 20 setup labels. Before evaluating profitability, inspect the eight disagreements and identify which wording caused different classifications.
Try it yourself
Design a revision process that improves rule clarity without deleting the disputed examples from the evaluation record.
Show the worked solution
Preserve original labels, document the revised wording and apply it to a new reserved sample. Keep the disputed cases as evidence of the earlier ambiguity rather than pretending the final rule was always clear.
Apply this to your course project
Annotate a fixed sample using written swing and break rules, including ambiguous examples.
Keep the calculation inputs, assumptions and decisions with your work. Practical exercises are self-reviewed; the scored knowledge checks assess the questions shown, not an independent certification of practical competence.