The competition illusion: how much can you trust a leaderboard
The top of a leaderboard is almost always worse than it looks. That is not cheating but a side effect of the act of selection itself.
4 steps
225 XP
A free account is needed
Start the lesson →
Sources
- Dwork, C. et al. 2015 · The Reusable Holdout: Preserving Validity in Adaptive Data Analysis · Science 349(6248)
- Recht, B. et al. 2019 · Do ImageNet Classifiers Generalize to ImageNet? · ICML 2019
- Blum, A. & Hardt, M. 2015 · The Ladder: A Reliable Leaderboard for Machine Learning Competitions · ICML 2015
- Demšar, J. 2006 · Statistical Comparisons of Classifiers over Multiple Data Sets · JMLR 7
ML Academy · an interactive machine learning course that runs in your browser ·
All lessons