Stopping overfitting

Dropout, weight decay, early stopping. All three do the same thing: stop the model from memorising. In this lesson we measure all of them.

Sources

• Srivastava, N. et al. 2014 · Dropout: A Simple Way to Prevent Neural Networks from Overfitting · JMLR, 15, 1929–1958
• Krogh, A. & Hertz, J. 1992 · A Simple Weight Decay Can Improve Generalization · NeurIPS 1991
• Prechelt, L. 1998 · Early Stopping, But When? · Neural Networks: Tricks of the Trade
• Loshchilov, I. & Hutter, F. 2019 · Decoupled Weight Decay Regularization (AdamW) · ICLR 2019