Weight initialisation: networks lost before training starts
If the initial scale of the weights is wrong, the signal either fades or explodes with depth. The network in this lesson learns nothing at all because of a single number.
4 steps
175 XP
A free account is needed
Start the lesson →
Sources
- Glorot, X. & Bengio, Y. 2010 · Understanding the Difficulty of Training Deep Feedforward Neural Networks · AISTATS 2010
- He, K., Zhang, X., Ren, S. & Sun, J. 2015 · Delving Deep into Rectifiers · ICCV 2015
- Goodfellow, I., Bengio, Y. & Courville, A. 2016 · Deep Learning, Bölüm 8.4 · MIT Press
ML Academy · an interactive machine learning course that runs in your browser ·
All lessons