Skip connections: making depth free
Deep plain networks train worse than shallow ones. The cause is not overfitting, it is the path the gradient takes. A single plus sign changes that.
4 steps
175 XP
A free account is needed
Start the lesson →
Sources
- He, K., Zhang, X., Ren, S. & Sun, J. 2016 · Deep Residual Learning for Image Recognition · CVPR 2016
- He, K. et al. 2016 · Identity Mappings in Deep Residual Networks · ECCV 2016
- Zhang, H., Dauphin, Y. & Ma, T. 2019 · Fixup Initialization: Residual Learning Without Normalization · ICLR 2019
ML Academy · an interactive machine learning course that runs in your browser ·
All lessons