Characterizing Training Dynamics for Finite-width Deep Neural Networks

Non-asymptotics bounds on the weights and activations of finite-width neural networks and results suggesting gradients have low rank.