Scaling up Natural Gradient by Sparsely Factorizing the Inverse Fisher Matrix

Second-order optimization methods, such as natural gradient, are difficult to apply to high-dimensional problems, because they require approximately solving large linear systems. We present FActorized