0.25
0.5
0.75
1.25
1.5
1.75
2
Net2Net: Accelerating Learning via Knowledge Transfer
Published on May 27, 20164111 Views
We introduce techniques for rapidly transferring the information stored in one neural net into another neural net. The main purpose is to accelerate the training of a significantly larger neural net.
Related categories
Chapter list
Net2Net: Rapidly Transferring Knowledge between Large Networks00:00
Outline - 100:08
Outline - 200:10
Neural nets are getting larger ...00:11
Deep Learning: Ideal vs Reality01:04
Motivation01:57
Outline - 302:51
Possible ways to Deal with an Old Net - 102:54
Initial Attempt: Learning from Old Model03:54
Possible ways to Deal with an Old Net - 204:51
Net2Net Workflow05:08
The Obstacle: (Partial) Random Initialized Components in the Net05:37
Motivated Solution to the Problem06:22
Two Ways to Expand Model Capacity06:53
Function-Preserving Transformation for Wider Nets07:28
Function-Preserving Transformations for Deeper Nets (General Idea)08:22
Function-Preserving Transformations for Deeper Nets: Add Identity Layer08:56
Outline - 409:51
Experimental Setup10:10
Experiment Results for Net2WiderNet10:38
Experiment Results for Net2DeeperNet12:30
Exploring New Design Space13:29
Take-aways14:46
Thank you15:39