0.25
0.5
0.75
1.25
1.5
1.75
2
Net2Net: Accelerating Learning via Knowledge Transfer
Published on 2016-05-274120 Views
We introduce techniques for rapidly transferring the information stored in one neural net into another neural net. The main purpose is to accelerate the training of a significantly larger neural net.
Related categories
Presentation
Net2Net: Rapidly Transferring Knowledge between Large Networks00:00
Outline - 100:08
Outline - 200:10
Neural nets are getting larger ...00:11
Deep Learning: Ideal vs Reality01:04
Motivation01:57
Outline - 302:51
Possible ways to Deal with an Old Net - 102:54
Initial Attempt: Learning from Old Model03:54
Possible ways to Deal with an Old Net - 204:51
Net2Net Workflow05:08
The Obstacle: (Partial) Random Initialized Components in the Net05:37
Motivated Solution to the Problem06:22
Two Ways to Expand Model Capacity06:53
Function-Preserving Transformation for Wider Nets07:28
Function-Preserving Transformations for Deeper Nets (General Idea)08:22
Function-Preserving Transformations for Deeper Nets: Add Identity Layer08:56
Outline - 409:51
Experimental Setup10:10
Experiment Results for Net2WiderNet10:38
Experiment Results for Net2DeeperNet12:30
Exploring New Design Space13:29
Take-aways14:46
Thank you15:39