NIPS '07 Workshop on Efficient Machine Learning

Who is Afraid of Non-Convex Loss Functions?

author: Yann LeCun, New York University

Description

The NIPS community has suffered of an acute convexivitis epidemic:
- ML applications seem to have trouble moving beyond logistic regression, SVMs, and exponential-family graphical models;
- For a new ML model, convexity is viewed as a virtue;
- Convexity is sometimes a virtue;
- But it is often a limitation.
ML theory has essentially never moved beyond convex models - the same way control theory has not really moved beyond linear systems.

Categories

Top: Computer Science: Machine Learning: Computational Learning Theory
Top: Computer Science: Machine Learning: Kernel Methods: Support Vector Machines

You might be experiencing some problems with Your Video player.

*Slides*
0:00	Who is afraid of nonconvex loss functions?
0:22	Convex Shmonvex - 1
3:54	Convex Shmonvex - 2
6:08	To solve complicated AI taks, ML will have to go nonconvex
8:46	Best results on MNIST (from raw images: no preprocessing)
9:11	Convexity is overrated
9:16	Normalized-uniform set: Error rates
10:15	Normalized-uniform set: Learning times
11:24	Experiment 2: Jittered-cluttered dataset
12:06	Jittered-cluttered dataset
12:33	Optimization algorithms for learning
12:49	Theoretical guarantees are overrated
12:52	Jittered-cluttered dataset
14:14	Theoretical guarantees are overrated
14:18	The visual system is “deep” and learned
14:19	Do we really need deep architectures?
14:21	Why are deep architectures more efficient?
14:25	Strategies (after Hinton 2007)
17:02	Deep learning is hard? - 1
19:20	- Questions
23:30	Shallow models
25:38	The problem with non-convex learning
27:30	Backprop learning is not as bad as it seems
28:27	Convolutional networks
29:18	“Only Yann can do it” (NOT!)
31:54	The basic idea for training deep feature hierarchies
32:10	The right tools: Automatic differentiation
36:44	A stochastic diagonal Levenberg-Marquardt method - 1
37:49	A stochastic diagonal Levenberg-Marquardt method - 2
38:44	On-line computation of Ψ
38:52	Recipe
40:10	Estimates of optimal learning rate - 1
40:40	Estimates of optimal learning rate - 2
40:52	- Questions
57:54	- Questions

Lecture rating

People found this lecture:
Worth seeing
because it is:
Valuable and informative
Well presented
Easily understandable
Acceptably recorded
You need to login to cast your vote.

Report a problem or upload files

If you have found a problem with this lecture or would like to send us extra material, articles, exercises, etc., please use our ticket system to describe your request and upload the data.
Enter your e-mail into the 'Cc' field, and we will keep you updated with your request's status.