Location: EU Supported » PASCAL - Pattern Analysis, Statistical Modelling and Computational Learning » NIPS '07 Workshop on Efficient Machine Learning

Who is Afraid of Non-Convex Loss Functions?

author: Yann LeCun, New York University
published: Dec. 29, 2007,   recorded: December 2007,   views: 1842
Categories
You might be experiencing some problems with Your Video player.

Slides

Slides
0:00 Who is afraid of nonconvex loss functions?
0:22 Convex Shmonvex - 1
3:54 Convex Shmonvex - 2
6:08 To solve complicated AI taks, ML will have to go nonconvex
8:46 Best results on MNIST (from raw images: no preprocessing)
9:11 Convexity is overrated
9:16 Normalized-uniform set: Error rates
10:15 Normalized-uniform set: Learning times
11:24 Experiment 2: Jittered-cluttered dataset
12:06 Jittered-cluttered dataset
12:33 Optimization algorithms for learning
12:49 Theoretical guarantees are overrated
12:52 Jittered-cluttered dataset
14:14 Theoretical guarantees are overrated
14:18 The visual system is “deep” and learned
14:19 Do we really need deep architectures?
14:21 Why are deep architectures more efficient?
14:25 Strategies (after Hinton 2007)
17:02 Deep learning is hard? - 1
19:20 - Questions
23:30 Shallow models
25:38 The problem with non-convex learning
27:30 Backprop learning is not as bad as it seems
28:27 Convolutional networks
29:18 “Only Yann can do it” (NOT!)
31:54 The basic idea for training deep feature hierarchies
32:10 The right tools: Automatic differentiation
36:44 A stochastic diagonal Levenberg-Marquardt method - 1
37:49 A stochastic diagonal Levenberg-Marquardt method - 2
38:44 On-line computation of Ψ
38:52 Recipe
40:10 Estimates of optimal learning rate - 1
40:40 Estimates of optimal learning rate - 2
40:52 - Questions
57:54 - Questions

Related content

Report a problem or upload files

If you have found a problem with this lecture or would like to send us extra material, articles, exercises, etc., please use our ticket system to describe your request and upload the data.
Enter your e-mail into the 'Cc' field, and we will keep you updated with your request's status.
Lecture popularity: You need to login to cast your vote.
Bibliography

Description

The NIPS community has suffered of an acute convexivitis epidemic:
- ML applications seem to have trouble moving beyond logistic regression, SVMs, and exponential-family graphical models;
- For a new ML model, convexity is viewed as a virtue;
- Convexity is sometimes a virtue;
- But it is often a limitation.
ML theory has essentially never moved beyond convex models - the same way control theory has not really moved beyond linear systems.

See Also:

Launch Windows Media PlayerLaunch in a standalone WM Player

WMedia PlayerSwitch to Windows Media Player

Download slides icon Download slides: eml07_lecun_wia_01.pdf (8.2 MB)

View slides View slides

Help icon Streaming Video Help

WebLink icon Windows Media Player Firefox Plugin - Download

  Join a Study Group

OpenStudy logo

You reached a lecture within the PASCAL NoE project video collection. Click on the logo and go to the Computer Science classroom on OpenStudy. Through this classroom, you can meet other students interested in the same problems and work together on assignments, ask each other questions or just discuss the topics of the lecture.

Link this page

Would you like to put a link to this lecture on your homepage?
Go ahead! Copy the HTML snippet !

Write your own review or comment:

make sure you have javascript enabled or clear this field: