event thumbnail image
The 7th International Symposium on Intelligent Data Analysis

Multiplicative Updates for L1-Regularized Linear and Logistic Regression

coauthor: Lawrence Saul, University of California, San Diego

Description

Multiplicative update rules have proven useful in many areas of machine learning. Simple to implement, guaranteed to converge, they account in part for the widespread popularity of algorithms such as nonnegative matrix factorization and Expectation-Maximization. In this paper, we show how to derive multiplicative updates for problems in L1-regularized linear and logistic regression. For L1–regularized linear regression, the updates are derived by reformulating the required optimization as a problem in nonnegative quadratic programming (NQP). The dual of this problem, itself an instance of NQP, can also be solved using multiplicative updates; moreover, the observed duality gap can be used to bound the error of intermediate solutions. For L1–regularized logistic regression, we derive similar updates using an iteratively reweighted least squares approach. We present illustrative experimental results and describe efficient implementations for large-scale problems of interest (e.g., with tens of thousands of examples and over one million features).

You might be experiencing some problems with Your Video player.
Slides
0:00 Multiplicative updates for L1-regularized regression
0:14 Trends in data analysis
0:57 How do we scale?
1:42 Searching for sparse models
2:31 An unexpected connection
2:59 This talk
3:32 Part I. Multiplicative updates
3:34 Nonnegative quadratic programming (NQP)
4:38 Matrix decomposition
5:30 Multiplicative update
5:55 Matrix decomposition (a)
6:01 Multiplicative update (a)
6:45 Fixed points
7:36 Attractive properties for NQP
8:13 Part II. Sparse regression
8:20 Linear regression
8:50 Regularization
9:13 L2 versus L1
9:58 Reformulation as NQP
10:15 L2 versus L1 (a)
10:32 Reformulation as NQP (a)
11:04 L1 norm as NQP
11:36 Why reformulate?
12:00 Logistic regression
12:41 Part III. Experimental results
12:54 Convergence to sparse solution
14:37 Primal-dual convergence
15:32 Large-scale implementation
17:18 Discussion
18:35 Large-scale implementation (a)

Lecture rating

People found this lecture:
Worth seeing
because it is:
 Valuable and informative
Well presented
Easily understandable
Acceptably recorded
You need to login to cast your vote.

Report a problem or upload files

If you have found a problem with this lecture or would like to send us extra material, articles, exercises, etc., please use our ticket system to describe your request and upload the data.
Enter your e-mail into the 'Cc' field, and we will keep you updated with your request's status.

Link this page

Would you like to put a link to this lecture on your homepage?
Go ahead! Copy the HTML snippet !

Write your own review or comment: