Multiplicative Updates for L1-Regularized Linear and Logistic Regression
Description
Multiplicative update rules have proven useful in many areas
of machine learning. Simple to implement, guaranteed to converge,
they account in part for the widespread popularity of algorithms such
as nonnegative matrix factorization and Expectation-Maximization. In
this paper, we show how to derive multiplicative updates for problems in
L1-regularized linear and logistic regression. For L1–regularized linear regression,
the updates are derived by reformulating the required optimization
as a problem in nonnegative quadratic programming (NQP). The
dual of this problem, itself an instance of NQP, can also be solved using
multiplicative updates; moreover, the observed duality gap can be used
to bound the error of intermediate solutions. For L1–regularized logistic
regression, we derive similar updates using an iteratively reweighted least
squares approach. We present illustrative experimental results and describe
efficient implementations for large-scale problems of interest (e.g.,
with tens of thousands of examples and over one million features).
Categories
Top: Computer Science: Machine Learning: RegressionTop: Computer Science: Machine Learning: Linear Models
| Slides | |
| 0:00 | Multiplicative updates for L1-regularized regression |
| 0:14 | Trends in data analysis |
| 0:57 | How do we scale? |
| 1:42 | Searching for sparse models |
| 2:31 | An unexpected connection |
| 2:59 | This talk |
| 3:32 | Part I. Multiplicative updates |
| 3:34 | Nonnegative quadratic programming (NQP) |
| 4:38 | Matrix decomposition |
| 5:30 | Multiplicative update |
| 5:55 | Matrix decomposition (a) |
| 6:01 | Multiplicative update (a) |
| 6:45 | Fixed points |
| 7:36 | Attractive properties for NQP |
| 8:13 | Part II. Sparse regression |
| 8:20 | Linear regression |
| 8:50 | Regularization |
| 9:13 | L2 versus L1 |
| 9:58 | Reformulation as NQP |
| 10:15 | L2 versus L1 (a) |
| 10:32 | Reformulation as NQP (a) |
| 11:04 | L1 norm as NQP |
| 11:36 | Why reformulate? |
| 12:00 | Logistic regression |
| 12:41 | Part III. Experimental results |
| 12:54 | Convergence to sparse solution |
| 14:37 | Primal-dual convergence |
| 15:32 | Large-scale implementation |
| 17:18 | Discussion |
| 18:35 | Large-scale implementation (a) |
Lecture rating
| People found this lecture: | ||
| Worth seeing | ||
| because it is: | ||
| Valuable and informative | ||
| Well presented | ||
| Easily understandable | ||
| Acceptably recorded | ||
| You need to login to cast your vote. | ||
Report a problem or upload files
If you have found a problem with this lecture or would like to send us extra material, articles, exercises, etc., please use our ticket system to describe your request and upload the data.Enter your e-mail into the 'Cc' field, and we will keep you updated with your request's status.
Related content
Link this page
Would you like to put a link to this lecture on your homepage?Go ahead! Copy the HTML snippet !




