On Primal and Dual Sparsity of Markov Networks

author: Jun Zhu, Department of Computer Science and Technology, Tsinghua University
published: Sept. 17, 2009,   recorded: June 2009,   views: 114

Report a problem or upload files

If you have found a problem with this lecture or would like to send us extra material, articles, exercises, etc., please use our ticket system to describe your request and upload the data.
Enter your e-mail into the 'Cc' field, and we will keep you updated with your request's status.

Description

Sparsity is a desirable property in high dimensional learning. The $\ell_1$-norm regularization can lead to primal sparsity, while max-margin methods achieve dual sparsity; but achieving both in a single structured prediction model remains difficult. This paper presents an $\ell_1$-norm max-margin Markov network ($\ell_1$-M$^3$N), which enjoys both primal and dual sparsity, and analyzes its connections to the Laplace max-margin Markov network (LapM$^3$N), which inherits the dual sparsity of max-margin models but is pseudo-primal sparse. We show that $\ell_1$-M$^3$N is an extreme case of LapM$^3$N when the regularization constant is infinity. We also show an equivalence between $\ell_1$-M$^3$N and an adaptive M$^3$N, from which we develop a robust EM-style algorithm for $\ell_1$-M$^3$N. We demonstrate the advantages of the simultaneously (pseudo-) primal and dual sparse models over the ones which enjoy either primal or dual sparsity on both synthetic and real data sets.