	Log in \| Register \|

Machine Learning Summer School 2003 - Tuebingen

Some Mathematical Tools for Machine Learning

author: Chris Burges, Microsoft Research

Description

Lecture contains:

Lagrange multipliers: * Lagrange the Mathematician * Lagrange multipliers: an indirect approach can be easier * Multiple Equality Constraints * Multiple Inequality Constraints * Two points on a d-sphere * The Largest Parallelogram * Resource allocation * A convex combination of numbers is maximized by choosing the largest * The Isoperimetric problem * For fixed mean and variance, which univariate distribution has maximum entropy? * An exact solution for an SVM living on a simplex
Notes on some Basic Statistics * Probabilities can be Counter-Intuitive (Simpson's paradox; the Monty Hall puzzle) * IID-ness: Measurement Error decreases as 1/sqrt{n} * Correlation versus Independence * The Ubiquitous Gaussian: o Product of Gaussians is Gaussian o Convolution of two Gaussians is a Gaussian o Projection of a Gaussian is a Gaussian o Sum of Gaussian random variables is a Gaussian random variables o Uncorrelated Gaussian variables are also independent o Maximum Likelihood Estimates for mean and covariance (prove required matrix identities) o Aside: For 1-dim Laplacian, max. likelihood gives the median * Using cumulative distributions to derive densities
Principal Component Analysis and Generalizations * Ordering by Variance * Does Grouping Change Things? * PCA Decorrelates the Samples * PCA gives Reconstruction with Minimal Mean Squared Error * PCA preserves Mutual Information on Gaussian data * PCA directions lie in the span of the data * PCA: second order moments only * The Generalized Rayleigh Quotient o Non-orthogonal principal directions o OPCA o Fisher Linear Discriminant o Multiple Discriminant Analysis
Elements of Functional Analysis * High Dimensional Spaces * Is Winning Transitive? * Most of the Volume is Near the Surface: Cubes * Spheres in n-dimensions * Banach Spaces, Hilbert Spaces, Compactness * Norms * Useful Inequalities (Minkowski and Holder) * Vector Norms * Matrix Norms * The Hamming Norm * L1, L2, L_infty norms - is L0 a norm? * Example: Using a Norm as a Constraint in Kernel Algorithms These are lectures on some fundamental mathematics underlying many approaches and algorithms in machine learning. They are not about particular learning algorithms; they are about the basic concepts and tools upon which such algorithms are built. Often students feel intimidated by such material: there is a vast amount of "classical mathematics", and it can be hard to find the wood for the trees. The main topics of these lectures are Lagrange multipliers, functional analysis, some notes on matrix analysis, and convex optimization. I've concentrated on things that are often not dwelt on in typical CS coursework. Lots of examples are given; if it's green, it's a puzzle for the student to think about. These lectures are far from complete: perhaps the most significant omissions are probability theory, statistics for learning, information theory, and graph theory. I hope eventually to turn all this into a series of short tutorials. Please let me know of any errors, etc.

(from Chris Burges homepage : http://research.microsoft.com/~cburges )

Categories

Top: Computer Science: Machine Learning

You might be experiencing some problems with Your Video player.

*Slides*
0:00	An indirect approach can be easieAn indirect approach can be easier
0:51	One equality constraint
2:22	One equality constraint, cont.
3:01	Multiple equality constraints
4:56	One inequality constraiOne inequality constraint
7:17	Multiple inequality constraints
9:15	A simple A simple example
11:44	Another simple example
13:08	Simple exerciseSimple exercises
14:03	Resource allocation
16:48	A variational probleA variational problem
19:23	Which univariate distribution has max entropy?
21:06	Which univariate distribution has maWhich univariate distribution has max entropy?
21:33	Max Entropy for Discrete Distribn + Linear Constraints
23:42	Basic Concepts in
23:57	Bibliography
24:13	What is a FielWhat is a Field?
25:03	Field : Examples
26:09	How Many Fields Are There?
27:21	What is a Vector Space?
28:24	Vector Spaces: Field MattersVector Spaces: Field Matters!
29:33	Vector Spaces: More Examples
31:16	What is an Inner ProduWhat is an Inner Product?
32:17	Inner Product: Examples
33:26	Inner Product: TracInner Product: Trace
34:26	Inner Product is General

Lecture rating

People found this lecture:
Worth seeing
because it is:
Valuable and informative
Well presented
Easily understandable
Acceptably recorded
You need to login to cast your vote.

Report a problem or upload files

If you have found a problem with this lecture or would like to send us extra material, articles, exercises, etc., please use our ticket system to describe your request and upload the data.
Enter your e-mail into the 'Cc' field, and we will keep you updated with your request's status.