23rd Annual Conference on Neural Information Processing Systems (NIPS), Vancouver 2009

Relative Entropy

author: Sergio Verdu, Princeton University
published: Jan. 25, 2010, recorded: December 2009, views: 2966

Slides

Slides
0:00	relative entropy
2:45	definition
3:35	alternative representation
3:52	discrete case
4:27	continuous case
4:59	units - 1
5:21	units - 2
5:53	D(P//Q)
7:27	first appearance
8:20	D(P//Q)+D(Q//P)
9:18	first definition - 1
10:15	first definition - 2
11:36	mutual information
13:52	entropy
15:09	conditional relative entropy
17:34	mutual information - 1
20:35	mutual information - 2
21:23	gibbs inequality - 1
22:50	gibbs inequality - 2
24:32	processing reduces D(P//Q)
26:25	conditioning increases D(P//Q)
27:24	chain rule
28:23	first definition
29:15	differential entropy - 1
30:01	differential entropy - 2
30:20	differential entropy - 3
30:40	differential entropy - 4
32:37	nongaussianness
33:47	maximum entropy: rationale
36:17	monotonic decrease of nongaussianness
39:24	memory
40:19	variational representation
41:23	parallelogram law
42:41	pythagoras
44:37	D(P//Q)
45:33	large deviations
50:20	law of large numbers
52:02	lossless data compression
53:55	channel capacity
55:21	estimation
56:07	mmse estimation
56:24	mmse estimator in AWGN - 1
56:51	mmse estimator in AWGN - 2
57:01	mmse estimator in AWGN - 3
57:06	mmse notation
57:17	mismatched estimation
57:52	mismatched estimation: example: Q Gaussian
58:26	relative entropy and mmse

Report a problem or upload files

If you have found a problem with this lecture or would like to send us extra material, articles, exercises, etc., please use our ticket system to describe your request and upload the data.
Enter your e-mail into the 'Cc' field, and we will keep you updated with your request's status.

Lecture popularity: You need to login to cast your vote.

Description

An overview of relative entropy (aka Kulback-Leibler divergence, etc.) and its multiple appearances in information theory, probability, and statistics, including recent results by the speaker.