Information Theory

author: David MacKay, University of Cambridge
published: Nov. 2, 2009,   recorded: August 2009,   views: 69946


Related Open Educational Resources

Related content

Report a problem or upload files

If you have found a problem with this lecture or would like to send us extra material, articles, exercises, etc., please use our ticket system to describe your request and upload the data.
Enter your e-mail into the 'Cc' field, and we will keep you updated with your request's status.
Lecture popularity: You need to login to cast your vote.

 Watch videos:   (click on thumbnail to launch)

Watch Part 1
Part 1 1:26:35
Watch Part 2
Part 2 1:35:39

See Also:

Download slides icon Download slides: mlss09uk_mackay_it.pdf (3.1┬áMB)

Help icon Streaming Video Help

Link this page

Would you like to put a link to this lecture on your homepage?
Go ahead! Copy the HTML snippet !

Reviews and comments:

Comment1 raghvendra, December 1, 2010 at 12:42 a.m.:

realy very good

Comment2 eliot, January 7, 2012 at 8:43 a.m.:

skillful lecturer

Comment3 Bart Alder, August 8, 2012 at 7:28 a.m.:


Comment4 mahdie, February 20, 2013 at 7:34 p.m.:

very good

Comment5 nuur, August 16, 2013 at 3:53 p.m.:


Comment6 Julien, April 16, 2018 at 7:59 p.m.:

I have issue with the example using cards. The correct answer is 1/2. Proof:

A card is a pair of colors:

c1 = (R, R)
c2 = (R, W)
c3 = (W, W)

We choose a card (x,y) at random.

I see red, so I know x = R. He then asks P(y = R | x = R)

P(y = R | x = R) = P(x = R, y = R) / P(x = R)
= p(card = c1) / p(card = c1 or card = c2)
= p(card = c1) / (p(card = c1) + p(card = c2))
= 1 / 2

Why does he say 2/3?

Comment7 Kris, September 13, 2020 at 3:05 p.m.:

Julien, if you still think your proof is correct 2 years later, then I hope the following exhaustive explanation convinces you (or other possibly confused readers) otherwise.

Using your notation (x=shown side,y=hidden side) and card labels, then as you say correctly from the product rule


You are also (accidentally?) correct that p(x=R,y=R)=2/6=1/3=p(card=c1). Where you misunderstand the problem is in not realizing that implicitly the orientations of the cards are also random. I can agree that this part of the problem is not explained well in the video (but see exercise 8.10 in MacKay's excellent ITILA book: This means that "unconditionally" (i.e. before seeing a red side, given this experimental setup) there are 4 possible unique outcomes for this "experiment": [x,y]={RR,WW,RW,WR}. Crucially, there are 2 ways in which outcome RR (and WW) can happen since c1 (c3) can be oriented in two possible ways, both giving the same outcome. On the other hand, RW (and WR) can only happen in one way for a specific orientation of c2. This means that the possible outcomes are not equally probable. We note that there are a total of possible 6 ways (3 cards x 2 orientations) that this experiment can generate an outcome. So for example p(x=R,y=R)=[# ways for RR]/(total # ways experiment can go]=2/6=1/3. If this doesn't make any sense, maybe this talk by McElreath is more pedagogical (

As such, you are making a common mistake in saying that p(x=R)=p(card=c1 or card=c2)=p(card=c1)+p(card=c2)=2/3. There are two ways to appreciate this. First, formally by the sum rule:


Where p(x=R,y=R)=2/6 because this outcome can be observed in two ways [both orientations of c1]. p(x=R,y=W)=1/6 because there is only one orientation of c2 (so 1/6 of the total possible ways) that could give this results.

Thereby formally p(y=R|x=R)=(1/3)/(1/2)=2/3

The simpler explanation is that (noting that the orientations are also random) given that there are 3 possible ways that could generate a red visible side [c1 oriented either way and c2 oriented a particular way]. Out of these 2 [c1 oriented either way] would have red on the other side. So p(y=R|x=R)=[#ways with R on hidden side and R on shown side]/[#ways with R on shown side].

Hope this was helpful.

Comment8 Albert Houstone, August 25, 2023 at 6:48 a.m.:

This is so informative! Thank you,

Write your own review or comment:

make sure you have javascript enabled or clear this field: