## Information Theory

author: David MacKay,
University of Cambridge

published: Nov. 2, 2009, recorded: August 2009, views: 69919

published: Nov. 2, 2009, recorded: August 2009, views: 69919

# Slides

# Related content

# Report a problem or upload files

If you have found a problem with this lecture or would like to send us extra material, articles, exercises, etc., please use our**to describe your request and upload the data.**

__ticket system__*Enter your e-mail into the 'Cc' field, and we will keep you updated with your request's status.*

# Link this page

Would you like to put a link to this lecture on your homepage?

Go ahead! Copy the HTML snippet !

## Reviews and comments:

raghvendra, December 1, 2010 at 12:42 a.m.:realy very good

eliot, January 7, 2012 at 8:43 a.m.:skillful lecturer

Bart Alder, August 8, 2012 at 7:28 a.m.:Excellent.

mahdie, February 20, 2013 at 7:34 p.m.:very good

nuur, August 16, 2013 at 3:53 p.m.:test

Julien, April 16, 2018 at 7:59 p.m.:I have issue with the example using cards. The correct answer is 1/2. Proof:

A card is a pair of colors:

c1 = (R, R)

c2 = (R, W)

c3 = (W, W)

We choose a card (x,y) at random.

I see red, so I know x = R. He then asks P(y = R | x = R)

P(y = R | x = R) = P(x = R, y = R) / P(x = R)

= p(card = c1) / p(card = c1 or card = c2)

= p(card = c1) / (p(card = c1) + p(card = c2))

= 1 / 2

Why does he say 2/3?

Kris, September 13, 2020 at 3:05 p.m.:Julien, if you still think your proof is correct 2 years later, then I hope the following exhaustive explanation convinces you (or other possibly confused readers) otherwise.

Using your notation (x=shown side,y=hidden side) and card labels, then as you say correctly from the product rule

p(y=R|x=R)=P(x=R,y=R)/P(x=R)

You are also (accidentally?) correct that p(x=R,y=R)=2/6=1/3=p(card=c1). Where you misunderstand the problem is in not realizing that implicitly the orientations of the cards are also random. I can agree that this part of the problem is not explained well in the video (but see exercise 8.10 in MacKay's excellent ITILA book: http://www.inference.org.uk/mackay/it...). This means that "unconditionally" (i.e. before seeing a red side, given this experimental setup) there are 4 possible unique outcomes for this "experiment": [x,y]={RR,WW,RW,WR}. Crucially, there are 2 ways in which outcome RR (and WW) can happen since c1 (c3) can be oriented in two possible ways, both giving the same outcome. On the other hand, RW (and WR) can only happen in one way for a specific orientation of c2. This means that the possible outcomes are not equally probable. We note that there are a total of possible 6 ways (3 cards x 2 orientations) that this experiment can generate an outcome. So for example p(x=R,y=R)=[# ways for RR]/(total # ways experiment can go]=2/6=1/3. If this doesn't make any sense, maybe this talk by McElreath is more pedagogical (https://www.youtube.com/watch?v=_NEMH...)

As such, you are making a common mistake in saying that p(x=R)=p(card=c1 or card=c2)=p(card=c1)+p(card=c2)=2/3. There are two ways to appreciate this. First, formally by the sum rule:

p(x=R)=p(x=R,y=R)+p(x=R,y=W)=(2/6)+(1/6)=1/2

Where p(x=R,y=R)=2/6 because this outcome can be observed in two ways [both orientations of c1]. p(x=R,y=W)=1/6 because there is only one orientation of c2 (so 1/6 of the total possible ways) that could give this results.

Thereby formally p(y=R|x=R)=(1/3)/(1/2)=2/3

The simpler explanation is that (noting that the orientations are also random) given that there are 3 possible ways that could generate a red visible side [c1 oriented either way and c2 oriented a particular way]. Out of these 2 [c1 oriented either way] would have red on the other side. So p(y=R|x=R)=[#ways with R on hidden side and R on shown side]/[#ways with R on shown side].

Hope this was helpful.

## Write your own review or comment: