## Information Theory

author: David MacKay, University of Cambridge
published: Nov. 2, 2009,   recorded: August 2009,   views: 69893
Categories

# Report a problem or upload files

If you have found a problem with this lecture or would like to send us extra material, articles, exercises, etc., please use our ticket system to describe your request and upload the data.
Enter your e-mail into the 'Cc' field, and we will keep you updated with your request's status.  Download slides: mlss09uk_mackay_it.pdf (3.1 MB) Streaming Video Help

Would you like to put a link to this lecture on your homepage?
Go ahead! Copy the HTML snippet !

I have issue with the example using cards. The correct answer is 1/2. Proof:

A card is a pair of colors:

c1 = (R, R)
c2 = (R, W)
c3 = (W, W)

We choose a card (x,y) at random.

I see red, so I know x = R. He then asks P(y = R | x = R)

P(y = R | x = R) = P(x = R, y = R) / P(x = R)
= p(card = c1) / p(card = c1 or card = c2)
= p(card = c1) / (p(card = c1) + p(card = c2))
= 1 / 2

Why does he say 2/3?

Julien, if you still think your proof is correct 2 years later, then I hope the following exhaustive explanation convinces you (or other possibly confused readers) otherwise.

Using your notation (x=shown side,y=hidden side) and card labels, then as you say correctly from the product rule

p(y=R|x=R)=P(x=R,y=R)/P(x=R)

You are also (accidentally?) correct that p(x=R,y=R)=2/6=1/3=p(card=c1). Where you misunderstand the problem is in not realizing that implicitly the orientations of the cards are also random. I can agree that this part of the problem is not explained well in the video (but see exercise 8.10 in MacKay's excellent ITILA book: http://www.inference.org.uk/mackay/it...). This means that "unconditionally" (i.e. before seeing a red side, given this experimental setup) there are 4 possible unique outcomes for this "experiment": [x,y]={RR,WW,RW,WR}. Crucially, there are 2 ways in which outcome RR (and WW) can happen since c1 (c3) can be oriented in two possible ways, both giving the same outcome. On the other hand, RW (and WR) can only happen in one way for a specific orientation of c2. This means that the possible outcomes are not equally probable. We note that there are a total of possible 6 ways (3 cards x 2 orientations) that this experiment can generate an outcome. So for example p(x=R,y=R)=[# ways for RR]/(total # ways experiment can go]=2/6=1/3. If this doesn't make any sense, maybe this talk by McElreath is more pedagogical (https://www.youtube.com/watch?v=_NEMH...)

As such, you are making a common mistake in saying that p(x=R)=p(card=c1 or card=c2)=p(card=c1)+p(card=c2)=2/3. There are two ways to appreciate this. First, formally by the sum rule:

p(x=R)=p(x=R,y=R)+p(x=R,y=W)=(2/6)+(1/6)=1/2

Where p(x=R,y=R)=2/6 because this outcome can be observed in two ways [both orientations of c1]. p(x=R,y=W)=1/6 because there is only one orientation of c2 (so 1/6 of the total possible ways) that could give this results.

Thereby formally p(y=R|x=R)=(1/3)/(1/2)=2/3

The simpler explanation is that (noting that the orientations are also random) given that there are 3 possible ways that could generate a red visible side [c1 oriented either way and c2 oriented a particular way]. Out of these 2 [c1 oriented either way] would have red on the other side. So p(y=R|x=R)=[#ways with R on hidden side and R on shown side]/[#ways with R on shown side].