Masood Mortazavi's Weblog

Sunday September 17, 2006

[ Mathematics ] Events, Probabilities and Information

The less probable an event is, the more information its occurrence conveys.

The value of the information (or the "message") an event brings is zero when the probability is equal to one "because receiving a message that we expect with certainty gives us, in effect, no new information" (Adam Drozdek, Elements of Data Compression).

On the other hand, as the probability of an event approaches zero, the message it carries will contain more valuable information.

As Dorzdek notes, "[t]his is in line with our intuition that rare events are more informative than unsurprising events."

2006-09-17 23:08:36.0 -- Comments [5] ; Permalink ; Trackback.

Comments:

Hmm. Interesting. My first thought about this was: "rare is not the same as improbable". The return of Halley's comet is comparatively rare (in human terms), but eminently predictable. But then I thought, in Zen terms, "every event is rare"... you cannot step twice into the same river. Objectively, it doesn't make any difference whether an event is rare/unpredictable or not. Subjectively, the important thing about unexpected or counter-intuitive events is that they (should) lead us to re-examine our frame of reference. Hmmmmm.

Posted by Robin Wilton on September 18, 2006 at 01:54 AM PDT #

Very good point!

I think by "rare" the author really means "unexpected." So, you put it well: "[T]he important thing about unexpected or counter-intuitive events is that they (should) lead us to re-examine our frame of reference."

Posted by M. Mortazavi on September 18, 2006 at 09:50 AM PDT #

... and the conclusion you draw is the main reason I posted this ...

Posted by M. Mortazavi on September 18, 2006 at 10:06 AM PDT #

Not sure I agree, at least in all cases. I can agree that there is little or no additional information in that which is certain, but not the inverse.

Consider a deck of cards that is riffle shuffled many times and is random. The odds of exactly this order of cards appearing is 52! or one in 80,658,175,170,943,878,571,660,636,856,403,766,975,289,505,440,883,277,824,000,000,000,000 . This is exceedingly small, and probably has never happened before and will likely never happen again, and yet it conveys no information at all.

Posted by Brian Utterback on September 19, 2006 at 06:08 AM PDT #

Concepts involving probabilities can be very deceiving, except in the most routine and, one may say, almost artificial cases.

You're right. In this case, the odds are very small and yet each rare event seems to have little information value.

However, how do we calculate odds and what do they mean?

Some events that seem to have large "odds" actually have very small odds.

What were the odds that there would be an Earth and that the Earth will be going around the Sun when the universe began?

What were the odds that I'll be writing this note and that you'll be reading it when the universe began?

When we put the question of "odds" aside, it still seems that the occurrence of what is unexpected will contain more valuable material / information for adjusting our frame of handling what it is then the occurence of what is expected. In this context, can we say that the randomness of the cards is unexpected? I imagine we all expect the shuffle to behave as it does. If it didn't, that event would be surprising and will contain high value information.

What is expected and what is rare may also depend on the context.

For example, what is the probability that 10 shuffles of a pack of cards will give the same run of card arrangements for the 52 cards for all of the 10 shuffles? It all depends on how I'm shuffling the cards and it can certainly be made to be much, much smaller than 1/52!.

Posted by M. Mortazavi on September 19, 2006 at 06:35 AM PDT #