[ Mathematics ]
Events, Probabilities and Information
The less probable an event is, the more information its occurrence conveys.
The value of the information (or the "message") an event brings is zero
when the probability is equal to one "because receiving a message that
we expect with certainty gives us, in effect, no new information" (Adam
Drozdek, Elements of Data Compression).
On the other hand, as the probability of an event approaches zero, the
message it carries will contain more valuable information.
As Dorzdek
notes, "[t]his is in line with our intuition that rare events are more
informative than unsurprising events."