Information content explained

In information theory, the information content, self-information, surprisal, or Shannon information is a basic quantity derived from the probability of a particular event occurring from a random variable. It can be thought of as an alternative way of expressing probability, much like odds or log-odds, but which has particular mathematical advantages in the setting of information theory.

The Shannon information can be interpreted as quantifying the level of "surprise" of a particular outcome. As it is such a basic quantity, it also appears in several other settings, such as the length of a message needed to transmit the event given an optimal source coding of the random variable.

The Shannon information is closely related to entropy, which is the expected value of the self-information of a random variable, quantifying how surprising the random variable is "on average". This is the average amount of self-information an observer would expect to gain about a random variable when measuring it.^[1]

The information content can be expressed in various units of information, of which the most common is the "bit" (more formally called the shannon), as explained below.

The term 'perplexity' has been used in language modelling to quantify the uncertainty inherent in a set of prospective events.

Definition

Claude Shannon's definition of self-information was chosen to meet several axioms:

An event with probability 100% is perfectly unsurprising and yields no information.
The less probable an event is, the more surprising it is and the more information it yields.
If two independent events are measured separately, the total amount of information is the sum of the self-informations of the individual events.

The detailed derivation is below, but it can be shown that there is a unique function of probability that meets these three axioms, up to a multiplicative scaling factor. Broadly, given a real number

b>1

and an event

with probability

, the information content is defined as follows:

\mathrm(x) := - \log_b = -\log_b.

The base b corresponds to the scaling factor above. Different choices of b correspond to different units of information: when, the unit is the shannon (symbol Sh), often called a 'bit'; when, the unit is the natural unit of information (symbol nat); and when, the unit is the hartley (symbol Hart).

Formally, given a discrete random variable

with probability mass function

p_X{\left(x\right)}

, the self-information of measuring

as outcome

is defined as^[2]

\operatorname I_X(x) := - \log = \log.

The use of the notation

I_X(x)

for self-information above is not universal. Since the notation

I(X;Y)

is also often used for the related quantity of mutual information, many authors use a lowercase

h_X(x)

for self-entropy instead, mirroring the use of the capital

H(X)

for the entropy.

Properties

Monotonically decreasing function of probability

For a given probability space, the measurement of rarer events are intuitively more "surprising", and yield more information content, than more common values. Thus, self-information is a strictly decreasing monotonic function of the probability, or sometimes called an "antitonic" function.

While standard probabilities are represented by real numbers in the interval

[0,1]

, self-informations are represented by extended real numbers in the interval

[0,infty]

. In particular, we have the following, for any choice of logarithmic base:

If a particular event has a 100% probability of occurring, then its self-information is

-log(1)=0

: its occurrence is "perfectly non-surprising" and yields no information.

If a particular event has a 0% probability of occurring, then its self-information is

-log(0)=infty

: its occurrence is "infinitely surprising".

From this, we can get a few general properties:

Intuitively, more information is gained from observing an unexpected event—it is "surprising".
- For example, if there is a one-in-a-million chance of Alice winning the lottery, her friend Bob will gain significantly more information from learning that she won than that she lost on a given day. (See also Lottery mathematics.)
This establishes an implicit relationship between the self-information of a random variable and its variance.

Relationship to log-odds

The Shannon information is closely related to the log-odds. In particular, given some event

, suppose that

p(x)

is the probability of

occurring, and that

p(lnotx)=1-p(x)

is the probability of

not occurring. Then we have the following definition of the log-odds:

\text(x) = \log\left(\frac\right)

This can be expressed as a difference of two Shannon informations: $\text(x) = \mathrm(\lnot x) - \mathrm(x)$

In other words, the log-odds can be interpreted as the level of surprise when the event doesn't happen, minus the level of surprise when the event does happen.

Additivity of independent events

The information content of two independent events is the sum of each event's information content. This property is known as additivity in mathematics, and sigma additivity in particular in measure and probability theory. Consider two independent random variables $X,\, Y$ with probability mass functions

p_X(x)

and

p_Y(y)

respectively. The joint probability mass function is

$p_\!\left(x, y\right) = \Pr(X = x,\, Y = y) = p_X\!(x)\,p_Y\!(y)$

(X,Y)=(x,y)

\begin\operatorname_(x, y) &= -\log_2\left[p_{X,Y}(x, y)\right] = -\log_2 \left[p_X\!(x)p_Y\!(y)\right] \\[5pt] &= -\log_2 \left[p_X{(x)}\right] -\log_2 \left[p_Y{(y)}\right] \\[5pt] &= \operatorname_X(x) + \operatorname_Y(y)\end

See below for an example.

The corresponding property for likelihoods is that the log-likelihood of independent events is the sum of the log-likelihoods of each event. Interpreting log-likelihood as "support" or negative surprisal (the degree to which an event supports a given model: a model is supported by an event to the extent that the event is unsurprising, given the model), this states that independent events add support: the information that the two events together provide for statistical inference is the sum of their independent information.

Relationship to entropy

The Shannon entropy of the random variable

above is defined as

\begin \Eta(X) &= \sum_ \\ &= \sum_ \\ & \ \operatorname,\end

by definition equal to the expected information content of measurement of

.^[3] ^[4] The expectation is taken over the discrete values over its support.

Sometimes, the entropy itself is called the "self-information" of the random variable, possibly because the entropy satisfies

Η(X)=\operatorname{I}(X;X)

, where

\operatorname{I}(X;X)

is the mutual information of

with itself.^[5]

For continuous random variables the corresponding concept is differential entropy.

Notes

This measure has also been called surprisal, as it represents the "surprise" of seeing the outcome (a highly improbable outcome is very surprising). This term (as a log-probability measure) was coined by Myron Tribus in his 1961 book Thermostatics and Thermodynamics.^[6] ^[7]

When the event is a random realization (of a variable) the self-information of the variable is defined as the expected value of the self-information of the realization.

Self-information is an example of a proper scoring rule.

Examples

Fair coin toss

. The probabilities of the events of the coin landing as heads

and tails

(see fair coin and obverse and reverse) are one half each,

p_X = p_X = \tfrac = 0.5

. Upon measuring the variable as heads, the associated information gain is

\operatorname_X(\text) = -\log_2 = -\log_2\! = 1,

so the information gain of a fair coin landing as heads is 1 shannon. Likewise, the information gain of measuring tails

\operatorname_X(T) = -\log_2 = -\log_2 = 1 \text.

Fair die roll

X\simDU[1,6]

with probability mass function

p_X(k) = \begin\frac, & k \in \ \\0, & \text\end

The probability of rolling a 4 is

p_X(4) = \frac

, as for any other valid roll. The information content of rolling a 4 is thus

\operatorname_(4) = -\log_2 = -\log_2 \approx 2.585\; \text

of information.

Two independent, identically distributed dice

Suppose we have two independent, identically distributed random variables $X,\, Y \sim \mathrm[1, 6]$ each corresponding to an independent fair 6-sided dice roll. The joint distribution of

and

\begin p_\!\left(x, y\right) & = \Pr(X = x,\, Y = y) = p_X\!(x)\,p_Y\!(y) \\ & = \begin \displaystyle, \ &x, y \in [1, 6] \cap \mathbb \\ 0 & \text \end\end

(X,Y)=(2,4)

\begin\operatorname_ &= -\log_2\! = \log_2\! = 2 \log_2\! \\ & \approx 5.169925 \text,\end

and can also be calculated by additivity of events

\begin\operatorname_ &= -\log_2\! = -\log_2\! -\log_2\! \\ & = 2\log_2\! \\ & \approx 5.169925 \text.\end

Information from frequency of rolls

If we receive information about the value of the dice without knowledge of which die had which value, we can formalize the approach with so-called counting variables $C_k := \delta_k(X) + \delta_k(Y) = \begin 0, & \neg\, (X = k \vee Y = k) \\ 1, & \quad X = k\, \veebar \, Y = k \\ 2, & \quad X = k\, \wedge \, Y = k\end$ for

k\in\{1,2,3,4,5,6\}

, then

\sum_^ = 2

and the counts have the multinomial distribution

\begin f(c_1,\ldots,c_6) & = \Pr(C_1 = c_1 \text \dots \text C_6 = c_6) \\ & = \begin, \ & \text \sum_^6 c_i=2 \\ 0 & \text \end \\ & = \begin, \ & \text c_k \text 1 \\, \ & \text c_k = 2 \\ 0, \ & \text \end\end

To verify this, the 6 outcomes $(X, Y) \in \left\_^ = \left\$ correspond to the event

C_k=2

and a total probability of . These are the only events that are faithfully preserved with identity of which dice rolled which outcome because the outcomes are the same. Without knowledge to distinguish the dice rolling the other numbers, the other

\binom = 15

combinations correspond to one die rolling one number and the other die rolling a different number, each having probability . Indeed,

6 \cdot \tfrac + 15 \cdot \tfrac = 1

, as required.

Unsurprisingly, the information content of learning that both dice were rolled as the same particular number is more than the information content of learning that one dice was one number and the other was a different number. Take for examples the events

A_k=\{(X,Y)=(k,k)\}

and

B_j,=\{c_j=1\}\cap\{c_k=1\}

for

j\nek,1\leqj,k\leq6

. For example,

A₂=\{X=2andY=2\}

and

B_3,=\{(3,4),(4,3)\}

The information contents are $\operatorname(A_2) = -\log_2\! = 5.169925 \text$ $\operatorname\left(B_\right) = - \log_2 \! \tfrac = 4.169925 \text$

Let $\text = \bigcup_^$ be the event that both dice rolled the same value and

Diff=\overline{Same

} be the event that the dice differed. Then

\Pr(\text) = \tfrac

and

\Pr(\text) = \tfrac

. The information contents of the events are

\operatorname(\text) = -\log_2\! = 2.5849625 \text

\operatorname(\text) = -\log_2\! = 0.2630344 \text.

Information from sum of die

The probability mass or density function (collectively probability measure) of the sum of two independent random variables is the convolution of each probability measure. In the case of independent fair 6-sided dice rolls, the random variable

Z=X+Y

has probability mass function

p_Z(z) = p_X(x) * p_Y(y) =

, where

represents the discrete convolution. The outcome

Z=5

has probability

p_Z(5) = \frac =

. Therefore, the information asserted is

\operatorname_Z(5) = -\log_2 = \log_2 \approx 3.169925 \text.

General discrete uniform distribution

Generalizing the example above, consider a general discrete uniform random variable (DURV)

X\simDU[a,b]; a,b\inZ, b\gea.

For convenience, define

N := b - a + 1

. The probability mass function is

p_X(k) = \begin \frac, & k \in [a, b] \cap \mathbb \\ 0, & \text.\end

In general, the values of the DURV need not be integers, or for the purposes of information theory even uniformly spaced; they need only be equiprobable. The information gain of any observation

X=k

\operatorname_X(k) = -\log_2 = \log_2 \text.

Special case: constant random variable

b=a

above,

degenerates to a constant random variable with probability distribution deterministically given by

X=b

and probability measure the Dirac measure

p_X(k) = \delta_(k)

. The only value

can take is deterministically

, so the information content of any measurement of

\operatorname_X(b) = - \log_2 = 0.

In general, there is no information gained from measuring a known value.

Categorical distribution

Generalizing all of the above cases, consider a categorical discrete random variable with support $\mathcal = \bigl\_^$ and probability mass function given by

$p_X(k) = \begin p_i, & k = s_i \in \mathcal \\ 0, & \text .\end$

For the purposes of information theory, the values

s\inl{S}

do not have to be numbers; they can be any mutually exclusive events on a measure space of finite measure that has been normalized to a probability measure

. Without loss of generality, we can assume the categorical distribution is supported on the set

[N] = \left\

; the mathematical structure is isomorphic in terms of probability theory and therefore information theory as well.

The information of the outcome

X=x

is given

$\operatorname_X(x) = -\log_2.$

From these examples, it is possible to calculate the information of any set of independent DRVs with known distributions by additivity.

Derivation

By definition, information is transferred from an originating entity possessing the information to a receiving entity only when the receiver had not known the information a priori. If the receiving entity had previously known the content of a message with certainty before receiving the message, the amount of information of the message received is zero. Only when the advance knowledge of the content of the message by the receiver is less than 100% certain does the message actually convey information.

For example, quoting a character (the Hippy Dippy Weatherman) of comedian George Carlin:

Weather forecast for tonight: dark.

Continued dark overnight, with widely scattered light by morning.^[8]

Assuming that one does not reside near the polar regions, the amount of information conveyed in that forecast is zero because it is known, in advance of receiving the forecast, that darkness always comes with the night.

Accordingly, the amount of self-information contained in a message conveying content informing an occurrence of event,

\omega_n

, depends only on the probability of that event.

$\operatorname I(\omega_n) = f(\operatorname P(\omega_n))$ for some function

f( ⋅ )

to be determined below. If

\operatornameP(\omega_n)=1

, then

\operatornameI(\omega_n)=0

. If

\operatornameP(\omega_n)<1

, then

\operatornameI(\omega_n)>0

Further, by definition, the measure of self-information is nonnegative and additive. If a message informing of event

is the intersection of two independent events

and

, then the information of event

occurring is that of the compound message of both independent events

and

occurring. The quantity of information of compound message

would be expected to equal the sum of the amounts of information of the individual component messages

and

respectively:

\operatorname I(C) = \operatorname I(A \cap B) = \operatorname I(A) + \operatorname I(B).

Because of the independence of events

and

, the probability of event

\operatorname P(C) = \operatorname P(A \cap B) = \operatorname P(A) \cdot \operatorname P(B).

However, applying function

f( ⋅ )

results in

\begin \operatorname I(C) & = \operatorname I(A) + \operatorname I(B) \\f(\operatorname P(C)) & = f(\operatorname P(A)) + f(\operatorname P(B)) \\ & = f\big(\operatorname P(A) \cdot \operatorname P(B)\big) \\\end

Thanks to work on Cauchy's functional equation, the only monotone functions

f( ⋅ )

having the property such that

f(x \cdot y) = f(x) + f(y)

are the logarithm functions

log_b(x)

. The only operational difference between logarithms of different bases is that of different scaling constants, so we may assume

$f(x) = K \log(x)$

where

log

is the natural logarithm. Since the probabilities of events are always between 0 and 1 and the information associated with these events must be nonnegative, that requires that

K<0

Taking into account these properties, the self-information

\operatornameI(\omega_n)

associated with outcome

\omega_n

with probability

\operatornameP(\omega_n)

is defined as:

\operatorname I(\omega_n) = -\log(\operatorname P(\omega_n)) = \log \left(\frac \right)

The smaller the probability of event

\omega_n

, the larger the quantity of self-information associated with the message that the event indeed occurred. If the above logarithm is base 2, the unit of

I(\omega_n)

is shannon. This is the most common practice. When using the natural logarithm of base

, the unit will be the nat. For the base 10 logarithm, the unit of information is the hartley.

As a quick illustration, the information content associated with an outcome of 4 heads (or any specific outcome) in 4 consecutive tosses of a coin would be 4 shannons (probability 1/16), and the information content associated with getting a result other than the one specified would be ~0.09 shannons (probability 15/16). See above for detailed examples.

External links

Notes and References

Jones, D.S., Elementary Information Theory, Vol., Clarendon Press, Oxford pp 11–15 1979
Book: McMahon, David M.. Quantum Computing Explained. Wiley-Interscience. 2008. 9780470181386 . Hoboken, NJ. 608622533.
Book: Fundamentals in Information Theory and Coding. Borda, Monica. Springer. 2011. 978-3-642-20346-6.
Book: Mathematics of Information and Coding. American Mathematical Society. 2002. 978-0-8218-4256-0. Han, Te Sun . Kobayashi, Kingo .
Thomas M. Cover, Joy A. Thomas; Elements of Information Theory; p. 20; 1991.
R. B. Bernstein and R. D. Levine (1972) "Entropy and Chemical Change. I. Characterization of Product (and Reactant) Energy Distributions in Reactive Molecular Collisions: Information and Entropy Deficiency", The Journal of Chemical Physics 57, 434–449 link.
http://www.eoht.info/page/Myron+Tribus Myron Tribus
Web site: A quote by George Carlin . 2021-04-01. www.goodreads.com.

Information content explained

Definition

Properties

Monotonically decreasing function of probability

Relationship to log-odds

Additivity of independent events

Relationship to entropy

Notes

Examples

Fair coin toss

Fair die roll

Two independent, identically distributed dice

Information from frequency of rolls

Information from sum of die

General discrete uniform distribution

Special case: constant random variable

Categorical distribution

Derivation

See also

Further reading

External links

Notes and References