Bernoulli distribution explained

In probability theory and statistics, the Bernoulli distribution, named after Swiss mathematician Jacob Bernoulli,^[1] is the discrete probability distribution of a random variable which takes the value 1 with probability

p and the value 0 with probability q = 1- p . Less formally, it can be thought of as a model for the set of possible outcomes of any single experiment that asks a yes–no question. Such questions lead to outcomes that are Boolean-valued: a single bit whose value is success/yes/true/one with probability p and failure/no/false/zero with probability q. It can be used to represent a (possibly biased) coin toss where 1 and 0 would represent "heads" and "tails", respectively, and p would be the probability of the coin landing on heads (or vice versa where 1 would represent tails and p would be the probability of tails). In particular, unfair coins would have p ≠ 1 / 2 .

The Bernoulli distribution is a special case of the binomial distribution where a single trial is conducted (so n would be 1 for such a binomial distribution). It is also a special case of the two-point distribution, for which the possible outcomes need not be 0 and 1.^[2]

Properties

If

X is a random variable with a Bernoulli distribution, then:\Pr(X =1 ) = p = 1 - \Pr(X =0 ) = 1 - q. f of this distribution, over possible outcomes k, isf(k;p) = \begin{cases} p & if k =1 , \\ q = 1- p & if k = 0 . \end{cases} ^[3]

This can also be expressed as

f(k;p) = p ^{k} ( 1- p) ^{1-k} for k\in\{ 0 , 1 \}

or as

f(k;p) = pk + ( 1- p)( 1- k) for k\in\{ 0 , 1 \}.

The Bernoulli distribution is a special case of the binomial distribution with

n = 1 . ^[4]

The kurtosis goes to infinity for high and low values of

p, but for p =1 / 2 the two-point distributions including the Bernoulli distribution have a lower excess kurtosis, namely −2, than any other probability distribution.

The Bernoulli distributions for

0 \le p \le 1 form an exponential family.

The maximum likelihood estimator of

p based on a random sample is the sample mean.

Mean

The expected value of a Bernoulli random variable

X is\operatorname{E}[X] = p

This is due to the fact that for a Bernoulli distributed random variable

X with \Pr(X =1 ) = p and \Pr(X =0 ) = q we find\operatorname{E}[X] = \Pr(X =1 ) ⋅ 1 + \Pr(X =0 ) ⋅ 0 = p ⋅ 1 + q ⋅ 0 = p.

Variance

The variance of a Bernoulli distributed

X is\operatorname{Var}[X] = pq = p( 1- p)

We first find

\operatorname{E}[X ^{2]} = \Pr(X =1 ) ⋅ 1 ^{2} + \Pr(X =0 ) ⋅ 0 ^{2} = p ⋅ 1 ^{2} + q ⋅ 0 ^{2} = p = \operatorname{E}[X]

From this follows

\operatorname{Var}[X] = \operatorname{E}[X ^{2]-\operatorname{E}[X]} ^{2} = \operatorname{E}[X] - \operatorname{E}[X] ^{2} = p - p ^{2} = p( 1- p) = pq

With this result it is easy to prove that, for any Bernoulli distribution, its variance will have a value inside

[ 0 , 1 / 4 ] .

Skewness

The skewness is

(q - p)/(\sqrt{pq) }=\frac. When we take the standardized Bernoulli distributed random variable (X - \operatorname{E)/([X]}{\sqrt{\operatorname{Var}[X]}}) we find that this random variable attains (q)/(\sqrt{pq) } with probability p and attains - (p)/(\sqrt{pq) } with probability q . Thus we get\begin{align} \gamma _{1} & = \operatorname{E} \left[\left( (X - \operatorname{E)/([X]}{\sqrt{\operatorname{Var}[X]}}\right)) ^{3\right]} \\ & = p ⋅ \left( (q)/(\sqrt{pq) }\right)^3 + q \cdot \left(-\frac\right)^3 \\&= \frac \left(pq^3-qp^3\right) \\&= \frac (q^2-p^2) \\&= \frac \\&= \frac = \frac.\end

Higher moments and cumulants

The raw moments are all equal due to the fact that

1 ^{k=1} and 0 ^{k=0} .\operatorname{E}[X ^{k]} = \Pr(X =1 ) ⋅ 1 ^{k} + \Pr(X =0 ) ⋅ 0 ^{k} = p ⋅ 1 + q ⋅ 0 = p = \operatorname{E}[X].

The central moment of order

k is given by\mu _{k} = ( 1- p)( - p) ^{k} + p( 1- p) ^{k.} The first six central moments are\begin{align} \mu _{1} & = 0 , \\ \mu _{2} & = p( 1- p), \\ \mu _{3} & = p( 1- p)( 1-2 p), \\ \mu _{4} & = p( 1- p)( 1-3 p( 1- p)), \\ \mu _{5} & = p( 1- p)( 1-2 p)( 1-2 p( 1- p)), \\ \mu _{6} & = p( 1- p)( 1-5 p( 1- p)( 1- p( 1- p))). \end{align} The higher central moments can be expressed more compactly in terms of \mu _{2} and \mu _{3} \begin{align} \mu _{4} & = \mu _{2} ( 1-3 \mu _{2} ), \\ \mu _{5} & = \mu _{3} ( 1-2 \mu _{2} ), \\ \mu _{6} & = \mu _{2} ( 1-5 \mu _{2} ( 1- \mu _{2} )). \end{align} The first six cumulants are\begin{align} \kappa _{1} & = p, \\ \kappa _{2} & = \mu _{2} , \\ \kappa _{3} & = \mu _{3} , \\ \kappa _{4} & = \mu _{2} ( 1-6 \mu _{2} ), \\ \kappa _{5} & = \mu _{3} ( 1-12 \mu _{2} ), \\ \kappa _{6} & = \mu _{2} ( 1-30 \mu _{2} ( 1-4 \mu _{2} )). \end{align}

Entropy and Fisher's Information

Entropy

Entropy is a measure of uncertainty or randomness in a probability distribution. For a Bernoulli random variable

X with success probability p and failure probability q = 1 - p , the entropy H(X) is defined as:\begin{align} H(X) & = E _{p} ln ( (1)/(P(X)) ) = - [P(X = 0 ) ln P(X = 0 ) + P(X = 1 ) ln P(X = 1 )] \\ H(X) & = - (q ln q + p ln p) , q = P(X = 0 ), p = P(X = 1 ) \end{align}

The entropy is maximized when

p = 0 . 5 , indicating the highest level of uncertainty when both outcomes are equally likely. The entropy is zero when p = 0 or p = 1 , where one outcome is certain.

Fisher's Information

Fisher information measures the amount of information that an observable random variable

X carries about an unknown parameter p upon which the probability of X depends. For the Bernoulli distribution, the Fisher information with respect to the parameter p is given by:\begin{align} I(p) = (1)/(pq) \end{align}

Proof:

• The Likelihood Function for a Bernoulli random variable

X is:\begin{align} L(p; X) = p ^{X} ( 1 - p) ^{1} \end{align} This represents the probability of observing X given the parameter p .

• The Log-Likelihood Function is:

\begin{align} ln L(p; X) = X ln p + ( 1 - X) ln ( 1 - p) \end{align}

• The Score Function (the first derivative of the log-likelihood w.r.t.

p is:\begin{align} (\partial)/(\partial p) ln L(p; X) = (X)/(p) - (1- X)/(1- p) \end{align}

• The second derivative of the log-likelihood function is:

\begin{align} (\partial ^{2})/(\partial p ^{2}) ln L(p; X) =- (X)/(p ^{2}) - (1- X)/(( 1- p) ^{2}) \end{align}

• Fisher information is calculated as the negative expected value of the second derivative of the log-likelihood:

\begin{align} I(p) =- E\left[ (\partial ^{2})/(\partial p ^{2}) ln L(p; X)\right] =- \left( - (p)/(p ^{2}) - (1- p)/(( 1- p) ^{2}) \right) = (1)/(p( 1- p)) = (1)/(pq) \end{align}

It is maximized when

p = 0 . 5 , reflecting maximum uncertainty and thus maximum information about the parameter p .

Related distributions

• If

X _{1,...,X} _{n} are independent, identically distributed (i.i.d.) random variables, all Bernoulli trials with success probability p, then their sum is distributed according to a binomial distribution with parameters n and p:\sum ^{n} _{k =1} X _{k} \sim \operatorname{B}(n,p) (binomial distribution).

The Bernoulli distribution is simply

\operatorname{B}( 1 , p) , also written as $\backslash mathrm\; (p).$

• The categorical distribution is the generalization of the Bernoulli distribution for variables with any constant number of discrete values.
• The Beta distribution is the conjugate prior of the Bernoulli distribution.^[5]
• The geometric distribution models the number of independent and identical Bernoulli trials needed to get one success.
• If $Y\; \backslash sim\; \backslash mathrm\backslash left(\backslash frac\backslash right)$, then $2Y\; -\; 1$ has a Rademacher distribution.

See also

• Bernoulli process, a random process consisting of a sequence of independent Bernoulli trials
• Bernoulli sampling
• Binary entropy function
• Binary decision diagram

Further reading

• Book: Johnson . N. L. . Kotz . S. . Kemp . A. . 1993 . Univariate Discrete Distributions . 2nd . Wiley . 0-471-54897-9 .
• Book: Peatman, John G. . Introduction to Applied Statistics . New York . Harper & Row . 1963 . 162–171 .

External links

• .
  • Interactive graphic: Univariate Distribution Relationships.

Notes and References

1. Book: Uspensky, James Victor . Introduction to Mathematical Probability . McGraw-Hill . New York . 1937 . 45 . 996937 .
2. Book: Dekking . Frederik . Kraaikamp . Cornelis . Lopuhaä . Hendrik . Meester . Ludolf . A Modern Introduction to Probability and Statistics . 9 October 2010 . Springer London . 9781849969529 . 43–48 . 1.
3. Book: Bertsekas, Dimitri P.. Introduction to Probability. Dimitri_Bertsekas. 2002. Athena Scientific. Tsitsiklis, John N., Τσιτσικλής, Γιάννης Ν.. 188652940X. Belmont, Mass.. 51441829.
4. Book: McCullagh, Peter . Peter McCullagh . Nelder, John . John Nelder . Generalized Linear Models, Second Edition . Boca Raton: Chapman and Hall/CRC . 1989 . 0-412-31760-5 . McCullagh1989 . Section 4.2.2 .
5. Web site: Orloff . Jeremy . Bloom . Jonathan . Conjugate priors: Beta and normal . October 20, 2023 . math.mit.edu.