The Theil index is a statistic primarily used to measure economic inequality[1] and other economic phenomena, though it has also been used to measure racial segregation.[2] [3] The Theil index TT is the same as redundancy in information theory which is the maximum possible entropy of the data minus the observed entropy. It is a special case of the generalized entropy index. It can be viewed as a measure of redundancy, lack of diversity, isolation, segregation, inequality, non-randomness, and compressibility. It was proposed by a Dutch econometrician Henri Theil (1924–2000) at the Erasmus University Rotterdam.
Henri Theil himself said (1967): "The (Theil) index can be interpreted as the expected information content of the indirect message which transforms the population shares as prior probabilities into the income shares as posterior probabilities."[4] Amartya Sen noted, "But the fact remains that the Theil index is an arbitrary formula, and the average of the logarithms of the reciprocals of income shares weighted by income is not a measure that is exactly overflowing with intuitive sense."
For a population of N "agents" each with characteristic x, the situation may be represented by the list xi (i = 1,...,N) where xi is the characteristic of agent i. For example, if the characteristic is income, then xi is the income of agent i.
The Theil T index is defined as
TT=T\alpha=1=
| 1 | |
| N |
| N | |
| \sum | |
| i=1 |
| xi | ln\left( | |
| \mu |
| xi | |
| \mu |
\right)
and the Theil L index is defined as
TL=T\alpha=0=
| 1 | |
| N |
| N | ||
| \sum | ln\left( | |
| i=1 |
| \mu | |
| xi |
\right)
where
\mu
| \mu= | 1 |
| N |
| N | |
| \sum | |
| i=1 |
xi
Theil-L is an income-distribution's dis-entropy per person, measured with respect to maximum entropy (...which is achieved with complete equality).
(In an alternative interpretation of it, Theil-L is the natural-logarithm of the geometric-mean of the ratio: (mean income)/(income i), over all the incomes. The related Atkinson(1) is just 1 minus the geometric-mean of (income i)/(mean income), over the income distribution.)
Because a transfer between a larger income & a smaller one will change the smaller income's ratio more than it changes the larger income's ratio, the transfer-principle is satisfied by this index.
Equivalently, if the situation is characterized by a discrete distribution function fk (k = 0,...,W) where fk is the fraction of the population with income k and W = Nμ is the total income, then
| W | |
| \sum | |
| k=0 |
fk=1
TT=\sum
| W | |
| k=0 |
fk
| k | ln\left( | |
| \mu |
| k | |
| \mu |
\right)
where
\mu
| W | |
| \mu=\sum | |
| k=0 |
kfk
Note that in this case income k is an integer and k=1 represents the smallest increment of income possible (e.g., cents).
if the situation is characterized by a continuous distribution function f(k) (supported from 0 to infinity) where f(k) dk is the fraction of the population with income k to k + dk, then the Theil index is:
TT=\int
| infty | |
| 0 |
f(k)
| k | ln\left( | |
| \mu |
| k | |
| \mu |
\right)dk
where the mean is:
| infty | |
| \mu=\int | |
| 0 |
kf(k)dk
Theil indices for some common continuous probability distributions are given in the table below:
| Income distribution function | PDF(x) (x ≥ 0) | Theil coefficient (nats) | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
\delta(x-x0),x0>0 | 0 | ||||||||||||||||
| Uniform distribution |
&a\lex\leb\ 0&otherwise \end{cases} | ln\left(
| |||||||||||||||
| Exponential distribution | λe-xλ,x>0 | 1- \gamma | |||||||||||||||
| Log-normal distribution |
|
| |||||||||||||||
| Pareto distribution |
&x\gek\\0&x<k \end{cases} | ln(1-\ | 1/\alpha)+\frac (α>1) | ||||||||||||||
| Chi-squared distribution |
| ln(2/k)+ \psi(0) (1+\ | k/2) | ||||||||||||||
| Gamma distribution[5] |
| \psi(0) (1+k)-ln(k) | |||||||||||||||
| Weibull distribution |
\right)k-1
|
\psi(0) (1+1/k)-ln\left(\Gamma(1+1/k)\right) |
If everyone has the same income, then TT equals 0. If one person has all the income, then TT gives the result
lnN
lnN
T[x\cupx]\neT[x]
The Theil index can be transformed into an Atkinson index, which has a range between 0 and 1 (0% and 100%), where 0 indicates perfect equality and 1 (100%) indicates maximum inequality. (See Generalized entropy index for the transformation.)
S
S=k
| N | |
| \sum | |
| i=1 |
\left(piloga\left({
| 1 | |
| pi |
where
i
pi
i
k
loga\left({x}\right)
a
When looking at the distribution of income in a population,
pi
STheil
STheil=
| N | |
| \sum | |
| i=1 |
\left(
| xi | |
| N\bar{x |
}ln\left({
| N\bar{x | |
| }{x |
i}}\right)\right)
where
xi
\left(N\bar{x}\right)
N
\bar{x}
ln\left(x\right)
x
\left(loge\left(x\right)\right)
TT
STheil
Smax=ln\left({N}\right)
TT=Smax-STheil=ln\left({N}\right)-STheil
When
x
STheil
The Theil index measures what is called redundancy in information theory.[9] It is the left over "information space" that was not utilized to convey information, which reduces the effectiveness of the price signal. The Theil index is a measure of the redundancy of income (or other measure of wealth) in some individuals. Redundancy in some individuals implies scarcity in others. A high Theil index indicates the total income is not distributed evenly among individuals in the same way an uncompressed text file does not have a similar number of byte locations assigned to the available unique byte characters.
| Notation | Information theory | Theil index TT | |
|---|---|---|---|
N | number of unique characters | number of individuals | |
i | a particular character | a particular individual | |
xi | count of ith character | income of ith individual | |
N\bar{x} | total characters in document | total income in population | |
TT | unused information space | unused potential in price mechanism | |
| data compression | progressive tax |
According to the World Bank,
"The best-known entropy measures are Theil’s T () and Theil’s L (TT
), both of which allow one to decompose inequality into the part that is due to inequality within areas (e.g. urban, rural) and the part that is due to differences between areas (e.g. the rural-urban income gap). Typically at least three-quarters of inequality in a country is due to within-group inequality, and the remaining quarter to between-group differences."[10]TL
If the population is divided into
m
si
i
N
Ni
i
Ti
\overline{x}i
i
\mu
then Theil's T index is
TT=
| m | |
| \sum | |
| i=1 |
siTi+
| m | |
| \sum | |
| i=1 |
siln{
| \overline{x | |
| i}{\mu}} |
si=
| Ni | |
| N |
| \overline{x | |
| i}{\mu} |
For example, inequality within the United States is the average inequality within each state, weighted by state income, plus the inequality between states.
Note: This image is not the Theil Index in each area of the United States, but of contributions to the Theil Index for the U.S. by each area. The Theil Index is always positive, although individual contributions to the Theil Index may be negative or positive.
The decomposition of the Theil index which identifies the share attributable to the between-region component becomes a helpful tool for the positive analysis of regional inequality as it suggests the relative importance of spatial dimension of inequality.[11]
Both Theil's T and Theil's L are decomposable. The difference between them is based on the part of the outcomes distribution that each is used for. Indexes of inequality in the generalized entropy (GE) family are more sensitive to differences in income shares among the poor or among the rich depending on a parameter that defines the GE index. The smaller the parameter value for GE, the more sensitive it is to differences at the bottom of the distribution.[12]
GE(0) = Theil's L and is more sensitive to differences at the lower end of the distribution. It is also referred to as the mean log deviation measure.
GE(1) = Theil's T and is more sensitive to differences at the top of the distribution.
The decomposability is a property of the Theil index which the more popular Gini coefficient does not offer. The Gini coefficient is more intuitive to many people since it is based on the Lorenz curve. However, it is not easily decomposable like the Theil.
In addition to multitude of economic applications, the Theil index has been applied to assess performance of irrigation systems[13] and distribution of software metrics.[14]
k
k
a
a
| N | |
| \sum | |
| i=1 |
\left(\left(
| xi | |
| \bar{x |
| N | |
| =\sum | |
| i=1 |
\left(
| 1 | |
| N |
ln\left({N}\right)\right)
=ln\left({N}\right)