In probability, a discrete-time Markov chain (DTMC) is a sequence of random variables, known as a stochastic process, in which the value of the next variable depends only on the value of the current variable, and not any variables in the past. For instance, a machine may have two states, A and E. When it is in state A, there is a 40% chance of it moving to state E and a 60% chance of it remaining in state A. When it is in state E, there is a 70% chance of it moving to A and a 30% chance of it staying in E. The sequence of states of the machine is a Markov chain. If we denote the chain by
X0,X1,X2,...
X0
X10
An example of a stochastic process which is not a Markov chain is the model of a machine which has states A and E and moves to A from either state with 50% chance if it has ever visited A before, and 20% chance if it has never visited A before (leaving a 50% or 80% chance that the machine moves to E). This is because the behavior of the machine depends on the whole history—if the machine is in E, it may have a 50% or 20% chance of moving to A, depending on its past values. Hence, it does not have the Markov property.
A Markov chain can be described by a stochastic matrix, which lists the probabilities of moving to each state from any individual state. From this matrix, the probability of being in a particular state n steps in the future can be calculated. A Markov chain's state space can be partitioned into communicating classes that describe which states are reachable from each other (in one transition or in many). Each state can be described as transient or recurrent, depending on the probability of the chain ever returning to that state. Markov chains can have properties including periodicity, reversibility and stationarity. A continuous-time Markov chain is like a discrete-time Markov chain, but it moves states continuously through time rather than as discrete time steps. Other stochastic processes can satisfy the Markov property, the property that past behavior does not affect the process, only the present state.
A discrete-time Markov chain is a sequence of random variables
X0,X1,X2,...
\Pr(Xn+1=x\midX1=x1,X2=x2,\ldots,Xn=xn)=\Pr(Xn+1=x\midXn=xn),
\Pr(X1=x1,\ldots,Xn=xn)>0.
The possible values of Xi form a countable set S called the state space of the chain.[1]
Markov chains are often described by a sequence of directed graphs, where the edges of graph n are labeled by the probabilities of going from one state at time n to the other states at time n + 1,
\Pr(Xn+1=x\midXn=xn).
These descriptions highlight the structure of the Markov chain that is independent of the initial distribution
\Pr(X1=x1).
\Pr(Xn=x\midX1=x1)
x1
\Pr(X1=y)=[x1=y]
[P]
\Pr(Xn+1=x\midXn=y)=\Pr(Xn=x\midXn-1=y)
for all n. The probability of the transition is independent of n.[1]
where m is finite, is a process satisfying
\begin{align} &\Pr(Xn=xn\midXn-1=xn-1,Xn-2=xn-2,...,X1=x1)\\ ={}&\Pr(Xn=xn\midXn-1=xn-1,Xn-2=xn-2,...,Xn-m=xn-m) forn>m \end{align}
In other words, the future state depends on the past m states. It is possible to construct a chain
(Yn)
(Xn)
Yn=\left(Xn,Xn-1,\ldots,Xn-m+1\right)
The probability of going from state i to state j in n time steps is
(n) | |
p | |
ij |
=\Pr(Xn=j\midX0=i)
and the single-step transition is
pij=\Pr(X1=j\midX0=i).
For a time-homogeneous Markov chain:
(n) | |
p | |
ij |
=\Pr(Xk+n=j\midXk=i)
and
pij=\Pr(Xk+1=j\midXk=i).
The n-step transition probabilities satisfy the Chapman–Kolmogorov equation, that for any k such that 0 < k < n,
(n) | |
p | |
ij |
=\sumr
(k) | |
p | |
ir |
(n-k) | |
p | |
rj |
where S is the state space of the Markov chain.[1]
The marginal distribution Pr(Xn = x) is the distribution over states at time n. The initial distribution is Pr(X0 = x). The evolution of the process through one time step is described by
\Pr(Xn=j)=\sumrprj\Pr(Xn-1=r)=\sumr
(n) | |
p | |
rj |
\Pr(X0=r).
(The superscript (n) is an index, and not an exponent).
A state j is said to be accessible from a state i (written i → j) if a system started in state i has a non-zero probability of transitioning into state j at some point. Formally, state j is accessible from state i if there exists an integer nij ≥ 0 such that
\Pr(X | |
nij |
=j\midX0=i)=
(nij) | |
p | |
ij |
>0.
A state i is said to communicate with state j (written i ↔ j) if both i → j and j → i. A communicating class is a maximal set of states C such that every pair of states in C communicates with each other. Communication is an equivalence relation, and communicating classes are the equivalence classes of this relation.[1]
A communicating class is closed if the probability of leaving the class is zero, namely if i is in C but j is not, then j is not accessible from i.[1] The set of communicating classes forms a directed, acyclic graph by inheriting the arrows from the original state space. A communicating class is closed if and only if it has no outgoing arrows in this graph.
A state i is said to be essential or final if for all j such that i → j it is also true that j → i. A state i is inessential if it is not essential.[2] A state is final if and only if its communicating class is closed.
A Markov chain is said to be irreducible if its state space is a single communicating class; in other words, if it is possible to get to any state from any state.[1] [3]
A state
i
k
i
k
i
k=\gcd\{n>0:\Pr(Xn=i\midX0=i)>0\}
(where
\gcd
k
k
\{6,~8,~10,~12,...\}
k
2
2
If
k=1
k>1
k
k
k
A state i is said to be transient if, given that we start in state i, there is a non-zero probability that we will never return to i. Formally, let the random variable Ti be the first return time to state i (the "hitting time"):
Ti=inf\{n\ge1:Xn=i\}.
The number
(n) | |
f | |
ii |
=\Pr(Ti=n\midX0=i)
is the probability that we return to state i for the first time after n steps.Therefore, state i is transient if
\Pr(Ti<infty\midX0=i)=
infty | |
\sum | |
n=1 |
(n) | |
f | |
ii |
<1.
State i is recurrent (or persistent) if it is not transient. Recurrence and transience are class properties, that is, they either hold or do not hold equally for all members of a communicating class.[1]
A state i is recurrent if and only if the expected number of visits to i is infinite:[1]
infty | |
\sum | |
n=0 |
(n) | |
p | |
ii |
=infty.
Even if the hitting time is finite with probability 1, it need not have a finite expectation. The mean recurrence time at state i is the expected return time Mi:
Mi=E[Ti]=\sum
infty | |
n=1 |
n ⋅
(n) | |
f | |
ii |
.
State i is positive recurrent (or non-null persistent) if Mi is finite; otherwise, state i is null recurrent (or null persistent). Positive and null recurrence are classes properties.[1]
A state i is called absorbing if it is impossible to leave this state. Therefore, the state i is absorbing if and only if
pii=1andpij=0fori\not=j.
If every state can reach an absorbing state, then the Markov chain is an absorbing Markov chain.[4] [5]
A Markov chain is said to be reversible if there is a probability distribution over its states such that
\pii\Pr(Xn+1=j\midXn=i)=\pij\Pr(Xn+1=i\midXn=j)
for all times n and all states i and j. This condition is known as the detailed balance condition (or local balance equation).
Considering a fixed arbitrary time n and using the shorthand
pij=\Pr(Xn+1=j\midXn=i),
the detailed balance equation can be written more compactly as
\piipij=\pijpji.
The single time-step from n to n + 1 can be thought of as each person i having i dollars initially and paying each person j a fraction pij of it. The detailed balance condition states that upon each payment, the other person pays exactly the same amount of money back.[6] Clearly the total amount of money each person has remains the same after the time-step, since every dollar spent is balanced by a corresponding dollar received. This can be shown more formally by the equality
\sumi\piipij=\sumi\pijpji=\pij\sumipji=\pij,
which essentially states that the total amount of money person j receives (including from himself) during the time-step equals the amount of money he pays others, which equals all the money he initially had because it was assumed that all money is spent (that is, pji sums to 1 over i). The assumption is a technical one, because the money not really used is simply thought of as being paid from person j to himself (that is, pjj is not necessarily zero).
As n was arbitrary, this reasoning holds for any n, and therefore for reversible Markov chains is always a steady-state distribution of Pr(Xn+1 = j | Xn = i) for every n.
If the Markov chain begins in the steady-state distribution, that is, if
\Pr(X0=i)=\pii
\Pr(Xn=i)=\pii
n
\Pr(Xn=i,Xn+1=j)=\Pr(Xn+1=i,Xn=j).
The left- and right-hand sides of this last equation are identical except for a reversing of the time indices n and n + 1.
Kolmogorov's criterion gives a necessary and sufficient condition for a Markov chain to be reversible directly from the transition matrix probabilities. The criterion requires that the products of probabilities around every closed loop are the same in both directions around the loop.
Reversible Markov chains are common in Markov chain Monte Carlo (MCMC) approaches because the detailed balance equation for a desired distribution necessarily implies that the Markov chain has been constructed so that is a steady-state distribution. Even with time-inhomogeneous Markov chains, where multiple transition matrices are used, if each such transition matrix exhibits detailed balance with the desired distribution, this necessarily implies that is a steady-state distribution of the Markov chain.
For any time-homogeneous Markov chain given by a transition matrix
P\inRn
|| ⋅ ||
Rn
\pi
P*
\pi
P
|| ⋅ ||.
P*
For example, consider the following Markov chain:This Markov chain is not reversible. According to the Frobenius Norm the closest reversible Markov chain according to
\pi=\left(
1 | , | |
3 |
1 | , | |
3 |
1 | |
3 |
\right)
\pi=\left(
1 | |
4 |
,
1 | |
4 |
,
1 | |
2 |
\right)
A distribution
\pi
P
\piP=\pi
\forallj\inS:\sumi\in\piipij=\pij
This condition implies that
\piPn=\pi
(Xn,n\inN)
X0=\pi
Xn=\pi
n\inN
If a Markov chain is irreducible then it has a stationary distribution if and only if it is positive recurrent, in which case the unique such distribution is given by
\pi | ||||
|
Mi=E(Ti)
Ci
\pii
\pii
\lim\nolimitsn
(n) | |
p | |
jj |
=\tfrac{C}{Mj}
\lim\nolimitsn
(n) | |
p | |
ij |
=C\tfrac{fij
If a state i is periodic with period k > 1 then the limit
\lim\nolimitsn
(n) | |
p | |
ii |
\lim\nolimitsn
(kn+r) | |
p | |
ii |
A Markov chain need not necessarily be time-homogeneous to have an equilibrium distribution. If there is a probability distribution over states
\boldsymbol{\pi}
\pij=\sumi\pii\Pr(Xn+1=j\midXn=i)
for every state j and every time n then
\boldsymbol{\pi}
See main article: Phase-type distribution. The hitting time is the time, starting in a given set of states until the chain arrives in a given state or set of states. The distribution of such a time period has a phase type distribution. The simplest such distribution is that of a single exponentially distributed transition.
For a subset of states A ⊆ S, the vector kA of hitting times (where element
A | |
k | |
i |
A | |
\begin{align} k | |
i |
=0&fori\inA\\ -\sumjqij
A | |
k | |
j |
=1&fori\notinA. \end{align}
An instance of ergodic theory, the ergodic theorem for states that for an irreducible aperiodic Markov chain, for any two states i and j,[1]
(n) | |
p | |
i,j |
→
1 | |
Mj |
n → infty