In statistics, econometrics, and signal processing, an autoregressive (AR) model is a representation of a type of random process; as such, it can be used to describe certain time-varying processes in nature, economics, behavior, etc. The autoregressive model specifies that the output variable depends linearly on its own previous values and on a stochastic term (an imperfectly predictable term); thus the model is in the form of a stochastic difference equation (or recurrence relation) which should not be confused with a differential equation. Together with the moving-average (MA) model, it is a special case and key component of the more general autoregressive–moving-average (ARMA) and autoregressive integrated moving average (ARIMA) models of time series, which have a more complicated stochastic structure; it is also a special case of the vector autoregressive model (VAR), which consists of a system of more than one interlocking stochastic difference equation in more than one evolving random variable.
Contrary to the moving-average (MA) model, the autoregressive model is not always stationary as it may contain a unit root.
Large language models are called autoregressive, but they are not a classical autoregressive model in this sense because they are not linear.
The notation
AR(p)
Xt=
p | |
\sum | |
i=1 |
\varphiiXt-i+\varepsilont
where
\varphi1,\ldots,\varphip
\varepsilont
Xt=
p | |
\sum | |
i=1 |
\varphiiBiXt+\varepsilont
so that, moving the summation term to the left side and using polynomial notation, we have
\phi[B]Xt=\varepsilont
An autoregressive model can thus be viewed as the output of an all-pole infinite impulse response filter whose input is white noise.
Some parameter constraints are necessary for the model to remain weak-sense stationary. For example, processes in the AR(1) model with
|\varphi1|\geq1
\Phi(z):=style1-
p | |
\sum | |
i=1 |
\varphiizi
zi
|zi|>1
In an AR process, a one-time shock affects values of the evolving variable infinitely far into the future. For example, consider the AR(1) model
Xt=\varphi1Xt-1+\varepsilont
\varepsilont
X1
\varepsilon1
X2
X1
X2
\varphi1\varepsilon1
X3
X2
X3
2 | |
\varphi | |
1 |
\varepsilon1
\varepsilon1
Because each shock affects X values infinitely far into the future from when they occur, any given value Xt is affected by shocks occurring infinitely far into the past. This can also be seen by rewriting the autoregression
\phi(B)Xt=\varepsilont
(where the constant term has been suppressed by assuming that the variable has been measured as deviations from its mean) as
Xt=
1 | |
\phi(B) |
\varepsilont.
When the polynomial division on the right side is carried out, the polynomial in the backshift operator applied to
\varepsilont
\varepsilont
The autocorrelation function of an AR(p) process can be expressed as
\rho(\tau)=
p | |
\sum | |
k=1 |
ak
-|\tau| | |
y | |
k |
,
where
yk
\phi(B)=1-
p | |
\sum | |
k=1 |
\varphikBk
where B is the backshift operator, where
\phi( ⋅ )
\varphik
The autocorrelation function of an AR(p) process is a sum of decaying exponentials.
The simplest AR process is AR(0), which has no dependence between the terms. Only the error/innovation/noise term contributes to the output of the process, so in the figure, AR(0) corresponds to white noise.
For an AR(1) process with a positive
\varphi
\varphi
\varphi
For an AR(2) process, the previous two terms and the noise term contribute to the output. If both
\varphi1
\varphi2
\varphi1
\varphi2
An AR(1) process is given by:where
\varepsilont
2 | |
\sigma | |
\varepsilon |
\varphi1
|\varphi|<1
\varphi=1
Xt
|\varphi|<1
\operatorname{E}(Xt)
\mu
\mu=0.
The variance is
rm{var}(Xt)=\operatorname{E}(X
2)-\mu | |
t |
| ||||||||||
,
\sigma\varepsilon
\varepsilont
rm{var}(Xt)=
2rm{var}(X | |
\varphi | |
t-1 |
)+
2, | |
\sigma | |
\varepsilon |
The autocovariance is given by
Bn=\operatorname{E}(Xt+n
| ||||||||||
X | ||||||||||
t)-\mu |
\varphi|n|.
It can be seen that the autocovariance function decays with a decay time (also called time constant) of
\tau=1-\varphi
The spectral density function is the Fourier transform of the autocovariance function. In discrete terms this will be the discrete-time Fourier transform:
\Phi(\omega)= | 1 |
\sqrt{2\pi |
This expression is periodic due to the discrete nature of the
Xj
\Deltat=1
\tau
Bn
B(t) ≈
| |||||||
1-\varphi2 |
\varphi|t|
which yields a Lorentzian profile for the spectral density:
\Phi(\omega)= | 1 |
\sqrt{2\pi |
where
\gamma=1/\tau
\tau
An alternative expression for
Xt
\varphiXt-2+\varepsilont-1
Xt-1
NX | |
X | |
t-N |
N-1 | |
+\sum | |
k=0 |
k\varepsilon | |
\varphi | |
t-k |
.
For N approaching infinity,
\varphiN
Xt=\sum
infty\varphi | |
k=0 |
k\varepsilon | |
t-k |
.
It is seen that
Xt
\varphik
\varepsilont
Xt
Xt
\varphi
For
\varepsilont=0
Xt=\varphiXt-1
Xt=a\varphit
a
The AR(1) model is the discrete-time analogy of the continuous Ornstein-Uhlenbeck process. It is therefore sometimes useful to understand the properties of the AR(1) model cast in an equivalent form. In this form, the AR(1) model, with process parameter
\theta\inR
Xt+1=Xt+(1-\theta)(\mu-Xt)+\varepsilont+1
|\theta|<1
\mu:=E(X)
\{\epsilont\}
\sigma
By rewriting this as
Xt+1=\thetaXt+(1-\theta)\mu+\varepsilont+1
Xt+n=\thetanXt+(1-\thetan)\mu+
n | |
\Sigma | |
i=1 |
\left(\thetan\epsilont\right)
\operatorname{E}(Xt+n|Xt)=\mu\left[1-\thetan\right]+
n | |
X | |
t\theta |
\operatorname{Var}(Xt+n|Xt)=\sigma2
1-\theta2n | |
1-\theta2 |
See main article: Partial autocorrelation function.
The partial autocorrelation of an AR(p) process equals zero at lags larger than p, so the appropriate maximum lag p is the one after which the partial autocorrelations are all zero.
There are many ways to estimate the coefficients, such as the ordinary least squares procedure or method of moments (through Yule–Walker equations).
The AR(p) model is given by the equation
Xt=
p | |
\sum | |
i=1 |
\varphiiXt-i+\varepsilont.
It is based on parameters
\varphii
The Yule–Walker equations, named for Udny Yule and Gilbert Walker,[5] [6] are the following set of equations.[7]
\gammam=
p | |
\sum | |
k=1 |
\varphik\gammam-k+
2\delta | |
\sigma | |
m,0 |
,
where, yielding equations. Here
\gammam
\sigma\varepsilon
\deltam,0
Because the last part of an individual equation is non-zero only if, the set of equations can be solved by representing the equations for in matrix form, thus getting the equation
\begin{bmatrix} \gamma1\\ \gamma2\\ \gamma3\\ \vdots\\ \gammap\\ \end{bmatrix} = \begin{bmatrix} \gamma0&\gamma-1&\gamma-2& … \\ \gamma1&\gamma0&\gamma-1& … \\ \gamma2&\gamma1&\gamma0& … \\ \vdots&\vdots&\vdots&\ddots\\ \gammap-1&\gammap-2&\gammap-3& … \\ \end{bmatrix} \begin{bmatrix} \varphi1\\ \varphi2\\ \varphi3\\ \vdots\\ \varphip\\ \end{bmatrix}
which can be solved for all
\{\varphim;m=1,2,...,p\}.
\gamma0=
p | |
\sum | |
k=1 |
\varphik\gamma-k+
2 | |
\sigma | |
\varepsilon |
,
which, once
\{\varphim;m=1,2,...,p\}
2 | |
\sigma | |
\varepsilon |
.
An alternative formulation is in terms of the autocorrelation function. The AR parameters are determined by the first p+1 elements
\rho(\tau)
\rho(\tau)=
p | |
\sum | |
k=1 |
\varphik\rho(k-\tau)
\gamma1=\varphi1\gamma0
\rho1=\gamma1/\gamma0=\varphi1
\gamma1=\varphi1\gamma0+\varphi2\gamma-1
\gamma2=\varphi1\gamma1+\varphi2\gamma0
\gamma-k=\gammak
\rho1=\gamma1/\gamma0=
\varphi1 | |
1-\varphi2 |
\rho2=\gamma2/\gamma0=
| ||||||||||||||||
1-\varphi2 |
The above equations (the Yule–Walker equations) provide several routes to estimating the parameters of an AR(p) model, by replacing the theoretical covariances with estimated values.[9] Some of these variants can be described as follows:
Xt=
p | |
\sum | |
i=1 |
\varphiiXt+i+
* | |
\varepsilon | |
t |
.
Here predicted values of Xt would be based on the p future values of the same series. This way of estimating the AR parameters is due to John Parker Burg, and is called the Burg method: Burg and later authors called these particular estimates "maximum entropy estimates", but the reasoning behind this applies to the use of any set of estimated AR parameters. Compared to the estimation scheme using only the forward prediction equations, different estimates of the autocovariances are produced, and the estimates have different stability properties. Burg estimates are particularly associated with maximum entropy spectral estimation.
Other possible approaches to estimation include maximum likelihood estimation. Two distinct variants of maximum likelihood are available: in one (broadly equivalent to the forward prediction least squares scheme) the likelihood function considered is that corresponding to the conditional distribution of later values in the series given the initial p values in the series; in the second, the likelihood function considered is that corresponding to the unconditional joint distribution of all the values in the observed series. Substantial differences in the results of these approaches can occur if the observed series is short, or if the process is close to non-stationarity.
The power spectral density (PSD) of an AR(p) process with noise variance
Var(Zt)=
2 | |
\sigma | |
Z |
S(f)=
| ||||||||
|
.
For white noise (AR(0))
S(f)=
2. | |
\sigma | |
Z |
For AR(1)
S(f)=
| |||||||
|1-\varphi1e-2|2 |
=
| ||||||||
|
\varphi1>0
\varphi1
\varphi1<0
The behavior of an AR(2) process is determined entirely by the roots of it characteristic equation, which is expressed in terms of the lag operator as:
1-\varphi1B-\varphi2B2=0,
or equivalently by the poles of its transfer function, which is defined in the Z domain by:
Hz=(1-\varphi1z-1-\varphi2z-2)-1.
It follows that the poles are values of z satisfying:
1-\varphi1z-1-\varphi2z-2=0
which yields:
z1,z2=
1 | |
2\varphi2 |
\left(\varphi1\pm
2 | |
\sqrt{\varphi | |
1 |
+4\varphi2}\right)
z1
z2
\begin{bmatrix}\varphi1&\varphi2\ 1&0\end{bmatrix}
AR(2) processes can be split into three groups depending on the characteristics of their roots/poles:
2 | |
\varphi | |
1 |
+4\varphi2<0
f*=
1 | |
2\pi |
\cos-1\left(
\varphi1 | |
2\sqrt{-\varphi2 |
with bandwidth about the peak inversely proportional to the moduli of the poles:
|z1|=|z2|=\sqrt{-\varphi2}.
The terms involving square roots are all real in the case of complex poles since they exist only when
\varphi2<0
Otherwise the process has real roots, and:
\varphi1>0
f=0
\varphi1<0
f=1/2
-1\le\varphi2\le1-|\varphi1|
The full PSD function can be expressed in real form as:
S(f)=
| ||||||||||||||
|
implementation in statsmodels.[16]
The impulse response of a system is the change in an evolving variable in response to a change in the value of a shock term k periods earlier, as a function of k. Since the AR model is a special case of the vector autoregressive model, the computation of the impulse response in vector autoregression#impulse response applies here.
Once the parameters of the autoregression
Xt=
p | |
\sum | |
i=1 |
\varphiiXt-i+\varepsilont
have been estimated, the autoregression can be used to forecast an arbitrary number of periods into the future. First use t to refer to the first period for which data is not yet available; substitute the known preceding values Xt-i for i=1, ..., p into the autoregressive equation while setting the error term
\varepsilont
There are four sources of uncertainty regarding predictions obtained in this manner: (1) uncertainty as to whether the autoregressive model is the correct model; (2) uncertainty about the accuracy of the forecasted values that are used as lagged values in the right side of the autoregressive equation; (3) uncertainty about the true values of the autoregressive coefficients; and (4) uncertainty about the value of the error term
\varepsilont