Errors-in-variables models explained
In statistics, errors-in-variables models or measurement error models are regression models that account for measurement errors in the independent variables. In contrast, standard regression models assume that those regressors have been measured exactly, or observed without error; as such, those models account only for errors in the dependent variables, or responses.
In the case when some regressors have been measured with errors, estimation based on the standard assumption leads to inconsistent estimates, meaning that the parameter estimates do not tend to the true values even in very large samples. For simple linear regression the effect is an underestimate of the coefficient, known as the attenuation bias. In non-linear models the direction of the bias is likely to be more complicated.[1] [2] [3]
Motivating example
Consider a simple linear regression model of the form
yt=\alpha+\beta
+\varepsilont, t=1,\ldots,T,
where
denotes the
true but
unobserved regressor. Instead we observe this value with an error:
where the measurement error
is assumed to be independent of the true value
.
If the
′s are simply regressed on the
′s (see
simple linear regression), then the estimator for the slope coefficient is
\hat{\beta}x=
t-\bar{y})}
| 2},
|
{\tfrac{1}{T}\sum | |
| t-\bar{x}) |
which converges as the sample size
increases without bound:
\hat{\beta}x\xrightarrow{p}
t,yt]}{\operatorname{Var}[xt]}
=
=
.
This is in contrast to the "true" effect of
, estimated using the
,:
Variances are non-negative, so that in the limit the estimated
is smaller than
, an effect which statisticians call
attenuation or
regression dilution.
[4] Thus the ‘naïve’ least squares estimator
is an
inconsistent estimator for
. However,
is a
consistent estimator of the parameter required for a best linear predictor of
given the observed
: in some applications this may be what is required, rather than an estimate of the ‘true’ regression coefficient
, although that would assume that the variance of the errors in the estimation and prediction is identical. This follows directly from the result quoted immediately above, and the fact that the regression coefficient relating the
′s to the actually observed
′s, in a simple linear regression, is given by
\betax=
t,yt]}{\operatorname{Var}[xt]}.
It is this coefficient, rather than
, that would be required for constructing a predictor of
based on an observed
which is subject to noise.
It can be argued that almost all existing data sets contain errors of different nature and magnitude, so that attenuation bias is extremely frequent (although in multivariate regression the direction of bias is ambiguous[5]). Jerry Hausman sees this as an iron law of econometrics: "The magnitude of the estimate is usually smaller than expected."[6]
Specification
Usually measurement error models are described using the latent variables approach. If
is the response variable and
are observed values of the regressors, then it is assumed there exist some latent variables
and
which follow the model's “true”
functional relationship
, and such that the observed quantities are their noisy observations:
\begin{cases}
y*=g(x*,w|\theta),\ y=y*+\varepsilon,\\
x=x*+η,\end{cases}
where
is the model's
parameter and
are those regressors which are assumed to be error-free (for example when linear regression contains an intercept, the regressor which corresponds to the constant certainly has no "measurement errors"). Depending on the specification these error-free regressors may or may not be treated separately; in the latter case it is simply assumed that corresponding entries in the variance matrix of
's are zero.
The variables
,
,
are all
observed, meaning that the statistician possesses a
data set of
statistical units
which follow the
data generating process described above; the latent variables
,
,
, and
are not observed however.
This specification does not encompass all the existing errors-in-variables models. For example in some of them function
may be
non-parametric or semi-parametric. Other approaches model the relationship between
and
as distributional instead of functional, that is they assume that
conditionally on
follows a certain (usually parametric) distribution.
Terminology and assumptions
may be called the
manifest,
indicator, or
proxy variable.
may be called the
latent or
true variable. It may be regarded either as an unknown constant (in which case the model is called a
functional model), or as a random variable (correspondingly a
structural model).
[7] - The relationship between the measurement error
and the latent variable
can be modeled in different ways:
the errors are
independent of the latent variable. This is the most common assumption, it implies that the errors are introduced by the measuring device and their magnitude does not depend on the value being measured.
\operatorname{E}[η|x*]=0,
the errors are mean-zero for every value of the latent regressor. This is a less restrictive assumption than the classical one,
[8] as it allows for the presence of
heteroscedasticity or other effects in the measurement errors.
the errors are independent of the
observed regressor
x.
[9] This assumption has very limited applicability. One example is round-off errors: for example if a person's
age* is a
continuous random variable, whereas the observed
age is truncated to the next smallest integer, then the truncation error is approximately independent of the observed
age. Another possibility is with the fixed design experiment: for example if a scientist decides to make a measurement at a certain predetermined moment of time
, say at
, then the real measurement may occur at some other value of
(for example due to her finite reaction time) and such measurement error will be generally independent of the "observed" value of the regressor.
is an indicator of a certain event or condition (such as person is male/female, some medical treatment given/not, etc.), then the measurement error in such regressor will correspond to the incorrect classification similar to
type I and type II errors in statistical testing. In this case the error
may take only 3 possible values, and its distribution conditional on
is modeled with two parameters:
\alpha=\operatorname{Pr}[η=-1|x*=1]
, and
\beta=\operatorname{Pr}[η=1|x*=0]
. The necessary condition for identification is that
, that is misclassification should not happen "too often". (This idea can be generalized to discrete variables with more than two possible values.)
Linear model
Linear errors-in-variables models were studied first, probably because linear models were so widely used and they are easier than non-linear ones. Unlike standard least squares regression (OLS), extending errors in variables regression (EiV) from the simple to the multivariable case is not straightforward.
Simple linear model
The simple linear errors-in-variables model was already presented in the "motivation" section:
\begin{cases}
yt=\alpha+\beta
+\varepsilont,\\
xt=
+ηt,
\end{cases}
where all variables are
scalar. Here
α and
β are the parameters of interest, whereas
σε and
ση—standard deviations of the error terms—are the
nuisance parameters. The "true" regressor
x* is treated as a random variable (
structural model), independent of the measurement error
η (
classic assumption).
This model is identifiable in two cases: (1) either the latent regressor x* is not normally distributed, (2) or x* has normal distribution, but neither εt nor ηt are divisible by a normal distribution.[10] That is, the parameters α, β can be consistently estimated from the data set
without any additional information, provided the latent regressor is not Gaussian.
Before this identifiability result was established, statisticians attempted to apply the maximum likelihood technique by assuming that all variables are normal, and then concluded that the model is not identified. The suggested remedy was to assume that some of the parameters of the model are known or can be estimated from the outside source. Such estimation methods include[11]
- Deming regression — assumes that the ratio δ = σ²ε/σ²η is known. This could be appropriate for example when errors in y and x are both caused by measurements, and the accuracy of measuring devices or procedures are known. The case when δ = 1 is also known as the orthogonal regression.
- Regression with known reliability ratio λ = σ²∗/ (σ²η + σ²∗), where σ²∗ is the variance of the latent regressor. Such approach may be applicable for example when repeating measurements of the same unit are available, or when the reliability ratio has been known from the independent study. In this case the consistent estimate of slope is equal to the least-squares estimate divided by λ.
- Regression with known σ²η may occur when the source of the errors in xs is known and their variance can be calculated. This could include rounding errors, or errors introduced by the measuring device. When σ²η is known we can compute the reliability ratio as λ = (σ²x − σ²η) / σ²x and reduce the problem to the previous case.
Estimation methods that do not assume knowledge of some of the parameters of the model, include
Multivariable linear model
The multivariable model looks exactly like the simple linear model, only this time β, ηt, xt and x*t are k×1 vectors.
\begin{cases}
yt=\alpha+
+\varepsilont,\\
xt=
+ηt.
\end{cases}
In the case when (εt,ηt) is jointly normal, the parameter β is not identified if and only if there is a non-singular k×k block matrix [''a A''], where a is a k×1 vector such that a′x* is distributed normally and independently of A′x*. In the case when εt, ηt1,..., ηtk are mutually independent, the parameter β is not identified if and only if in addition to the conditions above some of the errors can be written as the sum of two independent variables one of which is normal.[12]
Some of the estimation methods for multivariable linear models are
Non-linear models
A generic non-linear measurement error model takes form
\begin{cases}
yt=
+\varepsilont,\\
xt=
+ηt.
\end{cases}
Here function
g can be either parametric or non-parametric. When function
g is parametric it will be written as
g(
x*,
β).
For a general vector-valued regressor x* the conditions for model identifiability are not known. However in the case of scalar x* the model is identified unless the function g is of the "log-exponential" form [13]
and the latent regressor
x* has density
(x)=\begin{cases}
A
(eCx+E)-F,&if d>0\\
A
&if d=0
\end{cases}
where constants
A,
B,
C,
D,
E,
F may depend on
a,
b,
c,
d.
Despite this optimistic result, as of now no methods exist for estimating non-linear errors-in-variables models without any extraneous information. However there are several techniques which make use of some additional data: either the instrumental variables, or repeated observations.
Repeated observations
In this approach two (or maybe more) repeated observations of the regressor x* are available. Both observations contain their own measurement errors, however those errors are required to be independent:
\begin{cases}
x1t=
+η1t,\\
x2t=
+η2t,
\end{cases}
where
x* ⊥
η1 ⊥
η2. Variables
η1,
η2 need not be identically distributed (although if they are efficiency of the estimator can be slightly improved). With only these two observations it is possible to consistently estimate the density function of
x* using Kotlarski's
deconvolution technique.
[14] Further reading
- Book: Dougherty, Christopher . Stochastic Regressors and Measurement Errors . Introduction to Econometrics . Oxford University Press . Fourth . 2011 . 978-0-19-956708-9 . 300–330 . https://books.google.com/books?id=UXucAQAAQBAJ&pg=PA300 .
- Book: Kmenta, Jan . Jan Kmenta . Estimation with Deficient Data . 346–391 . Elements of Econometrics . New York . Macmillan . 1986 . Second . 978-0-02-365070-3 . https://books.google.com/books?id=Bxq7AAAAIAAJ&pg=PA346 . registration .
- Book: Schennach. Susanne. Daron. Acemoglu. Manuel. Arellano. Eddie. Dekel. Susanne Schennach. Measurement Error in Nonlinear Models – A Review . Advances in Economics and Econometrics . Cambridge University Press . 2013 . 296–337 . 9781107017214 . 10.1017/CBO9781139060035.009 . 10419/79526.
External links
Notes and References
- Griliches . Zvi . Ringstad . Vidar . 1970 . Errors-in-the-variables bias in nonlinear contexts . . 38 . 2 . 368–370 . 1913020 . 10.2307/1913020 .
- Chesher . Andrew . 1991 . The effect of measurement error . . 78 . 3 . 451–462 . 2337015 . 10.1093/biomet/78.3.451 .
- Book: Raymond J. . Carroll . David . Ruppert . Leonard A. . Stefanski . Ciprian . Crainiceanu . Measurement Error in Nonlinear Models: A Modern Perspective . Second . 978-1-58488-633-4 . 2006 .
- Book: Greene, William H. . 2003 . Econometric Analysis . 5th . Prentice Hall . New Jersey . 978-0-13-066189-0 . Chapter 5.6.1 .
- Book: 2000 . Measurement Error and Latent Variables . CITEREFWansbeek_and_Meijer2000 . Wansbeek . T. . Meijer . E. . Baltagi . B. H. . Blackwell . 9781405106764. 162–179 . https://books.google.com/books?id=xs55E7FsMHMC&pg=PA162 . 10.1111/b.9781405106764.2003.00013.x . A Companion to Theoretical Econometrics .
- 10.1257/jep.15.4.57 . Hausman . Jerry A. . 2001 . Mismeasured variables in econometric analysis: problems from the right and problems from the left . . 15 . 4 . 57–67 [p. 58] . 2696516 . free .
- Book: Fuller, Wayne A. . 1987 . Measurement Error Models . John Wiley & Sons . 978-0-471-86187-4 . 2 .
- Book: Hayashi, Fumio . Econometrics . Princeton University Press . 2000 . 7–8 . 978-1400823833 .
- Koul . Hira . Song . Weixing . 2008 . Regression model checking with Berkson measurement errors . Journal of Statistical Planning and Inference . 138 . 6 . 1615–1628 . 10.1016/j.jspi.2007.05.048 .
- Reiersøl . Olav . 1950 . Identifiability of a linear relation between variables which are subject to error . . 18 . 4 . 375–389 [p. 383] . 1907835 . 10.2307/1907835 . A somewhat more restrictive result was established earlier by R. C. . Geary . Inherent relations between random variables . . 47 . 1942 . 63–76 . 20488436 . He showed that under the additional assumption that (ε, η) are jointly normal, the model is not identified if and only if x*s are normal.
- Book: Fuller, Wayne A. . 1987 . A Single Explanatory Variable . Measurement Error Models . John Wiley & Sons . 978-0-471-86187-4 . 1–99 . https://books.google.com/books?id=Nalc0DkAJRYC&pg=PA1 .
- Ben-Moshe . Dan . 2020 . Identification of linear regressions with errors in all variables . . 37 . 4 . 1–31 . 10.1017/S0266466620000250. 1404.1473 . 225653359 .
- 2007 . Nonparametric identification of the classical errors-in-variables model without side information . Working Paper . Schennach . S. . Susanne Schennach . Hu . Y. . Lewbel . A. .
- Li . Tong . Vuong . Quang . 1998 . Nonparametric estimation of the measurement error model using multiple indicators . . 65 . 2 . 139–165 . 10.1006/jmva.1998.1741 . free .