Functional regression explained

Functional regression is a version of regression analysis when responses or covariates include functional data. Functional regression models can be classified into four types depending on whether the responses or covariates are functional or scalar: (i) scalar responses with functional covariates, (ii) functional responses with scalar covariates, (iii) functional responses with functional covariates, and (iv) scalar or functional responses with functional and scalar covariates. In addition, functional regression models can be linear, partially linear, or nonlinear. In particular, functional polynomial models, functional single and multiple index models and functional additive models are three special cases of functional nonlinear models.

__TOC__

Functional linear models (FLMs)

Functional linear models (FLMs) are an extension of linear models (LMs). A linear model with scalar response

Y\in R and scalar covariates X\in R ^{p} can be written aswhere \langle ⋅ , ⋅ \rangle denotes the inner product in Euclidean space, \beta _{0\inR} and \beta\in R ^{p} denote the regression coefficients, and \varepsilon is a random error with mean zero and finite variance. FLMs can be divided into two types based on the responses.

Functional linear models with scalar responses

Functional linear models with scalar responses can be obtained by replacing the scalar covariates

X and the coefficient vector \beta in model by a centered functional covariate X ^{c( ⋅ )} = X( ⋅ ) - E (X( ⋅ )) and a coefficient function \beta = \beta( ⋅ ) with domain l{T} , respectively, and replacing the inner product in Euclidean space by that in Hilbert space L ^{2} ,where \langle ⋅ , ⋅ \rangle here denotes the inner product in L ^{2} . One approach to estimating \beta _{0} and \beta( ⋅ ) is to expand the centered covariate X ^{c( ⋅ )} and the coefficient function \beta( ⋅ ) in the same functional basis, for example, B-spline basis or the eigenbasis used in the Karhunen - Loève expansion. Suppose \{\phi _{k\}} ^{inf ty} ^{k =1} is an orthonormal basis of L ^{2} . Expanding X ^{c} and \beta in this basis, X ^{c( ⋅ )} = \sum ^{inf ty} _{k =1} x _{k} \phi _{k( ⋅ )} , \beta( ⋅ ) = \sum ^{inf ty} _{k =1} \beta _{k} \phi _{k( ⋅ )} , model becomes$$Y\; =\; \backslash beta\_0\; +\; \backslash sum\_^\backslash infty\; \backslash beta\_k\; x\_k\; +\backslash varepsilon.$$For implementation, regularization is needed and can be done through truncation, L ^{2} penalization or L ^{1} penalization.^[1] In addition, a reproducing kernel Hilbert space (RKHS) approach can also be used to estimate \beta _{0} and \beta( ⋅ ) in model ^[2]

Adding multiple functional and scalar covariates, model can be extended towhere

Z _{1,\ldots,Z} _{q} are scalar covariates with Z _{1=1} , \alpha _{1,\ldots,\alpha} _{q} are regression coefficients for Z _{1,\ldots,Z} _{q} , respectively, X ^{c} _{j} is a centered functional covariate given by X ^{c( ⋅ )} _{j} = X _{j( ⋅ )} - E (X _{j( ⋅ ))} , \beta _{j} is regression coefficient function for X ^{c( ⋅ )} _{j} , and l{T} _{j} is the domain of X _{j} and \beta _{j} , for j =1 ,\ldots,p . However, due to the parametric component \alpha , the estimation methods for model cannot be used in this case^[3] and alternative estimation methods for model are available.^{[4] [5]}

Functional linear models with functional responses

For a functional response

Y( ⋅ ) with domain l{T} and a functional covariate X( ⋅ ) with domain l{S} , two FLMs regressing Y( ⋅ ) on X( ⋅ ) have been considered.^{[3] [6]} One of these two models is of the formwhere X ^{c( ⋅ )} = X( ⋅ ) - E (X( ⋅ )) is still the centered functional covariate, \beta _{0( ⋅ )} and \beta( ⋅ , ⋅ ) are coefficient functions, and \varepsilon( ⋅ ) is usually assumed to be a random process with mean zero and finite variance. In this case, at any given time t\in l{T} , the value of Y , i.e., Y(t) , depends on the entire trajectory of X . Model, for any given time t , is an extension of multivariate linear regression with the inner product in Euclidean space replaced by that in L ^{2} . An estimating equation motivated by multivariate linear regression is$$r\_\; =\; R\_\backslash beta,\; \backslash text\; \backslash beta\backslash in\; L^2(\backslash mathcal\backslash times\backslash mathcal),$$where r _{XY} (s,t) = cov (X(s),Y(t)) , R _{XX} : L ^{2(l{S} x l{S}) → L} ^{2(l{S} x l{T})} is defined as (R _{XX} \beta)(s,t) = \int _{l{S}r} _{XX} (s,w)\beta(w,t)dw with r _{XX} (s,w) = cov (X(s),X(w)) for s,w\in l{S} .^[3] Regularization is needed and can be done through truncation, L ^{2} penalization or L ^{1} penalization.^[1] Various estimation methods for model are available.^{[7] [8]}
When X and Y are concurrently observed, i.e., l{S}= l{T} ,^[9] it is reasonable to consider a historical functional linear model, where the current value of Y only depends on the history of X , i.e., \beta(s,t) =0 for s>t in model .^{[3] [10]} A simpler version of the historical functional linear model is the functional concurrent model (see below).
Adding multiple functional covariates, model can be extended towhere for j =1 ,\ldots,p , X ^{c( ⋅ ) = X} _{j( ⋅ )} - E (X _{j( ⋅ ))} is a centered functional covariate with domain l{S} _{j} , and \beta _{j( ⋅ , ⋅ )} is the corresponding coefficient function with the same domain, respectively.^[3] In particular, taking X _{j( ⋅ )} as a constant function yields a special case of model $$Y(t)\; =\; \backslash sum\_^p\; X\_j\; \backslash beta\_j(t)\; +\; \backslash varepsilon(t),\backslash \; \backslash text\backslash \; t\backslash in\backslash mathcal,$$which is a FLM with functional responses and scalar covariates.

Functional concurrent models

Assuming that

l{S} = l{T} , another model, known as the functional concurrent model, sometimes also referred to as the varying-coefficient model, is of the formwhere \alpha _{0} and \alpha are coefficient functions. Note that model assumes the value of Y at time t , i.e., Y(t) , only depends on that of X at the same time, i.e., X(t) . Various estimation methods can be applied to model .^{[11] [12] [13]}
Adding multiple functional covariates, model can also be extended to$$Y(t)\; =\; \backslash alpha\_0(t)\; +\; \backslash sum\_^p\backslash alpha\_j(t)X\_j(t)+\backslash varepsilon(t),\backslash \; \backslash text\backslash \; t\backslash in\backslash mathcal,$$where X _{1,\ldots,X} _{p} are multiple functional covariates with domain l{T} and \alpha _{0,\alpha} _{1,\ldots,\alpha} _{p} are the coefficient functions with the same domain.^[3]

Functional nonlinear models

Functional polynomial models

Functional polynomial models are an extension of the FLMs with scalar responses, analogous to extending linear regression to polynomial regression. For a scalar response

Y and a functional covariate X( ⋅ ) with domain l{T} , the simplest example of functional polynomial models is functional quadratic regression^[14] $$Y\; =\; \backslash alpha\; +\; \backslash int\_\backslash mathcal\backslash beta(t)X^c(t)\backslash ,dt\; +\; \backslash int\_\backslash mathcal\; \backslash int\_\backslash mathcal\; \backslash gamma(s,t)\; X^c(s)X^c(t)\; \backslash ,ds\backslash ,dt\; +\; \backslash varepsilon,$$where X ^{c( ⋅ )} = X( ⋅ ) - E (X( ⋅ )) is the centered functional covariate, \alpha is a scalar coefficient, \beta( ⋅ ) and \gamma( ⋅ , ⋅ ) are coefficient functions with domains l{T} and l{T} x l{T} , respectively, and \varepsilon is a random error with mean zero and finite variance. By analogy to FLMs with scalar responses, estimation of functional polynomial models can be obtained through expanding both the centered covariate X ^{c} and the coefficient functions \beta and \gamma in an orthonormal basis.^[14]

Functional single and multiple index models

A functional multiple index model is given by$$Y\; =\; g\backslash left(\backslash int\_\; X^c(t)\; \backslash beta\_1(t)\backslash ,dt,\; \backslash ldots,\; \backslash int\_\; X^c(t)\; \backslash beta\_p(t)\backslash ,dt\; \backslash right)\; +\; \backslash varepsilon.$$Taking

p =1 yields a functional single index model. However, for p> 1 , this model is problematic due to curse of dimensionality. With p> 1 and relatively small sample sizes, the estimator given by this model often has large variance.^[15] An alternative p -component functional multiple index model can be expressed as$$Y\; =\; g\_1\backslash left(\backslash int\_\; X^c(t)\; \backslash beta\_1(t)\backslash ,dt\backslash right)+\; \backslash cdots+\; g\_p\backslash left(\backslash int\_\; X^c(t)\; \backslash beta\_p(t)\backslash ,dt\; \backslash right)\; +\; \backslash varepsilon.$$Estimation methods for functional single and multiple index models are available.^{[15] [16]}

Functional additive models (FAMs)

Given an expansion of a functional covariate

X with domain l{T} in an orthonormal basis \{\phi _{k\}} ^{inf ty} ^{k =1} : X(t) = \sum ^{inf ty} _{k =1} x _{k} \phi _{k(t)} , a functional linear model with scalar responses shown in model can be written as$$\backslash mathbb(Y|X)=\backslash mathbb(Y)\; +\; \backslash sum\_^\backslash infty\; \backslash beta\_k\; x\_k.$$One form of FAMs is obtained by replacing the linear function of x _{k} , i.e., \beta _{k} x _{k} , by a general smooth function f _{k} ,$$\backslash mathbb(Y|X)=\backslash mathbb(Y)\; +\; \backslash sum\_^\backslash infty\; f\_k(x\_k),$$where f _{k} satisfies E (f _{k(x} _{k))=0} for k\in N .^{[3] [17]} Another form of FAMs consists of a sequence of time-additive models:$$\backslash mathbb(Y|X(t\_1),\backslash ldots,X(t\_p))=\backslash sum\_^p\; f\_j(X(t\_j)),$$where \{t _{1,\ldots,t} _{p\}} is a dense grid on l{T} with increasing size p\in N , and f _{j(x)} = g(t _{j,x)} with g a smooth function, for j =1 ,\ldots,p ^{[3] [18]}

Extensions

A direct extension of FLMs with scalar responses shown in model is to add a link function to create a generalized functional linear model (GFLM) by analogy to extending linear regression to generalized linear regression (GLM), of which the three components are:

1. Linear predictor

η = \beta _{0} + \int _{l{T} } X^c(t)\beta(t)\,dt;Var (Y|X) = V(\mu) , where \mu = E (Y|X) is the conditional mean;

1. Link function

g connecting the conditional mean and the linear predictor through \mu = g( η ) .

See also

• Functional data analysis
• Functional principal component analysis
• Karhunen - Loève theorem
• Generalized functional linear model
• Stochastic processes

References

1. 10.1146/annurev-statistics-010814-020413. Functional Regression. 2015. Morris. Jeffrey S.. Annual Review of Statistics and Its Application. 2. 1. 321–359. 1406.4068. 2015AnRSA...2..321M. 18637009.
2. Yuan and Cai (2010). "A reproducing kernel Hilbert space approach to functional linear regression". The Annals of Statistics. 38 (6):3412 - 3444. doi:10.1214/09-AOS772.
3. 10.1146/annurev-statistics-041715-033624. Functional Data Analysis. 2016. Wang. Jane-Ling. Chiou. Jeng-Min. Müller. Hans-Georg. Annual Review of Statistics and Its Application. 3. 1. 257–295. 2016AnRSA...3..257W. free.
4. Kong, Xue, Yao and Zhang (2016). "Partially functional linear regression in high dimensions". Biometrika. 103 (1):147 - 159. doi:10.1093/biomet/asv062.
5. Hu, Wang and Carroll (2004). "Profile-kernel versus backfitting in the partially linear models for longitudinal/clustered data". Biometrika. 91 (2): 251 - 262. doi:10.1093/biomet/91.2.251.
6. Ramsay and Silverman (2005). Functional data analysis, 2nd ed., New York: Springer, .
7. Ramsay and Dalzell (1991). "Some tools for functional data analysis". Journal of the Royal Statistical Society. Series B (Methodological). 53 (3):539 - 572. https://www.jstor.org/stable/2345586.
8. Yao, Müller and Wang (2005). "Functional linear regression analysis for longitudinal data". The Annals of Statistics. 33 (6):2873 - 2903. doi:10.1214/009053605000000660.
9. Grenander (1950). "Stochastic processes and statistical inference". Arkiv Matematik. 1 (3):195 - 277. doi:10.1007/BF02590638.
10. Malfait and Ramsay (2003). "The historical functional linear model". Canadian Journal of Statistics. 31 (2):115 - 128. doi:10.2307/3316063.
11. Fan and Zhang (1999). "Statistical estimation in varying coefficient models". The Annals of Statistics. 27 (5):1491 - 1518. doi:10.1214/aos/1017939139.
12. Huang, Wu and Zhou (2004). "Polynomial spline estimation and inference for varying coefficient models with longitudinal data". Biometrika. 14 (3):763 - 788. https://www.jstor.org/stable/24307415.
13. Şentürk and Müller (2010). "Functional varying coefficient models for longitudinal data". Journal of the American Statistical Association. 105 (491):1256 - 1264. doi:10.1198/jasa.2010.tm09228.
14. Yao and Müller (2010). "Functional quadratic regression". Biometrika. 97 (1):49 - 64. doi:10.1093/biomet/asp069.
15. Chen, Hall and Müller (2011). "Single and multiple index functional regression models with nonparametric link". The Annals of Statistics. 39 (3):1720 - 1747. doi:10.1214/11-AOS882.
16. Jiang and Wang (2011). "Functional single index models for longitudinal data". 39 (1):362 - 388. doi:10.1214/10-AOS845.
17. Müller and Yao (2008). "Functional additive models". Journal of the American Statistical Association. 103 (484):1534 - 1544. doi:10.1198/016214508000000751.
18. Fan, James and Radchenko (2015). "Functional additive regression". The Annals of Statistics. 43 (5):2296 - 2325. doi:10.1214/15-AOS1346.