Functional regression

Functional regression is a version of the regression analysis when responses or covariates include functional data. On the one hand, functional regression models can be classified into four types depending on whether the responses or covariates are functional or scalar: (i) scalar responses with functional covariates, (ii) functional responses with scalar covariates, (iii) functional responses with functional covariates, and (iv) scalar or functional responses with functional and scalar covariates. On the other hand, functional regression models can be linear, partially linear, or nonlinear. In particular, functional polynomial models, functional single and multiple single models and functional additive models are three special cases of functional nonlinear models.

Functional linear models (FLMs)

Functional linear models (FLMs) are an extension of linear models (LMs). A linear model with scalar response $Y\in \mathbb {R}$ and scalar covariates $X\in \mathbb {R} ^{p}$ can be written as

$$Y=\beta _{0}+\langle X,\beta \rangle +\epsilon ,$$ 1

where $\langle \cdot ,\cdot \rangle$ denotes the inner product in Euclidean space, $\beta _{0}\in \mathbb {R}$ and $\beta \in \mathbb {R} ^{p}$ denote the regression coefficients, and $\epsilon$ is a random error with mean zero and finite variance. FLMs can be divided into two types based on the responses.

Functional linear models with scalar responses

Functional linear models with scalar responses can be obtained by replacing the scalar covariates $X$ and the coefficient vector $\beta$ in model (1) by a centered functional covariate $X^{c}(\cdot )=X(\cdot )-\mathbb {E} (X(\cdot ))$ and a coefficient function $\beta =\beta (\cdot )$ with domain ${\mathcal {T}}$, respectively, and replacing the inner product in Euclidean space by that in $L^{2}$ space,

$$Y=\beta _{0}+\langle X^{c},\beta \rangle +\epsilon =\beta _{0}+\int _{\mathcal {T}}X^{c}(t)\beta (t)dt+\epsilon ,$$ 2

where $\langle \cdot ,\cdot \rangle$ here denotes the inner product in $L^{2}$ space. One approach to estimating $\beta _{0}$ and $\beta (\cdot )$ is to expand the centered covariate $X^{c}(\cdot )$ and the coefficient function $\beta (\cdot )$ in the same functional basis, for example, B-spline basis or the eigenbasis used in the Karhunen–Loève expansion. Suppose $\{\phi _{k}\}_{k=1}^{\infty }$ is an orthonormal basis of $L^{2}$ space. Expanding $X^{c}$ and $\beta$ in this basis, $X^{c}(\cdot )=\sum _{k=1}^{\infty }x_{k}\phi _{k}(\cdot )$, $\beta (\cdot )=\sum _{k=1}^{\infty }\beta _{k}\phi _{k}(\cdot )$, model (2) becomes $$Y=\beta _{0}+\sum _{k=1}^{\infty }\beta _{k}x_{k}+\epsilon .$$ In implementation, regularization is needed and can be done through trucation, $L^{2}$ penalization or $L^{1}$ penalization. In addition, a reproducing kernel Hilbert space (RKHS) approach can also be used to estimate $\beta _{0}$ and $\beta (\cdot )$ in model (2).

Adding multiple functional and scalar covariates, model (2) can be extended to

$$Y=\sum _{k=1}^{q}Z_{k}\alpha _{k}+\sum _{j=1}^{p}\int _{{\mathcal {T}}_{j}}X_{j}^{c}(t)\beta _{j}(t)dt+\epsilon ,$$ 3

where $Z_{1},\cdots ,Z_{q}$ are scalar covariates with $Z_{1}=1$, $\alpha _{1},\cdots ,\alpha _{q}$ are the coefficients corresponding to $Z_{1},\cdots ,Z_{q}$, respectively, $X_{j}^{c}$ is a centered functional covariate given by $X_{j}^{c}(\cdot )=X_{j}(\cdot )-\mathbb {E} (X_{j}(\cdot ))$, $\beta _{j}$ is the corresponding coefficient function, and ${\mathcal {T}}_{j}$ is the domain of $X_{j}$ and $\beta _{j}$, for $j=1,\cdots ,p$. However, due to the parametric component $\alpha$, the estimation methods for model (2) cannot be used in this case and various estimation methods for model (3) are also available.

Functional linear models with functional responses

For a functional response $Y(\cdot )$ with domain ${\mathcal {T}}$ and a functional covariate $X(\cdot )$ with domain ${\mathcal {S}}$, two FLMs regressing $Y(\cdot )$ on $X(\cdot )$ have been considered. One of these two models is of the form

$$Y(t)=\beta _{0}(t)+\int _{\mathcal {S}}\beta (s,t)X^{c}(s)ds+\epsilon (t),\ {\text{for}}\ t\in {\mathcal {T}},$$ 4

where $X^{c}(\cdot )=X(\cdot )-\mathbb {E} (X(\cdot ))$ is still the centered functional covariate, $\beta _{0}(\cdot )$ and $\beta (\cdot ,\cdot )$ are coefficient functions, and $\epsilon (\cdot )$ is usually assumed to be a random process with mean zero and finite variance. In this case, at any given time $t\in {\mathcal {T}}$, the value of $Y$, i.e. $Y(t)$, depends on the entire trajectory of $X$. Model (4), for any given time $t$, is an extension of multivariate linear regression with the inner product in Euclidean space replaced by that in $L^{2}$ space. An estimating equation motivated by multivariate linear regression is $$r_{XY}=R_{XX}\beta ,{\text{ for }}\beta \in L^{2}({\mathcal {S}}\times {\mathcal {S}}),$$ where $r_{XY}(s,t)={\text{cov}}(X(s),Y(t))$, $R_{XX}:L^{2}({\mathcal {S}}\times {\mathcal {S}})\rightarrow L^{2}({\mathcal {S}}\times {\mathcal {T}})$ is defined as $(R_{XX}\beta )(s,t)=\int _{\mathcal {S}}r_{XX}(s,w)\beta (w,t)dw$ with $r_{XX}(s,w)={\text{cov}}(X(s),X(w))$ for $s,w\in {\mathcal {S}}$. Regularization is needed and can be done through truncation, $L^{2}$ penalization or $L^{1}$ penalization. Various estimation methods for model (4) are also available.
When $X$ and $Y$ are concurrently observed, i.e. ${\mathcal {S}}={\mathcal {T}}$, it is sometimes reasonable to consider a historical functional linear model, where the current value of $Y$ only depends on the history of $X$, i.e. $\beta (s,t)=0$ for $s>t$ in model (4).
Adding multiple functional covariates, model (4) can be extended to

$$Y(t)=\beta _{0}(t)+\sum _{j=1}^{p}\int _{{\mathcal {S}}_{j}}\beta _{j}(s,t)X_{j}^{c}(s)ds+\epsilon (t),\ {\text{for}}\ t\in {\mathcal {T}},$$ 5

where for $j=1,\cdots ,p$, $X_{j}^{c}(\cdot )=X_{j}(\cdot )-\mathbb {E} (X_{j}(\cdot ))$ is a centered functional covariate with domain ${\mathcal {S}}_{j}$, and $\beta _{j}(\cdot ,\cdot )$ is the corresponding coefficient function with the same domain, respectively. In particular, taking $X_{j}(\cdot )$ as a constant function yields a special case of model (5) $$Y(t)=\sum _{j=1}^{p}X_{j}\beta _{j}(t)+\epsilon (t),\ {\text{for}}\ t\in {\mathcal {T}},$$ which is a FLM with functional responses and scalar covariates.

Concurrent models

Assuming that ${\mathcal {S}}={\mathcal {T}}$, another model, known as concurrent models, sometimes also referred to as varying-coefficient model, has been proposed, which is easier to handle with than the historical functional linear model. A concurrent model is of the form

$$Y(t)=\alpha _{0}(t)+\alpha (t)X(t)+\epsilon (t),\ {\text{for}}\ t\in {\mathcal {T}},$$ 6

where $\alpha _{0}$ and $\alpha$ are coefficient functions. Note that model (6) assumes the value of $Y$ at time $t$, i.e. $Y(t)$, only depends on that of $X$ at the same time, i.e. $X(t)$. Various estimation methods can be applied for model (6).
Adding multiple functional covariates, model (6) can also be extended to $$Y(t)=\alpha _{0}(t)+\sum _{j=1}^{p}\alpha _{j}(t)X_{j}(t)+\epsilon (t),\ {\text{for}}\ t\in {\mathcal {T}},$$ where $X_{1},\cdots ,X_{p}$ are multiple functional covariates with domain ${\mathcal {T}}$ and $\alpha _{0},\alpha _{1},\cdots ,\alpha _{p}$ are the coefficient functions with the same domain.

Functional nonlinear models

Functional polynomial models

Functional polynomial models are an extension of the FLMs, analogous to extending linear regression to polynomial regression. For a scalar response $Y$ and a functional covariate $X(\cdot )$ with domain ${\mathcal {T}}$, the simplest example of functional polynomial models is functional quadratic regression $$Y=\alpha +\int _{\mathcal {T}}\beta (t)X^{c}(t)dt+\int _{\mathcal {T}}\int _{\mathcal {T}}\gamma (s,t)X^{c}(s)X^{c}(t)dsdt+\epsilon$$ where $X^{c}(\cdot )=X(\cdot )-\mathbb {E} (X(\cdot ))$ is the centered functional covariate, $\alpha$ is a scalar coefficient, $\beta (\cdot )$ and $\gamma (\cdot ,\cdot )$ are coefficient functions with domains ${\mathcal {T}}$ and ${\mathcal {T}}\times {\mathcal {T}}$, respectively, and $\epsilon$ is a random error with mean zero and finite variance. By analogy to FLMs, estimation of functional polynomial models can be obtained through expanding both the centered covariate $X^{c}$ and the coefficient functions $\beta$ and $\gamma$ in an orthonormal basis.

Functional single and multiple index models

A functional multiple index model is given by $$Y=g\left(\int _{\mathcal {T}}X^{c}(t)\beta _{1}(t)dt,\cdots ,\int _{\mathcal {T}}X^{c}(t)\beta _{p}(t)dt\right)+\epsilon .$$ Taking $p=1$ yields a functional single index model. However, this model is problematic due to curse of dimensionality. In other words, with $p>1$ and relatively small sample sizes, this model often leads to high variability of the estimator. Alternatively, a preferable $p$-component functional multiple index model can be formed as $$Y=g_{1}\left(\int _{\mathcal {T}}X^{c}(t)\beta _{1}(t)dt\right)+\cdots +g_{p}\left(\int _{\mathcal {T}}X^{c}(t)\beta _{p}(t)dt\right)+\epsilon .$$ Various estimation methods for functional single and multiple index models are available.

Functional additive models (FAMs)

Given an expansion of a functional covariate $X$ with domain ${\mathcal {T}}$ in an orthonormal basis $\{\phi _{k}\}_{k=1}^{\infty }$: $X(t)=\sum _{k=1}^{\infty }x_{k}\phi _{k}(t)$, a functional linear model with scalar responses shown in model (2) can be written as $$\mathbb {E} (Y|X)=\mathbb {E} (Y)+\sum _{k=1}^{\infty }\beta _{k}x_{k}.$$ One form of FAMs is obtained by replacing the linear function of $x_{k}$, i.e. $\beta _{k}x_{k}$, by a general smooth function $f_{k}$,

$$\mathbb {E} (Y|X)=\mathbb {E} (Y)+\sum _{k=1}^{\infty }f_{k}(x_{k}),$$ 7

where $f_{k}$ satisfies $\mathbb {E} (f_{k}(x_{k}))=0$ for $k\in \mathbb {N}$. Apart from model (7), another form of FAMs consists of a sequence of time-additive models:

$$\mathbb {E} (Y|X(t_{1}),\cdots ,X(t_{p}))=\sum _{j=1}^{p}f_{j}(X(t_{j})),$$ 8

where $t_{1}\cdots ,t_{p}$ are a dense grid on ${\mathcal {T}}$ with increasing size $p\in \mathbb {N}$, and $f_{j}(x)=g(t_{j},x)$ with $g$ a smooth function, for $j=1,\cdots ,p$. Various estimation methods for models (7) and (8) are both available.

Extensions

A direct extension of FLMs with scalar responses shown in model (2) is to add a link function to create a generalized functional linear model (GFLM) by analogy to extending linear regression to generalized linear regression (GLM), of which the three components are:

1. Linear predictor $\eta =\beta _{0}+\int _{\mathcal {T}}X^{c}(t)\beta (t)dt$;
2. Variance function ${\text{Var}}(Y|X)=V(\mu )$, where $\mu =\mathbb {E} (Y|X)$ is the conditional mean;
3. Link function $g$ connecting the conditional mean and the linear predictor through $\mu =g(\eta )$.

See also

• Functional data analysis
• Functional principal component analysis
• Karhunen–Loève theorem
• Generalized functional linear model
• Stochastic processes