Invariant estimator

In statistics, an invariant estimator is a criterion that can be used to compare the properties of different estimators, formalising the idea that an estimator should have certain intuitively appealing qualities. Strictly speaking, "invariant" would mean that the estimates themselves are unchanged when both the measurements and the parameters are transformed in a compatible way, but the meaning has been extended to allow the estimates to change in appropriate ways with such transformations. In formal mathematics, the related term equivariant estimator is used as a precise description of the relation of how the estimator changes in response to changes to the dataset and parameterisation.

General setting

Background

In statistical inference, there are several approaches to estimation theory that can be used to decide immediately what estimators should be used according to those approaches. For example, ideas from Bayesian inference would lead directly to Bayesian estimators. Similarly, the theory of classical statistical inference can sometimes lead to strong conclusions about what estimator should be used. However, the usefulness of these theories depends on having a fully prescribed statistical model and may also depend on having a relevant loss function to determine the estimator. Thus a Bayesian analysis might be undertaken, leading to a posterior distribution for relevant parameters, but the use of a specific utility or loss function may be unclear. Ideas of invariance can then be applied to the task of summarising the posterior distribution. In other cases, statistical analyses are undertaken without a fully defined statistical model or the classical theory of statistical inference cannot be readily applied because the family of models being considered are not amenable to such treatment. In addition to these cases where general theory does not prescribe an estimator, the concept of invariance of an estimator can be applied when seeking estimators of alternative forms, either for the sake of simplicity of application of the estimator or so that the estimator is robust.

The concept of invariance is sometimes used on its own as a way of choosing between estimators, but this is not necessarily definitive. For example, the requirement of unbiasedness is often imposed on an estimator but this is counter-indicated by the possible requirement that the estimator should be invariant to transformations of the parameter being estimated.

One use of the concept of invariance is where a class or family of estimators is proposed and a particular formulation must be selected amongst these. One procedure is to impose relevant invariance properties and then to find the formulation within this class that has the best properties, leading to what is called the optimal invariant estimator.

Some classes of invariant estimators

There are several types of transformations that are usefully considered when dealing with invariant estimators: each gives rise to a class of estimators which are invariant to those particular types of transformation.

• Shift invariance. Notionally, estimates of a location parameter should be invariant to simple shifts of the data values: if all data values are increased by a given amount, the estimate should change by the same amount. When considering esimation using a weighted average, this invariance requirement immediately implies that the weights should sum to one. While the same result is often derived from a requirement for unbiasedness, the use of "invariance" does not require that a mean value exists and makes no use of any probability distribution at all.
• Scale invariance. Note that this is a topic not directly covered in scale invariance.
• Parameter-transformation invariance. Here the transformation applies to the parameters alone. The concept here is that essentially the same inference should be made from data and a model involving a parameter θ as would be made from the same data if the model used a parameter φ, where φ is a one-to-one transformation of θ, φ=h(θ). According to this type of invariance, results from transformation-invariant estimators should also be related by φ=h(θ). Maximum likelihood estimators have this property.
• Permutation invariance. Where a set of data values can be represented by a statistical model that they are outcomes from independent and identically distributed random variables, it is reasonable to impose the requirement that any estimator of any property of the common distribution should be permutation-invariant: specifically that the estimator, considered as a function the set of data-values, should not change if items of data are swapped within the dataset.

The combination of permutation invariance and location invariance for estimating a location parameter from an independent and identically distributed dataset using a weighted average implies that the weights should be identical and sum to one. Of course, estimators other than a weighted average may be preferable.

Optimal invariant estimators

Under this setting, we are given a measurement $x$ which contains information about an unknown parameter $\theta$. The measurement $x$ is modeled as a random variable having a probability density function $f(x|\theta )$ which depends on $\theta$.

We would like to estimate $\theta$ given $x$. The estimate, denoted by $a$, is a function of the measurements and belongs to a set $A$. The quality of the result is defined by a loss function $L=L(a,\theta )$ which determines a risk function $R=R(a,\theta )=E[L(a,\theta )|\theta ]$.

We denote the sets of possible values of $x$, $\theta$, and $a$ by $X$, $\Theta$, and $A$, respectively.

Mathematical setting

Definition

An invariant estimator is an estimator which obeys the following two rules:

1. Principle of Rational Invariance: The action taken in a decision problem should not depend on transformation on the measurement used
2. Invariance Principle: If two decision problems have the same formal structure (in terms of $X$, $\Theta$, $f(x|\theta )$ and $L$) then the same decision rule should be used in each problem

To define an invariant estimator formally we will first set some definitions about groups of transformations:

A group of transformations of $X$, to be denoted by $G$ is a set of (measurable) $1:1$ and onto transformations of $X$ into itself, which satisfies the following conditions:

1. If $g_{1}\in G$ and $g_{2}\in G$ then $g_{1}g_{2}\in G\,$
2. If $g\in G$ then $g^{-1}\in G$, where $g^{-1}(g(x))=x\,.$ (That is, each transformation has an inverse within the group.)
3. $e\in G$ (i.e there is an identity transformation $e(x)=x\,$)

Datasets $x_{1}$ and $x_{2}$ in $X$ are equivalent if $x_{1}=g(x_{2})$ for some $g\in G$. All the equivalent points form an equivalence class. Such an equivalence class is called an orbit (in $X$). The $x_{0}$ orbit, $X(x_{0})$, is the set $X(x_{0})=\{g(x_{0}):g\in G\}$. If $X$ consists of a single orbit then $g$ is said to be transitive.

A family of densities $F$ is said to be invariant under the group $G$ if, for every $g\in G$ and $\theta \in \Theta$ there exists a unique $\theta ^{*}\in \Theta$ such that $Y=g(x)$ has density $f(y|\theta ^{*})$. $\theta ^{*}$ will be denoted ${\bar {g}}(\theta )$.

If $F$ is invariant under the group $G$ then the loss function $L(\theta ,a)$ is said to be invariant under $G$ if for every $g\in G$ and $a\in A$ there exists an $a^{*}\in A$ such that $L(\theta ,a)=L({\bar {g}}(\theta ),a^{*})$ for all $\theta \in \Theta$. $a^{*}$ will be denoted ${\tilde {g}}(a)$.

${\bar {G}}=\{{\bar {g}}:g\in G\}$ is a group of transformations from $\Theta$ to itself and ${\tilde {G}}=\{{\tilde {g}}:g\in G\}$ is a group of transformations from $A$ to itself.

An estimation problem is invariant under $G$ if there exists three such groups $G,{\bar {G}},{\tilde {G}}$ as defined above.

For an estimation problem that is invariant under $G$, estimator $\delta (x)$ is invariant estimator under $G$ if for all $x\in X$ and $g\in G$ $\delta (g(x))={\tilde {g}}(\delta (x))$.

Properties

1. The risk function of an invariant estimator $\delta$ is constant on orbits of $\Theta$. Equivalently $R(\theta ,\delta )=R({\bar {g}}(\theta ),\delta )$ for all $\theta \in \Theta$ and ${\bar {g}}\in {\bar {G}}$.
2. The risk function of an invariant estimator with transitive ${\bar {g}}$ is constant.

For a given problem the invariant estimator with the lowest risk is termed the "best invariant estimator". Best invariant estimator cannot be achieved always. A special case for which it can be achieved is the case when ${\bar {g}}$ is transitive.

Example: Location parameter

$\theta$ is a location parameter if the density of $X$ is $f(x-\theta )$. For $\Theta =A={\mathbb {R}}^{1}$ and $L=L(a-\theta )$ the problem is invariant under $g={\bar {g}}={\tilde {g}}=\{g_{c}:g_{c}(x)=x+c,c\in {\mathbb {R}}\}$. The invariant estimator in this case must satisfy $\delta (x+c)=\delta (x)+c,\forall c\in {\mathbb {R}}$ thus it is of the form $\delta (x)=x+K$ ($K\in {\mathbb {R}}$). ${\bar {g}}$ is transitive on $\Theta$ so we have here constant risk: $R(\theta ,\delta )=R(0,\delta )=E[L(X+K)|\theta =0]$. The best invariant estimator is the one that brings the risk $R(\theta ,\delta )$ to minimum.

In the case that L is squared error $\delta (x)=x-E[X|\theta =0]$

Pitman estimator

Given the estimation problem: $X=(X_{1},\dots ,X_{n})$ that has density $f(x_{1}-\theta ,\dots ,x_{n}-\theta )$ and loss $L(|a-\theta |)$. This problem is invariant under $G=\{g_{c}:g_{c}(x)=(x_{1}+c,\dots ,x_{n}+c),c\in {\mathbb {R}}^{1}\}$, ${\bar {G}}=\{g_{c}:g_{c}(\theta )=\theta +c,c\in {\mathbb {R}}^{1}\}$ and ${\tilde {G}}=\{g_{c}:g_{c}(a)=a+c,c\in {\mathbb {R}}^{1}\}$ (additive groups).

The best invariant estimator $\delta (x)$ is the one that minimize ${\frac {\int _{-\infty }^{\infty }{L(\delta (x)-\theta )f(x_{1}-\theta ,\dots ,x_{n}-\theta )d\theta }}{\int _{-\infty }^{\infty }{f(x_{1}-\theta ,\dots ,x_{n}-\theta )d\theta }}}$ (Pitman's estimator, 1939).

For the square error loss case we get that $\delta (x)={\frac {\int _{-\infty }^{\infty }{\theta f(x_{1}-\theta ,\dots ,x_{n}-\theta )d\theta }}{\int _{-\infty }^{\infty }{f(x_{1}-\theta ,\dots ,x_{n}-\theta )d\theta }}}$

If $x\sim N(\theta 1_{n},I)\,\!$ (normal distribution) than $\delta _{pitman}=\delta _{ML}={\frac {\sum {x_{i}}}{n}}$

If $x\sim C(\theta 1_{n},I)\,\!$ (Cauchy distribution) than $\delta _{pitman}\neq \delta _{ML}$ and $\delta _{pitman}=\sum _{k=1}^{n}{x_{k}\left[{\frac {Re\{w_{k}\}}{\sum _{m=1}^{n}{Re\{w_{k}\}}}}\right]},n>1$ when $w_{k}=\prod _{j\neq k}\left[{\frac {1}{(x_{k}-x_{j})^{2}+4\sigma ^{2}}}\right]\left[1-{\frac {2\sigma }{(x_{k}-x_{j})}}i\right]$

References

• Berger, James O. (1985). Statistical decision theory and Bayesian Analysis (2nd ed.). New York: Springer-Verlag. ISBN 0-387-96098-8. MR0804611.
• The Pitman estimator of the Cauchy location parameter, Gabriela V. Cohen Freue, Journal of Statistical Planning and Inference 137 (2007) 1900 – 1913