Invariant estimator

Invariant Estimator is an intuitively appealing non Bayesian estimator. It is also sometimes called an "equivariant estimator". In the estimation problem we have random vector $x$ from space $X$ with density function $f(x|\theta )$ when $\theta$ is from the space $\Theta$. We want to estimate $\theta$ given set of measurements from the distribution $f(x|\theta )$. The estimation is denoted by $a$, is a function of the measurements and is in the space $A$. The quality of the result is defined by a loss function $L=L(a,\theta )$ which determine a risk function $R=R(a,\theta )=E[L(a,\theta )|\theta ]$.

Generally speaking invariant estimator is an estimator that obey the 2 following rules:

1. Principle of Rational Invariance: The action taken in a decision problem should not depend on transformation on the measurement used

2. Invariance Principle: If two decision problems have the same formal structure (in terms of $X$, $\Theta$, $f(x|\theta )$ and $L$) then the same decision rule should be used in each problem

To define invariant estimator formally we will first set some definitions about groups of transformations:

Invariant Estimation Problem and Invariant Estimator

A group of transformation of $X$, to be denoted by $G$ is a set of (measurable) $1:1$ and onto transformation of $X$ into itself, which satisfies the following conditions:

1. If $g_{1}\in G$ and $g_{2}\in G$ then $g_{1},g_{2}\in G$

2. If $g\in G$ then $g^{-1}\in G$ ($g^{-1}(g(x))=x)$

3. $e\in G$ ($e(x)=x$)

$x_{1}$ and $x_{2}$ in $X$ are equivalent if $x_{1}=g(x_{2})$ for some $g\in G$. All the equivalent points form an equivalence class. Such equivalence class is called orbit (in $X$). The $x_{0}$ orbit, $X(x_{0})$, is the set $X(x_{0})=\{g(x_{0}):g\in G\}$. If $X$ consist of a single orbit than $g$ is said to be transitive.

A family of densities $F$ is said to be invariant under the group $G$ if, for every $g\in G$ and $\theta \in \Theta$ there exists a unique $\theta ^{*}\in \Theta$ such that $Y=g(x)$ has density $f(y|\theta ^{*})$. $\theta ^{*}$ will be denoted ${\bar {g}}(\theta )$.

If $F$ is invariant under the group $G$ than the loss function $L(\theta ,a)$ is said to be invariant under $G$ if for every $g\in G$ and $a\in A$ there exists an $a^{*}\in A$ such that $L(\theta ,a)=L({\bar {g}}(\theta ),a^{*})$ for all $\theta \in \Theta$. $a^{*}$ will be denoted ${\tilde {g}}(a)$.

${\bar {G}}=\{{\bar {g}}:g\in G\}$ is a group of transformations from $\Theta$ to itself and ${\tilde {G}}=\{{\tilde {g}}:g\in G\}$ is a group of transformations from $A$ to itself.

An estimation problem is invariant under $G$ if there exists such three groups $G,{\bar {G}},{\tilde {G}}$.

For an estimation problem that is invariant under $G$, estimator $\delta (x)$ is invariant estimator under $G$ if for all $x\in X$ and $g\in G$ $\delta (g(x))={\tilde {g}}(\delta (x))$.

Properties of Invariant Estimators

1. The risk function of an invariant estimator $\delta$ is constant on orbits of $\Theta$. Equivalently $R(\theta ,\delta )=R({\bar {g}}(\theta ),\delta )$ for all $\theta \in \Theta$ and ${\bar {g}}\in {\bar {G}}$.

2. The risk function of an invariant estimator with transitive ${\bar {g}}$ is constant.

For a given problem the invariant estimator with the lowest risk is termed the "best invariant estimator". Best invariant estimator cannot be achieved always. A special case for which it can be achieved is the case when ${\bar {g}}$ is transitive.

Location Parameter Problem Example

$\theta$ is a location parameter if the density of $X$ is $f(x-\theta )$. For $\Theta =A={\mathbb {R}}^{1}$ and $L=L(a-\theta )$ the problem is invariant under $g={\bar {g}}={\tilde {g}}=\{g_{c}:g_{c}(x)=x+c,c\in {\mathbb {R}}\}$. The invariant estimator in this case must satisfy $\delta (x+c)=\delta (x)+c,\forall c\in {\mathbb {R}}$ thus it is of the form $\delta (x)=x+K$ ($K\in {\mathbb {R}}$). ${\bar {g}}$ is transitive on $\Theta$ so we have here constant risk: $R(\theta ,\delta )=R(0,\delta )=E[L(X+K)|\theta =0]$. The best invariant estimator is the one that bring the risk $R(\theta ,\delta )$ to minimum.

In the case that L is squared error $\delta (x)=x-E[X|\theta =0]$

Pitman Estimator

Given the estimation problem: $X=(X_{1},\dots ,X_{n})$ that has density $f(x_{1}-\theta ,\dots ,x_{n}-\theta )$ and loss $L(|a-\theta |)$. This problem is invariant under $G=\{g_{c}:g_{c}(x)=(x_{1}+c,\dots ,x_{n}+c),c\in {\mathbb {R}}^{1}\}$, ${\bar {G}}=\{g_{c}:g_{c}(\theta )=\theta +c,c\in {\mathbb {R}}^{1}\}$ and ${\tilde {G}}=\{g_{c}:g_{c}(a)=a+c,c\in {\mathbb {R}}^{1}\}$ (additive groups).

The best invariant estimator $\delta (x)$ is the one that minimize ${\frac {\int _{-\infty }^{\infty }{L(\delta (x)-\theta )f(x_{1}-\theta ,\dots ,x_{n}-\theta )d\theta }}{\int _{-\infty }^{\infty }{f(x_{1}-\theta ,\dots ,x_{n}-\theta )d\theta }}}$ (Pitman's estimator, 1939).

For the square error loss case we get that $\delta (x)={\frac {\int _{-\infty }^{\infty }{\theta f(x_{1}-\theta ,\dots ,x_{n}-\theta )d\theta }}{\int _{-\infty }^{\infty }{f(x_{1}-\theta ,\dots ,x_{n}-\theta )d\theta }}}$

If $x\sim N(\theta 1_{n},I)\,\!$ than $\delta _{pitman}=\delta _{ML}={\frac {\sum {x_{i}}}{n}}$

If $x\sim C(\theta 1_{n},I)\,\!$ than $\delta _{pitman}\neq \delta _{ML}$ and $\delta _{pitman}=\sum _{k=1}^{n}{x_{k}[{\frac {Re\{w_{k}\}}{\sum _{m=1}^{n}{Re\{w_{k}\}}}}]},n>1$ when $w_{k}=\prod _{j\neq k}[{\frac {1}{(x_{k}-x_{j})^{2}+4\sigma ^{2}}}][1-{\frac {2\sigma }{(x_{k}-x_{j})}}i]$

References

• James O. Berger Statistical Decision Theory and Bayesian Analysis. 1980. Springer Series in Statistics. ISBN 0-387-90471-9.
• The Pitman estimator of the Cauchy location parameter, Gabriela V. Cohen Freue, Journal of Statistical Planning and Inference 137 (2007) 1900 – 1913