Generalized p-value

In Statistics, generalized p-values is an extended version the classical p-values, which except in a limited number of applications, provide only approximate solutions.

Conventional statistical methods do not provide exact solutions to many statistical problems, especially when the problem involves many nuisance parameters. As a result, practitioners often resort to approximate statistical methods or asymptotic statistical methods that are valid only with large samples. With small samples, such methods often have poor performance. Use of approximate and asymptotic methods may lead to misleading conclusions or may fail to detect truly significant results from experiments.

Tests based on generalized p-values are exact statistical methods in that they are based on exact probability statements. While conventional statistical methods do not provide exact solutions to such problems as testing variance components or ANOVA under unequal variances, the references below provide exact tests based on generalized p-values..

In order to overcome the shortcomings of the classical p-values, Tsui and Weerahandi (1989) extended the definition of the classical p-values so that one can obtain exact solutions for problems such as the Behrens–Fisher problem.

To describe the idea of generalized p-values in a simple example, consider a situation of sampling from a normal population with mean $\mu$, and variance $\sigma ^{2}$, suppose ${\overline {X}}$ and $S^{2}$ are the sample mean and the sample variance. Inferences on all unknown parameters can be based on the distributional results

  $Z={\sqrt {n}}({\overline {X}}-\mu )/\sigma \sim N(0,1)$ 

and

  $U=nS^{2}/\sigma ^{2}\sim \chi _{n-1}^{2}$. 

Now suppose we need to test the coefficient of variation, $\rho =\mu /\sigma$. While the problem is not trivial with conventional p-values, the task can be easily accomplished based on the generalized test variable

  $R={\frac {{\overline {x}}S}{s\sigma }}-{\frac {{\overline {X}}-\mu }{\sigma }}={\frac {\overline {x}}{s}}{\frac {\sqrt {U}}{\sqrt {n}}}~-~{\frac {Z}{\sqrt {n}}}$, 

where ${\overline {x}}$ is the observed value of ${\overline {X}}$ and $S$ is the observed value of $s$. Note that the distribution of $R$ and its observed value are both free of nuisance parameters. Therefore, one-sided hypotheses such as $H_{0}:\rho <\rho _{0}$ can be tested based on the generalized p-value $p=Pr(R\geq \rho _{0})$, a quantity that can be evaluated via Monte Carlo simulation or using the non-central t-distribution.

References

[1] Tsui, K. and Weerahandi, S. (1989): Generalized p-values in significance testing of hypotheses in the presence of nuisance parameters. Journal of the American Statistical Association, 84, 602-607 (1989). [1]

[2] Weerahandi, S. 1995. Exact Statistical Method for Data Analysis. Springer-Verlag, New York.

[3] XPro, Free software package for exact parametric statistics