Talk:Statistical hypothesis test

Template:Vital article

Common test statistics

I corrected the erroneous last test, ("regression t-test") to a correct F-test. Harald Lang, 2015-11-29.

relationships

Does a mans financial responsibility only start when the couple gets married?

Clairvoyant example...

I could be completely wrong about this but looking at the clairvoyant example...

The probability of getting every guess correct (clairvoyantly) is said to be

(1/4)^25 ~= 10^-15

This is basically the 1/4 probability that a card will be of a chosen suit rasied to the power of the number of correctly chosen cards right?

So then the probability of getting between 10 and 25 of the choices correct is the sum of getting exactly 10,11,12,13, etc.. up to 25 choices correct so if I put that into Wolfram's summation widget I get something like 1.26*10^-6, NOT ~= .07 as stated in the article?

Am I missing something here?

http://www.wolframalpha.com/input/?i=sum+[%2F%2Fmath:%281%2F4%29^k%2F%2F],+[%2F%2Fmath:k%2F%2F],+[%2F%2Fmath:10%2F%2F],+[%2F%2Fmath:25%2F%2F] — Preceding unsigned comment added by 132.45.121.6‎ (talk) 28 October 2016

What you said in words is correct, but your translation of that into maths isn't. The probability of getting k cards right (and hence 25–k cards wrong) is $^{25}C_{k}$ $(1/4)^{k}(3/4)^{25-k}$. Wolfram Alpha gives 0.071328... [1]. See Binomial distribution#Probability mass function. —Qwfp (talk) 10:08, 29 October 2016 (UTC)

The p-value doesn't have to be strictly lower than the significance level to reject the null hypothesis.

The significance level “alpha” is defined as the risk of rejecting a true null hypothesis (risk of type 1 error, or false positive). The p-value is defined as the probability of getting a test statistic at least as extreme as observed, under the null hypothesis. The page says one should reject the null hypothesis when the p-value is less than alpha. This rule appears to contract the two definitions. If we reject H0 only when a sample yields a p-value that is strictly lower than alpha, the rejection rate of a true H0 might be lower than alpha, while it should equal alpha, by definition.

To illustrate: H0 is “this coin is fair” and H1 is “there is a probability >1/2 of getting a head” (one-sided test). We toss the coin 10 times. Our test statistic X is the number of heads observed in 10 trials. X follows Bi(10, 1/2) under H0. We get 5 heads. The p-value is P(X ≥ 5) = 0.6230469. You can check with R using binom.test(5, 10, 1/2, “greater”).

If we choose alpha = P(X ≥ 5) = 0.6230469, and decide to reject H0 when the p-value is strictly lower than alpha, we would reject H0 only if there are 6 heads of more, because if we get 5 heads, the p-value equals alpha. Getting 6 heads or more under H0 has a probably P(X ≥ 6) = 0.3769531. This is the rate at which we would reject the true H0. As you can see, it does not equal alpha.

If I’m right, the wiki page is wrong. Jpeccoud (talk) 05:41, 29 August 2019 (UTC)jpeccoud

I agree, it should be "less than or equal to". This is meaningless for continuous test statistics for which there will be no difference. The issue comes up for discrete distributions. I'm changing the phrasing. Tal Galili (talk) 14:21, 29 August 2019 (UTC)

Thanks. This implies many corrections to the Statistical significance article, which I'd rather not do myself (not being a statistician nor a native English speaker) — Preceding unsigned comment added by Jpeccoud (talk • contribs) 08:02, 30 August 2019 (UTC)

Thanks for the heads up. I've now made modifications to Statistical significance based on this. Tal Galili (talk) 07:05, 1 September 2019 (UTC)