Hypothesis testing (仮説検定) uses data from a sample to decide, on the basis of probability, whether a hypothesis about the population can be set aside.
State the null and alternative hypotheses
Set the significance level
Express how unlikely the sample result is as a probability
Decide whether to reject
If that probability falls below the significance level, reject the null hypothesis and adopt the alternative; otherwise, do not reject it.
4 / 4
First, state the hypothesis you want to test for rejection (the null hypothesis, such as “there is no difference in average spending between the two sales floors”) and its opposite, the alternative hypothesis. Next, express how unlikely the sample result would be if the null hypothesis were true, and if that probability falls below a cutoff fixed in advance (the significance level), reject the null hypothesis and adopt the alternative. Failing to reject does not prove the null hypothesis true, and the conclusion stays at “we cannot say there is a difference”.
AnalogyIt works like a criminal trial: the defendant is presumed innocent, and a guilty verdict comes only when the evidence is too strong to square with that presumption. When the evidence falls short and there is no conviction, innocence has not been proven either.
How to rememberThe Japanese 帰無 means “return to nothing”. The hypothesis is set up in the expectation that it will be rejected, hence the name.
Exam points
Mix-upThe conclusion of hypothesis testing (仮説検定) is a probabilistic judgment based on a sample, so it can be wrong even when every step is followed correctly.
The significance level (有意水準) is the probability that marks the cutoff for rejecting the null hypothesis, and it is set before the test is run.
Values such as 5% or 1% are common, and when the probability of the sample result or a more extreme one under a true null hypothesis drops below this value, the null hypothesis is rejected as “hard to put down to chance”. The significance level also fixes how much probability of rejecting a null hypothesis that is in fact true you are willing to accept. The smaller you set it, the stricter the condition for rejection becomes.
AnalogyIt is like a pass line drawn in advance: anything rarer than this will not be treated as chance. Moving the line after seeing the result lets the judgment bend to suit you.
How to remember有意 means “having a meaning that chance cannot explain away”. A result below the cutoff is called a “significant difference” (有意な差).
Exam points
Mix-upThe significance level (有意水準) is not the probability that the null hypothesis (帰無仮説) is true. It is a cutoff fixed in advance for deciding whether to reject.
A Type I error (第1種の誤り) is rejecting the null hypothesis when it is actually true.
It amounts to concluding “there is a difference” when in fact there is none. The probability of this error is kept from exceeding the significance level set before the test. Judging a new advertisement “effective” because a sample happened to show rising sales, when the ad actually did nothing, is one case of it.
AnalogyIt is like a healthy person being told at a checkup that they need further tests. Nothing is wrong, yet an abnormality is reported.
How to rememberIn Japanese it is nicknamed the “hasty person's error” (あわてものの誤り). Picture someone rushing to announce “there is a difference”.
A Type II error (第2種の誤り) is letting slip a null hypothesis that is actually false, by failing to reject it.
Decision \ Reality
Null hypothesis true
Null hypothesis false
Do not reject
Correct decision
Type II error
Reject
Type I error
Correct decision
It amounts to concluding “we cannot say there is a difference” when a difference really exists. Suppose a new study material does help, but the sample was too small for the difference to show, so its effect goes unnoticed. With other conditions equal, a larger sample makes this error less likely.
AnalogyIt is like a person whose illness has already begun being told at a checkup that nothing is wrong. A change that is really happening gets overlooked.
How to rememberIn Japanese it is nicknamed the “absent-minded person's error” (ぼんやりものの誤り). Picture someone daydreaming and missing a difference that is really there.
Exam points
Mix-upA Type I error (第1種の誤り) rejects a true null hypothesis (帰無仮説), while a Type II error (第2種の誤り) fails to reject a false one, so whether a rejection (棄却) was made tells them apart.
Type I error vs. Type II error
Aspect
Type I error
Type II error
Null hypothesis in reality
True
False
Key differenceDecision made
Reject
Do not reject
Lowering the significance level with the same sample size
Becomes less likely
Becomes more likely
Nickname
Hasty person's error
Absent-minded person's error
Key points
Hypothesis testing expresses how unlikely the sample result would be if the null hypothesis were true, then compares that probability with the significance level to decide on rejection.
The significance level is a cutoff set before the test, stating how much probability of rejecting a true null hypothesis is acceptable.
Failing to reject the null hypothesis does not prove that it is true.
With the same sample size, lowering the significance level to cut one kind of error tends to raise the other.