AP Statistics · Lesson 2

Decisions, errors and power

Know Know it

A test ends in one of two decisions, and each can be wrong. A Type I error rejects H0H_0 when H0H_0 is true. A Type II error fails to reject H0H_0 when HaH_a is true.

The probability of a Type I error, when H0H_0 is true, is the significance level α\alpha. Power is the probability of rejecting H0H_0 when a particular alternative value is true, so P(Type II error)=1−powerP(\text{Type II error}) = 1 - \text{power} for that value.

Power goes up with a larger sample size, a larger α\alpha, and a true value further from the null value. Lowering α\alpha makes a Type I error less likely but a Type II error more likely, when nothing else changes.

Choose α\alpha by weighing the costs. If a Type I error is the more serious mistake, use a smaller α\alpha. If a Type II error is the more serious mistake, a larger α\alpha (or a larger sample) helps.

The four outcomes of a significance test
DecisionH0H_0 is trueHaH_a is true
Reject H0H_0Type I errorCorrect decision (power)
Fail to reject H0H_0Correct decisionType II error

Worked example Easy

A fictional chip factory tests H0:p=0.02H_0: p = 0.02 against Ha:p>0.02H_a: p > 0.02, where p is the proportion of defective chips made on a line. If it rejects H0H_0, it halts the line for repairs.

Which describes a Type I error in this setting?

  1. Deciding the defect rate is not above 0.02 and keeping the line running when the rate really is above 0.02
  2. Finding that the sample proportion of defective chips is above 0.02
  3. Halting the line when the defect rate really is above 0.02
  4. Concluding the defect rate is above 0.02 and halting the line when the rate really is 0.02Answer

How to solve it

  1. A Type I error rejects H0H_0 when H0H_0 is true.
  2. Rejecting here means concluding the defect rate is above 0.02 and halting the line.
  3. H0H_0 true means the rate really is 0.02, so the halt was not needed.

Why each choice is right or wrong

  • A. Incorrect. Keeping the line running when the rate really is above 0.02 is failing to reject a false null, which is a Type II error.
  • B. Incorrect. A sample proportion above 0.02 is just a result from the data; an error is a wrong decision about the population.
  • C. Incorrect. Halting the line when the rate really is above 0.02 is rejecting a false null, which is a correct decision.
  • D. Correct. A Type I error rejects a true null: the factory concludes there is a problem and halts the line when the rate really is 0.02.

Trap Fail to reject is not accept

A large P-value means the data are consistent with H0H_0, not that H0H_0 is true. The sample may simply be too small to detect a real difference. Write "fail to reject H0H_0" and "there is not convincing evidence that" the alternative is true; never write "accept H0H_0" or "the data prove H0H_0".

Explain Explain it

Task verbs: describe, explain

To describe an error in context, use the words of the setting, not the symbols: say what the test would conclude, what is really true, and what action follows.

To explain a consequence, follow the decision to its result for the people involved: a recall that was not needed, a problem that goes unfixed, a treatment that is dropped although it works.

Worked example Medium

A researcher plans to test H0:p=0.5H_0: p = 0.5 against Ha:p>0.5H_a: p > 0.5 at α=0.05\alpha = 0.05 with a random sample of 200. If the true proportion is 0.56, the power of this test is about 0.52.

Which single change would increase the power against a true proportion of 0.56?

  1. Using a significance level of 0.01 instead of 0.05
  2. Using a random sample of 400 instead of 200Answer
  3. Finding the power against a true proportion of 0.52 instead of 0.56
  4. Using the two-sided alternative Ha:p≠0.5H_a: p \neq 0.5 at the same significance level

How to solve it

  1. Power rises with a larger n, a larger alpha, or a true value farther from the null.
  2. Doubling n to 400 shrinks the standard deviation of p^\hat{p}, so it increases the power.
  3. The other choices make it harder to reject: a smaller alpha, a true value closer to 0.5, or splitting alpha between two tails.

Why each choice is right or wrong

  • A. Incorrect. A significance level of 0.01 makes rejection harder, which lowers the power, to about 0.26 here.
  • B. Correct. A sample of 400 shrinks the spread of the sample proportion, so a true value of 0.56 is easier to detect; the power rises to about 0.78.
  • C. Incorrect. A true value of 0.52 is closer to the null value 0.5, so it is harder to detect and the power falls, to about 0.14.
  • D. Incorrect. A two-sided test at 0.05 splits alpha across two tails, so its cutoff on the high side is farther out and the power against 0.56 goes down.

Describe Your turn

A fictional water utility tests H0:p=0.05H_0: p = 0.05 against Ha:p>0.05H_a: p > 0.05, where p is the proportion of homes in its area with lead above the safe limit. If it rejects H0H_0, it replaces the pipes. Describe a Type II error in this setting and give one consequence of making it.

Write it, then compare. Nothing you type here is saved or graded.

Compare with a model answer

A full answer does this

  • Describes the Type II error in context: the test does not find convincing evidence that more than 5 percent of homes exceed the limit, when in fact more than 5 percent do
  • Gives a consequence for residents that follows from that decision

Model answer

A Type II error would happen if more than 5 percent of the homes really have lead above the safe limit, but the utility fails to reject the null hypothesis because its sample does not give convincing evidence of that. As a consequence, the utility would not replace the pipes, and residents would keep using water with lead above the safe limit.

Apply Apply it

Practice these topics. AP Practice questions on exactly what this lesson covers, with an explanation for every answer.

Practice these topics