Estimation and Hypothesis Testing I

• We want to test if our estimates are statistically significant and they describe the population
  • This is the "bread and butter" of inferential statistics and the purpose of regression

• All modern science is built upon statistical hypothesis testing, so understand it well!

Estimation and Hypothesis Testing II

^† With a model this simple, it's almost certainly not causal, but this is the ultimate direction we are heading...

Null and Alternative Hypotheses I

• All scientific inquiries begin with a null hypothesis (H0) that proposes a specific value of a population parameter
  • Notation: add a subscript 0: β1,0 (or μ0, p0, etc)

• We suggest an alternative hypothesis (Ha), often the one we hope to verify
  • Note, can be multiple alternative hypotheses: H1,H2,…,Hn

• Ask: "Does our data (sample) give us sufficient evidence to reject H0 in favor of Ha?"
  • Note: the test is always about H0!
  • See if we have sufficient evidence to reject the status quo

Null and Alternative Hypotheses II

• Null hypothesis assigns a value (or a range) to a population parameter

  • e.g. β1=2 or β1≤20
  • Most common is β1=0 ⟹ X has no effect on Y (no slope for a line)
  • Note: always an equality!

• Alternative hypothesis must mathematically contradict the null hypothesis

  • e.g. β1≠2 or β1>20 or β1≠0
  • Note: always an inequality!

• Alternative hypotheses come in two forms:
  1. One-sided alternative: β1>H0 or β1<H0
  2. Two-sided alternative: β1≠H0
     • Note this means either β1<H0 or β1>H0

Components of a Valid Hypothesis Test

• All statistical hypothesis tests have the following components:

1. A null hypothesis, H0

2. An alternative hypothesis, Ha

3. A test statistic to determine if we reject H0 when the statistic reaches a "critical value"

   • Beyond the critical value is the "rejection region", sufficient evidence to reject H0

4. A conclusion whether or not to reject H0 in favor of Ha

Type I and Type II Errors I

Type I and Type II Errors II

Type I and Type II Errors III

• Depending on context, committing one type of error may be more serious than the other

Type I and Type II Errors IV

Type I and Type II Errors V

Blackstone, William, 1765-1770, Commentaries on the Laws of England

Type I and Type II Errors VI

Type I and Type II Errors VI

Significance Level, α, and Confidence Level 1−α

• The significance level, α, is the probability of a Type I error

α=P(Reject H0|H0 is true)

• The confidence level is defined as (1−α)

  • Specify in advance an α-level (0.10, 0.05, 0.01) with associated confidence level (90%, 95%, 99%)

• The probability of a Type II error is defined as β:

β=P(Don't reject H0|H0 is false)

α and β

Power and p-values

• The statistical power of the test is (1−β): the probability of correctly rejecting H0 when H0 is in fact false (e.g. not convicting an innocent person)

Power=1−β=P(Reject H0|H0 is false)

• The p-value or significance probability is the probability that, if the null hypothesis were true, the test statistic from any sample will be at least as extreme as the test statistic from our sample

p(δ≥δi|H0 is true)

• where δ represents some test statistic
• δi is the test statistic we observe in our sample
• More on this in a bit

p-Values and Statistical Significance

• After running our test, we need to make a decision between the competing hypotheses

• Compare p-value with pre-determined α (commonly, α=0.05, 95% confidence level)

• If p<α: statistically significant evidence sufficient to reject H0 in favor of Ha

  • Note this does not mean Ha is true! We merely have rejected H0!

• If p≥α: insufficient evidence to reject H0

  • Note this does not mean H0 is true! We merely have failed to reject H0!

Hypothesis Testing and the Philosophy of Science I

Hypothesis Testing and the Philosophy of Science I

Hypothesis Testing and p-Values

• Hypothesis testing, confidence intervals, and p-values are probably the hardest thing to understand in statistics

Hypothesis Testing: Which Test? I

• Rigorous course on statistics (ECMG 212 or MATH 112) will spend weeks going through different types of tests:

  • Sample mean; difference of means
  • Proportion; difference of proportions
  • Z-test vs t-test
  • 1 sample vs. 2 samples
  • χ2 test

• See today's class notes page for more

Hypothesis Testing: Which Test? II

There is Only One Test

• Fortunately, some clever statisticians realized "there is only one test" and built a nice R package called infer

1. Calculate a statistic, δi^†, from a sample of data

2. Simulate a world where δ is null (H0)

3. Examine the distribution of δ across the null world

4. Calculate the probability that δi could exist in the null world

5. Decide if δi is statistically significant

^† δ can stand in for any test-statistic in any hypothesis test! For our purposes, δ is the slope of our regression sample, ˆβ1.

Elements of a Hypothesis Test

Hypothesis Testing with the infer Package I

• R naturally runs the following hypothesis test on any regression as part of lm():

H0:β1=0H1:β1≠0

• infer allows you to run through these steps manually to understand the process:

1. specify() a model

2. hypothesize() the null

3. generate() simulations of the null world

4. calculate() the p-value

5. visualize() with a histogram (optional)

Hypothesis Testing with the infer Package II

Hypothesis Testing with the infer Package II

Hypothesis Testing with the infer Package II

Hypothesis Testing with the infer Package II

Hypothesis Testing with the infer Package II

Hypothesis Testing with the infer Package II

Classical Inference: Critical Values of Test Statistic

• Test statistic (δ): measures how far what we observed in our sample (^β1) is from what we would expect if the null hypothesis were true (β1=0)

  • Calculated from a sampling distribution of the estimator (i.e. ^β1)
  • In econometrics, we use t-distributions which have n−k−1 degrees of freedom^†

• Rejection region: if the test statistic reaches a "critical value" of δ, then we reject the null hypothesis

^† Again, see today's class notes for more on the t-distribution. k is the number of independent variables our model has, in this case, with just one X, k=1. We use two degrees of freedom to calculate ^β0 and ^β1, hence we have n−2 df.

Imagine a Null World, where H0 is True

Comparing the Worlds I

• From that null world where H0:β1=0 is true, we simulate another sample and calculate OLS estimators again

Comparing the Worlds II

Prepping the infer Pipeline

• Before I show you how to do this, let's first save our estimated slope from our actual sample
  • We'll want this later!

# save as obs_slope
sample_slope <- school_reg_tidy %>% # this is the regression tidied with broom
  filter(term=="str") %>%
  pull(estimate)
# confirm what it is
sample_slope

## [1] -2.279808

The infer Pipeline: Specify

The infer Pipeline: Specify

data %>%
  specify(y ~ x)

CASchool %>%
  specify(testscr ~ str)

The infer Pipeline: Hypothesize

The infer Pipeline: Hypothesize

CASchool %>%
  specify(testscr ~ str) %>%
  hypothesize(null = "independence")

The infer Pipeline: Generate I

The infer Pipeline: Generate I

CASchool %>%
  specify(testscr ~ str) %>%
  hypothesize(null = "independence") %>%
  generate(reps = 1000,
           type = "permute")

The infer Pipeline: Generate II

The infer Pipeline: Calculate I

The infer Pipeline: Calculate I

CASchool %>%
  specify(testscr ~ str) %>%
  hypothesize(null = "independence") %>%
  generate(reps = 1000,
           type = "permute") %>%
  generate(reps = 1000, type = "permute") %>%
  calculate(stat = "slope")

The infer Pipeline: Calculate II

The infer Pipeline: Get p Value

CASchool %>%
  specify(testscr ~ str) %>%
  hypothesize(null = "independence") %>%
  generate(reps = 1000,
           type = "permute") %>%
  calculate(stat = "slope") %>%
  get_p_value(obs_stat = sample_slope,
              direction = "both")

The infer Pipeline: Visualize I

The infer Pipeline: Visualize I

CASchool %>%
  specify(testscr ~ str) %>%
  hypothesize(null = "independence") %>%
  generate(reps = 1000,
           type = "permute") %>%
  calculate(stat = "slope") %>%
visualize()

The infer Pipeline: Visualize II

CASchool %>%
  specify(testscr ~ str) %>%
  hypothesize(null = "independence") %>%
  generate(reps = 1000,
           type = "permute") %>%
  calculate(stat = "slope") %>%
  visualize(obs_stat = sample_slope)

The infer Pipeline: Visualize p-value

CASchool %>%
  specify(testscr ~ str) %>%
  hypothesize(null = "independence") %>%
  generate(reps = 1000,
           type = "permute") %>%
  calculate(stat = "slope") %>%
  visualize(obs_stat = sample_slope)+
  shade_p_value(obs_stat = sample_slope,
                direction = "two_sided")

The infer Pipeline: Visualize Confidence Intervals

simulations %>%
 visualize(obs_stat = sample_slope)+
  shade_confidence_interval(ci_values)

The infer Pipeline: Visualize is a Wrapper of ggplot

• infer's visualize() function is just a wrapper function for ggplot()
  • you can take your simulations tibble and just ggplot a normal histogram

simulations %>%
  ggplot(data = .)+
  aes(x = stat)+
  geom_histogram(color="white", fill="indianred")+
  geom_vline(xintercept = sample_slope,
             color = "blue",
             size = 2,
             linetype = "dashed")+
  labs(x = expression(paste("Distribution of ", hat(beta[1]), " under ", H[0], " that ", beta[1]==0)),
       y = "Samples")+
    theme_classic(base_family = "Fira Sans Condensed",
           base_size=20)

What R Does: Classical Statistical Inference I

^† k is the number of X variables.

What R Does: Classical Statistical Inference II

What R Does: Classical Statistical Inference III

^† Think of our simulated distribution, the center was 0.

1-Sided vs. 2-Sided p-values I

1-Sided vs. 2-Sided p-values I

Ha:β1≠0

p-value: 2×Prob(t>|ti|)

Hypothesis Tests in Regression Output I

summary(school_reg)

## 
## Call:
## lm(formula = testscr ~ str, data = CASchool)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -47.727 -14.251   0.483  12.822  48.540 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept) 698.9330     9.4675  73.825  < 2e-16 ***
## str          -2.2798     0.4798  -4.751 2.78e-06 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 18.58 on 418 degrees of freedom
## Multiple R-squared:  0.05124,    Adjusted R-squared:  0.04897 
## F-statistic: 22.58 on 1 and 418 DF,  p-value: 2.783e-06

Hypothesis Tests in Regression Output II

• In broom's tidy() (with confidence intervals)

tidy(school_reg, conf.int=TRUE)

Conclusions

H0:β1=0Ha:βa≠0

• Because the hypothesis test's p-value < α (0.05)...

• We have sufficient evidence to reject H0 in favor of our alternative hypothesis. Our sample suggests that there is a relationship between class size and test scores.

• Using the confidence intervals:

• We are 95% confident that the true marginal effect of class size on test scores is between −3.22 and −1.34.

Hypothesis Testing vs. Confidence Intervals

• Confidence intervals are all two-sided by nature CI0.95=([^β1−2×se(^β1)],[^β1+2×se(^β1]))

• Hypothesis test (t-test) of H0:β1=0 computes a t-value of¹ t=^β1se(^β1)

  and p<0.05 when t≥2

• If a confidence interval contains the H0 value (i.e. 0, for our test), then we fail to reject H0.

¹ Since our null hypothesis is that β1,0=0, the test statistic simplifies to this neat fraction.

Common Misconceptions about p-values

Abusing p-Values I

Source: SMBC

Abusing p-Values II

Wasserstein, Ronald L. and Nicole A. Lazar, (2016), "The ASA's Statement on p-Values: Context, Process, and Purpose," The American Statistician 30(2): 129-133

Abusing p-Values II

“No economist has achieved scientific success as a result of a statistically significant coefficient. Massed observations, clever common sense, elegant theorems, new policies, sagacious economic reasoning, historical perspective, relevant accounting, these have all led to scientific success. Statistical significance has not.”

McCloskey, Dierdre N and Stephen Ziliak, 1996, The Cult of Statistical Significance, p. 112)

p-value Clarification

• Again, p-value is the probability that, if the null hypothesis were true, we obtain (by pure random chance) a test statistic at least as extreme as the one we estimated for our sample

• A low p-value means either (and we can't distinguish which):

  1. H0 is true and a highly improbable event has occurred OR
  2. H0 is false

Significance In Regression Tables