Chapter 29 Comparing means in R

29.1 Testing the value of a single mean (Section 28.1)

In this example, we will show multiple ways to test a hypothesis about the value of a single mean. As an example, let’s test whether the mean systolic blood pressure (BP) in the NHANES dataset (averaged over the three measurements that were taken for each person) is greater than 120 mm, which is the standard value for normal systolic BP.

First let’s perform a power analysis to see how large our sample would need to be in order to detect a small difference (Cohen’s d = .2).

## 
##      One-sample t test power calculation 
## 
##               n = 156
##               d = 0.2
##       sig.level = 0.05
##           power = 0.8
##     alternative = greater

Based on this, we take a sample of 156 individuals from the dataset.

## [1] "Mean BP:"
## [1] 123

First let’s perform a sign test to see whether the observed mean of 123.11 is significantly different from zero. To do this, we count the number of values that are greater than the hypothesized mean, and then use a binomial test to ask how surprising that number is if the true proportion is 0.5 (as it would be if the distribution were centered at the hypothesized mean).

## 
##  Exact binomial test
## 
## data:  nOver120 and nrow(NHANES_BP_sample)
## number of successes = 84, number of trials = 155, p-value = 0.2
## alternative hypothesis: true probability of success is greater than 0.5
## 95 percent confidence interval:
##  0.47 1.00
## sample estimates:
## probability of success 
##                   0.54

This shows no significant difference. Next let’s perform a one-sample t-test:

## 
##  One Sample t-test
## 
## data:  NHANES_BP_sample$BPSysAve
## t = 2, df = 154, p-value = 0.01
## alternative hypothesis: true mean is greater than 120
## 95 percent confidence interval:
##  121 Inf
## sample estimates:
## mean of x 
##       123

Here we see that the difference is not statistically signficant. Finally, we can perform a randomization test to test the hypothesis. Under the null hypothesis we would expect roughly half of the differences from the expected mean to be positive and half to be negative (assuming the distribution is centered around the mean), so we can cause the null hypothesis to be true on average by randomly flipping the signs of the differences.

## [1] 0.014

This gives us a very similar p value to the one observed with the standard t-test.

We might also want to quantify the degree of evidence in favor of the null hypothesis, which we can do using the Bayes Factor:

## Bayes factor analysis
## --------------
## [1] Alt., r=0.707 -Inf<d<0    : 0.029 ±0.29%
## [2] Alt., r=0.707 !(-Inf<d<0) : 1.8   ±0%
## 
## Against denominator:
##   Null, mu = 120 
## ---
## Bayes factor type: BFoneSample, JZS

This tells us that our result doesn’t provide particularly strong evidence for either the null or alternative hypothesis; that is, it’s inconclusive.

29.2 Comparing two means (Section 28.2)

To compare two means from independent samples, we can use the two-sample t-test. Let’s say that we want to compare blood pressure of smokers and non-smokers; we don’t have an expectation for the direction, so we will use a two-sided test. First let’s perform a power analysis, again for a small effect:

## 
##      Two-sample t test power calculation 
## 
##               n = 393
##               d = 0.2
##       sig.level = 0.05
##           power = 0.8
##     alternative = two.sided
## 
## NOTE: n is number in *each* group

This tells us that we need 394 subjects in each group, so let’s sample 394 smokers and 394 nonsmokers from the NHANES dataset, and then put them into a single data frame with a variable denoting their smoking status.

Let’s test our hypothesis using a standard two-sample t-test. We can use the formula notation to specify the analysis, just like we would for lm().

## 
##  Welch Two Sample t-test
## 
## data:  BPSysAve by SmokeNow
## t = 4, df = 775, p-value = 3e-05
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  2.9 7.8
## sample estimates:
##  mean in group No mean in group Yes 
##               125               120

This shows us that there is a significant difference, though the direction is surprising: Smokers have lower blood pressure!

Let’s look at the Bayes factor to quantify the evidence:

## Bayes factor analysis
## --------------
## [1] Alt., r=0.707 : 440 ±0%
## 
## Against denominator:
##   Null, mu1-mu2 = 0 
## ---
## Bayes factor type: BFindepSample, JZS

This shows that there is very strong evidence against the null hypothesis of no difference.

29.3 The t-test as a linear model (Section 28.3)

We can also use lm() to implement these t-tests.

The one-sample t-test is basically a test for whether the intercept is different from zero, so we use a model with only an intercept and apply this to the data after subtracting the null hypothesis mean (so that the expectation under the null hypothesis is an intercept of zero):

## 
## Call:
## lm(formula = BPSysAveDiff ~ 1, data = NHANES_BP_sample)
## 
## Residuals:
##    Min     1Q Median     3Q    Max 
## -36.11 -13.11  -1.11   9.39  67.89 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)  
## (Intercept)     3.11       1.41     2.2    0.029 *
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 18 on 154 degrees of freedom

You will notice that this p-value is twice as big as the one obtained from the one-sample t-test above; this is because that was a one-tailed test, while lm() is performing a two-tailed test.

We can also run the two-sample t-test using lm():

## 
## Call:
## lm(formula = BPSysAve ~ SmokeNow, data = sample_df)
## 
## Residuals:
##    Min     1Q Median     3Q    Max 
## -45.16 -11.16  -2.16   8.84 101.18 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept)  125.160      0.897  139.54  < 2e-16 ***
## SmokeNowYes   -5.341      1.269   -4.21  2.8e-05 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 18 on 784 degrees of freedom
## Multiple R-squared:  0.0221, Adjusted R-squared:  0.0209 
## F-statistic: 17.7 on 1 and 784 DF,  p-value: 2.84e-05

This gives the same p-value for the SmokeNowYes variable as it did for the two-sample t-test above.

29.4 Comparing paired observations (Section 28.5)

Let’s look at how to perform a paired t-test in R. In this case, let’s generate some data for a set of individuals on two tests, where each indivdual varies in their overall ability, but there is also a practice effect such that performance on the second test is generally better than the first.

First, let’s see how big of a sample we will require to find a medium (d=0.5) sized effect. Let’s say that we want to be extra sure in our results, so we will find the sample size that gives us 95% power to find an effect if it’s there:

## 
##      Paired t test power calculation 
## 
##               n = 45
##               d = 0.5
##       sig.level = 0.05
##           power = 0.95
##     alternative = greater
## 
## NOTE: n is number of *pairs*

Now let’s generate a dataset with the required number of subjects:

Let’s perform a paired t-test on these data. To do that, we need to separate the first and second test data into separate variables, which we can do by converting our long data frame into a wide data frame.

## Observations: 44
## Variables: 4
## $ subject_id <int> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12,…
## $ subMean    <dbl> 116, 95, 103, 91, 97, 91, 89, 97, 99, …
## $ test1      <dbl> 121, 108, 102, 94, 105, 111, 110, 89, …
## $ test2      <dbl> 104, 101, 102, 107, 108, 101, 157, 126…

Now we can pass those new variables into the t.test() function:

## 
##  Welch Two Sample t-test
## 
## data:  paired_data_wide$test1 and paired_data_wide$test2
## t = -1, df = 73, p-value = 0.2
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  -10.5   2.3
## sample estimates:
## mean of x mean of y 
##       108       112

This analysis is a bit trickier to perform using the linear model, because we need to estimate a separate intercept for each subject in order to account for the overall differences between subjects. We can’t do this using lm() but we can do it using a function called lmer() from the lme4 package. To do this, we need to add (1|subject_id) to the formula, which tells lmer() to add a separate intercept (“1”) for each value of subject_id.

## Linear mixed model fit by REML. t-tests use Satterthwaite's method [
## lmerModLmerTest]
## Formula: score ~ test_id + (1 | subject_id)
##    Data: paired_data
## 
## REML criterion at convergence: 719
## 
## Scaled residuals: 
##     Min      1Q  Median      3Q     Max 
## -2.5424 -0.6214 -0.0929  0.7349  2.9793 
## 
## Random effects:
##  Groups     Name        Variance Std.Dev.
##  subject_id (Intercept)   0       0.0    
##  Residual               228      15.1    
## Number of obs: 88, groups:  subject_id, 44
## 
## Fixed effects:
##             Estimate Std. Error     df t value Pr(>|t|)    
## (Intercept)   107.59       2.28  86.00   47.26   <2e-16 ***
## test_id         4.12       3.22  86.00    1.28      0.2    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Correlation of Fixed Effects:
##         (Intr)
## test_id -0.707
## convergence code: 0
## boundary (singular) fit: see ?isSingular

This gives a similar answer to the standard paired t-test. The advantage is that it’s more flexible, allowing us to perform repeated measures analyses, as we will see below.

29.5 Analysis of variance (Section 28.6.1)

Often we want to compare several different means, to determine whether any of them are different from the others. In this case, let’s look at the data from NHANES to determine whether Marital Status is related to sleep quality. First we clean up the data:

In this case we are going to treat the full NHANES dataset as our sample, with the goal of generalizing to the entire US population (from which the NHANES dataset is mean to be a representative sample). First let’s look at the distribution of the different values of the MaritalStatus variable:

MaritalStatus n
Divorced 437
LivePartner 370
Married 2434
NeverMarried 889
Separated 134
Widowed 329

There are reasonable numbers of most of these categories, but let’s remove the Separated category since it has relatively few members:

Now let’s use lm() to perform an analysis of variance. Since we also suspect that Age is related to the amount of sleep, we will also include Age in the model.

## 
## Call:
## lm(formula = SleepHrsNight ~ MaritalStatus + Age, data = NHANES_sleep_marriage)
## 
## Residuals:
##    Min     1Q Median     3Q    Max 
## -5.016 -0.880  0.107  1.082  5.282 
## 
## Coefficients:
##                           Estimate Std. Error t value Pr(>|t|)    
## (Intercept)                6.51758    0.09802   66.49  < 2e-16 ***
## MaritalStatusLivePartner   0.14373    0.09869    1.46  0.14536    
## MaritalStatusMarried       0.23494    0.07094    3.31  0.00093 ***
## MaritalStatusNeverMarried  0.25172    0.08404    3.00  0.00276 ** 
## MaritalStatusWidowed       0.26304    0.10327    2.55  0.01090 *  
## Age                        0.00318    0.00141    2.25  0.02464 *  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 1.4 on 4453 degrees of freedom
## Multiple R-squared:  0.00458,    Adjusted R-squared:  0.00347 
## F-statistic:  4.1 on 5 and 4453 DF,  p-value: 0.00102

This tells us that there is a highly significant effect of marital status (based on the F test), though it accounts for a very small amount of variance (less than 1%).

It’s also useful to look in more detail at which groups differ from which others, which we can do by examining the estimated marginal means for each group using the emmeans() function.

##  MaritalStatus emmean    SE   df lower.CL upper.CL .group
##  Divorced         6.7 0.066 4453      6.5      6.8  a    
##  LivePartner      6.8 0.073 4453      6.7      7.0  ab   
##  Married          6.9 0.028 4453      6.8      7.0   b   
##  NeverMarried     6.9 0.050 4453      6.8      7.0   b   
##  Widowed          6.9 0.082 4453      6.8      7.1  ab   
## 
## Confidence level used: 0.95 
## P value adjustment: tukey method for comparing a family of 5 estimates 
## significance level used: alpha = 0.05

The letters in the group column tell us which individual conditions differ from which others; any pair of conditions that don’t share a group identifier (in this case, the letters a and b) are significantly different from one another. In this case, we see that Divorced people sleep less than Married or Widowed individuals; no other pairs differ significantly.

29.5.1 Repeated measures analysis of variance

The standard analysis of variance assumes that the observations are independent, which should be true for different people in the NHANES dataset, but may not be true if the data are based on repeated measures of the same individual. For example, the NHANES dataset involves three measurements of blood pressure for each individual. If we want to test whether there are any differences between those, then we would need to use a repeated measures analysis of variance. We can do this using lmer() as we did above. First, we need to create a “long” version of the dataset.

Then we fit a model that includes a separate intercept for each individual.

## Linear mixed model fit by REML. t-tests use Satterthwaite's method [
## lmerModLmerTest]
## Formula: BPsys ~ test + (1 | ID)
##    Data: NHANES_bp_all
## 
## REML criterion at convergence: 89301
## 
## Scaled residuals: 
##    Min     1Q Median     3Q    Max 
## -4.547 -0.513 -0.005  0.495  4.134 
## 
## Random effects:
##  Groups   Name        Variance Std.Dev.
##  ID       (Intercept) 280.9    16.8    
##  Residual              16.8     4.1    
## Number of obs: 12810, groups:  ID, 4270
## 
## Fixed effects:
##              Estimate Std. Error        df t value Pr(>|t|)    
## (Intercept)  122.0037     0.2641 4605.7049   462.0   <2e-16 ***
## testBPSys2    -0.9283     0.0887 8538.0000   -10.5   <2e-16 ***
## testBPSys3    -1.6215     0.0887 8538.0000   -18.3   <2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Correlation of Fixed Effects:
##            (Intr) tsBPS2
## testBPSys2 -0.168       
## testBPSys3 -0.168  0.500

This shows us that the second and third tests are significant different from the first test (which was automatically assigned as the baseline by lmer()). We might also want to know whether there is an overall effect of test. We can determine this by comparing the fit of our model to the fit of a model that does not include the test variable, which we will fit here. We then compare the models using the anova() function, which performs an F test to compare the two models.

## Data: NHANES_bp_all
## Models:
## repeated_lmer_baseline: BPsys ~ (1 | ID)
## repeated_lmer: BPsys ~ test + (1 | ID)
##                        Df   AIC   BIC logLik deviance Chisq Chi Df Pr(>Chisq)
## repeated_lmer_baseline  3 89630 89652 -44812    89624                        
## repeated_lmer           5 89304 89341 -44647    89294   330      2     <2e-16
##                           
## repeated_lmer_baseline    
## repeated_lmer          ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

This shows that blood pressure differs significantly across the three tests.