SUPP 5.3.(From Introduction to the Practice of Statistics, Moore and McCabe)

"Every user of statistics should understand the distinction between statistical significance and practical importance. A sufficiently large sample will declare very small effects statistically significant."

In general, a population of elite female Canadian athletes were believed to be consuming an average of 2403.7 kcal/day with population standard deviation = sigma= 880 kcal/day. A nutritionist implemented a new diet among a sample of these athletes. Under the new diet, the sample of athletes had average daily intake of xbar=2453.7 kcal/day. The standard deviation is assumed to remain the same with=880 kcal/day.

Conduct a statistical test to test whether we have evidence that true average intake is larger than2403.7 kcal/day for athletes on the new diet. To save you some time, we've abbreviated some of the steps to the right so you can concentrate on the topic... how sample size impacts p values.

Use these sample sizes:

a) n=100

b) n=1000

c) No alpha level was mentioned for the above problems, so we used the default level of alpha= 0.05. If the alpha level were given as 0.1 or 0.01, would statistical significance be impacted in Parts a-b?

d) Why did we use a "one-sided" test for this problem?

WHAT DOES THIS PROBLEM SHOW?:

The same sample average (xbar = 2453.7 kcal/day) can either be statistically significant or statistically NOT significant - depending on the sample size!

So as noted many times, p-values and statistical significance have issues - and easily are misinterpreted.

Small differences may be statistically significant if, say the sample size is large.

Large difference may be statistically non significant if, say the sample size is small.

We should not be overly dependent on pvalues - they mask important information.