Assessment: Confounding

For this set of exercises, we examine the data from a2014 PNAS paper that analyzed success rates from funding agencies in the NetherlandsExternal link

and concluded:

"our results reveal gender bias favoring male applicants over female applicants in the prioritization of their "quality of researcher" (but not "quality of proposal") evaluations and success rates, as well as in the language used in instructional and evaluation materials."

A response was published a few months later titledNo evidence that gender contributes to personal research funding success in The Netherlands: A reaction to Van der Lee and EllemersExternal link

, which concluded:

However, the overall gender effect borders on statistical significance, despite the large sample. Moreover, their conclusion could be a prime example of Simpson's paradox; if a higher percentage of women apply for grants in more competitive scientific disciplines (i.e., with low application success rates for both men and women), then an analysis across all disciplines could incorrectly show "evidence" of gender inequality.

Who is right here: the original paper or the response? Here, you will examine the data and come to your own conclusion.

The main evidence for the conclusion of the original paper comes down to a comparison of the percentages. The information we need was originally in Table S1 in the paper, which we include indslabs:

library(dslabs) data("research_funding_rates") research_funding_rates 

Question 1

Construct a two-by-two table of gender (men/women) by award status (awarded/not) using the total numbers across all disciplines.

What is the number of men not awarded?

What is the number of women not awarded?

Question 2

Use the two-by-two table from Question 1 to compute the percentages of men awarded versus women awarded.

What is the percentage of men awarded?Report a percentage between 0 and 100.

What is the percentage of women awarded?Report a percentage between 0 and 100.

Question 3

Run achi-squared testExternal link

on the two-by-two table to determine whether the difference in the two success rates is significant. (You can usetidy()to turn the output ofchisq.test()into a data frame as well.)

What is the p-value of the difference in funding rate?

Question4

There may be an association between gender and funding. But can we infer causation here? Is gender bias causing this observed difference? The response to the original paper claims that what we see here is similar to the UC Berkeley admissions example. Specifically they state that this "could be a prime example of Simpson's paradox; if a higher percentage of women apply for grants in more competitive scientific disciplines, then an analysis across all disciplines could incorrectly show 'evidence' of gender inequality."

To settle this dispute, use this dataset with number of applications, awards, and success rate for each gender:

 dat <- research_funding_rates %>% mutate(discipline = reorder(discipline, success_rates_total)) %>% rename(success_total = success_rates_total, success_men = success_rates_men, success_women = success_rates_women) %>% gather(key, value, -discipline) %>% separate(key, c("type", "gender")) %>% spread(type, value) %>% filter(gender != "total") dat 

To check if this is a case of Simpson's paradox, plot the success rates versus disciplines, which have been ordered by overall success, with colors to denote the genders and size to denote the number of applications.

In which fields do men have a higher success rate than women?Select ALL that apply.

Chemical sciences

Earth/life sciences

Humanities

Interdisciplinary

Medical sciences

Physical sciences

Physics

Social sciences

Technical sciences

Which two fields have the most applications from women?

Select TWO.

Chemical sciences

Earth/life sciences

Humanities

Interdisciplinary

Medical sciences

Physical sciences

Physics

Social sciences

Technical sciences

Which two fields have the lowest overall funding rates?

Select TWO.

Chemical sciences

Earth/life sciences

Humanities

Interdisciplinary

Medical sciences

Physical sciences

Physics

Social sciences

Technical sciences