Modern statistics emerged at the end of the $19$th century and was formalized as a methodology for the analysis of data with the publication of Statistical Methods for Research Workers by Ronald Fisher in $1925.$ Fisher described a methodology, previously proposed by Karl Pearson, in which we pose an hypothesis and measure how far the data deviate from the hypothesis. Using probability, if we decide that the discrepancy is too large to be due to chance, then we $$reject$$ the hypothesis. This is similar to a trial in which the defendant is $$assumed$($hypothesized$)$ to be innocent. The prosecution then presents evidence $($data$).$ If we cannot reconcile the observed evidence with the hypothesis, then we reject that the defendant is innocent. If we reject innocence, then we conclude guilt.

Where does probability come into the trial? Maybe the evidence just happened by coincidence. If the probability of this happening is very small, then we say that we have $$proven$$ guilt $$beyond reasonable doubt$.$ In statistics, when we reject an hypothesis, we conclude that the $$alternative$$ is $$statistically significant$.$

Over the next decades after Fisher$$s book was published, $$beyond reasonable doubt$$ or $$statistically significant$$ came to be standardized as either a $5\%$ or $1\%$ chance that the observed discrepancy happened by chance, when the original hypothesis was true. This is like saying that if a defendant is innocent, there is a $5\%($or $1\%$ if you want to set a higher standard$)$ chance s$/$he is found guilty just because the evidence seemed to suggest guilt, but s$/$he really wasn$$t$.$

P$\{$conclude guilty by mistake $|$ innocent$\}=5\%.$

This approach was seen as taking the subjectivity out of research and letting the data speak for itself. Although this methodology has been the standard for almost $100$ years, it has not been without critics and the criticism has been increasing markedly in the last $20$ years. You will study this methodology in practice in MGSC $2207.$

This methodology is similar to using a classification model with a sensitivity rate of $95\%,$ but all other performance measures are unknown.

Let us look at one of the reasons why it is being challenged. Suppose that we are looking at research that is try to determine whether one $$group$$ is better than another $($e$.$g$.,$ A$/$B testing in online marketing$).$ We are trying to prove that a new treatment is better than a placebo $($no treatment$).$ We do the research by randomly assigning individuals to the treatment and placebo groups. We work from the hypothesis that the treatment is not better than the placebo $($the opposite of what we are trying to $$prove$).$ If the data shows convincingly that the treatment is superior to the placebo, we $$reject$$ our initial hypothesis and conclude that we have $$proven$$ the treatment is better. There is a chance that we will see improvement just by chance, but this probability is small.

$1.$Suppose we require $$proof beyond reasonable doubt$$ to be a $$significance level$$ of $5\%.$ This means, P$\{$conclude better $|$ not effective$\}$ is $5\%($or less$).$ In many applications we cannot accurately estimate P$\{$conclude better $|$ effective$\}$ unless we actually know how much better the treatment really is$.$ Let us suppose that P$\{$conclude better $|$ effective$\}$ is $60\%.$ Imagine that there are thousands of researchers doing these experiments, each testing some hypothesis they have proposed. Let us assume that only in $4\%$ of cases is the treatment really effective $($it is not easy to find good treatments$).$

a$.$ Construct a tree diagram that will show all possible outcomes of effective $($not effective$)$ and conclusions $($better or not better$).$ Label each branch with respect to the event associated with the branch and the probability of travelling along the branch.

b$.$ Summarize the outcomes in a probability table such as

Treatment not effective Treatment effective Total

Conclude treatment not better

Conclude treatment better

Total $100\%$

c$.$ What is the probability that you will conclude that the treatment is effective?

d$.$ Among cases where you conclude the treatment is effective, what proportion really are effective?