[5] 3. (a) You are collaborating with a group of vaccine development researchers. They are using a machine learning package for a binary classification task and wish to analyse their results using a confusion matrix, with a test set of 200 cases. Provide an example of a confusion matrix consistent with this, and make up values of your own for each entry (numbers, not percentages). Explain how every entry would be obtained in practice. For your confusion matrix, compute the accuracy and True Positive rate. (b) The research group next wish to compute a ROC curve for the problem. Can they do this for their task, or do they have to modify the task first? In your own words, describe in detail the process of constructing the ROC curve, including an illustration of one with axes and all other parts clearly labelled. (c) The research team have sent you a new message. They have a new training set of 500 cases, and they would like to perform repeated 5-fold cross-validation. In your own words, explain the process, including details of how many cases will be in each fold, how the data in each fold is used, and what it means to do it repeatedly. Also, how many classifiers will they build in total, and how do they estimate the performance of the final classifier? [5] [8] [5] 3. (a) You are collaborating with a group of vaccine development researchers. They are using a machine learning package for a binary classification task and wish to analyse their results using a confusion matrix, with a test set of 200 cases. Provide an example of a confusion matrix consistent with this, and make up values of your own for each entry (numbers, not percentages). Explain how every entry would be obtained in practice. For your confusion matrix, compute the accuracy and True Positive rate. (b) The research group next wish to compute a ROC curve for the problem. Can they do this for their task, or do they have to modify the task first? In your own words, describe in detail the process of constructing the ROC curve, including an illustration of one with axes and all other parts clearly labelled. (c) The research team have sent you a new message. They have a new training set of 500 cases, and they would like to perform repeated 5-fold cross-validation. In your own words, explain the process, including details of how many cases will be in each fold, how the data in each fold is used, and what it means to do it repeatedly. Also, how many classifiers will they build in total, and how do they estimate the performance of the final classifier? [5] [8]