Trimming Data

Nabilah Deen

Joaquin was hired as a senior researcher for a medical research facility. His department has been developing a new medical device to detect HIV faster and cheaper, and the facility has promised investors a prototype would be ready in a year. However, the preliminary results have been somewhat disappointing as only 60% of the case samples have tested positive for HIV, whereas the expected sensitivity was aimed at 80%. Product development had been extended for another six months, but additional funding is needed to continue the project.

Concerned about the low true positives detected in HIV cases, senior managers decide to convene with their scientific advisors (who are under non-disclosure agreement) to determine if 60% is sufficient, if the study is adequately powered, and if there is a better way to evaluate the data. Before the meeting, Joaquin's boss asks him to trim the data by reporting results only from certain samples, and filter out samples with low yields through observations made during meta-analysis of the clinical data.

Joaquin is uncomfortable with trimming, and argues with his boss that the scientific advisory board (SAB) should see all of their analyses, unfiltered and filtered, because it's valuable information that could be telling them something about disease stage or test performance. His boss states that as long as they don't lie about the numbers, there's no harm in filtering. Joaquin counters they should show a flowchart of the numbers they started with, clinical metadata groupings, analysis methods and filtering steps so that all the information is laid out openly. In this way, the SAB has the opportunity to ask about the methods used to achieve their receiver operating curves, and better understand the clinical cohort subclassification applied to the analysis.

The more they argue over the discussion, the more his boss is unwilling to relent. Joaquin knows that the scientific advisors will expect to see sensitivity and specificity data, ROC curves, false positives and false negatives, and figures of all unfiltered data. While the device is on its way of determining positive HIV results, it is not there yet, so he understands his boss's reluctance to share the information. How should Joaquin and his boss address this problem?

Nabilah Deen was a 2014-2015 Hackworth Fellow in Engineering Ethics at the Markkula Center for Applied Ethics at Santa Clara University.