A bank manager working for a large American bank is interested in reducing their workload, and turn to an AI consulting group to help them set up an automated system for managing loan applications. The consultants ask for a dataset containing a large selection of approved and disapproved loan applications, for which the bank manager provides them all the historical loan data from the bank, going back almost $100$ years. The data includes information about the loan applicants, the type of loan, and whether or not the loan was approved or denied.

The consulting group trains a complicated machine learning algorithm to predict which loans will be approved and which will be denied. Upon receiving the algorithm, the bank manager decides that all loan applications will be determined using the new system.

An employee at the bank is concerned that the new system is approving and denying loans in a biased way. The bank manager insists that the system is objective and data driven, and therefor free of any bias.