In a small ski resort village in Utah, a fully-loaded jet airplane carrying 100 people on board crashed into the side of a mountain on its approach to a small airport that serves over 100,000 flyers a year. No one survived. This occurred during broad daylight and the weather was fine except for some unusually strong winds. This model of the airplane has been in service for over 10 years with a perfect safety record. The black box was found and showed that all equipment was working properly and there were no obvious signs of pilot error. However, the guidance system was using a new auto-pilot program that flies the plane automatically, especially during landings.

The guidance system was developed by Gaff, Inc., a company that specializes in creating predictive analytics software for guiding airplanes in flight. The following is known about how the model (that was used to fly the plane) was developed:

The training data used for the model was based on 10,000 flights of this same plane in the New York area. Data collected included wind speed and direction, altitude, the weight of the plane (accounting for fuel remaining and the number of passengers), rain, and length of the runway. This data was then used to predict how to accelerate/decelerate the plane, manage the wing flaps, manage the tail rudder, and other mechanics based on how the pilots flew the planes during those 10,000 flights.

The training data consisted of over 100 petabytes of data so the team decided to remove all outliers and missing values from the data.

Also, due to the size of the training data, the team chose to reduce the training set by randomly sampling across all of the data. The data was composed of the following:

1% of the values in the training data consisted of flights where high winds were present and the remaining 99% were flights with normal wind patterns.

33% of the flights had less than 30 passengers, 33% of the flights had between 30 and 75 passengers and 33% of the flights were completely full (100).

100% of the flights used a runway of the same length.

The model algorithm used was nave Bayes. Model accuracy was used to assess the model (not F1 score). (Note that this is continuous that was data bucketed into classes.)

The model was tested on over 1000 flights where the plane landed safely and without incident (pilots were, of course, there to ensure proper operation). In a rush to get the model approved, these 1000 flights were the first, sequential flights immediately after the development of the model.

EXPLAIN IN DETAIL - DO YOU SEE ANY ISSUES WITH HOW THE MODEL WAS TRAINED ? HOW IT WAS TESTED?