You have been employed as a data scientist by a large data analytics company and your first project has gone well. Your first project involved supervised learning and you were able to apply the methods that you covered in Assessment 1 of this course.

However, the second project that you have been assigned involves datasets wherelabels are not relevant to the problem at hand or they involve too much timecommitment from domain specialists to label them. You realise the only method ofsolution is to apply unsupervised learning, specifically clustering.

As you have been assigned datasets from four very different applicationenvironments you have decided that the best approach is to explore three widely

used clustering algorithms and deploy each of them on the different datasets.The three algorithms that you have decided to explore are 

1) K Means 

2) DBSCAN

and 3) Agglomerative.

The four datasets that you have been given are:

1) Dow Jones Index https://archive.ics.uci.edu/ml/datasets/Dow+Jones+Index#

2) Facebook Live Sellers in Thailand

https://archive.ics.uci.edu/ml/datasets/Facebook+Live+Sellers+in+Thailand

3) Sales Transactions

https://archive.ics.uci.edu/ml/datasets/Sales_Transactions_Dataset_Weekly

4) Water Treatment Plant https://archive.ics.uci.edu/ml/datasets/Water+Treatment+Plant

You will need to complete three tasks as detailed below.

Task 1

For each activity in this task you must apply a suitable feature selection

algorithm before deploying each clustering algorithm. Your clustering results

should include the following measures:

Time taken, Sum of Squares Errors (SSE), Cluster Silhouette Measure (CSM)

Submit Python code used for parts a) to c) below. You only need to submit the code

for one of the 4 datasets.

a) Run the K means algorithm on each of the four datasets. Obtain the best

value of K using either SSE and/or CSM. Tabulate your results in a 4 by 3

table, with each row corresponding to a dataset and each column

corresponding to one of the three measures mentioned above. Display the

CSM plot for the best value of the K parameter for each dataset.

b) Repeat the same activity for DBSCAN and tabulate your results once again,

just as you did for part a). Display the CSM plot and the 4 by 3 table for each

dataset.

c) Finally, use the Agglomerative algorithm and document your results as you

did for parts a) and b). Display the CSM plot and the 4 by 3 table for each

dataset.

Task 2

a) For each dataset identify which clustering algorithm performed best. Justify

your answer.

In the event that no single algorithm performs best on all three performance

measures you will need to carefully consider how you will rate each of the

measures and then decide how you will produce an overall measure that will

enable you to rank the algorithms.

b) For each winner algorithm and for each dataset explain why it produced the

best value for the CSM measure. This explanation must refer directly to the

conceptual design details of the algorithm. There is no need to produce any

further experimental evidence for this part of the question.

c) Based on what you produced in a) above, which clustering algorithm would

you consider to be the overall winner (i.e. after taking into consideration

performance across all four datasets). Justify your answer.

Task 3

This task requires you to do some further research on your own. The t-sne algorithm

(https://lvdmaaten.github.io/tsne/) was designed to visualize high dimensional data after

reducing dimensionality.

a) After gaining an understanding of how it works identify one important potential

advantage of t-sne over Principal Components Analysis (PCA). You may use

one or more sources from the machine learning literature to support your

answer.

b) Select the Sales Transactions dataset that you experimented with:

1. apply t-sne to reduce dimensionality to two components and then

visualize the data using a suitable plot. Submit the Python code and

your plot.

2. is the potential advantage of t-sne over PCA that you mentioned in part

a) present itself in this dataset? Justify your answer with suitable

experimental evidence.

3. does the 2D visualization give insights into the structure of the data?

Explain your answer.

4. if so, does it inform the choice of which clustering algorithm to use? On

the other hand, if it does not narrow down the choice of clustering

the algorithm then explains why the visual is insufficient on its own draw a

definite conclusion.