Question
(Data Mining short Python code only) I just need some code to start bellow. Please help me write some Python code. Only fill the YOUR
(Data Mining short Python code only)
I just need some code to start bellow. Please help me write some Python code. Only fill the YOUR CODE HERE part .
I already asked step 1 and step 2 before. you could answer only step 3 part.
Frequent Pattern Mining
implementing the APRIORI algorithm for Frequent Pattern Mining using Python. In order to implement the full algorithm, you will need to write two helper functions: a candidate generator and a support counter. Once you have a working candidate generator and support counter, you can implement the APRIORI algorithm and run it on the three provided datasets (described below). Each function has type hints to help you understand what the expected inputs/outputs are.
As in Assignment 2, this notebook povides a template of the functions you will need to implement. Your task is to read and understand each step of this assignment and to fill in the areas marked YOUR CODE HERE. DO NOT alter any code outside of the marked areas or your work may not run or generate the correct output.
Data
3 files each including a transaction dataset; each row is a seperate transaction so transaction ID's are omitted.
dataset1.txt -> 100 transactions
dataset2.txt -> 400 transactions
dataset3.txt -> 800 transactions
Your code shoule be able to handle all 3 datasets in a reasonable timeframe (<1min each at most)
##Python code###
# Imports - DO NOT CHANGE
from itertools import combinations
from time import time
from typing import List, Dict
from gcsfs import GCSFileSystem
# Download datasets - may take a few seconds
fs = GCSFileSystem(project='csci4800-dm', token='anon', access='read_only')
fs.get('csci4800-data/assignment_3/dataset1.txt', './dataset1.txt')
fs.get('csci4800-data/assignment_3/dataset2.txt', './dataset2.txt')
fs.get('csci4800-data/assignment_3/dataset3.txt', './dataset3.txt')
# Declare timed decorator for timing functions
"""
NOTE: This is a helper function (called a decorator) that outputs the runtime of a function, it is used by placing
@timed above the declaration of a function.
"""
def timed(f):
def time_wrap(*args, **kwargs):
t_start = time()
result = f(*args, **kwargs)
t_end = time()
if f.__name__ == 'gen_candidates':
print("func: {} took: {:E} sec and generated {} candidates for k = {}".format(
f.__name__, t_end-t_start, len(result), args[1]))
else:
print("func: {} took: {:E} sec".format(f.__name__, t_end-t_start))
return result
return time_wrap
Step 1 - Candidate Generator
Write the function that generates potential frequent candidates at each level of the iterative APRIORI algorithm. Use the lecture notes for a description of how the algorithm works, if you need specific implementation help: first read the python documentation, if that does not solve the issue then you may stop by for the TA's office hours.
Notes:
Even though you are not using SQL, the SQL implementation in the lecture notes (slide 15) provides a good blueprint for designing this function using Python
A good way to store the candidates is as a dictionary (Python dict) with key -> values pairs being candidate -> count setting the initial counts all to 0.
Python dicts cannot store sets or lists as keys, but you can convert a string to a set/list and vice-versa with ' '.join() and .split(' ')
You can iterate through dict keys using a for statement (e.g. for itemset in L.keys())
The Python itertools module provides a function (combinations) for generating all subsets of length k of a set
Code gose here:
@timed
def gen_candidates(L: Dict[str, int], k: int) -> Dict[str, int]:
"""
Generates candidate itemsets from the k-1 frequent itemsets
:param L: frequent itemsets at k-1 as a dict of itemset -> count pairs
:param k: length of itemsets being generated
:returns: dict of candidate itemsets
"""
C = {}
"""
*
* YOUR CODE HERE
*
"""
Step 2 - Support Counter
Write the function that counts the support for each candidates generated by your algorithm in Step 1.
Notes:
This function only needs to generate the counts, it does not handle pruning
Python has functions all() and any() that check if all or any elements of a list are true respectively
As with before the itemsets are whitespace delimited strings that should be split to form arrays
Code goes here:
@timed
def count_support(candidates: Dict[str, int], transactions: List[str]) -> Dict[str, int]:
"""
Counts support for each candidate itemset
:param candidates: candidate itemsets generated using the candidate generator
:param transactions: list of transactions, delimited by whitespaces
:returns: a dict of candidate -> support pairs
"""
supports = candidates.copy()
"""
*
* YOUR CODE HERE
*
"""
Step 3 - Main Loop
Write the main loop that runs the APRIORI algorithm using the Candidate Generator and Support Counter implemented in steps 1 and 2.
Notes:
You may return a dict of itemset -> count pairs rather than just the sets of frequent items
A useful way to generate L_k is by using dict comprehension (see docs for dict above) on the itemset -> count pairs
The first set of candidates has been generated for you (itemsets of length 1, i.e. k = 1)
@timed
def APRIORI(transactions: List[str], min_support: int) -> List[set]:
"""
Runs the APRIORI algorithm on a set of transactions
:param transactions: list of transactions, delimited by whitespace
:param min_support: minimum support
:return: list of frequent itemsets for each value of k
"""
# Generate the first set of candidates
C0 = {}
for line in transactions:
for t in line.rstrip().split(' '):
C0[t] = 0
# Count support for each candidate
C0 = count_support(C0, transactions)
# Generate the first frequent itemset (k = 1)
L_k = { c:count for c, count in C0.items() if count > min_support }
# Add the first frequent itemset to the union for k = 1
L_union = [L_k]
# Begin at k = 2
k = 2
while len(L_k) > 0:
"""
*
* YOUR CODE HERE
*
"""
return L_union
Step by Step Solution
There are 3 Steps involved in it
Step: 1
Get Instant Access to Expert-Tailored Solutions
See step-by-step solutions with expert insights and AI powered tools for academic success
Step: 2
Step: 3
Ace Your Homework with AI
Get the answers you need in no time with our AI-driven, step-by-step assistance
Get Started