Answered step by step
Verified Expert Solution
Question
1 Approved Answer
Introduction In this project, you will do work similar to an ETL engine using Linux tools and utilities. All of the following should be automated
Introduction In this project, you will do work similar to an ETL engine using Linux tools and utilities. All of the following should be automated with one driving bash script that runs each step or multiple steps Use a variety of different Linux utilities/filters to accomplish the tasks (tr, awk, sed, bash, sort, cut etc). You may only use standard Linux utilities, bash scripts that act as filters, and c programs that act as filters. Do not use programming languages we have not covered in class (python, perl, ruby, etc) and try not to use the same program to solve each problem. However, do use the program that makes the most sense for your problem. For example, awk is a good utility for reports and a good candidate for several other steps as well Each step should be written using a separate script or command. Do not write one big progranm to perform all the steps. The goal is to have separate programs that could be reused later for other ETL problems The script should trap reasonable errors and print useful error messages to help the user resolve The script should print messages to standard output to indicate which steps of the ETL process If no parameters are passed to your script, it should print a usage statement that makes it easy problems have completed for the user to figure out how to run the script. IfI can't figure out how to run your script, I can't grade it! The bash script should accept parameters for the following Remote file transfer parameters o remote-server: server name or ip address o remove-userid: userid to login to the remote machine. (assume you are using ssh keys so that a password is not required) remote-file: full path to the remote file on the remote server local file transfer location (optional, may assume current working directory) o o MariaDB parameters o mysql-user-id o mysql-database Keep in mind that for grading, this script will be tested with a different file from the test file. Try to think about what type of errors you could run into and handle them in your script Introduction In this project, you will do work similar to an ETL engine using Linux tools and utilities. All of the following should be automated with one driving bash script that runs each step or multiple steps Use a variety of different Linux utilities/filters to accomplish the tasks (tr, awk, sed, bash, sort, cut etc). You may only use standard Linux utilities, bash scripts that act as filters, and c programs that act as filters. Do not use programming languages we have not covered in class (python, perl, ruby, etc) and try not to use the same program to solve each problem. However, do use the program that makes the most sense for your problem. For example, awk is a good utility for reports and a good candidate for several other steps as well Each step should be written using a separate script or command. Do not write one big progranm to perform all the steps. The goal is to have separate programs that could be reused later for other ETL problems The script should trap reasonable errors and print useful error messages to help the user resolve The script should print messages to standard output to indicate which steps of the ETL process If no parameters are passed to your script, it should print a usage statement that makes it easy problems have completed for the user to figure out how to run the script. IfI can't figure out how to run your script, I can't grade it! The bash script should accept parameters for the following Remote file transfer parameters o remote-server: server name or ip address o remove-userid: userid to login to the remote machine. (assume you are using ssh keys so that a password is not required) remote-file: full path to the remote file on the remote server local file transfer location (optional, may assume current working directory) o o MariaDB parameters o mysql-user-id o mysql-database Keep in mind that for grading, this script will be tested with a different file from the test file. Try to think about what type of errors you could run into and handle them in your script
Step by Step Solution
There are 3 Steps involved in it
Step: 1
Get Instant Access to Expert-Tailored Solutions
See step-by-step solutions with expert insights and AI powered tools for academic success
Step: 2
Step: 3
Ace Your Homework with AI
Get the answers you need in no time with our AI-driven, step-by-step assistance
Get Started