Answered step by step
Verified Expert Solution
Link Copied!

Question

1 Approved Answer

Introduction In this project, you will do work similar to an ETL engine using Linux tools and utilities. All of the following should be automated

image text in transcribed

image text in transcribed

Introduction In this project, you will do work similar to an ETL engine using Linux tools and utilities. All of the following should be automated with one driving bash script that runs each step or multiple steps Use a variety of different Linux utilities/filters to accomplish the tasks (tr, awk, sed, bash, sort, cut etc). You may only use standard Linux utilities, bash scripts that act as filters, and c programs that act as filters. Do not use programming languages we have not covered in class (python, perl, ruby, etc) and try not to use the same program to solve each problem. However, do use the program that makes the most sense for your problem. For example, awk is a good utility for reports and a good candidate for several other steps as well Each step should be written using a separate script or command. Do not write one big progranm to perform all the steps. The goal is to have separate programs that could be reused later for other ETL problems The script should trap reasonable errors and print useful error messages to help the user resolve The script should print messages to standard output to indicate which steps of the ETL process If no parameters are passed to your script, it should print a usage statement that makes it easy problems have completed for the user to figure out how to run the script. IfI can't figure out how to run your script, I can't grade it! The bash script should accept parameters for the following Remote file transfer parameters o remote-server: server name or ip address o remove-userid: userid to login to the remote machine. (assume you are using ssh keys so that a password is not required) remote-file: full path to the remote file on the remote server local file transfer location (optional, may assume current working directory) o o MariaDB parameters o mysql-user-id o mysql-database Keep in mind that for grading, this script will be tested with a different file from the test file. Try to think about what type of errors you could run into and handle them in your script Introduction In this project, you will do work similar to an ETL engine using Linux tools and utilities. All of the following should be automated with one driving bash script that runs each step or multiple steps Use a variety of different Linux utilities/filters to accomplish the tasks (tr, awk, sed, bash, sort, cut etc). You may only use standard Linux utilities, bash scripts that act as filters, and c programs that act as filters. Do not use programming languages we have not covered in class (python, perl, ruby, etc) and try not to use the same program to solve each problem. However, do use the program that makes the most sense for your problem. For example, awk is a good utility for reports and a good candidate for several other steps as well Each step should be written using a separate script or command. Do not write one big progranm to perform all the steps. The goal is to have separate programs that could be reused later for other ETL problems The script should trap reasonable errors and print useful error messages to help the user resolve The script should print messages to standard output to indicate which steps of the ETL process If no parameters are passed to your script, it should print a usage statement that makes it easy problems have completed for the user to figure out how to run the script. IfI can't figure out how to run your script, I can't grade it! The bash script should accept parameters for the following Remote file transfer parameters o remote-server: server name or ip address o remove-userid: userid to login to the remote machine. (assume you are using ssh keys so that a password is not required) remote-file: full path to the remote file on the remote server local file transfer location (optional, may assume current working directory) o o MariaDB parameters o mysql-user-id o mysql-database Keep in mind that for grading, this script will be tested with a different file from the test file. Try to think about what type of errors you could run into and handle them in your script

Step by Step Solution

There are 3 Steps involved in it

Step: 1

blur-text-image

Get Instant Access to Expert-Tailored Solutions

See step-by-step solutions with expert insights and AI powered tools for academic success

Step: 2

blur-text-image

Step: 3

blur-text-image

Ace Your Homework with AI

Get the answers you need in no time with our AI-driven, step-by-step assistance

Get Started

Recommended Textbook for

Pro Database Migration To Azure Data Modernization For The Enterprise

Authors: Kevin Kline, Denis McDowell, Dustin Dorsey, Matt Gordon

1st Edition

1484282299, 978-1484282298

More Books

Students also viewed these Databases questions