Answered step by step
Verified Expert Solution
Link Copied!

Question

1 Approved Answer

You are working on an application which creates lots of log every day. Sample logs are shown in the table below. You need to write

You are working on an application which creates lots of log every day. Sample logs are shown in the table below. You need to write the code to analyse the logs and share insights. Build the following system in Python 3:

1. A Publisher publishes each line of the attached log data to a Stream Bus or Queue. (You can use Redis or Kafka) 2. A Consumer reads the logs from the queue, processes it, and stores in a Database. (You can use any SQL DB or MongoDB) 3. A Flask REST API that connects to the DB and returns the following stats: a. Number of logs at a certain time. b. Peak time for GET and POST requests. c. Hour wise frequency of logs. d. The ratio of Status code 200 vs other status code for a given URL. e. The 5 most requested URLs.

4. Dockerize the Publisher, Consumer, Database, and the Flask application. 5. The System should have efficient Time and Space Complexity. 6. Extra points for Documentation and Deployment Automation (ease-of-deployment with scripts).

Sample log file :-

IP Time URL Staus
10.128.2.1 [29/Nov/2017:06:58:55 GET /login.php HTTP/1.1 200
10.128.2.1 [29/Nov/2017:06:59:02 POST /process.php HTTP/1.1 302
10.128.2.1 [29/Nov/2017:06:59:03 GET /home.php HTTP/1.1 200
10.131.2.1 [29/Nov/2017:06:59:04 GET /js/vendor/moment.min.js HTTP/1.1 200
10.130.2.1 [29/Nov/2017:06:59:06 GET /bootstrap-3.3.7/js/bootstrap.js HTTP/1.1 200
10.130.2.1 [29/Nov/2017:06:59:19 GET /profile.php?user=bala HTTP/1.1 200
10.128.2.1 [29/Nov/2017:06:59:19 GET /js/jquery.min.js HTTP/1.1 200
10.131.2.1 [29/Nov/2017:06:59:19 GET /js/chart.min.js HTTP/1.1 200
10.131.2.1 [29/Nov/2017:06:59:30 GET /edit.php?name=bala HTTP/1.1 200
10.131.2.1 [29/Nov/2017:06:59:37 GET /logout.php HTTP/1.1 302
10.131.2.1 [29/Nov/2017:06:59:37 GET /login.php HTTP/1.1 200
10.130.2.1 [29/Nov/2017:07:00:19 GET /login.php HTTP/1.1 200
10.130.2.1 [29/Nov/2017:07:00:21 GET /login.php HTTP/1.1 200
10.130.2.1 [29/Nov/2017:13:31:27 GET / HTTP/1.1 302
10.130.2.1 [29/Nov/2017:13:31:28 GET /login.php HTTP/1.1 200
10.129.2.1 [29/Nov/2017:13:38:03 POST /process.php HTTP/1.1 302
10.131.0.1 [29/Nov/2017:13:38:04 GET /home.php HTTP/1.1 200
10.131.0.1 [29/Nov/2017:13:38:07 GET /contestproblem.php?name=RUET%20OJ%20Server%20Testing%20Contest HTTP/1.1 200
10.130.2.1 [29/Nov/2017:13:38:19 GET / HTTP/1.1 302
10.131.2.1 [29/Nov/2017:13:38:20 GET /login.php HTTP/1.1 200
10.131.2.1 [29/Nov/2017:13:38:20 GET /css/bootstrap.min.css HTTP/1.1 200
10.128.2.1 [29/Nov/2017:13:38:20 GET /css/font-awesome.min.css HTTP/1.1 200
10.131.0.1 [29/Nov/2017:13:38:20 GET /css/normalize.css HTTP/1.1 200
10.128.2.1 [29/Nov/2017:13:38:20 GET /css/style.css HTTP/1.1 200
10.131.0.1 [29/Nov/2017:13:38:20 GET /js/vendor/modernizr-2.8.3.min.js HTTP/1.1 200
10.129.2.1 [29/Nov/2017:13:38:20 GET /css/main.css HTTP/1.1 200
10.131.0.1 [29/Nov/2017:13:38:21 GET /js/vendor/jquery-1.12.0.min.js HTTP/1.1 200
10.128.2.1 [29/Nov/2017:13:38:21 GET /bootstrap-3.3.7/js/bootstrap.min.js HTTP/1.1 200
10.131.2.1 [29/Nov/2017:13:38:23 GET /fonts/fontawesome-webfont.woff2?v=4.6.3 HTTP/1.1 200
10.131.2.1 [29/Nov/2017:13:46:46 GET / HTTP/1.1 200
10.131.2.1 [29/Nov/2017:13:46:50 GET /contestproblem.php?name=RUET%20OJ%20Server%20Testing%20Contest HTTP/1.1 200
10.131.2.1 [29/Nov/2017:13:46:53 GET /logout.php HTTP/1.1 302
10.131.2.1 [29/Nov/2017:13:46:54 GET /login.php HTTP/1.1 200
10.131.2.1 [29/Nov/2017:13:47:04 POST /process.php HTTP/1.1 302

Step by Step Solution

There are 3 Steps involved in it

Step: 1

blur-text-image

Get Instant Access to Expert-Tailored Solutions

See step-by-step solutions with expert insights and AI powered tools for academic success

Step: 2

blur-text-image_2

Step: 3

blur-text-image_3

Ace Your Homework with AI

Get the answers you need in no time with our AI-driven, step-by-step assistance

Get Started

Recommended Textbook for

More Books

Students also viewed these Databases questions