Question
You are working on an application which creates lots of log every day. Sample logs are shown in the table below. You need to write
You are working on an application which creates lots of log every day. Sample logs are shown in the table below. You need to write the code to analyse the logs and share insights. Build the following system in Python 3:
1. A Publisher publishes each line of the attached log data to a Stream Bus or Queue. (You can use Redis or Kafka) 2. A Consumer reads the logs from the queue, processes it, and stores in a Database. (You can use any SQL DB or MongoDB) 3. A Flask REST API that connects to the DB and returns the following stats: a. Number of logs at a certain time. b. Peak time for GET and POST requests. c. Hour wise frequency of logs. d. The ratio of Status code 200 vs other status code for a given URL. e. The 5 most requested URLs.
4. Dockerize the Publisher, Consumer, Database, and the Flask application. 5. The System should have efficient Time and Space Complexity. 6. Extra points for Documentation and Deployment Automation (ease-of-deployment with scripts).
Sample log file :-
IP | Time | URL | Staus |
10.128.2.1 | [29/Nov/2017:06:58:55 | GET /login.php HTTP/1.1 | 200 |
10.128.2.1 | [29/Nov/2017:06:59:02 | POST /process.php HTTP/1.1 | 302 |
10.128.2.1 | [29/Nov/2017:06:59:03 | GET /home.php HTTP/1.1 | 200 |
10.131.2.1 | [29/Nov/2017:06:59:04 | GET /js/vendor/moment.min.js HTTP/1.1 | 200 |
10.130.2.1 | [29/Nov/2017:06:59:06 | GET /bootstrap-3.3.7/js/bootstrap.js HTTP/1.1 | 200 |
10.130.2.1 | [29/Nov/2017:06:59:19 | GET /profile.php?user=bala HTTP/1.1 | 200 |
10.128.2.1 | [29/Nov/2017:06:59:19 | GET /js/jquery.min.js HTTP/1.1 | 200 |
10.131.2.1 | [29/Nov/2017:06:59:19 | GET /js/chart.min.js HTTP/1.1 | 200 |
10.131.2.1 | [29/Nov/2017:06:59:30 | GET /edit.php?name=bala HTTP/1.1 | 200 |
10.131.2.1 | [29/Nov/2017:06:59:37 | GET /logout.php HTTP/1.1 | 302 |
10.131.2.1 | [29/Nov/2017:06:59:37 | GET /login.php HTTP/1.1 | 200 |
10.130.2.1 | [29/Nov/2017:07:00:19 | GET /login.php HTTP/1.1 | 200 |
10.130.2.1 | [29/Nov/2017:07:00:21 | GET /login.php HTTP/1.1 | 200 |
10.130.2.1 | [29/Nov/2017:13:31:27 | GET / HTTP/1.1 | 302 |
10.130.2.1 | [29/Nov/2017:13:31:28 | GET /login.php HTTP/1.1 | 200 |
10.129.2.1 | [29/Nov/2017:13:38:03 | POST /process.php HTTP/1.1 | 302 |
10.131.0.1 | [29/Nov/2017:13:38:04 | GET /home.php HTTP/1.1 | 200 |
10.131.0.1 | [29/Nov/2017:13:38:07 | GET /contestproblem.php?name=RUET%20OJ%20Server%20Testing%20Contest HTTP/1.1 | 200 |
10.130.2.1 | [29/Nov/2017:13:38:19 | GET / HTTP/1.1 | 302 |
10.131.2.1 | [29/Nov/2017:13:38:20 | GET /login.php HTTP/1.1 | 200 |
10.131.2.1 | [29/Nov/2017:13:38:20 | GET /css/bootstrap.min.css HTTP/1.1 | 200 |
10.128.2.1 | [29/Nov/2017:13:38:20 | GET /css/font-awesome.min.css HTTP/1.1 | 200 |
10.131.0.1 | [29/Nov/2017:13:38:20 | GET /css/normalize.css HTTP/1.1 | 200 |
10.128.2.1 | [29/Nov/2017:13:38:20 | GET /css/style.css HTTP/1.1 | 200 |
10.131.0.1 | [29/Nov/2017:13:38:20 | GET /js/vendor/modernizr-2.8.3.min.js HTTP/1.1 | 200 |
10.129.2.1 | [29/Nov/2017:13:38:20 | GET /css/main.css HTTP/1.1 | 200 |
10.131.0.1 | [29/Nov/2017:13:38:21 | GET /js/vendor/jquery-1.12.0.min.js HTTP/1.1 | 200 |
10.128.2.1 | [29/Nov/2017:13:38:21 | GET /bootstrap-3.3.7/js/bootstrap.min.js HTTP/1.1 | 200 |
10.131.2.1 | [29/Nov/2017:13:38:23 | GET /fonts/fontawesome-webfont.woff2?v=4.6.3 HTTP/1.1 | 200 |
10.131.2.1 | [29/Nov/2017:13:46:46 | GET / HTTP/1.1 | 200 |
10.131.2.1 | [29/Nov/2017:13:46:50 | GET /contestproblem.php?name=RUET%20OJ%20Server%20Testing%20Contest HTTP/1.1 | 200 |
10.131.2.1 | [29/Nov/2017:13:46:53 | GET /logout.php HTTP/1.1 | 302 |
10.131.2.1 | [29/Nov/2017:13:46:54 | GET /login.php HTTP/1.1 | 200 |
10.131.2.1 | [29/Nov/2017:13:47:04 | POST /process.php HTTP/1.1 | 302 |
Step by Step Solution
There are 3 Steps involved in it
Step: 1
Get Instant Access to Expert-Tailored Solutions
See step-by-step solutions with expert insights and AI powered tools for academic success
Step: 2
Step: 3
Ace Your Homework with AI
Get the answers you need in no time with our AI-driven, step-by-step assistance
Get Started