I am working on one sample log file from one vending machine. (pretty new to the pandas). Every day the machine will generate one .log file.

Q: How to use python and pandas to extract the info from the .log file, and eventually save the info into a data framework for the next step analysis? like the below pattern. (Note: message info should also be one complete piece info)

You can find my sample code and sample .log file below:

import pandas as pd filePath = "~/sample.log" with open(filePath,'r',encoding='utf16') as fp : line = fp.read() print(lines)

Sample logs info:

From the above samples, we can clearly see that the log file contains multiple similar pattern info. the log file starts with Index and ends with UserName.

a couple of things noticed: 1. the log file might start with multiple empty lines 2. the Message might contain multiple empty lines and could contain ":" (colons) as well

Below are a couple of my thoughts:

the most difficult part to analyze the above log is the "Message" part, which could have colons as well.

the log file is repeating the process from Index, EntryTpye..... until Message.

Can someone please provide me with some code to solve this problem?

Thank you so much!

UserName. (Column names below are the corresponding info) Index 112078 112079 112080 112081 EntryType Error Error Error Error Index EntryType InstanceId Message : 112090 : Error : 21224610 : The program wond.exe version 14.12.134.13 stopped working with Windows and was closed. To see if more information feel free to check the problem history. Process ID: 111111 Start Time: 1607841118 Termination Time: 12 Application Path: C:\windows\system64\testing\wond.exe Report Id: aac3-433-3333-223dd-33322deee Testing full name: Testing possble-happen application ID: Category : (200) CategoryNumber : 200 ReplacementStrings : {wond.exe, 14.12.134.13, 11111, 1607841118...} Source : someone where Time Generated : 12/11/2020 7:28:59 AM TimeWritten : 12/11/2020 7:28:59 AM UserName :| Index : 112078 EntryType : Error InstanceId : 21224490 Message : this is can be a sample test info for machine '-20933422' in the city 'nowhere'. The following information is part of the event:'testing', "C:\Windows\system64\testing\sample.dll', '10' Category : (0) CategoryNumber : 0 ReplacementStrings : {sample, C:\Windows\system64\testing\sample.dll, 10} Source : unkonwn Time Generated : 12/11/2020 15:02:20 PM TimeWritten : 12/11/2020 15:03:20 PM UserName : Index : 112079 EntryType : Error InstanceId : 21224500 Message : this is can be a sample test info for machine '-20933423' in the city 'where'. The following information is part of the event:'testing', 'C:\windows\system64\testing\sample.dll', '10': Category : (0) CategoryNumber : 0 ReplacementStrings : {} Source : eventi Time Generated : 12/11/2020 16:03:20 PM TimeWritten : 12/11/2020 16:02:21 PM UserName : UserName. (Column names below are the corresponding info) Index 112078 112079 112080 112081 EntryType Error Error Error Error Index EntryType InstanceId Message : 112090 : Error : 21224610 : The program wond.exe version 14.12.134.13 stopped working with Windows and was closed. To see if more information feel free to check the problem history. Process ID: 111111 Start Time: 1607841118 Termination Time: 12 Application Path: C:\windows\system64\testing\wond.exe Report Id: aac3-433-3333-223dd-33322deee Testing full name: Testing possble-happen application ID: Category : (200) CategoryNumber : 200 ReplacementStrings : {wond.exe, 14.12.134.13, 11111, 1607841118...} Source : someone where Time Generated : 12/11/2020 7:28:59 AM TimeWritten : 12/11/2020 7:28:59 AM UserName :| Index : 112078 EntryType : Error InstanceId : 21224490 Message : this is can be a sample test info for machine '-20933422' in the city 'nowhere'. The following information is part of the event:'testing', "C:\Windows\system64\testing\sample.dll', '10' Category : (0) CategoryNumber : 0 ReplacementStrings : {sample, C:\Windows\system64\testing\sample.dll, 10} Source : unkonwn Time Generated : 12/11/2020 15:02:20 PM TimeWritten : 12/11/2020 15:03:20 PM UserName : Index : 112079 EntryType : Error InstanceId : 21224500 Message : this is can be a sample test info for machine '-20933423' in the city 'where'. The following information is part of the event:'testing', 'C:\windows\system64\testing\sample.dll', '10': Category : (0) CategoryNumber : 0 ReplacementStrings : {} Source : eventi Time Generated : 12/11/2020 16:03:20 PM TimeWritten : 12/11/2020 16:02:21 PM UserName