Log File Overview

Log files (also spelled logfiles), are files used by computers to log events. Software programs, background processes, services, or transactions between services, including the operating system itself, may generate such events. Log files are dependent on the application that generates them. It is up to the application developer to conform to log file convention. Software documentation should include information on its log files.

Step 1: Web server log file example

Because log files are essentially a way to track specific events, the type of information stored varies depending of the application or services generating the events.

a. Consider the single log entry below. It was generated by Apache, a popular web server.

[Wed Mar 22 11:23:12.207022 2017] [core:error] [pid 3548:tid 4682351596] [client 209.165.200.230] File does not exist: /var/www/apache/htdocs/favicon.ico

The single log entry above represents a web event recorder by Apache. A few pieces of information are important in web transactions, including client IP address, time and details of the transaction. The entry above can be broken down into five main parts:

Timestamp: This part records when the event took place. It is very important that the server clock is correctly synchronized as it allows for accurately cross-referencing and tracing back events.

 Operating system log file example

Any software can keep log files, including the operating system itself. Conventionally, Linux uses the /var/log directory to stores various log files, including operating system logs. Modern operating systems are complex pieces of software and therefore, use several different files to log events. This section takes a quick look at the /var/log/messages file.

a. Stored under /var/log, the messages file stores various system events. The connection of new USB drive, a network card becoming available, and too many missed root login attempts, are a few examples of events logged to the /var/log/messages file. Use the more command to display the contents of the /var/log/messages file. Unlike the cat command, more allows for a paced navigation through the file.
Press ENTER to advance line-by-line or SPACE to advance an entire page. Press q or CTRL + C to abort and exit more.

Note: the sudo command is required because the messages file belongs to the root user.

[analyst@secOps ~]$ sudo more /var/log/messages [sudo] password for analyst: Mar 20 08:34:38 secOps kernel: [ 6.149910] random: crng init done Mar 20 08:34:40 secOps kernel: [ 8.280667] floppy0: no floppy controllers found Mar 20 08:34:40 secOps kernel: [ 8.280724] work still pending Mar 20 08:35:16 secOps kernel: [ 44.414695] hrtimer: interrupt took 5346452 ns Mar 20 14:28:29 secOps kernel: [21239.566409] pcnet32 0000:00:03.0 enp0s3: link down Mar 20 14:28:33 secOps kernel: [21243.404646] pcnet32 0000:00:03.0 enp0s3: link up, 100Mbps, full-duplex Mar 20 14:28:35 secOps kernel: [21245.536961] pcnet32 0000:00:03.0 enp0s3: link down Mar 20 14:28:43 secOps kernel: [21253.427459] pcnet32 0000:00:03.0 enp0s3: link up, 100Mbps, full-duplex Mar 20 14:28:53 secOps kernel: [21263.449480] pcnet32 0000:00:03.0 enp0s3: link down Mar 20 14:28:57 secOps kernel: [21267.500152] pcnet32 0000:00:03.0 enp0s3: link up, 100Mbps, full-duplex Mar 20 14:29:01 secOps kernel: [21271.551499] pcnet32 0000:00:03.0 enp0s3: link down Mar 20 14:29:05 secOps kernel: [21275.389707] pcnet32 0000:00:03.0 enp0s3: link up, 100Mbps, full-duplex Mar 22 06:01:40 secOps kernel: [ 0.000000] Linux version 4.8.12-2-ARCH (builduser@andyrtr) (gcc version 6.2.1 20160830 (GCC) ) #1 SMP PREEMPT Fri Dec 2 20:41:47 CET 2016 Mar 22 06:01:40 secOps kernel: [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' Mar 22 06:01:40 secOps kernel: [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' Mar 22 06:01:40 secOps kernel: [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' Mar 22 06:01:40 secOps kernel: [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 Mar 22 06:01:40 secOps kernel: [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. Mar 22 06:01:40 secOps kernel: [ 0.000000] x86/fpu: Using 'eager' FPU context switches. 

Notice that the events listed above are very different from the web server events. Because the operating system itself is generating this log, all recorded events are in relation to the OS itself.

b. If necessary, enter Ctrl + C to exit out of the previous command.

c. Log files are very important for troubleshooting. Assume that a user of that specific system reported that all network operations were slow around 2:30pm. Can you find evidence of that in the log entries shown above? If so in what lines? Explain.

Type: This is the type of event. In this case, it was an error.

PID: This contains information about the process ID used by Apache at the moment.

Client: This records the IP address of the requesting client.

Description: This contains a description of the event.

Based on the log entry above, describe what happened.
On Wednesday, March 22nd, 11:23:12.207022 am of 2017, a client with IP address of 209.165.200.230 requested a non-existent file named favicon.ico. The file should have been located in the following path /var/www/apache/htdocs/favicon.ico, but because it could not be found, it triggered an error.

Use the cat command below to list a web server sample log file. The sample file is located at /var/log:

[analyst@secOps ~]$ cat /var/log/logstash-tutorial.log 83.149.9.216  

Is the output above still considered a web transaction? Explain why the output of the cat command is in a different format than the single entry shown in item (a).

Step 2: Operating system log file example

Any software can keep log files, including the operating system itself. Conventionally, Linux uses the /var/log directory to stores various log files, including operating system logs. Modern operating systems are complex pieces of software and therefore, use several different files to log events. This section takes a quick look at the /var/log/messages file.

a. Stored under /var/log, the messages file stores various system events. The connection of new USB drive, a network card becoming available, and too many missed root login attempts, are a few examples of events logged to the /var/log/messages file. Use the more command to display the contents of the /var/log/messages file. Unlike the cat command, more allows for a paced navigation through the file.
Press ENTER to advance line-by-line or SPACE to advance an entire page. Press q or CTRL + C to abort and exit more.

Note: the sudo command is required because the messages file belongs to the root user.

[analyst@secOps ~]$ sudo more /var/log/messages [sudo] password for analyst: Mar 20 08:34:38 secOps kernel: [ 6.149910] random: crng init done Mar 20 08:34:40 secOps kernel: [ 8.280667] floppy0: no floppy controllers found Mar 20 08:34:40 secOps kernel: [ 8.280724] work still pending Mar 20 08:35:16 secOps kernel: [ 44.414695] hrtimer: interrupt took 5346452 ns Mar 20 14:28:29 secOps kernel: [21239.566409] pcnet32 0000:00:03.0 enp0s3: link down Mar 20 14:28:33 secOps kernel: [21243.404646] pcnet32 0000:00:03.0 enp0s3: link up, 100Mbps, full-duplex Mar 20 14:28:35 secOps kernel: [21245.536961] pcnet32 0000:00:03.0 enp0s3: link down Mar 20 14:28:43 secOps kernel: [21253.427459] pcnet32 0000:00:03.0 enp0s3: link up, 100Mbps, full-duplex Mar 20 14:28:53 secOps kernel: [21263.449480] pcnet32 0000:00:03.0 enp0s3: link down Mar 20 14:28:57 secOps kernel: [21267.500152] pcnet32 0000:00:03.0 enp0s3: link up, 100Mbps, full-duplex Mar 20 14:29:01 secOps kernel: [21271.551499] pcnet32 0000:00:03.0 enp0s3: link down Mar 20 14:29:05 secOps kernel: [21275.389707] pcnet32 0000:00:03.0 enp0s3: link up, 100Mbps, full-duplex Mar 22 06:01:40 secOps kernel: [ 0.000000] Linux version 4.8.12-2-ARCH (builduser@andyrtr) (gcc version 6.2.1 20160830 (GCC) ) #1 SMP PREEMPT Fri Dec 2 20:41:47 CET 2016 Mar 22 06:01:40 secOps kernel: [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' Mar 22 06:01:40 secOps kernel: [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' Mar 22 06:01:40 secOps kernel: [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' Mar 22 06:01:40 secOps kernel: [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 Mar 22 06:01:40 secOps kernel: [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. Mar 22 06:01:40 secOps kernel: [ 0.000000] x86/fpu: Using 'eager' FPU context switches. 

Notice that the events listed above are very different from the web server events. Because the operating system itself is generating this log, all recorded events are in relation to the OS itself.

b. If necessary, enter Ctrl + C to exit out of the previous command.

c. Log files are very important for troubleshooting. Assume that a user of that specific system reported that all network operations were slow around 2:30pm. Can you find evidence of that in the log entries shown above? If so in what lines? Explain.