What are logs and why monitor them?

It seems incredible that a text file can be important for the proper functioning of a company.

Because a log, in its most basic form, is nothing more than that.

However, it is not that they are important, it is that they are vital.

Let’s see it.

What is a log and what is it for?

In computing, when generally speaking of “log”, we refer to information belonging to a more or less low level reported by the operating system or a specific application that helps identify what is being done, including errors, problems or minor warnings, and when that happens, it indicates the date and time. In some cases, the source, the user, the IP address and other interesting fields from the point of view of what has happened can be identified.

Each application and each operating system can have a different log format. In fact, although there is a certain standard for unifying log records – generically called syslog – the truth is that most applications still have their own formats.

Keep in mind that the term “log” is so generic that it does not even define how that information is stored or where it is stored. In fact, Microsoft Windows systems store information in what they call “system events”, which are accessed through the system event viewer. Each system event in Windows has a series of fields that define the event, its severity, the type of information, category, etc.

Monitoring software feed on logs

Pandora and any other monitoring software uses and generates logs to give you vital information about your systems in real time.

How?

You can see for yourself using the Open Source version of Pandora:

Log examples

This is what a Windows event looks like, which would be the Windows “log”:

While in Windows, the information log or system and application log is almost always unified in the system event viewer, in Unix systems (Linux included in this category) it is much more chaotic, and you will find that there is no single log but rather there are several and with different formats.

This is an example Linux log (messages):

Nov 17 04:19:52 cylon3 systemd: Stopping The Apache HTTP Server…
Nov 17 04:19:53 cylon3 systemd: Stopped The Apache HTTP Server.
Nov 17 04:19:54 cylon3 systemd: Starting The Apache HTTP Server…
Nov 17 04:19:54 cylon3 httpd: AH00558: httpd: Could not reliably determine the server’s fully qualified domain name, using fe80::4637:e6ff:fedd:fa27. Set the ‘ServerName’ directive globally to suppress this message
Nov 17 04:19:54 cylon3 systemd: Started The Apache HTTP Server.
Nov 17 04:20:41 cylon3 yum[26424]: Installed: php-imap-7.3.24-1.el7.remi.x86_64
Nov 17 04:20:42 cylon3 yum[26424]: Installed: php-ldap-7.3.24-1.el7.remi.x86_64
Nov 17 04:20:44 cylon3 yum[26424]: Installed: php-mbstring-7.3.24-1.el7.remi.x86_64
Nov 17 04:20:52 cylon3 systemd: Stopping The Apache HTTP Server…
Nov 17 04:20:53 cylon3 systemd: Stopped The Apache HTTP Server.
Nov 17 04:20:53 cylon3 systemd: Starting The Apache HTTP Server…
Nov 17 04:20:53 cylon3 httpd: AH00558: httpd: Could not reliably determine the server’s fully qualified domain name, using fe80::4637:e6ff:fedd:fa27. Set the ‘ServerName’ directive globally to suppress this message
Nov 17 04:20:53 cylon3 systemd: Started The Apache HTTP Server.
Nov 17 05:01:01 cylon3 systemd: Started Session 71 of user root.
Nov 17 06:01:01 cylon3 systemd: Started Session 72 of user root.
Nov 17 07:01:01 cylon3 systemd: Started Session 73 of user root.
Nov 17 07:20:32 cylon3 systemd: Starting Cleanup of Temporary Directories…

On Linux systems, logs are stored in files located in /var/log directory, although many programs handle their own logs and save them in /var/log/. The advantage of Linux is that most logs are plain text log files, accessible and visible from any text handling tool, unlike Windows events that can only be accessed through the event viewer and/or low level API.

Some of the most important logs in Linux and that should be in all systems are the following:

• /var/log/messages – Here you will find the system’s generic logs. Many applications dump their logs here through the syslog program that collects log events from other applications to put them in this file.
• /var/log/secure – This log records system logins, the times you do su, etc. Failed attempts are recorded in lines with information such as invalid password or authentication failure.
• /var/log/dmesg – This file stores the information generated by the kernel during system startup and other events it may generate during execution. These are usually low-level messages not captured by the syslog subsystem.

Some typical application logs that you may optionally find are the following:

• /var/log/maillog – Mailing log, generally related to the local mailing subsystem (SMTP, Postfix, sendmail).
• /var/log/pandora/pandora/pandora_agent.log – Pandora FMS monitoring agent log, which informs you of each execution and if it has suffered any issues.
• /var/log/httpd/access_log – Log from an HTTP server, such as Apache.

What are logs for?

What are logs for? It will depend on what you need, but it is important that you know that logs can be essential from a security point of view, as well as that of the analysis of system usage, performance or failure detection.

Logs are sometimes the “food of choice” for Machine Learning techniques to detect patterns that help you make decisions.

Logs are essential for managing and controlling access to resources; this is related to security audit and derivatives.

Logs can be monitored and rules may be set to warn when something happens. It is one of the things that systems like Pandora FMS do.

To sum up, logs can help you understand how your system is used to prevent information leaks, as well as inappropriate behavior that causes errors.

There are different technologies that help centrally manage logs; the best known worldwide is Splunk, and another one is Pandora FMS for log collection, which allows log storage, collection and management.

What is Amazon CloudWatch Logs?

Amazon CloudWatch Logs is used to monitor and store log files (logs) from Amazon Elastic Compute Cloud (Amazon EC2), AWS CloudTrail, Route 53, and other Amazon sources.

CloudWatch Logs lets you centralize logs for all AWS systems, applications, and services you use, into a single, highly scalable service. It allows you to access and see them to look for specific error codes or patterns in them, set filters based on the content of certain fields, or securely archive them for future analysis.

CloudWatch Logs enables you to see all of your logs regardless of their source, as a single consistent stream of events sorted by date and time, as well as query and sort them based on other dimensions, group them by specific fields, and more. It also allows you to export these data so that other systems (such as Splunk or Pandora FMS) can manage them externally.