Introduction to Centralized Logging

In the vast and often chaotic world of software development and system administration, logging is the unsung hero that helps us make sense of what’s happening behind the scenes. Imagine a world where every error, every warning, and every piece of diagnostic information is scattered across multiple servers and devices, making it a nightmare to troubleshoot issues. This is where Graylog steps in, offering a powerful and flexible solution for centralized logging.

What is Graylog?

Graylog is an open-source platform designed for the centralized collection, storage, visualization, filtering, and search of logs. It supports a wide range of log sources, including Linux servers, Windows hosts, network devices, and more. With Graylog, you can collect terabytes of logs and still perform searches almost instantaneously.

Components of the Graylog Stack

Before diving into the setup, it’s essential to understand the components that make up the Graylog stack:

  • Graylog Server: This is the core component responsible for collecting, processing, and analyzing logs. It provides a web interface for configuration and visualization.
  • MongoDB: Used for storing metadata.
  • OpenSearch (formerly Elasticsearch): Acts as the search engine and NoSQL storage for the logs.
  • Graylog Web Interface: Provides a user-friendly interface for managing and visualizing logs.

Setting Up Graylog

Prerequisites

Before you start, ensure your system meets the necessary requirements. Here are some key steps to prepare your environment:

Adjust Virtual Memory Settings

To run OpenSearch, you need to adjust the virtual memory settings:

$ sudo sysctl -w vm.max_map_count=262144
$ sudo echo 'vm.max_map_count=262144' >> /etc/sysctl.conf

Install OpenSearch

OpenSearch is a critical component of the Graylog stack. Here’s how you can install and start it:

$ sudo systemctl enable --now opensearch

Installing Graylog Server

Graylog offers two versions: the free Graylog Open and the enterprise version Graylog Operations. Here, we will focus on installing Graylog Open:

$ wget https://packages.graylog2.org/repo/packages/graylog-5.2-repository_latest.deb
$ sudo dpkg -i graylog-5.2-repository_latest.deb
$ sudo apt-get update && sudo apt-get install graylog-server

Configuring Graylog Server

To configure the Graylog server, you need to generate passwords for password_secret and root_password_sha2. Here’s how you can do it:

$ pwgen -N 1 -s 96
$ echo -n "Enter Password: " && head -1 </dev/stdin | tr -d '\n' | sha256sum | cut -d" " -f1

Update the /etc/graylog/server/server.conf file with these values and specify the http_bind_address:

http_bind_address = 0.0.0.0:9000
password_secret = your_generated_password_secret
root_password_sha2 = your_generated_root_password_sha2

Starting Graylog Server

Now, you can start the Graylog server:

$ sudo systemctl enable --now graylog-server

Initial Configuration

The first time you start Graylog, you’ll need to use a temporary password found in the server logs:

$ cat /var/log/graylog-server/server.log

Access the initial configuration interface using the temporary credentials provided in the logs:

http://admin:[email protected]:9000

Follow the setup wizard to complete the initial configuration.

Configuring Data Inputs

To start collecting logs, you need to set up data inputs. Here’s an example of setting up a Syslog UDP input for Linux servers:

  1. Create Input:

    • Go to System -> Inputs and create a new input of type Syslog UDP.
    • Specify the name and port (e.g., 514) where the server will receive data.
    • Leave other settings as default.

  2. Create Index:

    • Go to System -> Indices and create a new index for the Linux logs.
    • Specify the name, description, and prefix (e.g., linux_indx).
    • Configure how long to keep old logs and when to delete old indices.

  3. Create Stream:

    • Go to Streams -> Create Stream and specify the name of the stream.
    • Select the index you created for Linux logs.
    • Add a new rule to the stream to match the input you created.
graph TD A("Linux Server") -->|Syslog UDP| B("Graylog Server") B -->|Input| C("Stream") C -->|Rule| D("Index") D -->|Storage| B("OpenSearch")

Configuring Log Forwarding from Clients

To send logs from your devices to Graylog, you can use various tools such as rsyslog, Filebeat, or Winlogbeat.

Using rsyslog for Linux Servers

Here’s how you can configure rsyslog to forward logs to Graylog:

$ sudo apt install rsyslog
$ sudo systemctl status rsyslog
$ sudo nano /etc/rsyslog.d/60-graylog.conf

# Add the following line to the file
*.* @192.168.14.146:20514;RSYSLOG_SyslogProtocol23Format

Replace 192.168.14.146 with the IP address of your Graylog server and 20514 with the port number of the input you created.

Strategies for Log Collection

When setting up your logging strategy, you need to consider several factors:

Minimal Strategy

  • Collect only what is necessary.
  • This approach reduces storage costs and minimizes noise, allowing you to focus on critical events.
  • However, it may miss important information if not carefully planned.

Maximal Strategy

  • Collect all possible logs.
  • This approach ensures you have all data, which is particularly valuable for forensic analysis.
  • However, it can be resource-intensive and may not be practical due to budget constraints.

User Management and Access Control

Graylog allows you to manage users and their access levels. Here are some steps to set up users:

  1. Create a New User:

    • Go to System -> Users and Teams and create a new user.
    • Assign the admin role and specify an email address.

  2. Access Control:

    • Ensure each user group has the necessary privileges based on the principle of least privilege.
    • Typical user groups include security analysts, engineers, management, and support teams.

Storage and Retention

Proper planning of log storage is crucial. Here are some considerations:

  • Online Storage: Logs are stored in OpenSearch and are accessible via the Graylog interface.
  • Archived Storage: Logs are stored in a compressed format and can be restored for searching.
  • Retention Period: Determine how long to keep logs based on regulatory requirements or practical needs. A simple rule is to calculate the daily log volume, multiply it by the number of days to retain, and then multiply by 1.3 to account for metadata.

Conclusion

Setting up a centralized logging system with Graylog is a powerful way to manage and analyze logs from various sources. By following these steps and considering your logging strategy, user management, and storage needs, you can create a robust and efficient logging system that enhances your ability to monitor and troubleshoot your infrastructure.

With Graylog, you’re not just collecting logs; you’re building a window into the heart of your system, allowing you to see what’s happening in real-time and make informed decisions. So, go ahead and log in – the logs are waiting