Rsyslog Configuration Role

Overview

This role configures rsyslog on Linux systems to forward system logs to a remote Graylog server and optionally monitor custom application log files. It installs rsyslog, configures remote log forwarding via UDP or TCP, sets up file monitoring for application-specific logs, and restarts the service to activate changes. This enables centralized logging for system events and application logs in Graylog for analysis, alerting, and troubleshooting.

Purpose

Centralized Logging: Forward all system logs to Graylog
Application Monitoring: Monitor custom application log files
Remote Forwarding: Send logs via UDP or TCP to Graylog
File Tailing: Watch log files and forward new entries
Flexible Configuration: Define which logs to monitor per host
Automated Setup: Deploy consistent logging across all servers
SIEM Integration: Enable security monitoring and alerting

Requirements

Ansible 2.9 or higher
Target system: Debian/Ubuntu (systemd-based)
Root or sudo privileges
Graylog server with syslog input configured
Network connectivity from target to Graylog
Rsyslog installed (role will install if missing)

What is Rsyslog?

Rsyslog is a high-performance log processing system for Linux:

Key features:

Collects system logs (kernel, services, applications)
Forwards logs to remote servers
Monitors log files (tail functionality)
Filters and transforms logs
Supports TCP, UDP, TLS protocols

Why use remote logging:

Centralization: All logs in one place
Retention: Long-term storage on dedicated server
Search: Powerful query capabilities in Graylog
Alerting: Notify on specific log patterns
Security: Logs survive if host compromised
Compliance: Meet audit requirements

Role Variables

Optional Variables

Variable	Default	Description
`rsyslog_configuration_server_ip`	`hostvars['graylog']['ip_vlan12']`	Graylog server IP
`rsyslog_configuration_server_port`	`514`	Syslog port on Graylog
`rsyslog_configuration_protocol`	`udp`	Transport protocol (udp/tcp)
`rsyslog_configuration_custom_log_files`	`[]`	Custom log files to monitor
`rsyslog_configuration_polling_interval`	`10`	File polling interval (seconds)
`rsyslog_configuration_default_severity`	`info`	Default log severity
`rsyslog_configuration_default_facility`	`local0`	Default log facility
`rsyslog_configuration_service_name`	`rsyslog`	Rsyslog service name
`rsyslog_configuration_config_dir`	`/etc/rsyslog.d`	Config directory

Variable Details

rsyslog_configuration_server_ip

IP address of the Graylog server to receive logs.

Default: "{{ hostvars['graylog']['ip_vlan12'] }}"

Example: "192.168.x.x"

rsyslog_configuration_server_port

UDP or TCP port for syslog on Graylog.

Default: 514 (standard syslog port)

Common values:

514: Standard syslog
1514: Alternative port
6514: Syslog over TLS (requires additional config)

rsyslog_configuration_protocol

Transport protocol for log forwarding.

Default: udp

Values:

udp: Fast, may lose packets, lower overhead
tcp: Reliable, guaranteed delivery, connection-based

UDP vs TCP:

UDP: Better for high-volume, non-critical logs
TCP: Better for critical logs requiring delivery guarantee

Configuration:

# UDP (single @)
*.* @192.168.x.x:514

# TCP (double @@)
*.* @@192.168.x.x:514

rsyslog_configuration_custom_log_files

List of custom application log files to monitor.

Default: [] (empty, no custom monitoring)

Structure:

rsyslog_configuration_custom_log_files:
  - file: /path/to/logfile.log
    tag: application-name
    severity: info
    facility: local0

Example:

rsyslog_configuration_custom_log_files:
  - file: /var/log/myapp/application.log
    tag: myapp
    severity: info
    facility: local1

  - file: /var/log/nginx/access.log
    tag: nginx-access
    severity: info
    facility: local2

  - file: /var/log/apache2/error.log
    tag: apache-error
    severity: err
    facility: local3

Field descriptions:

Field	Required	Default	Description
`file`	Yes	-	Full path to log file
`tag`	Yes	-	Identifier tag for logs
`severity`	No	`info`	Log severity level
`facility`	No	`local0`	Syslog facility

rsyslog_configuration_polling_interval

How often (in seconds) rsyslog checks monitored files for new content.

Default: 10 seconds

Lower values: More responsive, higher CPU usage Higher values: Less responsive, lower CPU usage

Recommended: 10-30 seconds for most applications

rsyslog_configuration_default_severity

Default severity for monitored log files if not specified.

Default: info

Severity levels (highest to lowest):

emerg: Emergency (system unusable)
alert: Alert (immediate action)
crit: Critical
err: Error
warn: Warning
notice: Notice
info: Informational
debug: Debug

rsyslog_configuration_default_facility

Default facility for monitored log files if not specified.

Default: local0

Facilities:

local0 through local7: Custom applications (recommended)
user: User-level messages
daemon: System daemons
auth: Authentication
syslog: Syslog internal
kern: Kernel messages

Best practice: Use local0-local7 for application logs

Dependencies

No Ansible role dependencies, but requires:

Graylog server with syslog input
Network connectivity to Graylog
Target system running rsyslog (or compatible)

Often used with:

deploy_graylog: Deploy Graylog server
opnsense_syslog_configuration: Forward OPNsense logs
Application roles: Any role with log files to monitor

Example Playbook

Basic System Log Forwarding

---
- name: Configure Rsyslog Forwarding
  hosts: all
  become: true

  roles:
    - rsyslog_configuration

Custom Graylog Server

---
- name: Configure Rsyslog with Custom Server
  hosts: servers
  become: true

  vars:
    rsyslog_configuration_server_ip: "192.168.x.x"
    rsyslog_configuration_server_port: 1514
    rsyslog_configuration_protocol: tcp

  roles:
    - rsyslog_configuration

Monitor Application Logs

---
- name: Configure Rsyslog with Application Monitoring
  hosts: webservers
  become: true

  vars:
    rsyslog_configuration_custom_log_files:
      - file: /var/log/nginx/access.log
        tag: nginx-access
        severity: info
        facility: local1

      - file: /var/log/nginx/error.log
        tag: nginx-error
        severity: err
        facility: local1

      - file: /var/www/myapp/logs/application.log
        tag: myapp
        severity: info
        facility: local2

  roles:
    - rsyslog_configuration

High-Volume Logs (TCP)

---
- name: Configure Rsyslog with TCP for Reliability
  hosts: critical-servers
  become: true

  vars:
    rsyslog_configuration_protocol: tcp
    rsyslog_configuration_custom_log_files:
      - file: /var/log/app/transactions.log
        tag: transactions
        severity: notice
        facility: local3

  roles:
    - rsyslog_configuration

What This Role Does

Install rsyslog:
- Ensures rsyslog package installed
- Updates package cache if needed
Configure log forwarding:
- Creates forwarding config in /etc/rsyslog.d/
- Template: forward_to_graylog.conf.j2
- Forwards all system logs to Graylog
- Uses UDP or TCP based on configuration
Configure file monitoring (if custom logs defined):
- Loads imfile module for file tailing
- Creates monitoring config per host
- Template: custom_log_monitoring.conf.j2
- Monitors each specified log file
- Tags logs for identification in Graylog
Restart rsyslog:
- Restarts service via systemd
- Activates new configuration
- Begins forwarding logs immediately

Configuration Files Created

Forward to Graylog Config

Location: /etc/rsyslog.d/99-forward-to-graylog.conf

Content (UDP):

# Graylog Remote Forwarding Configuration
# This file is managed by Ansible - do not edit manually
# Forwards all system logs to Graylog server

# Forward all logs to Graylog via UDP
*.* @192.168.x.x:514

Content (TCP):

# Forward all logs to Graylog via TCP
*.* @@192.168.x.x:514

Syntax:

*.*: All facilities, all severities
@: UDP protocol (single @)
@@: TCP protocol (double @)
ip:port: Graylog server destination

Custom Log Monitoring Config

Location: /etc/rsyslog.d/90-<hostname>-custom-logs.conf

Example content:

# Custom Application Log Monitoring for webserver1
# This file is managed by Ansible - do not edit manually
# Monitors application-specific log files and forwards them to Graylog

# Load imfile module for file monitoring
module(load="imfile" PollingInterval="10")

# Monitor nginx-access
input(type="imfile"
      File="/var/log/nginx/access.log"
      Tag="nginx-access"
      Severity="info"
      Facility="local1")

# Monitor myapp
input(type="imfile"
      File="/var/www/myapp/logs/application.log"
      Tag="myapp"
      Severity="info"
      Facility="local2")

Rsyslog imfile Module

imfile monitors text log files and forwards new entries.

How It Works

1. rsyslog monitors file inode
2. Polls for changes every PollingInterval seconds
3. Reads new lines appended to file
4. Tags and forwards to destinations
5. Tracks read position (survives restart)

State Tracking

State file: /var/spool/rsyslog/statefile

Purpose: Remembers last read position

Prevents duplicate log entries after restart
Handles log rotation gracefully
Tracks multiple files independently

Rotation Handling

Rsyslog automatically detects log rotation:

1. Application writes to app.log
2. Logrotate moves app.log → app.log.1
3. Application creates new app.log
4. Rsyslog detects inode change
5. Finishes reading old file
6. Switches to new file

No configuration needed for standard logrotate.

Log Format in Graylog

System Logs

Example message:

<facility*8+severity>timestamp hostname process[pid]: message

Parsed fields:

source: Hostname sending logs
facility: Syslog facility (user, daemon, kern, etc.)
level: Severity level (0-7)
message: Log message content
timestamp: Log generation time

Custom Application Logs

Example message:

<facility*8+severity>timestamp hostname tag: log content

Parsed fields:

source: Hostname
facility: Configured facility (local0-local7)
level: Configured severity
tag: Application identifier (nginx-access, myapp, etc.)
message: Log line from file

Searching in Graylog:

# All nginx access logs
tag:nginx-access

# Application errors
tag:myapp AND level:3

# Logs from specific host
source:webserver1 AND tag:nginx-access

Syslog Message Structure

Priority Calculation

Formula: priority = (facility * 8) + severity

Examples:

local0 (16) + info (6) = 134
local1 (17) + err (3) = 139
auth (4) + crit (2) = 34

Severity Levels

Level	Name	Description
0	emerg	System unusable
1	alert	Action must be taken immediately
2	crit	Critical conditions
3	err	Error conditions
4	warn	Warning conditions
5	notice	Normal but significant
6	info	Informational
7	debug	Debug-level messages

Facility Codes

Code	Facility	Description
0	kern	Kernel messages
1	user	User-level messages
2	mail	Mail system
3	daemon	System daemons
4	auth	Security/authorization
16-23	local0-7	Local use (applications)

Network Requirements

Firewall rules needed:

# On target servers: Allow outbound to Graylog
- action: pass
  direction: out
  destination_net: graylog_ip
  destination_port: "514"
  protocol: udp  # or tcp

# On Graylog: Allow inbound from all servers
- action: pass
  source_net: server_networks
  destination_port: "514"
  protocol: udp  # or tcp

Test connectivity:

# UDP test
nc -u -zv 192.168.x.x 514

# TCP test
nc -zv 192.168.x.x 514

# Send test syslog message
logger -n 192.168.x.x -P 514 "Test message from $(hostname)"

Graylog Input Configuration

Graylog must have syslog input:

System → Inputs
Select input type:
- “Syslog UDP” for UDP
- “Syslog TCP” for TCP
Configure:
- Bind address: 0.0.0.0 or specific IP
- Port: 514
- Allow override date: yes (recommended)
Save and Start

Security Considerations

Unencrypted: Standard syslog (UDP/TCP) is not encrypted
- Logs transmitted in plain text
- Consider TLS for sensitive environments (requires additional config)
Network Security: Ensure firewall rules allow only trusted sources
Log Privacy: Logs may contain sensitive information
Graylog Access: Restrict access to Graylog UI
Retention: Configure appropriate retention in Graylog
Authentication: Standard syslog has no authentication
- Use network segmentation
- Consider syslog-ng with TLS and certificates for high-security

Notes

Role runs on target systems (not localhost)
become: true required for package installation and config
Configuration files in /etc/rsyslog.d/ with numeric prefixes
Higher numbers = later processing (99 for forwarding ensures all logs captured)
Rsyslog service restarted automatically on config change
State files track file read positions
Compatible with systemd-based systems

Troubleshooting

Logs not appearing in Graylog

Check rsyslog running:

systemctl status rsyslog

Check configuration syntax:

rsyslogd -N1
# Should show "rsyslogd: version X, config validation run"

Test connectivity to Graylog:

# Send test message
logger -n 192.168.x.x -P 514 "Test from $(hostname)"

# Check if received in Graylog
# Search → Last 5 minutes → Filter: "Test from"

Check rsyslog logs:

# Debian/Ubuntu
tail -f /var/log/syslog | grep rsyslog

# Look for connection errors

Permission denied on log files

Symptom: Rsyslog can’t read application log file

Error in logs:

imfile: error opening file '/var/log/myapp/app.log': Permission denied

Solution: Adjust file permissions

# Option 1: Make readable by rsyslog
chmod 644 /var/log/myapp/app.log

# Option 2: Add rsyslog user to app group
usermod -a -G appgroup syslog

# Option 3: Use ACLs
setfacl -m u:syslog:r /var/log/myapp/app.log

File not being monitored

Check configuration:

cat /etc/rsyslog.d/90-*-custom-logs.conf

# Verify:
# - File path correct
# - File exists
# - imfile module loaded

Check state file:

ls -la /var/spool/rsyslog/
# Should show state files for monitored files

Restart rsyslog:

systemctl restart rsyslog

High CPU usage

Cause: Very frequent polling or large log files

Solutions:

# Increase polling interval
rsyslog_configuration_polling_interval: 30  # Check every 30s instead of 10s

# Or use inotify mode (more efficient)
# In custom template, change:
# module(load="imfile" mode="inotify")

Connection refused (TCP)

Symptom: rsyslog logs show connection refused

Causes:

Graylog not listening on TCP
Firewall blocking connection
Wrong IP or port

Solutions:

# Check Graylog listening
netstat -tln | grep 514

# Check from client
telnet 192.168.x.x 514

# Verify Graylog input running
# Graylog UI → System → Inputs

Logs delayed or batched

Cause: Rsyslog batching for performance

Explanation: Normal behavior, logs sent in batches

If immediate delivery needed:

# Add to forwarding config
*.* @@192.168.x.x:514
$ActionQueueType LinkedList
$ActionQueueFileName srvrfwd
$ActionResumeRetryCount -1
$ActionQueueSaveOnShutdown on

Testing the Role

Verify Configuration Files

# Check forwarding config
cat /etc/rsyslog.d/99-forward-to-graylog.conf

# Check custom monitoring config
cat /etc/rsyslog.d/90-*-custom-logs.conf

Test Configuration Syntax

# Validate rsyslog config
rsyslogd -N1

# Should output: "rsyslogd: version X, config validation run"
# No errors = valid config

Send Test Messages

# Test system log forwarding
logger "Test message from $(hostname)"

# Test custom severity
logger -p local0.info "Test custom facility"

# Test with specific tag
logger -t myapp "Test application log"

Verify in Graylog

Graylog UI → Search
Set timeframe: Last 5 minutes
Search queries:

# All logs from host
source:hostname

# Test message
message:Test

# Specific tag
tag:nginx-access

# Specific facility
facility:local0

Monitor Rsyslog Stats

# Check rsyslog status
systemctl status rsyslog

# Check message counts
rsyslogd -i /var/run/rsyslogd.pid

# View rsyslog debug output (verbose)
rsyslogd -dn

Best Practices

Use TCP for critical logs: Guaranteed delivery for important systems
Use UDP for high-volume: Better performance for less critical logs
Tag application logs: Use descriptive tags for easy filtering
Set appropriate severity: Match severity to log importance
Use facilities consistently: Assign facilities by application or purpose
Monitor disk space: On both clients and Graylog server
Test after deployment: Verify logs flowing to Graylog
Document monitored files: Keep inventory of what’s being logged
Regular rotation: Ensure logrotate configured on clients
Graylog retention: Set appropriate retention policy for log volume

Performance Tuning

High Volume Environments

Queue configuration:

# Add to forwarding config for buffering
$ActionQueueType LinkedList
$ActionQueueFileName forward
$ActionQueueMaxDiskSpace 1g
$ActionResumeRetryCount -1
$ActionQueueSaveOnShutdown on

Benefits:

Buffers logs during network outage
Prevents log loss
Resumes forwarding when connection restored

File Monitoring Optimization

Use inotify instead of polling:

module(load="imfile" mode="inotify")