Server Log Management and Why It Matters

Every server keeps a running diary: who logged in, which pages were requested, what errors occurred, which services restarted. Those logs are the first place to look when something breaks and the main evidence after a security incident. Yet on many servers they are left to grow until the disk fills, are deleted too soon to be useful, or are never read at all. Good log management means collecting the right logs, keeping them for the right length of time, protecting them, and actually using them.

Why log management matters

• Troubleshooting: error logs explain why a page returns 500, why a service crashed, or why an email bounced.
• Security: authentication logs reveal brute-force attempts and suspicious logins; web logs show scanning and exploitation attempts.
• Investigation: after an incident, logs reconstruct what happened, when, and from where. Without them, you are guessing.
• Compliance: many security standards and contracts require that certain events are logged and retained.
• Performance insight: access logs show slow endpoints, traffic patterns and bot activity.

Where Linux logs live

Minimal installs and containers may not write text files at all; on systemd-based systems everything also goes to the journal, managed by journald.

Reading logs with journalctl

journalctl -u nginx --since "1 hour ago"     # one service, recent entries
journalctl -p err -b                         # errors since last boot
journalctl -f                                # follow live, like tail -f
journalctl --disk-usage                      # how much space the journal uses

By default some distributions keep the journal only in memory, losing it at reboot. To keep it on disk and cap its size, set in /etc/systemd/journald.conf:

[Journal]
Storage=persistent
SystemMaxUse=1G

Then restart with sudo systemctl restart systemd-journald.

Rotating log files with logrotate

Text logs grow forever unless rotated. logrotate renames the current file, starts a fresh one, compresses old copies and deletes the oldest. Most packages install a sensible rule in /etc/logrotate.d/, but custom application logs are often forgotten. An example for an application writing to /var/log/myapp/:

/var/log/myapp/*.log {
    daily
    rotate 14
    compress
    delaycompress
    missingok
    notifempty
    create 0640 myapp adm
    sharedscripts
    postrotate
        systemctl reload myapp >/dev/null 2>&1 || true
    endscript
}

Test a rule without waiting a day using sudo logrotate -d /etc/logrotate.d/myapp (dry run) or force a rotation with -f. The postrotate step tells the application to reopen its log file; without it, some programs keep writing to the renamed file.

Centralised logging

Once you have more than one or two servers, logging into each to read files becomes impractical. Centralised logging ships logs from every server to one place where they can be searched, correlated and alerted on. It also protects evidence: an attacker who gains root on a server can edit local logs, but not copies already sent elsewhere.

Common approaches:

• rsyslog forwarding to a central syslog server: simple, built in, and well suited to small setups.
• Grafana Loki with an agent such as Grafana Alloy or Promtail: lightweight, indexes labels rather than full text, pairs well with Grafana dashboards.
• Elasticsearch or OpenSearch with Logstash, Fluent Bit or Beats: powerful full-text search, heavier to run.
• Cloud services such as Amazon CloudWatch Logs, Azure Monitor Logs and Google Cloud Logging, or commercial log platforms.

A one-line rsyslog forwarding rule, in /etc/rsyslog.d/90-forward.conf, sending everything over TCP:

*.* @@logs.internal.example.com:514

Plain syslog over the network is unencrypted; use TLS or a private network for anything sensitive.

Retention: how long to keep logs

Keep logs long enough to investigate incidents, which are often discovered weeks after they start, but no longer than you need. A common pattern is 30 to 90 days searchable, with older logs archived to cheaper storage for a year or more where security or compliance requirements call for it. Check any legal, contractual or industry requirements that apply to you.

Logs contain personal data

IP addresses, email addresses, usernames and sometimes form contents end up in logs. Treat logs as sensitive data: restrict who can read them, avoid logging passwords, tokens or payment details, mask what you do not need, and apply the same data protection rules you apply elsewhere.

What to alert on

Logs are most valuable when they trigger action automatically. Good starting alerts include:

• Successful logins by root or from unexpected countries or networks.
• Sudden spikes in failed logins or HTTP 5xx errors.
• New user accounts or changes to sudo rules.
• Out-of-memory events, disk errors and service crash loops.
• Silence: a server that stops sending logs at all.

Logging pipelines, retention and alerting are part of our server management work, and our open source solutions page covers self-hosted stacks such as Loki and OpenSearch.

Key takeaways

• Log management covers collection, rotation, central storage, retention, protection and alerting.
• Make the journal persistent and size-limited, and rotate every text log, including custom application logs.
• Ship logs off the server to a central system so they survive failures and tampering.
• Treat logs as personal data, keep them only as long as needed, and alert on the events that matter.