Logs are the first thing you reach for during an incident and the first thing that quietly causes one. Unbounded logs fill disks. Over-aggressive cleanup deletes the evidence you needed. Good log management is deciding in advance how much you keep, for how long, and where.

Two Logging Worlds on a Modern Server

Most current Linux servers have two logging systems running side by side.

The systemd journal. Services managed by systemd write to the journal, which stores logs in a structured, indexed format. You query it with journalctl. Its size is governed by its own configuration, with settings for maximum disk usage and, depending on setup, whether it persists across reboots.

Plain text log files. Applications such as web servers and databases commonly write their own files under /var/log or their own directories. Rotation of these is typically handled by logrotate, run on a schedule by cron or a systemd timer.

Knowing which system holds the logs you need saves time, and it determines which tool controls retention.

How logrotate Works

Logrotate reads configuration from a main file and from per-application snippets, usually in /etc/logrotate.d/. For each log file pattern, a stanza defines what to do. The common directives:

• Rotation frequency or size trigger: daily, weekly, monthly, or when a file exceeds a size.
• Number of rotated files to keep.
• Compression of older files, often with a delay so the most recent archive stays easily readable.
• Missing or empty file handling, so absent logs do not generate errors.
• Post-rotation actions, such as signaling a service to reopen its log file.

That last item is where most breakage occurs. When logrotate renames a log file, the application still holds the old file open and keeps writing to the renamed file unless told to reopen. The usual solutions are a post-rotate command that signals the service, or the copytruncate option, which copies the file and truncates the original. Copytruncate is simpler but can lose lines written during the copy, so prefer a proper reopen where the application supports it.

Setting Retention Deliberately

Retention is a policy decision, not a technical default. Ask:

1. How far back do you investigate? Many issues are noticed days later, so a few weeks of detail is often a practical minimum for operational logs.
2. Are there legal or contractual requirements? Some environments require keeping certain logs for defined periods. Requirements vary by jurisdiction and industry, so consult the relevant compliance or legal advisers rather than guessing.
3. How much disk can logs consume? Estimate daily volume and multiply by retention, with room for spikes.
4. Do logs contain personal data? If so, retention should be as short as your needs allow, and access should be restricted.

For anything beyond a single server, shipping logs to a central location is worth considering. It protects evidence if a server is lost or compromised, and it lets you search across machines. Keep local retention short and central retention longer, and make sure the shipping itself is monitored.

The Journal’s Limits

Journal size is capped in its configuration file, and you can trim existing data with journalctl --vacuum-size or --vacuum-time. Decide whether you want persistent storage; without it, journal contents are lost at reboot, which is painful when you are investigating why a machine rebooted. Check your distribution’s current defaults, as they differ.

Pitfalls That Bite Later

• Logs that are not covered by any rotation rule, typically a custom application writing to a new path.
• Rotation that succeeds but the service keeps writing to the old file, leaving a new empty file and a huge hidden one.
• Permissions after rotation: the new file is created with the wrong owner, and the service can no longer write.
• Verbose debug logging left on, producing volume rotation cannot keep up with.
• Compressing too soon, making the freshest logs awkward to read during an incident.
• Logrotate itself failing silently, because nobody checks its status.

Configuration Decisions at a Glance

Decision	Options	Consideration
Rotation trigger	Time-based or size-based	Use size as a safety net for noisy logs
Reopen method	Signal service or copytruncate	Signal avoids losing lines
Compression	Immediate or delayed	Delay keeps recent logs easy to read
Local retention	Days to weeks	Match disk capacity and investigation needs
Central retention	Longer, policy-driven	Cost, privacy and compliance
Journal persistence	Volatile or persistent	Persistent helps post-reboot analysis

A Realistic Example

A team runs a custom service that writes to a directory nobody added to logrotate. Months later the disk fills. The fix has three parts. First, they add a logrotate snippet that rotates the file daily, keeps two weeks, compresses older archives with a one-cycle delay, and signals the service to reopen its log in a post-rotate step. Second, they test it with logrotate’s debug mode, which shows what would happen without changing anything, and then force one rotation to confirm the service writes to the fresh file. Third, they add a check that alerts if any log file under the application directory grows past a chosen size, catching the next unconfigured path before it fills the disk.

Frequently Asked Questions

Is it safe to delete old log files manually? Rotated archives, generally yes once retention needs are met. For a file a service is actively writing, truncate it instead of deleting, or the space will not be freed until the process closes the file.

What is the difference between copytruncate and a post-rotate signal? Copytruncate copies the log then empties the original, so the service needs no change but a few lines may be lost. A signal makes the service reopen its log file, which is cleaner but depends on application support.

How long should I keep logs? It depends on how far back you investigate, disk capacity, privacy and any legal or contractual obligations. Confirm requirements with appropriate advisers if you are in a regulated environment.

Does the journal need logrotate? No. The journal manages its own size through its configuration and vacuum commands.

How do I test a logrotate change safely? Run logrotate in debug mode against the config to see what it would do, then force a single rotation and verify the service continues writing.

Next Step

List every directory under /var/log and your application paths, and check each has a rotation rule or journal cap. Add one for any that do not, then run logrotate in debug mode to confirm it behaves as intended.