A full disk is rarely a quiet failure. Databases refuse writes, services fail to start, logins get strange, and package installs break halfway. The good news is that the investigation is mechanical, and the same four or five steps solve most cases.

Step One: Which Filesystem Is Full?

Run df -h. It lists each mounted filesystem with size, used and available space. Many servers have separate mounts for /, /var, /home or a data volume, and only one of them is usually the problem. Knowing which one narrows the search enormously.

Also run df -i. That shows inode usage, which is the other way a filesystem can be “full.”

Step Two: Where Is the Space Going?

Use du to walk down from the full mount. A practical pattern is to summarize one level at a time and sort:

du -xh --max-depth=1 /var | sort -h

The -x flag keeps du on a single filesystem so it does not wander into other mounts. Take the biggest directory, repeat one level deeper, and within a few commands you usually land on the culprit. If you have it installed, an interactive tool such as ncdu makes this faster, but du is always there.

Typical offenders:

• Application and web server logs that were never rotated.
• Database directories, including write-ahead logs and binary logs that accumulate when replication or archiving falls behind.
• Container images, stopped containers and build cache on Docker hosts.
• The systemd journal, if its size is not capped.
• Old kernels and package caches.
• Backups written to the same disk they are supposed to protect.
• Core dumps from a crashing service.

Step Three: When df and du Disagree

Sometimes df says the disk is nearly full but du cannot find the files. Two causes dominate.

Deleted but still open files. If a process holds a file open and you delete it, the name disappears but the space is not released until the process closes it or exits. This often happens when someone removes a large log file while the service is still writing to it. Running lsof +L1 lists open files with a link count of zero. Restarting the owning service, or truncating a file instead of deleting it in the future (for example with truncate -s 0), releases the space.

Hidden under a mount point. If data was written to a directory and a volume was later mounted over it, the original files are still consuming space on the underlying filesystem but are invisible. Unmounting temporarily, or bind-mounting the parent filesystem somewhere else, reveals them.

Step Four: The Inode Problem

If df -i shows inode use near the limit while byte usage is fine, the filesystem is full of tiny files. Session directories, mail queues, and cache directories with millions of entries are the classic sources. Find the directory with the most entries (counting files per directory with find and a short pipeline works), then clear it using the application’s own cleanup mechanism where one exists. The number of inodes is often fixed when the filesystem is created, so prevention matters more than cure.

Cleaning Up Safely

Source	Usually safe action	Caution
Rotated, compressed logs	Delete the oldest	Confirm retention requirements first
Active log file	Truncate, do not delete	Deleting leaves space held by the process
Systemd journal	Vacuum by size or age with journalctl	Keep enough for incident investigation
Package cache	Use the package manager’s clean command	None significant
Docker images and cache	Use Docker’s prune commands deliberately	Check what is in use before pruning volumes
Database logs	Fix the cause (replication, archiving)	Never delete files by hand inside a live data directory
Backups on same disk	Move to separate storage	Verify the copy before deleting the original

A Realistic Example

Suppose a small application server alerts at midnight because its root filesystem is at 100 percent. df -h shows the /var mount is full. Walking down with du points at a directory of application logs where one file is enormous: a debug logging flag was left on after a deployment. The administrator truncates the file rather than deleting it, which instantly frees space, then turns off debug logging and confirms rotation covers that path. A follow-up configuration change caps the journal and sets an alert at a lower threshold, so the next warning arrives while there is still time to react calmly.

Prevent the Repeat

Cleanup solves tonight. These habits solve next quarter:

• Alert on disk usage at a level that leaves hours or days of runway, and also alert on growth rate if your monitoring supports it.
• Alert on inode usage too; almost nobody does until it bites them.
• Make sure log rotation covers every application, and cap the journal size.
• Keep backups and databases on separate volumes from the operating system where practical.
• Reserve headroom. Running a filesystem close to full also degrades performance and complicates maintenance tasks.

Frequently Asked Questions

Why does df show a full disk but du totals are much smaller? Most often a deleted file is still held open by a running process, or files are hidden beneath a mount point. Use lsof +L1 to find open deleted files and check whether a volume was mounted over a non-empty directory.

Is it safe to delete files in /var/log? Rotated and compressed archives are usually safe to remove once you have confirmed retention needs. For a log a service is actively writing, truncate it rather than deleting, then fix the rotation.

What does “No space left on device” mean when df shows free space? Usually inode exhaustion, which df -i reveals. It can also be a quota limit or a reserved-block setting, depending on the filesystem.

Can I just grow the volume instead of cleaning up? Often yes, especially on cloud and virtual hosts, and sometimes that is the right call. But if something is growing without bound, a bigger disk only delays the same outage.

How much free space should I keep? There is no universal figure. Keep enough headroom that your alerts leave time to act, and more for volumes that hold databases or are subject to bursts. Check your filesystem and database documentation for specific guidance.

Next Step

Add two alerts today if you lack them: one for disk usage on each mount and one for inode usage. Then run du against your largest volume once, so you know what normal looks like before the next incident.