Disk space and mounts

A full disk stops a service from writing anything. The error is always the same, "No space left on device", but the cause can be one of three things. df -h only reveals the first. A mount is how a disk, network share or memory-backed filesystem is attached to a folder, so the disk's contents appear under that path.

Walking through it

Start with two commands. df reports each filesystem. du reports how big each folder is.

df -h                     # free blocks (space) per filesystem
df -i                     # free inodes per filesystem
du -sh /var/* | sort -h   # which folders are big

Then check the three causes in this order:

1. Out of blocks. df -h shows 100%. Usually logs, core dumps or Docker images. Check with docker system df.
2. Out of inodes. Every file uses one inode, a fixed metadata record, and the filesystem allocates a fixed number of them when it is formatted. Ten million tiny session files can use them all while df -h still shows 60% free. df -i reveals it.
3. Deleted but still open. A process holds a 50 GB log open. Someone runs rm on it. The space is not freed, because the process still holds the file. df says full, but du cannot find the files. lsof +L1 lists open files with zero links. The fix is to restart the process, or to empty the file in place with : > app.log.

Mounts are configured in /etc/fstab. On a cloud VM, you typically attach a block volume, format it with mkfs.ext4 or mkfs.xfs, and mount it at /var/lib/<service> so the data survives replacing the machine. Two rules keep the machine bootable:

• Refer to the disk by UUID= in fstab, not by device name like /dev/nvme1n1, because device names can change between boots.
• Add nofail, so a missing volume does not stop the machine from booting.

Use findmnt to see what is mounted where, and lsblk to see the disks.

Common mistakes

• Deleting a live log file to free space. Truncate it with : > app.log, or use logrotate's copytruncate.
• Trusting df -h alone. It cannot show inode exhaustion or deleted-but-open files.
• Using device names in fstab. Use the UUID.

Check yourself

1. df -h shows 60% free, but writes fail with "no space left". What do you run next? df -i. The filesystem has run out of inodes.
2. You deleted a 50 GB log and df still shows the disk full. Why? A process still has the file open. The space comes back when the process closes it.

Key takeaways

• "No space left" can mean blocks, inodes, or deleted files still held open.
• Check df -h, then df -i, then lsof +L1.
• Mount volumes by UUID with nofail, and truncate live logs instead of deleting them.

Further learning

The filesystem layer underneath these numbers is covered in file systems and journaling. The next page covers the rules for running scripts that touch these disks: safe bash scripts.