Linux filesystem; difference in calculating size using df & du

Question

When I run df it shows the root device is full.

Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             9.9G  9.4G     0 100% /

I looked at the inode usage and there is pretty much space available for root device

Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1               640K    103K    538K   16% /

But, when I run the du command, it shows I have used only 2G out of 9.9G.

ip-XXX-XXX-XXX-XXX:/$ du -xh --max-depth=1
14M ./etc
4.0K ./mnt
96K ./tmp
3.5M ./bin
0   ./sys
964K ./boot
4.0K ./srv
0   ./dev
55M ./lib
25M ./root
1.1G ./usr
4.0K ./opt
846M ./var
4.3M ./sbin
23M ./home
16K ./lost+found
0   ./proc
2.0G .

It just driving me crazy and interesting too. This is big problem for us since the root disk / is full and some of the function in our site is failing.

Please help me resolve (also understand) this problem.

Thanks.

Answer 1

When files are deleted in *nix, they continue to live on disk (and take up disk space) for as long as a process has them open. It's fairly common to take advantage of this to "secure" temp files by creating them with a small size, deleting them, and then using the deleted file to store data without having to worry about other processes (easily) getting access to it, so the amount of space in deleted files can grow pretty large if, say, a temp database or multimedia editing session is being handled in this way. Another possibility for how you could have so much "lost" space would be if the system has been upgraded (multiple times) without rebooting or restarting programs, resulting in all your old .so libraries being held open by programs which were started prior to the upgrade and are still running.

df sees the space used by these files because it just looks at how much space is allocated on the device, but du doesn't see them because there aren't any corresponding directory entries.

"Hidden" used space like this can only be freed up when processes that have deleted files open close them. You can find these processes with the fuser command and terminate them (or, for many daemons, send a signal telling them to close and re-open any open files).

Answer 2

There have been times when if the disk gets full, it can then be confused till reboot/remount that the disk is still full, even when you've deleted a load of files.

Answer 3

On my side, I simply restarted syslogd to get the disk space back. I had 3GB missing ! My server was up and running for 250 days old.

Answer 4

There is a way to cleanup space without restarting application. Here are the details:

1. Let's say you have process foo running and creating a 2 GB file named abc.log. Now say this abc.log is deleted by someone else.

2. Get foo's pid (let's say 123). So /proc/123/fd will show list of file descriptors opened by foo. One with abc.log will show as deleted. Let's say fd of abs.log is 111. If you run less /proc/123/fd/111, it will still show you all that 2 GB of data.

3. Run echo " " > /proc/123/fd/111. This will overwrite the contents with an empty string. After this command if you try df it will show an additional 2 GB recovered by cleaning abc.log.

That's it. I tried this on CentOS and it works.