2

I have a 115GB partition on my hard disk (output of cgdisk /dev/sda is below):

Part. #     Size        Partition Type            Partition Name
----------------------------------------------------------------
            1007.0 KiB  free space
   1        499.0 MiB   Windows RE                Basi
   2        100.0 MiB   EFI System                EFI 
   3        16.0 MiB    Microsoft reserved        Micr
   5        43.9 GiB    Linux filesystem          ubuntu-root
   6        43.9 GiB    Linux filesystem          ubuntu-home
   4        114.9 GiB   Linux filesystem          data         <--- this partition
   7        29.5 GiB    Linux filesystem

And I have mounted that partition on /data in my /etc/fstab:

UUID=<drive-uuid>  /data  ext4  defaults  0  0

When I do df -h /data, I have the following output:

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda4       113G   96G   11G  90% /data

And when I use du like this: du /data -h --max-depth=1 | sort -hr, I see this:

51G    /data
40G    /data/virtual-box
4.4G   /data/temp
4.1G   /data/manjaro-minikube
1.9G   /data/.nuget
764M   /data/OneDrive
62M    /data/manjaro-lxd
40K    /data/.minikube
16K    /data/lost+found

which, if I'm not mistaken, is showing that /data is taking 51G and then I have the virtual-box, temp, manjaro-minikube, and .nuget directories taking space (the other ones don't take significant space)

If I do a long listing of my directory (ls -alh /data):

total 68K
drwxr-xr-x  10 farzad farzad 4.0K Aug 13 21:47 .
drwxr-xr-x  19 root   root   4.0K Jul 13 10:32 ..
drwx------   2 farzad farzad  16K Mar 22 18:22 lost+found
drwx--x--x  15 root   root   4.0K Aug 20 17:47 manjaro-lxd
drwxr-xr-x   3 farzad farzad 4.0K Jul  3 18:16 manjaro-minikube
drwxrwxr-x   9 farzad farzad 4.0K Jul 30 17:38 .minikube
drwxr-xr-x 202 farzad farzad  16K Aug 17 10:00 .nuget
drwxr-xr-x   3 farzad farzad 4.0K Aug 13 21:47 OneDrive
drwxrwxr-x  16 farzad farzad 4.0K Jun  3 21:45 temp
drwxr-xr-x   6 farzad farzad 4.0K Aug 20 20:21 virtual-box

I don't see any file or something contributing to that 51G reported for /data, so I expect my drive to have almost 65G empty space, but for some reason, the /data parent directory is taking 51G by itself!

I tried researching the Internet, but I couldn't find anything. Can someone please let me know what's happening?

UPDATE

As suggested in the answers, I ran lsof /data | grep deleted (as root), but I don't have any result, although I see a warning, which I'm not sure is relevant:

lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs
      Output information may be incomplete.
Farzad
  • 155

3 Answers3

2

Your /data is taking 51G. This includes all subdirectories and files within it. If you add sizes reported by du for subdirectories (and consider rounding issues), you will get about 51G. If there were regular files directly in /data, they would contribute to the value reported for /data as well.

So du does not report high usage. It's df what reports high usage: 96G is in use in the filesystem.

Since /data is a mountpoint, you may expect the two values to be the same. But the two tools work differently: du traverses directories and adds sizes of encountered objects, df queries the filesystem about its knowledge of its own state.

Such big discrepancy may be because:

  • du was not able to get to (or get info about) all objects. Was there any permission denied error?
  • There is inconsistency in the filesystem; fsck.ext4 aka e2fsck may help.

  • (most likely) There is at least one file that was deleted (all directory entries pointing to the respective inode was removed, the file doesn't appear in any directory listing so du cannot know about it) but it's still in use by some process (so the filesystem keeps the data and takes it into account while reporting to df). See this answer, this question.

    The following command should find such files and processes using them:

    lsof /data | grep deleted

    Example output:

    some_daemon  …  …  …  …  …  …  …  /data/temp/huge_file (deleted)

    This means the filesystem will truly delete the huge_file only after some_daemon stops using it. Note in general the process can still append to the file or truncate it, so the file may grow or shrink. This will affect what df says, but not du.

Kamil Maciorowski
  • 74,068
  • Thank you @Kamil for your reply, I tried your command and updated the question. Do you have any idea how I can track down what's happening? – Farzad Aug 22 '19 at 01:28
  • @Farzad You should check the filesystem. Run e2fsck -f /dev/sda4 after unmounting; or schedule the check with tune2fs (e.g. this answer) so it happens during reboot. After you confirm the filesystem is healthy and in a consistent state, run df and du again. Please report if the discrepancy is still there, edit the question if needed. – Kamil Maciorowski Aug 22 '19 at 11:41
  • Thank you @Kamil, I modified my /etc/fstab to add fsck to the file system, and I'm somehow "shocked" by how much more available space it reports now :). I didn't want to take credit for what you suggested, so I didn't "answer" my question. Please let me know what we can do for the answer. – Farzad Aug 23 '19 at 19:22
  • @Farzad Frankly I did not notice there is 0 in the sixth field in the fstab entry. If your answer is going to be like "0 is bad, it makes fsck not run; see man 5 fstab and change accordingly (to 2 in this case); after I did this…" (post new output of df, du) – then yes!, add your own answer, because this is new and useful, brings attention to fstab where the primary culprit was; it's not an ad-hoc fix, it will prevent similar situations in the future. In your answer you may admit you misinterpreted du (I think you did). Accept your answer and I will gladly upvote it. – Kamil Maciorowski Aug 23 '19 at 20:03
1

So, it turns out that there were a few issues here (thank you @Kamil for helping to find them):

Although this was my initial intuition that the amount reported by du for /data is the sum of all its child directories (with some rounding), I guess I was trying to justify the discrepancies between df and du and thought that I should sum up du's output for /data and all its child directories to get the same result as df.

The other issue, which is the main one causing the discrepancies, was because of how I had configured my /etc/fstab:

UUID=<uuid>  /data  ext4  defaults  0  0
                                       ^
                                     ISSUE

When I created my /etc/fstab, I thought that I wouldn't need to prolong my startup by enabling file system checks (fsck) on my mount, hence 0 for the sixth field, but as it turned out, is was causing inodes not be cleaned up, hence causing the BIG difference between df and du.

So, looking at man 5 fstab, we can see that, in order to enable checks, the root file system should have value 1, and other file systems should have value 2, so I changed the line to be:

UUID=<uuid>  /data  ext4  defaults  0  2

And after restart, a lot of issues were reported by fsck, I chose to fix them, and now the output of du /data -h --max-depth=1 | sort -hr:

28G    /data
16G    /data/virtual-box
4.5G   /data/temp
4.1G   /data/manjaro-minikube
1.9G   /data/.nuget
824M   /data/OneDrive
64M    /data/manjaro-lxd
40K    /data/.minikube
16K    /data/lost+found

And output of df /data -h:

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda4       113G   28G   80G  26% /data

It's worth noting that, compared to my original question, I have removed a few files (hence 28G usage instead of 51G), but the good thing is that both du and df report the same value :)

Farzad
  • 155
0

Your command is sorting the results from highest to lowest. The highest value is the total space used by /data, not the space it has used by itself.

ajgringo619
  • 306