7

I am facing an issue with wrong fs space reported by df.

We are talking about bare metal Ubuntu 14.04 server, with the / partition managed by LVM.

Size reported by df is different from the size reported by fs super-block. Also calculating size from du -skh (on directories related to / partition) proves that df is showing incorrect results.

This is what reported by df:

Filesystem                  Size  Used Avail Use% Mounted on
/dev/mapper/rootvg-lv_root   20G   16G  2.8G  86% /

This is what super-block reports (3616906*4096/1024**2=14128 MB free):

Last mounted on:          /
Filesystem magic number:  0xEF53
Filesystem state:         clean
Filesystem OS type:       Linux
Inode count:              1290240
Block count:              5242880 << exactly 20 GB
Reserved block count:     227170
Free blocks:              3616906 << around 14 GB free
Free inodes:              1201391
First block:              0
Block size:               4096
Fragment size:            4096

I did a strace for df and found that it uses the statfs syscall to get information about a particular fs (962132*4096/1024**2=3758 MB free):

statfs("/", {f_type="EXT2_SUPER_MAGIC", f_bsize=4096, f_blocks=5129119, f_bfree=**962132**, f_bavail=730866, f_files=1290240, f_ffree=1091851, f_fsid={-456623966, 1549023591}, f_namelen=255, f_frsize=4096}) = 0

Probably the problem with statfs or with a way how it fetches fs information.

To solve it I tried to remount / partition (mount / -orw,remount), but it didn't help.

My question - how can I fix df information without rebooting or disturbing normal operation of this server?

P.S.

  • there are no fd on hold by process (lsof +L1 | grep deleted)
  • I aware that 883 MB is reserved for user with UID 0
  • This is not a mount point masking issue (when mounted fs 'hide' files in mount point directory)
Tom Lime
  • 201
  • What is the problem you want to fix? – reinierpost Aug 07 '16 at 11:26
  • Our monitoring is rely on df output. All I want - correct information to be reported by df. – Tom Lime Aug 07 '16 at 11:31
  • 4
    Possible duplicate of du vs. df difference – user9517 Aug 07 '16 at 14:01
  • Lain, thanks for pointing to masking (when you have files in directory which is used as mount point) issue, but this is not the case. It is seen by free blocks report in super-block (14 GB free), but df report only 2.8GB free. Probably the issue is more related to statfs and LVM. – Tom Lime Aug 07 '16 at 14:22
  • @TomLime Where do you get that super-block info? I can't find a command called super-block... You using dumpe2fs or something? – Ryan Babchishin Aug 07 '16 at 16:22
  • @RyanBabchishin, yes exactly I ran command dumpe2fs /dev/mapper/rootvg-lv_root. – Tom Lime Aug 07 '16 at 16:44
  • @TomLime I get 19,941MB from df and 82,860MB from the free blocks calculation on my system. Could your problem be due to sparse files? I don't really understand, it's just a stab in the dark. e.g. Is your filesystem actually storing 16G of data but not using all the blocks? – Ryan Babchishin Aug 07 '16 at 16:56
  • @RyanBabchishin, my system actually storing 6GB, but df wrongly reports 16GB. I tend to believe to SB information, as this is only the source of truth (how much blocks are free). – Tom Lime Aug 08 '16 at 06:55
  • Check for unlinked (inode deleted, but file descriptor still in use) files: lsof -a +L1 / | grep deleted – Karen B Aug 09 '16 at 07:57
  • 2
    @KarenB Unlinked files would cause du-vs-df inconsistency, but shouldn't cause df to be wrong. – Barmar Aug 09 '16 at 19:00
  • @KarenB, I already did this check, nothing significant was found. – Tom Lime Aug 10 '16 at 07:47
  • 1
    What does du -skh --apparent-size / would show? Can you allocate a 3gb file while only 2.8GB reported free by df on / with fallocate -l 3GB /test.file or will it say fallocate failed: No space left on device? – Anubioz Aug 10 '16 at 08:25
  • @Anubioz, du -skh / gave 8.4G, du -skh --apparent-size / gave 8.2G. I tried to allocate a file (3GB) and it gave me a message fallocate failed: No space left on device. I checked amount of free blocks with dumpe2fs and it had same value as before fallocate. – Tom Lime Aug 10 '16 at 18:47

1 Answers1

1

in sake of fellows who stuck with same issue:

i solved it by issuing unmount /dev/mapper/rootvg-lv_root. it gave me an error, but after df reported correctly.

looks like umount refreshed some internal structs.

hope it help someone.

Tom Lime
  • 201