0

I am in a pickle.

I have a server (physicals machine) that has a link between a directory and a partition. I don't see a symbolic link.

The problem is that the root partition shows 0% space. Even if I delete data from other directories it doesn't effect the root partition correctly.

I deleted 200MB and it showed 2MB of free space.
There is nothing in fastab.

The link is between /path/backups/ftp to /home/ftp/public_html

This is my df -h

devtmpfs                  63G     0   63G   0% /dev
tmpfs                     63G     0   63G   0% /dev/shm
tmpfs                     63G  4.1G   59G   7% /run
tmpfs                     63G     0   63G   0% /sys/fs/cgroup
/dev/mapper/centos-root   45G   45G  208K 100% /
/dev/sda1               1014M  194M  821M  20% /boot
/dev/sdb                 4.3T   89M  4.1T   1% /path/vms
/dev/sdc                  11T  7.5T  2.9T  73% /path/backups
tmpfs                     13G     0   13G   0% /run/user/0

this is fastab file:

#
# /etc/fstab
# Created by anaconda on Tue Nov 17 22:49:51 2020
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/centos-root / xfs rw,seclabel,relatime,attr2,inode64,noquota 0 0
UUID=dca34673-80bb-4c10-a1ca-cd76167ebcf4 /boot                   xfs     defaults        0 0
/dev/mapper/centos-swap swap                    swap    defaults        0 0
/dev/sdc        /path/backups    ext4    defaults    0 2
/dev/sdb        /path/vms        ext4    defaults    0 2

This is my lsblk:

NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda               8:0    0   50G  0 disk
├─sda1            8:1    0    1G  0 part /boot
└─sda2            8:2    0   49G  0 part
  ├─centos-root 253:0    0   45G  0 lvm  /
  └─centos-swap 253:1    0    4G  0 lvm  [SWAP]
sdb               8:16   0  4.3T  0 disk /path/vms
sdc               8:32   0 10.9T  0 disk /path/backups

I thought maybe it can be a hard link, but I can't find any proof of that, and also it should not be possible to hard link 2 directories.

Any ideas?

Edit:

The mtab contenct:

sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
devtmpfs /dev devtmpfs rw,nosuid,size=65906808k,nr_inodes=16476702,mode=755 0 0
securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,nodev,mode=755 0 0
tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0
cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0
pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0
cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_prio,net_cls 0 0
cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0
cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpuacct,cpu 0 0
cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0
cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
configfs /sys/kernel/config configfs rw,relatime 0 0
/dev/mapper/centos-root / xfs rw,relatime,attr2,inode64,noquota 0 0
systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=25,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=12844 0 0
debugfs /sys/kernel/debug debugfs rw,relatime 0 0
mqueue /dev/mqueue mqueue rw,relatime 0 0
hugetlbfs /dev/hugepages hugetlbfs rw,relatime 0 0
/dev/sda1 /boot xfs rw,relatime,attr2,inode64,noquota 0 0
/dev/sdb /path/vms ext4 rw,relatime,data=ordered 0 0
/dev/sdc /path/backups ext4 rw,relatime,data=ordered 0 0
sunrpc /var/lib/path/rpc_pipefs rpc_pipefs rw,relatime 0 0
nfsd /proc/fs/nfsd nfsd rw,relatime 0 0
/dev/sdc /home/ftp/public_html ext4 rw,relatime,data=ordered 0 0
binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0
tmpfs /run/user/0 tmpfs rw,nosuid,nodev,relatime,size=13183736k,mode=700 0 0

I have /dev/sdc mounted twice. It makes stuff work funny.

I see this in this lines:

/dev/sdc /path/backups ext4 rw,relatime,data=ordered 0 0
/dev/sdc /home/ftp/public_html ext4 rw,relatime,data=ordered 0 0

The storage problem and also I see that it shows 97% use of inodes, even though there is not so many files actually there.

matisa
  • 131
  • 2
    Directories can't be hardlinked, period. Also show us mount, there could be bind mounts that actually really allow share storage without links; however, I suspect you just faced with ordinary file system overflow, nothing special. This also may be caused by lost inodes; perform a file system check (this is root fs, you will need to reboot the machine; assure it has enoungh space to return). This may be caused by deleted files still opened (so there is no name, but there is still an inode), to release them, kill processes that are keeping them. – Nikita Kipriyanov Oct 13 '21 at 10:08
  • By the way, there could not be any hard link between anything in /path/backups and /, because those are different file systems. Hard linking is only possible within a single file system. – Nikita Kipriyanov Oct 13 '21 at 10:20
  • Yes! Thank you. There is a mount to 2 different directories. I am not sure how to proceed without losing data. – matisa Oct 13 '21 at 10:23
  • Don't panic. This is exactly how a bind mount looks. This can be frustrating, but first check findmnt (check https://unix.stackexchange.com/questions/295525/how-is-findmnt-able-to-list-bind-mounts/346460). Also, this has nothing to do with your root file system overflow. Investigate that with du -sh * descend, as the answer below suggests. – Nikita Kipriyanov Oct 13 '21 at 11:36
  • Actually I fixed it. I unmounted the mount, deleted all the files that took space. AS I suspected there were files on the original path. Restarted the machine which freed all the old inodes and remounted the path (As we still need the mount there).

    Thanks a lot. Please add this as an answer so I can except it.

     – matisa Oct 13 '21 at 13:12

1 Answers1

-1

why don't you use "du" instead of "df". Basically, df reads the superblock only. du reads each object and sums them up. so you may use "du -sh" to find the size of each directory with all its contents inside. Also you may have a look at du vs. df difference

Zareh Kasparian
  • 765
  • Thanks. I used du... it doesn't help me to know the directory size, or the object size. That is not the problem. The problem is that there is some weird link between 2 directories that effect the whole partition in a way that is very problematic. – matisa Oct 13 '21 at 09:27
  • whatever it is, du command can report it back to you. you just need to traverse to the directories step by step to find the desired file. – Zareh Kasparian Oct 13 '21 at 09:33
  • I have a very good understanding of the du command. I did use it to locate data I can delete outside of the linked directories. I deleted over 300 MB now. It doesn't help. What I need is to locate the type of link that links this paths and fix this. – matisa Oct 13 '21 at 10:03
  • 1
    Why are you so sure there are some mysterious "links"? And why do you think this will ever help you? Don't waste time to that, search for large directories with du -sh *, descending from the root down to largest directories, often /home and /var. – Nikita Kipriyanov Oct 13 '21 at 10:05
  • Look at the correspondence on the original question. You will see what happened. – matisa Oct 13 '21 at 13:13