It's a terrible mistake, I remove the disk file of lxc.
I know linux rm will not remove file when it's open by other processes. So I login ssh of that lxc, it's running, and the files in that disk is still there. That means the disk file is still at somewhere.
And I found this question. It told me to find the fd of process which it opening the file.
So I check the conf of lxc
arch: amd64
cores: 2
features: mount=nfs
hostname: frp
memory: 1024
mp0: /opt/nfs,mp=/opt
net0: name=eth0,bridge=vmbr0,gw=192.168.0.1,hwaddr=2A:CA:4F:D5:86:12,ip=192.168.0.205/24,type=veth
onboot: 1
ostype: debian
rootfs: local:205/vm-205-disk-0.raw,size=4G
swap: 0
I try to find process of that lxc:
# ps aux | grep lxc-start
root 27462 0.0 0.0 5024 2560 ? Ss Sep27 3:18 /usr/bin/lxc-start -F -n 205
and lsof
# lsof -p 27462
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
lxc-start 27462 root cwd DIR 8,4 4096 2 /
lxc-start 27462 root rtd DIR 8,4 4096 2 /
lxc-start 27462 root txt REG 8,4 1121592 1611382 /usr/bin/lxc-start
lxc-start 27462 root mem REG 8,4 1922136 1572889 /usr/lib/x86_64-linux-gnu/libc.so.6
lxc-start 27462 root mem REG 8,4 125312 1572875 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
lxc-start 27462 root mem REG 8,4 47288 1573323 /usr/lib/x86_64-linux-gnu/libcap.so.2.66
lxc-start 27462 root mem REG 8,4 80600 1577578 /usr/lib/x86_64-linux-gnu/libapparmor.so.1.8.4
lxc-start 27462 root mem REG 8,4 129312 1575206 /usr/lib/x86_64-linux-gnu/libseccomp.so.2.5.4
lxc-start 27462 root mem REG 8,4 1116504 1611390 /usr/lib/x86_64-linux-gnu/liblxc.so.1.7.0
lxc-start 27462 root mem REG 8,4 210968 1572885 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
lxc-start 27462 root 0r CHR 1,3 0t0 5 /dev/null
lxc-start 27462 root 1w CHR 1,3 0t0 5 /dev/null
lxc-start 27462 root 2w REG 0,23 0 1972 /run/pve/ct-205.stderr
lxc-start 27462 root 3u unix 0x0000000084bdb3e3 0t0 149082 @/var/lib/lxc/205/command type=STREAM (LISTEN)
lxc-start 27462 root 4u DIR 0,26 0 10771 /sys/fs/cgroup/lxc/205
lxc-start 27462 root 5u a_inode 0,14 0 1060 [signalfd]
lxc-start 27462 root 6u DIR 0,51 0 1 /dev/pts
lxc-start 27462 root 7u DIR 0,26 0 1 /sys/fs/cgroup
lxc-start 27462 root 8u a_inode 0,14 0 1060 bpf-prog
lxc-start 27462 root 9u CHR 5,2 0t0 2 /dev/pts/ptmx
lxc-start 27462 root 10r REG 7,0 0 1633 /var/lib/lxc/205/rootfs/.lxc_keep (deleted)
lxc-start 27462 root 11u a_inode 0,14 0 1060 [eventpoll:3,5,13,26]
lxc-start 27462 root 12u a_inode 0,14 0 1060 [eventpoll]
lxc-start 27462 root 13u unix 0x0000000067bfd7d3 0t0 150861 @/var/lib/lxc/205/command type=STREAM (CONNECTED)
lxc-start 27462 root 14u DIR 0,26 0 10838 /sys/fs/cgroup/lxc/205/ns
lxc-start 27462 root 15u DIR 0,26 0 10771 /sys/fs/cgroup/lxc/205
lxc-start 27462 root 16u a_inode 0,14 0 1060 [pidfd:27485]
lxc-start 27462 root 17r REG 0,4 0 4026532464 mnt
lxc-start 27462 root 18r REG 0,4 0 4026532467 pid
lxc-start 27462 root 19r REG 0,4 0 4026532465 uts
lxc-start 27462 root 20r REG 0,4 0 4026532466 ipc
lxc-start 27462 root 21r REG 0,4 0 4026532469 net
lxc-start 27462 root 22r REG 0,4 0 4026532468 cgroup
lxc-start 27462 root 23u CHR 136,1 0t0 4 /dev/pts/1
lxc-start 27462 root 24u CHR 5,2 0t0 2 /dev/pts/ptmx
lxc-start 27462 root 25u CHR 136,2 0t0 5 /dev/pts/2
lxc-start 27462 root 26u CHR 5,2 0t0 2 /dev/pts/ptmx
lxc-start 27462 root 27u CHR 136,0 0t0 3 /dev/pts/0
I can't see any fd to the disk file : vm-205-disk-0.raw, and /proc//fd does NOT help either
# ls -l /proc/27462/fd
total 0
lr-x------ 1 root root 64 Sep 27 15:08 0 -> /dev/null
l-wx------ 1 root root 64 Sep 27 15:08 1 -> /dev/null
lr-x------ 1 root root 64 Oct 7 08:55 10 -> '/var/lib/lxc/205/rootfs/.lxc_keep (deleted)'
lrwx------ 1 root root 64 Oct 7 08:55 11 -> 'anon_inode:[eventpoll]'
lrwx------ 1 root root 64 Oct 7 08:55 12 -> 'anon_inode:[eventpoll]'
lrwx------ 1 root root 64 Oct 7 08:55 13 -> 'socket:[150861]'
l--------- 1 root root 64 Oct 7 08:55 14 -> /sys/fs/cgroup/lxc/205/ns
l--------- 1 root root 64 Oct 7 08:55 15 -> /sys/fs/cgroup/lxc/205
lrwx------ 1 root root 64 Oct 7 08:55 16 -> 'anon_inode:[pidfd]'
lr-x------ 1 root root 64 Oct 7 08:55 17 -> 'mnt:[4026532464]'
lr-x------ 1 root root 64 Oct 7 08:55 18 -> 'pid:[4026532467]'
lr-x------ 1 root root 64 Oct 7 08:55 19 -> 'uts:[4026532465]'
l-wx------ 1 root root 64 Sep 27 15:08 2 -> /run/pve/ct-205.stderr
lr-x------ 1 root root 64 Oct 7 08:55 20 -> 'ipc:[4026532466]'
lr-x------ 1 root root 64 Oct 7 08:55 21 -> 'net:[4026532469]'
lr-x------ 1 root root 64 Oct 7 08:55 22 -> 'cgroup:[4026532468]'
lrwx------ 1 root root 64 Oct 7 08:55 23 -> /dev/pts/1
lrwx------ 1 root root 64 Oct 7 08:55 24 -> /dev/pts/ptmx
lrwx------ 1 root root 64 Oct 7 08:55 25 -> /dev/pts/2
lrwx------ 1 root root 64 Oct 7 08:55 26 -> /dev/pts/ptmx
lrwx------ 1 root root 64 Oct 7 08:55 27 -> /dev/pts/0
lrwx------ 1 root root 64 Sep 27 15:08 3 -> 'socket:[149082]'
l--------- 1 root root 64 Sep 27 15:08 4 -> /sys/fs/cgroup/lxc/205
lrwx------ 1 root root 64 Oct 7 08:55 5 -> 'anon_inode:[signalfd]'
l--------- 1 root root 64 Oct 7 08:55 6 -> /dev/pts
l--------- 1 root root 64 Oct 7 08:55 7 -> /sys/fs/cgroup
lrwx------ 1 root root 64 Oct 7 08:55 8 -> anon_inode:bpf-prog
lrwx------ 1 root root 64 Oct 7 08:55 9 -> /dev/pts/ptmx
I even check pstree
# pstree -p 27462
lxc-start(27462)───systemd(27485)─┬─agetty(27654)
├─agetty(27659)
├─agetty(28399)
├─cron(27637)
├─dbus-daemon(27638)
├─frp(30957)─┬─{frp}(30962)
│ ├─{frp}(30963)
│ ├─{frp}(30964)
│ ├─{frp}(30965)
│ ├─{frp}(239046)
│ └─{frp}(359586)
├─master(27820)─┬─pickup(843315)
│ └─qmgr(27822)
├─rpcbind(27634)
├─rsyslogd(27640)─┬─{rsyslogd}(27650)
│ ├─{rsyslogd}(27651)
│ └─{rsyslogd}(27652)
├─sshd(27669)───sshd(870247)───sshd(870386)───bash(870387)
└─systemd-journal(27580)
But none of them open vm-205-disk-0.raw.
So where can I find which process opening vm-205-disk-0.raw? Any suggestions will be appreciated.
=character between parameter and value, not:. This could be LXD or something deploying LXC within a VM. But not plain LXC. To be helped, you have to figure out what is the exact tool you're using (if the question is still in need of an answer...) – A.B Jan 08 '24 at 19:32