2

On a Xen 4.8 hypervisor, running on Debian Stretch, I am trying to remove the logical volume of a domU that is no longer used.

I stopped the domU and successfully removed the logical volume for the swap file.

However, when trying to remove the logical volume of the main disk, I get the error:

Logical volume xen3-vg/vmXX-disk in use.

As I need the disk space on the hypervisor for other domUs, I successfully resized the logical volume to 4 MB.

To make it obvious which logical volume needs to be deleted, I renamed the logical volume to "xen3-vg/deleteme".

Nevertheless:

> lvremove -vf /dev/xen3-vg/deleteme

Returns:

Logical volume xen3-vg/deleteme in use.

On searching, I have tried all the following, but to no avail.

> lvdisplay

--- Logical volume ---
LV Path                /dev/xen3-vg/deleteme
LV Name                deleteme
VG Name                xen3-vg
LV UUID                iL2tB4-4tjc-0dlj-ibBX-nQup-oZsX-g81XXO
LV Write Access        read/write
LV Creation host, time xen3, 2017-01-04 07:16:26 +0100
LV Status              available
# open                 1
LV Size                4.00 MiB
Current LE             1
Segments               1
Allocation             inherit
Read ahead sectors     auto
- currently set to     256
Block device           254:7

> dmsetup info -c | grep deleteme

xen3--vg-deleteme   254   7 L--w    1    1      3 LVM-aAW4aSeLjqJPPWlF1s1WxAgzeXAjWmXiiL2tB44tjc0dljibBXnQupoZsXg81XXO

> lvchange -an -v /dev/xen3-vg/deleteme

Logical volume xen3-vg/deleteme in use.

> lvremove -vf /dev/xen3-vg/deleteme

Logical volume xen3-vg/deleteme in use.

> umount /dev/xen3-vg/deleteme

umount: /dev/xen3-vg/deleteme: not mounted

> lsof | grep "254,7"

[no output]

> lsof | grep deleteme

[no output]

> fuser /dev/xen3-vg/deleteme

[no output]

If someone could offer advice on how to remove xen3-vg/deleteme I would be very happy.

Rebooting the hypervisor is the final option (to be executed out of office hours), but I would prefer not rebooting.

Thank you in advance.

Thank you, Brandon for your suggestion.

Following the instructions at http://naveen161089.blogspot.com/2014/03/forcefully-remove-lvm.html I tried the following:

> dmsetup ls

[..]
xen3--vg-deleteme   (254:7)
[..]

> dmsetup info -c xen3--vg-deleteme

Name              Maj Min Stat Open Targ Event  UUID                                                                
xen3--vg-deleteme 254   7 L--w    1    1      3 LVM-aAW4aSeLjqJPPWlF1s1WxAgzeXAjWmXiiL2tB44tjc0dljibBXnQupoZsXg81XXO

> dmsetup remove xen3--vg-deleteme

device-mapper: remove ioctl on xen3--vg-deleteme failed: Device or resource busy
Command failed

> lvremove -f /dev/xen3-vg/deleteme

Logical volume xen3-vg/deleteme in use.

I already followed the instructions at: Can't remove open logical volume as stated in my original post.

Neither approach solved the problem.

Does anyone have any further suggestions?

MacMartin
  • 378
Jonathan Maron
  • 121
  • These might be helpful: http://naveen161089.blogspot.com/2014/03/forcefully-remove-lvm.html (more to the point) or this one https://serverfault.com/questions/266697/cant-remove-open-logical-volume (more options to try) – Brandon Xavier Jul 25 '17 at 05:51
  • Brandon, thanks for your assistance. I updated the original post with the results of your suggestions. – Jonathan Maron Jul 25 '17 at 08:21
  • I'm out of suggestions. It's interesting that you could both rename and resize the volume, but not remove it. Please do keep us posted if you find a solution – Brandon Xavier Jul 26 '17 at 04:26
  • Indeed, that puzzles me too. When I have the solution, I will update my original post. – Jonathan Maron Jul 26 '17 at 04:37
  • A running process is has an "hold" on the device. To find the process use dmsetup and lsof. In my case if was libvirt-qemu – MrCalvin May 24 '19 at 12:29

5 Answers5

1

I think the main mistake was to execute lsof/fuser on the device-file "/dev/xen3-vg/deleteme" and not on the mount-point (e.g.: "/mnt/myDataMount").

(Thats misleading and I also tripped into it: but it's the same, as if you wanted to run lsof on "/dev/sda3" - that does not work. But you can run lsof on /mount/somedrive)

The steps to do in short:

  1. search the "LV path" in the mounts with mount |grep deleteme # where "deleteme" stands for the lv-name. this exemplary outputs something like "/my/mountpoint"
  2. lsof /my/mountpoint to find out which process is using it
  3. kill the process (kill -9 naggingprocess) or stop the service (service something stop)
  4. check (2.) again
  5. umount /dev/xen3-vg/deleteme
  6. lvremove /dev/xen3-vg/deleteme (or lvremove -f)
MacMartin
  • 378
1

I had managed to forget I had associated a loop device with the LV:

lrwxrwxrwx 1 root root       7 Feb  1 14:17 /dev/mapper/vg--main-test -> ../dm-8
#losetup -a
<snip>
/dev/loop7: [0005]:3088 (/dev/dm-8)

by doing:

#losetup -d /dev/loop7

I could then remove the LV

root@virtual3:/dev# lvremove vg-main/test
Do you really want to remove and DISCARD active logical volume vg-main/test? [y/n]: y
  Logical volume "test" successfully removed

Stupid I know.. but it took me half a day to realise. lsof doesn't discover this problem.

David
  • 81
0

I had a similar problem, there was an rsync process stuck on using a snapshot, that was already lazily unmounted. So:

  • Because being unmounted, it did not show up on a mount command.
  • Because the umount was lazy, the process that had an open file could continue to have it open. However, it was not connected to the mount point (/mnt/snapshot) anymore, so I did not wind it with lsof | grep snapshot.
  • It seems that because the snapshot used up all allocated space, I/O was totally blocked and rsync was stuck waiting, thus keeping the file descriptor open, thus not letting the lazy unmount finish.

Of course I did not know this back then, but I happened to try to get the device ID using lvdisplay:

lvdisplay big/db_snapshot_20231004.0505 | grep Block.device
   Block device           253:3

And then search for the culprit with lsof:

lsof |grep 253,3
rsync     30508                             root  cwd       DIR              253,3       4096      45059 /postgresql/14/main
rsync     30508                             root    3r      REG              253,3 1073741824      46124 /postgresql/14/main/base/16466/735757683.4

(Those are not valid paths at the end, by the way.)

So after killing the rsync process, lvremove worked!

P.Péter
  • 639
0

I had the same problem and I have used this command and fixed it. Give it a try.

kpartx -dv /dev/mapper/VG-XXX_rootimg
Then lvremove -f /dev/mapper/VG-XXX_rootimg
Hannan
  • 29
  • 1
  • I appreciate your response, but this did not solve the issue for me. The logical volume still cannot be deleted. – Jonathan Maron Jan 02 '19 at 13:56
  • These did not work for me either. Nothing is using this disk. The LUKS encryption broke (boot repair corrupted it), and these logical drives are stuck there in limbo. Really annoying, had to reinstall, and still while booting from another disk I can't get rid of these logical disks. Can't even just deletion the partition! – CoderGuy123 Apr 15 '20 at 17:59
0

In the recent Christmas holidays, I had the opportunity of rebooting the Xen 4.8 hypervisor.

Following a successful reboot, I was able to delete the problematic logical volume without issue.

I can only speculate as to what the problem was, but I think that the hypervisor – despite apparently having deleted the domU – still had a reference to the logical volume in memory, hence, it could not be deleted.

Jonathan Maron
  • 121