[Bug 1680997] [NEW] Container file system corruption on libvirtd restart

[Eugen Rieck]

Public bug reported:

A data corruption bug exists in the LXC driver for libvirt, that has
just cost me a MySQL server.

Steps to reproduce:
- (for visualization only) In virt-manager add a connection to local lxc:// 
- create an LXC container, that has a loop-mounted image file and start it
- (for visualization only) the container shows as running in virt-manager
- systemctl stop libvirtd ; sleep 2 ; sync ; systemctl start libvirtd
- (for visualization only) the container shows as shut off in virt-manager
- The container no longer responds to network requests, has no attachable console
- The loop mount does no longer show up on host-side "mount" output
      BUT: losetup -a reveals, that a loop device is still attached to the image file
      BUT: In reality this loop device is still mounted, processes in the container still access the file system
      BUT: There is no way to unmount or free it - losetup -d ends without an error but does nothing
- restart the container (virsh -c lxc:// start name-of-container or via virt-manager)
      THIS SHOULD NOT BE ALLOWED
- The image file is now twice mounted and corruption starts creeping in
- Depending on how long this state persists (in terms of IO), the damage can be significant

When finally discovering the problem, the only way to unstick the
container is a reboot. This is the final nail in the coffin: The hidden
instance syncs AFTER the new instance, effectivly pushing back the past.

This can be quite nasty, if a libvirt restart results from an unattended
upgrade.

I do understand, that libvirt/LXC is deprecated - this strikes me as a rather unsubtle way to push users to the newest incarnation, though.
In non-enterprisy environments (read SMB or NGO) virt-manager is often used as a "power user" tool, and those end users are unwilling if not unable to use different toolsets for containers and full-fledged VMs. And disabling unattended upgrades in such an environment is inviting trouble.

** Affects: libvirt (Ubuntu)
     Importance: Undecided
         Status: New

** Package changed: udev (Ubuntu) => libvirt (Ubuntu)

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to udev in Ubuntu.
https://bugs.launchpad.net/bugs/1680997

Title:
  Container file system corruption on libvirtd restart

Status in libvirt package in Ubuntu:
  New

Bug description:
  A data corruption bug exists in the LXC driver for libvirt, that has
  just cost me a MySQL server.

  Steps to reproduce:
  - (for visualization only) In virt-manager add a connection to local lxc:// 
  - create an LXC container, that has a loop-mounted image file and start it
  - (for visualization only) the container shows as running in virt-manager
  - systemctl stop libvirtd ; sleep 2 ; sync ; systemctl start libvirtd
  - (for visualization only) the container shows as shut off in virt-manager
  - The container no longer responds to network requests, has no attachable console
  - The loop mount does no longer show up on host-side "mount" output
        BUT: losetup -a reveals, that a loop device is still attached to the image file
        BUT: In reality this loop device is still mounted, processes in the container still access the file system
        BUT: There is no way to unmount or free it - losetup -d ends without an error but does nothing
  - restart the container (virsh -c lxc:// start name-of-container or via virt-manager)
        THIS SHOULD NOT BE ALLOWED
  - The image file is now twice mounted and corruption starts creeping in
  - Depending on how long this state persists (in terms of IO), the damage can be significant

  When finally discovering the problem, the only way to unstick the
  container is a reboot. This is the final nail in the coffin: The
  hidden instance syncs AFTER the new instance, effectivly pushing back
  the past.

  This can be quite nasty, if a libvirt restart results from an
  unattended upgrade.

  I do understand, that libvirt/LXC is deprecated - this strikes me as a rather unsubtle way to push users to the newest incarnation, though.
  In non-enterprisy environments (read SMB or NGO) virt-manager is often used as a "power user" tool, and those end users are unwilling if not unable to use different toolsets for containers and full-fledged VMs. And disabling unattended upgrades in such an environment is inviting trouble.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1680997/+subscriptions