[Bug 1834875] Re: cloud-init growpart race with udev

[Dimitri John Ledkov]

flock will not block inotify or udev events being emitted.

See
https://github.com/systemd/systemd/blob/master/src/udev/udevd.c#L322
https://github.com/systemd/systemd/blob/master/src/udev/udevd.c#L409

it will prevent udevd from running the rules against it. Thus
effectively the event will be fired and done, but nothing actually
executed for it.

" /* apply rules, create node, symlinks */"

Also, most importantly udevd also takes locks on devices, thus if one
cannot take a lock on it, it means it shouldn't be touched yet as udevd
is still processing it.

Yeah, i read code and ioctls both call, and I have no idea which one is
better, or what the difference is. If partx works, cool. But also, I
thought udev should be updating partition tables all by itself. I
somehow wonder if we even need partx call, if we properly flock the
device and trigger udev after everything is done.

Udev rules should be safe to be retriggered like that. But also, we
should retrigger the expected events which have been skipped. So does
growpart create partition? move it? delete/recreate one? i.e does ADD
happen? or like REMOVE & ADD? or maybe it's like just MOVE or CHANGE? Do
we have logs of the emitted events already?

Don't like flags, as then we'll have to supported forever =) maybe env
variable? or like simply change in focal and compare focal vs eoan?

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1834875

Title:
  cloud-init growpart race with udev

Status in cloud-init:
  Incomplete
Status in cloud-utils:
  New
Status in linux-azure package in Ubuntu:
  New
Status in systemd package in Ubuntu:
  Incomplete

Bug description:
  On Azure, it happens regularly (20-30%), that cloud-init's growpart
  module fails to extend the partition to full size.

  Such as in this example:

  ========================================

  2019-06-28 12:24:18,666 - util.py[DEBUG]: Running command ['growpart', '--dry-run', '/dev/sda', '1'] with allowed return codes [0] (shell=False, capture=True)
  2019-06-28 12:24:19,157 - util.py[DEBUG]: Running command ['growpart', '/dev/sda', '1'] with allowed return codes [0] (shell=False, capture=True)
  2019-06-28 12:24:19,726 - util.py[DEBUG]: resize_devices took 1.075 seconds
  2019-06-28 12:24:19,726 - handlers.py[DEBUG]: finish: init-network/config-growpart: FAIL: running config-growpart with frequency always
  2019-06-28 12:24:19,727 - util.py[WARNING]: Running module growpart (<module 'cloudinit.config.cc_growpart' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_growpart.py'>) failed
  2019-06-28 12:24:19,727 - util.py[DEBUG]: Running module growpart (<module 'cloudinit.config.cc_growpart' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_growpart.py'>) failed
  Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 812, in _run_modules
      freq=freq)
    File "/usr/lib/python3/dist-packages/cloudinit/cloud.py", line 54, in run
      return self._runners.run(name, functor, args, freq, clear_on_fail)
    File "/usr/lib/python3/dist-packages/cloudinit/helpers.py", line 187, in run
      results = functor(*args)
    File "/usr/lib/python3/dist-packages/cloudinit/config/cc_growpart.py", line 351, in handle
      func=resize_devices, args=(resizer, devices))
    File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2521, in log_time
      ret = func(*args, **kwargs)
    File "/usr/lib/python3/dist-packages/cloudinit/config/cc_growpart.py", line 298, in resize_devices
      (old, new) = resizer.resize(disk, ptnum, blockdev)
    File "/usr/lib/python3/dist-packages/cloudinit/config/cc_growpart.py", line 159, in resize
      return (before, get_size(partdev))
    File "/usr/lib/python3/dist-packages/cloudinit/config/cc_growpart.py", line 198, in get_size
      fd = os.open(filename, os.O_RDONLY)
  FileNotFoundError: [Errno 2] No such file or directory: '/dev/disk/by-partuuid/a5f2b49f-abd6-427f-bbc4-ba5559235cf3'

  ========================================

  @rcj suggested this is a race with udev. This seems to only happen on
  Cosmic and later.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-init/+bug/1834875/+subscriptions