[Bug 1851737] [NEW] PermissionError for lock file

[Newell Jensen]

Public bug reported:

MAAS uses twisted and we have ran into an issue where it seems to be
that the MAAS's networking monitoring services has a lock file to ensure
that only one processes updates the networking information. If the
processes gets killed, the lock file stays, pointing to the PID the
killed regiond process had.

Now what normally happens is that another process tries to acquire the
lock, sees that the lock points to a killed PID , and recreates the
lock.

This normally works, but what can happen is that the killed PID gets
recycled, so that the lock now points to a PID which the maas user isn't
allowed to kill. Now a PermissionError is raised, that the lock file
implementation doesn't handle this case, and the networking monitoring
service can never start.

Here is the traceback:

2019-10-31 08:12:13 twisted.scripts: [info] twistd 17.9.0 (/usr/bin/python3 3.6.8) starting up.
2019-10-31 08:12:13 twisted.scripts: [info] reactor class: twisted.internet.asyncioreactor.AsyncioSelectorReactor.
2019-10-31 08:12:14 -: [critical] Unhandled Error
        Traceback (most recent call last):
          File "/usr/lib/python3/dist-packages/twisted/internet/task.py", line 194, in start
            self()
          File "/usr/lib/python3/dist-packages/twisted/internet/task.py", line 239, in __call__
            d = defer.maybeDeferred(self.f, *self.a, **self.kw)
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 150, in maybeDeferred
            result = f(*args, **kw)
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1532, in unwindGenerator
            return _inlineCallbacks(None, gen, Deferred())
        --- <exception caught here> ---
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1386, in _inlineCallbacks
            result = g.send(result)
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/services.py", line 1001, in updateInterfaces
            responsible = self._assumeSoleResponsibility()
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/services.py", line 1077, in _assumeSoleResponsibility
            self._lock.acquire()
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/fs.py", line 467, in acquire
            if not self._fslock.lock():
          File "/usr/lib/python3/dist-packages/twisted/python/lockfile.py", line 185, in lock
            kill(int(pid), 0)
        builtins.PermissionError: [Errno 1] Operation not permitted

** Affects: twisted (Ubuntu)
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to twisted in Ubuntu.
https://bugs.launchpad.net/bugs/1851737

Title:
  PermissionError for lock file

Status in twisted package in Ubuntu:
  New

Bug description:
  MAAS uses twisted and we have ran into an issue where it seems to be
  that the MAAS's networking monitoring services has a lock file to
  ensure that only one processes updates the networking information. If
  the processes gets killed, the lock file stays, pointing to the PID
  the killed regiond process had.

  Now what normally happens is that another process tries to acquire the
  lock, sees that the lock points to a killed PID , and recreates the
  lock.

  This normally works, but what can happen is that the killed PID gets
  recycled, so that the lock now points to a PID which the maas user
  isn't allowed to kill. Now a PermissionError is raised, that the lock
  file implementation doesn't handle this case, and the networking
  monitoring service can never start.

  Here is the traceback:

  2019-10-31 08:12:13 twisted.scripts: [info] twistd 17.9.0 (/usr/bin/python3 3.6.8) starting up.
  2019-10-31 08:12:13 twisted.scripts: [info] reactor class: twisted.internet.asyncioreactor.AsyncioSelectorReactor.
  2019-10-31 08:12:14 -: [critical] Unhandled Error
          Traceback (most recent call last):
            File "/usr/lib/python3/dist-packages/twisted/internet/task.py", line 194, in start
              self()
            File "/usr/lib/python3/dist-packages/twisted/internet/task.py", line 239, in __call__
              d = defer.maybeDeferred(self.f, *self.a, **self.kw)
            File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 150, in maybeDeferred
              result = f(*args, **kw)
            File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1532, in unwindGenerator
              return _inlineCallbacks(None, gen, Deferred())
          --- <exception caught here> ---
            File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1386, in _inlineCallbacks
              result = g.send(result)
            File "/usr/lib/python3/dist-packages/provisioningserver/utils/services.py", line 1001, in updateInterfaces
              responsible = self._assumeSoleResponsibility()
            File "/usr/lib/python3/dist-packages/provisioningserver/utils/services.py", line 1077, in _assumeSoleResponsibility
              self._lock.acquire()
            File "/usr/lib/python3/dist-packages/provisioningserver/utils/fs.py", line 467, in acquire
              if not self._fslock.lock():
            File "/usr/lib/python3/dist-packages/twisted/python/lockfile.py", line 185, in lock
              kill(int(pid), 0)
          builtins.PermissionError: [Errno 1] Operation not permitted

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/twisted/+bug/1851737/+subscriptions