[Bug 1530518] Re: /etc/cron.weekly/apt-xapian-index reports TypeError

[Barry Warsaw]

Almost positive what's happening is that Python shutdown process is
interacting badly with ServerProgress's __del__().

__del__() methods are not the best way to reclaim resources, especially
when you leave it to Python's shutdown process to implicitly free
objects, like what happens in update-apt-xapian-index.   Here's what
happens:

When Python shutdown, after it's run the atexit handlers, it then tries
to free up all objects that are still implicitly alive because of
reference counts.  It does this first by emptying the __dict__s of all
modules, which will decref any still referenced globals.  It actually
does this first by setting all keys in the module's globals to None.
Because Python can't know what's a safe order to do this, it will do it
randomly.

Notice a few things.  In update-a-x-i, indexer is a module global.  When
its .lock() method is called, it sets self.progress to a ServerProgress
instance and neither instance is explicitly freed, so they live until
Python shutdown.  Now, when Python gets around to clearing the module
dicts as described above, it will set u-a-x-i's indexer instance to
None, which will decref it to zero, and so freeing the indexer instance.

Because the ServerProgress instance is kept alive by the reference from
the indexer's self.progress attribute, once the indexer is freed, the
ServerProgress instance probably decrefs to zero and then *it* gets
freed.  This will call ServerProgress.__del__().  Line 306 is the
os.unlink() call and you'll notice that you get an exception that None
is not callable.  Why is that?

Well, because if the os module's dictionary got cleared *before* the
ServerProgress instance's __del__() got called, the os module is still
valid, and it still has a name bound in it called 'unlink' but that name
has been bound to None!  If you're lucky, the ServerProgress instance
gets decrefed away before os gets cleared, but as you've observed in
IRC, about every third time, you're unlucky, and os gets cleared first
(remember, the order is random).  In those cases... boom!

I don't like the design here, since relying on __del__ methods to free
resources isn't good Python style, especially when those resources can
be freed explicitly, but fixing that is probably more work than we want
to do for a package that is abandoned in Debian.  A bandaid might be
easy though: before the sys.exit() call in u-a-x-i, let's try to
explicitly `del indexer`.  That will cause all the cascading decrefs
before Python's shutdown machinery clears out the os module.  I'll
attach a diff to try.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to apt-xapian-index in Ubuntu.
https://bugs.launchpad.net/bugs/1530518

Title:
  /etc/cron.weekly/apt-xapian-index reports TypeError

Status in apt-xapian-index package in Ubuntu:
  Triaged
Status in apt-xapian-index source package in Xenial:
  Triaged

Bug description:
  Received the following email from cron:

  To: glitsj16 at gmail.com
  Subject: Anacron job 'cron.weekly' on lab16
  Date: Sat,  2 Jan 2016 05:53:34 +0100 (CET)

  /etc/cron.weekly/apt-xapian-index:
  Exception ignored in: <bound method ServerProgress.__del__ of <axi.indexer.ServerProgress object at 0x7f226fad1710>>
  Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/axi/indexer.py", line 306, in __del__
  TypeError: 'NoneType' object is not callable

  $ lsb_release -rd
  Description:	Ubuntu Xenial Xerus (development branch)
  Release:	16.04

  $ apt-cache policy apt-xapian-index
  apt-xapian-index:
    Installed: 0.47ubuntu6
    Candidate: 0.47ubuntu6
    Version table:
   *** 0.47ubuntu6 500
          500 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages
          500 http://archive.ubuntu.com/ubuntu xenial/main i386 Packages
          100 /var/lib/dpkg/status

  $ sudo /usr/sbin/update-apt-xapian-index
  The index /var/lib/apt-xapian-index is up to date
  Exception ignored in: <bound method ServerProgress.__del__ of <axi.indexer.ServerProgress object at 0x7f5b2a3736d8>>
  Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/axi/indexer.py", line 306, in __del__
  TypeError: 'NoneType' object is not callable

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/apt-xapian-index/+bug/1530518/+subscriptions