[Bug 1876034] Re: apt update crashes with "KeyError: 'suite'" in cnf-update-db if index compression is enabled

[John Freeman]

The fix works for me. It's relatively easy to check for yourself with
Docker.

To reproduce the issue (version 20.04.4):
docker run --rm ubuntu:focal bash -c 'export DEBIAN_FRONTEND=noninteractive && apt-get update && apt-get -y install --no-install-recommends command-not-found && apt-get update ; apt-cache policy command-not-found'

To test the fix (version 20.04.5):
docker run --rm ubuntu:focal bash -c 'echo "deb http://archive.ubuntu.com/ubuntu/ focal-proposed restricted main multiverse universe" > /etc/apt/sources.list.d/ubuntu-focal-proposed.list && export DEBIAN_FRONTEND=noninteractive && apt-get update && apt-get -y install --no-install-recommends command-not-found && apt-get update ; apt-cache policy command-not-found'

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to command-not-found in Ubuntu.
https://bugs.launchpad.net/bugs/1876034

Title:
  apt update crashes with "KeyError: 'suite'" in cnf-update-db if index
  compression is enabled

Status in command-not-found package in Ubuntu:
  Fix Released
Status in command-not-found source package in Focal:
  Fix Committed
Status in command-not-found source package in Hirsute:
  Fix Committed
Status in command-not-found source package in Impish:
  Fix Released

Bug description:
  [Impact]
  On all Ubuntu releases since at least 19.10, enabling compressed APT package list storage with command-not-found installed leads to this crash in the cnf-update-db hook called by apt update:

  Traceback (most recent call last):
    File "/usr/lib/cnf-update-db", line 26, in <module>
      col.create(db)
    File "/usr/lib/python3/dist-packages/CommandNotFound/db/creator.py", line 93, in create
      self._fill_commands(con)
    File "/usr/lib/python3/dist-packages/CommandNotFound/db/creator.py", line 127, in _fill_commands
      self._parse_single_commands_file(con, fp)
    File "/usr/lib/python3/dist-packages/CommandNotFound/db/creator.py", line 165, in _parse_single_commands_file
      suite=tagf.section["suite"]
  KeyError: 'suite'

  (To enable compressed APT list storage, add Acquire::GzipIndexes
  "true"; to an apt.conf fragment.)

  This appears to be caused by apt downloading an lz4-compressed command
  list for command-not-found and cnf-update-db passing the
  /var/lib/apt/lists/*Commands-*.lz4 files to db/creator.py, which does
  not know about the compression and attempts to parse them as plain
  text.

  
  [Test plan]
  Enable the option, purge old lists, and update to populate lists directory with compressed files; this will run cnf-update-db for you.

  [Where problems could occur]
  Replacing the open() call with subprocess.Popen for apt-helper seems like a minimal change. If apt-helper has problems, we check the return code and fail, keeping the old db in place.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/command-not-found/+bug/1876034/+subscriptions