[Bug 2105503] Re: Uncontrolled memory usage

[Launchpad Bug Tracker]

** Merge proposal unlinked:
   https://code.launchpad.net/~hyask/command-not-found-extractor/+git/command-not-found-extractor/+merge/492711

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to python-apt in Ubuntu.
https://bugs.launchpad.net/bugs/2105503

Title:
  Uncontrolled memory usage

Status in python-apt package in Ubuntu:
  Triaged

Bug description:
  Here is a simplified snippet of the `filelist` function in the
  `debfile.py` file. I got there while debugging OOM kill triggering on
  cnf-extractor, that makes use of that function.

  ```
  $ cat apt_file_0ad-data.py
  import apt_inst

  files = []
  debfile = apt_inst.DebFile("0ad-data_0.27.0-1_all.deb")
  debfile.data.go(lambda item, data: files.append(item.name))
  print(files)
  ```

  To run this, you need the latest `0ad-data` package downloaded next to
  the script: `apt download 0ad-data`.

  When running this "normally", I can see very high memory usage, sometimes leading to OOM kill (how I digged into that in the first place):
  python3 apt_file_0ad-data.py  & ; htop -p $!

  Interestingly, when wrapping that with some kind of constraints, the issue doesn't appear anymore at all, and the memory usage remains low:
  prlimit -v536870912 python3 apt_file_0ad-data.py  & ; htop -p $!
  With that previous command, even the time to finish is divided by two (~consistently 8s vs 15s without wrapping on my machine), but I didn't take precise measurements there, so might be noise, but still...

  And then one final interesting result:
  systemd-run --scope -p MemoryMax=500M --user python3 apt_file_0ad-data.py
  Sometimes this works just as with `prlimit`, with slightly higher memory usage, and sometimes it correctly triggers oom kill, after more than one minute. Again, the timing benchmark is inaccurate, but still...

  ProblemType: Bug
  DistroRelease: Ubuntu 25.04
  Package: python3-apt 2.9.9build1
  ProcVersionSignature: Ubuntu 6.12.0-12.12-generic 6.12.4
  Uname: Linux 6.12.0-12-generic x86_64
  NonfreeKernelModules: zfs
  ApportVersion: 2.32.0-0ubuntu3
  Architecture: amd64
  CasperMD5CheckResult: unknown
  CurrentDesktop: sway
  Date: Mon Mar 31 15:58:25 2025
  InstallationDate: Installed on 2023-08-02 (607 days ago)
  InstallationMedia: Ubuntu 23.04 "Lunar Lobster" - Release amd64 (20230418)
  ProcEnviron:
   LANG=fr_FR.UTF-8
   PATH=(custom, no user)
   SHELL=/bin/zsh
   TERM=alacritty
   XDG_RUNTIME_DIR=<set>
  SourcePackage: python-apt
  UpgradeStatus: Upgraded to plucky on 2024-06-12 (292 days ago)

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/python-apt/+bug/2105503/+subscriptions