[Bug 2105503] Re: Uncontrolled memory usage

[Julian Andres Klode]

Setting a size limit works around this, because the go() method calls
the callback with None when the object does not fit into memory (and no
specific member was requested in the go call).

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to python-apt in Ubuntu.
https://bugs.launchpad.net/bugs/2105503

Title:
  Uncontrolled memory usage

Status in python-apt package in Ubuntu:
  Triaged

Bug description:
  Here is a simplified snippet of the `filelist` function in the
  `debfile.py` file. I got there while debugging OOM kill triggering on
  cnf-extractor, that makes use of that function.

  ```
  $ cat apt_file_0ad-data.py
  import apt_inst

  files = []
  debfile = apt_inst.DebFile("0ad-data_0.27.0-1_all.deb")
  debfile.data.go(lambda item, data: files.append(item.name))
  print(files)
  ```

  To run this, you need the latest `0ad-data` package downloaded next to
  the script: `apt download 0ad-data`.

  When running this "normally", I can see very high memory usage, sometimes leading to OOM kill (how I digged into that in the first place):
  python3 apt_file_0ad-data.py  & ; htop -p $!

  Interestingly, when wrapping that with some kind of constraints, the issue doesn't appear anymore at all, and the memory usage remains low:
  prlimit -v536870912 python3 apt_file_0ad-data.py  & ; htop -p $!
  With that previous command, even the time to finish is divided by two (~consistently 8s vs 15s without wrapping on my machine), but I didn't take precise measurements there, so might be noise, but still...

  And then one final interesting result:
  systemd-run --scope -p MemoryMax=500M --user python3 apt_file_0ad-data.py
  Sometimes this works just as with `prlimit`, with slightly higher memory usage, and sometimes it correctly triggers oom kill, after more than one minute. Again, the timing benchmark is inaccurate, but still...

  ProblemType: Bug
  DistroRelease: Ubuntu 25.04
  Package: python3-apt 2.9.9build1
  ProcVersionSignature: Ubuntu 6.12.0-12.12-generic 6.12.4
  Uname: Linux 6.12.0-12-generic x86_64
  NonfreeKernelModules: zfs
  ApportVersion: 2.32.0-0ubuntu3
  Architecture: amd64
  CasperMD5CheckResult: unknown
  CurrentDesktop: sway
  Date: Mon Mar 31 15:58:25 2025
  InstallationDate: Installed on 2023-08-02 (607 days ago)
  InstallationMedia: Ubuntu 23.04 "Lunar Lobster" - Release amd64 (20230418)
  ProcEnviron:
   LANG=fr_FR.UTF-8
   PATH=(custom, no user)
   SHELL=/bin/zsh
   TERM=alacritty
   XDG_RUNTIME_DIR=<set>
  SourcePackage: python-apt
  UpgradeStatus: Upgraded to plucky on 2024-06-12 (292 days ago)

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/python-apt/+bug/2105503/+subscriptions