[Bug 1998265] Re: OverFlow error when using cnf-extractor

[Julian Andres Klode]

** Description changed:

+ [Impact]
+ Large files can't be downloaded as we use a 32-bit integer in AcquireFile constructor, this breaks e.g. command-not-found generator:
+ 
  I'm trying to update the command-not-found indexes (bin/cnf-extract.py
  $MIRROR $suite $component $arch) on the production cnf extractor system
  (running bionic) and have encountered Tracebacks when generating them
  for Jammy and Lunar. This is causing the command not found indexes to be
  out of date for all(?) releases of Ubuntu.
  
  The lunar failure:
  
  Get:1 dotnet-sdk-6.0_6.0.111-0ubuntu3_amd64.deb [79.1 MB]
  Fetched 79.1 MB in 0s (0 B/s)
  Traceback (most recent call last):64... 7%
    File "./bin/cnf-extract.py", line 54, in <module>
      mirror_root, suite, component, arch)
    File "/srv/cnf-extractor/extractor/extractor.py", line 174, in command_not_found_extract
      debpath = pkg.candidate.fetch_binary(destdir=tmpdir)
    File "/usr/lib/python3/dist-packages/apt/package.py", line 883, in fetch_binary
      self.size, base, destfile=destfile)
  OverflowError: signed integer is greater than maximum
  
  I'll get more context for the Jammy failure but it wouldn't surprise me
  if it was the same package.
+ 
+ [Test plan]
+ I have included a test for the AcquireFile constructor to accept a large file size, and then a test that ensures it can read the file size.
+ 
+ Run the autopkgtest suite.
+ 
+ [Where problems could occur]
+ It could fail in other places now.
+ 
+ [Other info]
+ Also including some auxiliary changes as we move back to proper upstream releases to enable testing, mostly around
+ 
+ d/gbp.conf, .gitlab-ci.yml - being updated to point at right places
+ d/t/control - adding a missing test dependency
+ data/ - updated mirror lists
+ 
+ The mirror lists are standard updates; the debian/tests/control change
+ is needed to enable testing on more minimal autopkgtest images (our images
+ accidentally include binutils); the other changes have zero impact on the Ubuntu infra or end user side as they only run during development.

** Changed in: python-apt (Ubuntu Focal)
       Status: New => In Progress

** Description changed:

  [Impact]
  Large files can't be downloaded as we use a 32-bit integer in AcquireFile constructor, this breaks e.g. command-not-found generator:
  
  I'm trying to update the command-not-found indexes (bin/cnf-extract.py
  $MIRROR $suite $component $arch) on the production cnf extractor system
  (running bionic) and have encountered Tracebacks when generating them
  for Jammy and Lunar. This is causing the command not found indexes to be
  out of date for all(?) releases of Ubuntu.
  
  The lunar failure:
  
  Get:1 dotnet-sdk-6.0_6.0.111-0ubuntu3_amd64.deb [79.1 MB]
  Fetched 79.1 MB in 0s (0 B/s)
  Traceback (most recent call last):64... 7%
    File "./bin/cnf-extract.py", line 54, in <module>
      mirror_root, suite, component, arch)
    File "/srv/cnf-extractor/extractor/extractor.py", line 174, in command_not_found_extract
      debpath = pkg.candidate.fetch_binary(destdir=tmpdir)
    File "/usr/lib/python3/dist-packages/apt/package.py", line 883, in fetch_binary
      self.size, base, destfile=destfile)
  OverflowError: signed integer is greater than maximum
  
  I'll get more context for the Jammy failure but it wouldn't surprise me
  if it was the same package.
  
  [Test plan]
  I have included a test for the AcquireFile constructor to accept a large file size, and then a test that ensures it can read the file size.
  
  Run the autopkgtest suite.
  
  [Where problems could occur]
  It could fail in other places now.
  
+ For kinetic, the package was copied in from lunar without changes. This
+ upload is actually built in kinetic and hence there is additional
+ regression potential from toolchain changes.
+ 
  [Other info]
  Also including some auxiliary changes as we move back to proper upstream releases to enable testing, mostly around
  
  d/gbp.conf, .gitlab-ci.yml - being updated to point at right places
  d/t/control - adding a missing test dependency
  data/ - updated mirror lists
  
  The mirror lists are standard updates; the debian/tests/control change
  is needed to enable testing on more minimal autopkgtest images (our images
  accidentally include binutils); the other changes have zero impact on the Ubuntu infra or end user side as they only run during development.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to python-apt in Ubuntu.
https://bugs.launchpad.net/bugs/1998265

Title:
  OverFlow error when using cnf-extractor

Status in python-apt package in Ubuntu:
  In Progress
Status in python-apt source package in Bionic:
  In Progress
Status in python-apt source package in Focal:
  In Progress
Status in python-apt source package in Jammy:
  New
Status in python-apt source package in Kinetic:
  New

Bug description:
  [Impact]
  Large files can't be downloaded as we use a 32-bit integer in AcquireFile constructor, this breaks e.g. command-not-found generator:

  I'm trying to update the command-not-found indexes (bin/cnf-extract.py
  $MIRROR $suite $component $arch) on the production cnf extractor
  system (running bionic) and have encountered Tracebacks when
  generating them for Jammy and Lunar. This is causing the command not
  found indexes to be out of date for all(?) releases of Ubuntu.

  The lunar failure:

  Get:1 dotnet-sdk-6.0_6.0.111-0ubuntu3_amd64.deb [79.1 MB]
  Fetched 79.1 MB in 0s (0 B/s)
  Traceback (most recent call last):64... 7%
    File "./bin/cnf-extract.py", line 54, in <module>
      mirror_root, suite, component, arch)
    File "/srv/cnf-extractor/extractor/extractor.py", line 174, in command_not_found_extract
      debpath = pkg.candidate.fetch_binary(destdir=tmpdir)
    File "/usr/lib/python3/dist-packages/apt/package.py", line 883, in fetch_binary
      self.size, base, destfile=destfile)
  OverflowError: signed integer is greater than maximum

  I'll get more context for the Jammy failure but it wouldn't surprise
  me if it was the same package.

  [Test plan]
  I have included a test for the AcquireFile constructor to accept a large file size, and then a test that ensures it can read the file size.

  Run the autopkgtest suite.

  [Where problems could occur]
  It could fail in other places now.

  For kinetic, the package was copied in from lunar without changes.
  This upload is actually built in kinetic and hence there is additional
  regression potential from toolchain changes.

  [Other info]
  Also including some auxiliary changes as we move back to proper upstream releases to enable testing, mostly around

  d/gbp.conf, .gitlab-ci.yml - being updated to point at right places
  d/t/control - adding a missing test dependency
  data/ - updated mirror lists

  The mirror lists are standard updates; the debian/tests/control change
  is needed to enable testing on more minimal autopkgtest images (our images
  accidentally include binutils); the other changes have zero impact on the Ubuntu infra or end user side as they only run during development.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/python-apt/+bug/1998265/+subscriptions