[Bug 1764220] Re: [SRU] dpkg zstd support

Balint Reczey 1764220 at bugs.launchpad.net
Wed Apr 21 15:50:17 UTC 2021 

Verified 1.18.4ubuntu1.7 on Xenial:

root at x-zstd:~# wget https://people.canonical.com/~rbalint/zstd-debs/glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb
--2021-04-21 15:37:36--  https://people.canonical.com/~rbalint/zstd-debs/glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb
Resolving people.canonical.com (people.canonical.com)... 91.189.89.62
Connecting to people.canonical.com (people.canonical.com)|91.189.89.62|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5618446 (5.4M) [application/x-debian-package]
Saving to: ‘glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb’

glibc-doc-reference_2.33-0ubuntu2~zstd1_all
100%[=========================================================================================>]
5.36M  4.76MB/s    in 1.1s

2021-04-21 15:37:38 (4.76 MB/s) - ‘glibc-doc-
reference_2.33-0ubuntu2~zstd1_all.deb’ saved [5618446/5618446]

root at x-zstd:~# dpkg-deb -R glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb glibc-doc-extracted
dpkg-deb: error: archive 'glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb' uses unknown compression for member 'control.tar.zst', giving up
root at x-zstd:~# sed -i s/backports/proposed/ /etc/apt/sources.list
root at x-zstd:~# apt update -qq
5 packages can be upgraded. Run 'apt list --upgradable' to see them.
root at x-zstd:~# apt install -qq dpkg 
The following package was automatically installed and is no longer required:
  libfreetype6
Use 'apt autoremove' to remove it.
The following additional packages will be installed:
  libzstd1
The following NEW packages will be installed:
  libzstd1
The following packages will be upgraded:
  dpkg
1 upgraded, 1 newly installed, 0 to remove and 4 not upgraded.
Need to get 2,237 kB of archives.
After this operation, 399 kB of additional disk space will be used.
Do you want to continue? [Y/n] 
Selecting previously unselected package libzstd1.
(Reading database ... 25816 files and directories currently installed.)
Preparing to unpack .../libzstd1_1.3.1+dfsg-1~ubuntu0.16.04.1_amd64.deb ...
Unpacking libzstd1 (1.3.1+dfsg-1~ubuntu0.16.04.1) ...
Processing triggers for libc-bin (2.23-0ubuntu11.2) ...
Setting up libzstd1 (1.3.1+dfsg-1~ubuntu0.16.04.1) ...
Processing triggers for libc-bin (2.23-0ubuntu11.2) ...
(Reading database ... 25821 files and directories currently installed.)
Preparing to unpack .../dpkg_1.18.4ubuntu1.7_amd64.deb ...
Unpacking dpkg (1.18.4ubuntu1.7) over (1.18.4ubuntu1.6) ...
Setting up dpkg (1.18.4ubuntu1.7) ...
Processing triggers for man-db (2.7.5-1) ...
root at x-zstd:~#  time dpkg-deb -R glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb glibc-doc-extracted

real	0m0.031s
user	0m0.009s
sys	0m0.037s
root at x-zstd:~#  dpkg -i glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb 
Selecting previously unselected package glibc-doc-reference.
(Reading database ... 25821 files and directories currently installed.)
Preparing to unpack glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb ...
Unpacking glibc-doc-reference (2.33-0ubuntu2~zstd1) ...
Setting up glibc-doc-reference (2.33-0ubuntu2~zstd1) ...
Processing triggers for install-info (6.1.0.dfsg.1-5) ...
root at x-zstd:~# dpkg -l glibc-doc-reference
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                  Version                 Architecture            Description
+++-=====================================-=======================-=======================-================================================================================
ii  glibc-doc-reference                   2.33-0ubuntu2~zstd1     all                     GNU C Library: Documentation
root at x-zstd:~# sed -i 's/# deb-src/deb-src/' /etc/apt/sources.list
root at x-zstd:~# apt update -qq
4 packages can be upgraded. Run 'apt list --upgradable' to see them.
root at x-zstd:~# apt build-dep hello
Reading package lists... Done
Reading package lists... Done
...
Setting up libdpkg-perl (1.18.4ubuntu1.7) ...
Setting up dpkg-dev (1.18.4ubuntu1.7) ...
...
Processing triggers for libc-bin (2.23-0ubuntu11.2) ...
root at x-zstd:~# apt source hello
Reading package lists... Done
Need to get 733 kB of source archives.
Get:1 http://archive.ubuntu.com/ubuntu xenial/main hello 2.10-1 (dsc) [1,323 B]
...
'_apt'. - pkgAcquire::Run (13: Permission denied)
root at x-zstd:~# cd hello-2.10/
root at x-zstd:~/hello-2.10# dpkg-buildpackage 
dpkg-buildpackage: source package hello
...
dpkg-deb: building package 'hello' in '../hello_2.10-1_amd64.deb'.
 dpkg-genchanges  >../hello_2.10-1_amd64.changes
dpkg-genchanges: including full source code in upload
 dpkg-source --after-build hello-2.10
dpkg-buildpackage: full upload (original source is included)
root at x-zstd:~/hello-2.10# cd ..
root at x-zstd:~# rm -rf hello-2.10/

root at x-zstd:~# apt source hello
...
root at x-zstd:~# cd hello-2.10/
root at x-zstd:~/hello-2.10# vi debian/rules
root at x-zstd:~/hello-2.10# cat debian/rules
#!/usr/bin/make -f
%:
	dh $@

override_dh_auto_clean:
	[ ! -f Makefile ] || $(MAKE) distclean

override_dh_installdocs:
	dh_installdocs NEWS

override_dh_builddeb:
	dh_builddeb -- -Zzstd
root at x-zstd:~/hello-2.10# dpkg-buildpackage 
dpkg-buildpackage: source package hello
...
dh_builddeb -- -Zzstd
dpkg-deb: error: only decompression is supported for 'zstd'!

Type dpkg-deb --help for help about manipulating *.deb files;
Type dpkg --help for help about installing and deinstalling packages.
dh_builddeb: dpkg-deb -Zzstd --build debian/hello .. returned exit code 2
debian/rules:12: recipe for target 'override_dh_builddeb' failed
make[1]: *** [override_dh_builddeb] Error 1
make[1]: Leaving directory '/root/hello-2.10'
debian/rules:3: recipe for target 'binary' failed
make: *** [binary] Error 2
dpkg-buildpackage: error: debian/rules binary gave error exit status 2
root at x-zstd:~/hello-2.10# dpkg -l dpkg dpkg-dev
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                  Version                 Architecture            Description
+++-=====================================-=======================-=======================-================================================================================
ii  dpkg                                  1.18.4ubuntu1.7         amd64                   Debian package management system
ii  dpkg-dev                              1.18.4ubuntu1.7         all                     Debian package development tools
root at x-zstd:~/hello-2.10# 


** Tags removed: verification-needed verification-needed-xenial
** Tags added: verification-done verification-done-xenial

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to dpkg in Ubuntu.
https://bugs.launchpad.net/bugs/1764220

Title:
  [SRU] dpkg zstd support

Status in dpkg:
  Incomplete
Status in dpkg package in Ubuntu:
  Fix Released
Status in dpkg source package in Xenial:
  Fix Committed

Bug description:
  [Impact]

  * Xenial's dpkg can't decompress zstd-compressed binary packages
  preventing some systems of Launchpad from processing packages with
  such compression. This blocks publishing zstd-compressed binary
  packages through Launchpad for later Ubuntu releases as well.

  [Test Plan]

  * https://people.canonical.com/~rbalint/zstd-debs/ contains a .deb built on Hirsute having both data and control members of the .deb being compressed with zstd.
  * Download and unpack it. With unfixed dpkg an error should be shown.

  $ wget https://people.canonical.com/~rbalint/zstd-debs/glibc-doc-
  reference_2.33-0ubuntu2~zstd1_all.deb

  # unfixed:
  $ dpkg-deb -R glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb  glibc-doc-extracted
  dpkg-deb: error: archive 'glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb' uses unknown compression for member 'control.tar.zst', giving up

  # fixed
  $ time dpkg-deb -R glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb  glibc-doc-extracted

  real	0m0.148s
  user	0m0.041s
  sys	0m0.124s

  * Also install the package:

  root at x-zstd:~# dpkg -i glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb
  Selecting previously unselected package glibc-doc-reference.
  (Reading database ... 25816 files and directories currently installed.)
  Preparing to unpack glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb ...
  Unpacking glibc-doc-reference (2.33-0ubuntu2~zstd1) ...
  Setting up glibc-doc-reference (2.33-0ubuntu2~zstd1) ...
  Processing triggers for install-info (6.1.0.dfsg.1-5) ...
  root at x-zstd:~#

  * Build the hello package, it should work
  * Build the hello package overriding the compression to zstd, this should fail:

  $ cat debian/rules
  ...

  override_dh_builddeb:
   dh_builddeb -- -Zzstd

  ...
  make[1]: Entering directory '/root/hello-2.10'
  dh_builddeb -- -Zzstd
  dpkg-deb: error: only decompression is supported for 'zstd'!

  Type dpkg-deb --help for help about manipulating *.deb files;
  Type dpkg --help for help about installing and deinstalling packages.
  dh_builddeb: dpkg-deb -Zzstd --build debian/hello .. returned exit code 2
  debian/rules:12: recipe for target 'override_dh_builddeb' failed

  [Where problems could occur]

  * The fix is isolated and is a backport from Bionic with the compression part omitted. Crashes could happen due to coding errors should they exist.
  Only decompression should be supported and this is verified in the test plan.

  * Incompabilities between libzstd present in Xenial and present in
  later releases could prevent dpkg running on Xenial from successfully
  processing packages built on later releases. The package for Xenial is
  built with libzstd1 (build-depending on libzstd1-dev) which is at
  version 1.3.1+dfsg-1~ubuntu0.16.04.1. Bionic's version is 1.3.3+dfsg-
  2ubuntu1.2 and there were no format breaking changes between those
  versions, not in any later version. See
  https://github.com/facebook/zstd/issues/999#issuecomment-359538229 .

  [Original Bug Text]

  As discussed previously, we want to have zstd support in 18.04 to
  evaluate and potentially enable it in later releases.

  The zstd support adds a dependency on libzstd1 to dpkg. This should
  not have any effect on live images, since libzstd1 is part of the
  various live tasks, as btrfs-progs need it. For installed systems,
  this might be a new dependency (if they do not use btrfs, tor, or some
  other tools), so an increase of ~520 KB, as that's the size of the
  library and the library only depends on libc6.

  The change is isolated, it adds the compressor and decompressor to
  dpkg, please see the attached patch for the details.

  The change is being discussed here:
  https://lists.ubuntu.com/archives/ubuntu-devel/2018-March/040211.html
  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=892664

To manage notifications about this bug go to:
https://bugs.launchpad.net/dpkg/+bug/1764220/+subscriptions

More information about the foundations-bugs mailing list