[Bug 2075337] Re: py3clean fails when using alternate character set
Dave Jones
2075337 at bugs.launchpad.net
Mon Aug 5 00:41:04 UTC 2024
Having spent some time digging into this (at the request of the SRU
team), I'll summarise my findings:
Firstly, the analysis is correct: py3clean is ultimately the issue;
packages that both use py3clean and have diversions are affected
(diversions are the only circumstances that produce localised output in
dpkg-query -L).
Secondly, the proposed fix upstream is also fine, though personally I'd
also like to see the stdout decode use errors='replace' (there's no good
reason to fail here in the event of dodgy UTF-8 in a translation, for
instance). I'll attach debdiffs for noble and oracular to illustrate my
intent.
Thirdly, the proposed workaround for affected packages: using Breaks on
python3-minimal with a version strictly less than the fixed version also
appears to be sufficient (in all the cases I've tested this causes
python3-minimal to be at least unpacked prior to the prerm script of the
affected package being executed).
So, if the fix for python3-minimal can be uploaded, it simply remains to
determine which packages are affected. This is where things get tricky.
Benjamin's efforts are much appreciated here, and will likely be
decisive in light of the following: determining which packages use
py3clean is relatively simple (pretty much anything that installs a
python module). Determining which packages have diversions turns out to
be extremely difficult.
It's not enough to detect if a package *itself* uses dpkg-divert. Take
the cloud-init case where this was first detected: the postinst calls
dpkg-divert but only to *remove* an old diversion. The diversion that
actually affects this package comes from usr-merge (because cloud-init
still has files under the unmerged paths for various reasons). In
essence, any package (foo) can cause a diversion in another package
(bar) without the affected package (bar) showing any sign of this in
either its source or its binary artefacts. Further, the diversion may or
may not affect the package (bar) as the source of the diversion (foo)
may be optional to install.
Hence, there are two approaches. The thorough, but likely impractical,
approach would be to apply the "Breaks" fix to all packages using
py3clean: i.e. all packages that install python modules. Then there's
(Benjamin's) empirical approach: attempt to install everything from the
former set and see what fails. As noted above, this cannot guarantee
correctness as we cannot be certain that all packages that may divert
files in our target package are installed, but in practice it's probably
(hopefully!) "good enough" given the rarity of diversions and avoids
updates to (presumably) several thousand packages.
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to python3-defaults in Ubuntu.
https://bugs.launchpad.net/bugs/2075337
Title:
py3clean fails when using alternate character set
Status in dh-python package in Ubuntu:
New
Status in python3-defaults package in Ubuntu:
New
Bug description:
We noticed this in bug 2060769 but think it is a separate issue.
dh_python3 is putting a py3clean invocation into package prerm
maintainer scripts. When that runs, it explodes if the system
character set is for example ISO-8859-1.
This appears to affect every package for which dh_python3 has inserted
a call to py3clean into the package prerm.
python3-minimal 3.12.3-0ubuntu1
In this reproducer, cloud-init is at 24.1.3-0ubuntu3.3.
Steps to reproduce:
$ lxc launch ubuntu:noble rbasak-pro-l8n
$ lxc shell rbasak-pro-l8n
# locale-gen fr_FR
Generating locales (this might take a while)...
fr_FR.ISO-8859-1... done
Generation complete.
# update-locale LANG=fr_FR
# exit
# adjust terminal to ISO-8859-1
$ lxc shell rbasak-pro-l8n
# apt update && apt install --reinstall cloud-init
Expected results: success
Actual results:
Lecture des listes de paquets... Fait
Construction de l'arbre des dépendances... Fait
Lecture des informations d'état... Fait
0 mis à jour, 0 nouvellement installés, 1 réinstallés, 0 à enlever et 5 non mis à jour.
Il est nécessaire de prendre 598 ko dans les archives.
Après cette opération, 0 o d'espace disque supplémentaires seront utilisés.
Réception de :1 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 cloud-init all 24.1.3-0ubuntu3.3 [598 kB]
598 ko réceptionnés en 1s (926 ko/s)
Préconfiguration des paquets...
(Lecture de la base de données... 34406 fichiers et répertoires déjà installés.)
Préparation du dépaquetage de .../cloud-init_24.1.3-0ubuntu3.3_all.deb ...
Traceback (most recent call last):
File "/usr/bin/py3clean", line 210, in <module>
main()
File "/usr/bin/py3clean", line 196, in main
pfiles = set(dpf.from_package(options.package))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/share/python3/debpython/files.py", line 54, in from_package
stdout = str(stdout, 'utf-8')
^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 2417: invalid continuation byte
dpkg: avertissement: le sous-processus ancien paquet cloud-init script pre-removal a renvoyé un état de sortie d'erreur 1
dpkg: tentative d'exécution du script du nouveau paquet à la place...
Traceback (most recent call last):
File "/usr/bin/py3clean", line 210, in <module>
main()
File "/usr/bin/py3clean", line 196, in main
pfiles = set(dpf.from_package(options.package))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/share/python3/debpython/files.py", line 54, in from_package
stdout = str(stdout, 'utf-8')
^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 2417: invalid continuation byte
dpkg: erreur de traitement de l'archive /var/cache/apt/archives/cloud-init_24.1.3-0ubuntu3.3_all.deb (--unpack) :
le sous-processus nouveau cloud-init paquet pre-removal script a renvoyé un état de sortie d'erreur 1
dmesg: read kernel buffer failed: Operation not permitted
Traceback (most recent call last):
File "/usr/bin/py3compile", line 323, in <module>
main()
File "/usr/bin/py3compile", line 302, in main
compile(files, versions,
File "/usr/bin/py3compile", line 185, in compile
for fn, versions_to_compile in filter_files(files, e_patterns, versions):
File "/usr/bin/py3compile", line 128, in filter_files
for fpath in files:
File "/usr/share/python3/debpython/files.py", line 71, in filter_public
for fn in files:
File "/usr/share/python3/debpython/files.py", line 54, in from_package
stdout = str(stdout, 'utf-8')
^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 2417: invalid continuation byte
dpkg: erreur lors du nettoyage:
le sous-processus paquet cloud-init script post-installation installé a renvoyé un état de sortie d'erreur 1
Des erreurs ont été rencontrées pendant l'exécution :
/var/cache/apt/archives/cloud-init_24.1.3-0ubuntu3.3_all.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/dh-python/+bug/2075337/+subscriptions
More information about the foundations-bugs
mailing list