[Bug 2138890] Re: llvm-21-tools: shtest-encoding.py missing encoding declaration breaks Python 3.14 installation

[Richard Scott McNew]

** Also affects: rust-toolchain
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to llvm-toolchain-21 in Ubuntu.
https://bugs.launchpad.net/bugs/2138890

Title:
  llvm-21-tools: shtest-encoding.py missing encoding declaration breaks
  Python 3.14 installation

Status in Rust Toolchain:
  New
Status in dh-python package in Ubuntu:
  New
Status in llvm-toolchain-21 package in Ubuntu:
  In Progress
Status in dh-python package in Debian:
  Fix Released

Bug description:
  The llvm-21-tools package ships a test file that contains non-UTF-8
  characters without an encoding declaration, causing package
  installation to fail with Python 3.14.

  ** Affected Package **
  llvm-21-tools 1:21.1.8-1ubuntu1

  ** Problematic File **
  /usr/lib/llvm-21/build/utils/lit/tests/shtest-encoding.py

  ** Error **
  When Python 3.14 is the default interpreter, installing llvm-21-tools triggers:
    SyntaxError: Non-UTF-8 code starting with '\xc2' on line 3, but no encoding declared

  This occurs during package post-installation when Python attempts to byte-compile the file. The error causes llvm-21-tools configuration to fail, which blocks installation of:
  - clang-tidy-21
  - python3-pygments (via rtupdate hooks)
  - Any package build-depending on LLVM 21 toolchain

  ** Impact **
  This is a Python 3.14 transition blocker affecting the entire Ubuntu archive. Any package that build-depends on clang-21, clang-tidy-21, clang-tools-21, llvm-21-dev, or llvm-21-tools will fail to build when Python 3.14 is default.

  ** Technical Details **
  Line 3 of shtest-encoding.py contains byte 0xc2 (non-UTF-8 character) but lacks a PEP 263 encoding declaration (e.g., # -*- coding: utf-8 -*-). Python 3.14 enforces PEP 263 strictly and fails to compile such files, while Python 3.13 is more lenient.

  Verification:
  $ python3.14 -m py_compile /usr/lib/llvm-21/build/utils/lit/tests/shtest-encoding.py
    File "/usr/lib/llvm-21/build/utils/lit/tests/shtest-encoding.py", line 3
  SyntaxError: Non-UTF-8 code starting with '\xc2'

  $ sed -n '3p' /usr/lib/llvm-21/build/utils/lit/tests/shtest-encoding.py | xxd
  (shows 0xc2 byte)

  ** Proposed Solutions **

  Option 1 (minimal fix):
  Add encoding declaration to shtest-encoding.py:
    # -*- coding: utf-8 -*-

  Option 2:
  Don't ship test files in binary packages. Modify debian/llvm-21-tools.install to exclude:
    /usr/lib/llvm-21/build/utils/lit/tests/

  Option 3 (new):
  Add a bcep (Byte-compilation exception patterns) in debian/ directory.
    d/llvm-X.Y-tools.bcep.in

  ** Reproduction **
  1. Install Python 3.14 as default interpreter
  2. Attempt to install llvm-21-tools
  3. Observe package configuration failure during Python byte-compilation

  ** Build Log Evidence **
  Failed build (Python 3.14): https://launchpadlibrarian.net/844369461/buildlog_ubuntu-resolute-amd64.hipify_7.1.0+dfsg-0ubuntu3_BUILDING.txt.gz
  Successful build (Python 3.13): https://launchpadlibrarian.net/844233787/buildlog_ubuntu-resolute-amd64.hipify_7.1.0+dfsg-0ubuntu3_BUILDING.txt.gz

  ** References **
  PEP 263: Defining Python Source Code Encodings
  https://www.python.org/dev/peps/pep-0263/

To manage notifications about this bug go to:
https://bugs.launchpad.net/rust-toolchain/+bug/2138890/+subscriptions