[Bug 2076269] Comment bridged from LTC Bugzilla

[bugproxy]

------- Comment From iii at de.ibm.com 2024-10-07 10:58 EDT-------
I've extracted the compression logic into a stand-alone script, and it doesn't fail:

#!/usr/bin/env python3
import gzip
import struct
import zlib

value = b"ABABABABABABABABABAB\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00Z"
compressed = b"\037\213\010\010\000\000\000\000\002\377File\000"
bc = zlib.compressobj(6, zlib.DEFLATED, -zlib.MAX_WBITS, zlib.DEF_MEM_LEVEL, 0)
compressed += bc.compress(value)
compressed += bc.flush()
compressed += struct.pack("<L", zlib.crc32(value) & 0xFFFFFFFF)
compressed += struct.pack("<L", len(value) & 0xFFFFFFFF)
print(compressed)
assert gzip.decompress(compressed) == value

I think the problem is rather here:

@staticmethod
def _write_binary_item_base64_encoded(
[...]
for chunk in chunks:
file.write(b"\n ")
file.write(base64.b64encode(chunk))

Example:

>>> base64.b64decode(base64.b64encode(b"a")+base64.b64encode(b"b"))
b'a'

Sometimes this works though:

>>> base64.b64decode(base64.b64encode(b"aaa")+base64.b64encode(b"b"))
b'aaab'

Seems like s390x was unlucky and hit the problematic case. I would
suggest to call b64decode() on each line individually.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to apport in Ubuntu.
https://bugs.launchpad.net/bugs/2076269

Title:
  invalid base64 encoded gzip data on s390x causes autopkgtest failures

Status in Ubuntu on IBM z Systems:
  New
Status in apport package in Ubuntu:
  New
Status in python3.12 package in Ubuntu:
  New

Bug description:
  The following tests all fail with an encoding/decoding issue:

  tests/unit/test_problem_report.py::T:test_modify
  tests/unit/test_report.py::T::test_report_from_systemd_coredump_storage_journal
  tests/integration/test_problem_report::T::test_write_file

  Example failure:

  564s =================================== FAILURES ===================================
  564s ________________________________ T.test_modify _________________________________
  564s
  564s self = <tests.unit.test_problem_report.T testMethod=test_modify>
  564s
  564s     def test_modify(self):
  564s         """reading, modifying fields, and writing back."""
  564s         report = textwrap.dedent(
  564s             """\
  564s             ProblemType: Crash
  564s             Date: now!
  564s             Long:
  564s              xxx
  564s              .
  564s              yyy
  564s             Short: Bar
  564s             File: base64
  564s              H4sICAAAAAAC/0ZpbGUA
  564s              c3RyxIAMcBAFAK/2p9MfAAAA
  564s             """
  564s         ).encode()
  564s
  564s         pr = problem_report.ProblemReport()
  564s         pr.load(io.BytesIO(report))
  564s
  564s         self.assertEqual(pr["Long"], "xxx\n.\nyyy")
  564s
  564s         # write back unmodified
  564s         out = io.BytesIO()
  564s         pr.write(out)
  564s >       self.assertEqual(out.getvalue(), report)
  564s E       AssertionError: b'Pro[73 chars]e64\n H4sICAAAAAAC/0ZpbGUA\n cnTChAxwEA==\n BRgAr/an0x8AAAA=\n' != b'Pro[73 chars]e64\n H4sICAAAAAAC/0ZpbGUA\n c3RyxIAMcBAFAK/2p9MfAAAA\n'
  564s
  564s tests/unit/test_problem_report.py:509: AssertionError

  Autopkgtest noble log: https://autopkgtest.ubuntu.com/results/autopkgtest-noble/noble/s390x/a/apport/20240807_023039_8850a@/log.gz
  oracular log: https://autopkgtest.ubuntu.com/results/autopkgtest-oracular/oracular/s390x/a/apport/20240805_132135_d3a80@/log.gz

  This failure was seen on noble (log above) but also on oracular.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu-z-systems/+bug/2076269/+subscriptions