[Bug 1766337] Re: uncaught python exception : UnicodeDecodeError: 'utf-8' codec can't decode byte; invalid continuation byte

[Brian Murray]

** Description changed:

+ [Impact]
+ Ubuntu crash reports and package installation failures may not include DpkgTerminalLog.txt if there is a UnicodeDecodeError trying to read the dpkg log file.
+ 
+ [Test Case]
+ Because the fix is simple and its complicated to have a dpkg log file with unicode in it we'll just test that truncating of a dpkg log file still works.
+ 
+ 1) Install 2vcard (so we have a recent dpkg log entry)
+ 2) Run update-manager
+ 3) Kill it with signal 11 e.g. kill -11 $PID of update-manager
+ 4) Run ubuntu-bug /var/crash/_usr_bin_update-manager.1000.crash
+ 5) Observe DpkgTerminalLog.txt only contains a log entry regarding the installation of 2vcard and matches the content in /var/log/apt/term.log.
+ 
+ [Regression Potential]
+ Its possible that the code to replace the DpkgTerminalLog contents of the report is wrong but the test case above ensures that it isn't.
+ 
+ [Original Description]
  Hi,
  
  here is a python exception I got while reporting a bug:
  
  Traceback (most recent call last):
-   File "/usr/lib/python3/dist-packages/apport/report.py", line 198, in _run_hook
-     symb['add_info'](report, ui)
-   File "/usr/share/apport/general-hooks/ubuntu.py", line 89, in add_info
-     check_attachment_for_errors(report, attachment)
-   File "/usr/share/apport/general-hooks/ubuntu.py", line 208, in check_attachment_for_errors
-     trim_dpkg_log(report)
-   File "/usr/share/apport/general-hooks/ubuntu.py", line 508, in trim_dpkg_log
-     report['DpkgTerminalLog'] = '\n'.join([str(line.decode('utf-8')) for line in lines])
-   File "/usr/share/apport/general-hooks/ubuntu.py", line 508, in <listcomp>
-     report['DpkgTerminalLog'] = '\n'.join([str(line.decode('utf-8')) for line in lines])
+   File "/usr/lib/python3/dist-packages/apport/report.py", line 198, in _run_hook
+     symb['add_info'](report, ui)
+   File "/usr/share/apport/general-hooks/ubuntu.py", line 89, in add_info
+     check_attachment_for_errors(report, attachment)
+   File "/usr/share/apport/general-hooks/ubuntu.py", line 208, in check_attachment_for_errors
+     trim_dpkg_log(report)
+   File "/usr/share/apport/general-hooks/ubuntu.py", line 508, in trim_dpkg_log
+     report['DpkgTerminalLog'] = '\n'.join([str(line.decode('utf-8')) for line in lines])
+   File "/usr/share/apport/general-hooks/ubuntu.py", line 508, in <listcomp>
+     report['DpkgTerminalLog'] = '\n'.join([str(line.decode('utf-8')) for line in lines])
  UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe2 in position 3070: invalid continuation byte
  
  It can also be seen here:
  
-   https://launchpadlibrarian.net/367002573/HookError_ubuntu.txt
+   https://launchpadlibrarian.net/367002573/HookError_ubuntu.txt
  
- ProblemType: Bug
- DistroRelease: Ubuntu 18.04
+ ProblemType: BugDistroRelease: Ubuntu 18.04
  Package: apport 2.20.9-0ubuntu6
  Uname: Linux 4.16.3-041603-generic x86_64
  ApportLog:
-  
+ 
  ApportVersion: 2.20.9-0ubuntu6
  Architecture: amd64
  CurrentDesktop: MATE
  Date: Mon Apr 23 20:13:24 2018
  PackageArchitecture: all
  ProcEnviron:
-  TERM=xterm
-  PATH=(custom, no user)
-  XDG_RUNTIME_DIR=<set>
-  LANG=en_US.UTF-8
-  SHELL=/bin/bash
- SourcePackage: apport
+  TERM=xterm
+  PATH=(custom, no user)
+  XDG_RUNTIME_DIR=<set>
+  LANG=en_US.UTF-8
+  SHELL=/bin/bashSourcePackage: apport
  UpgradeStatus: No upgrade log present (probably fresh install)

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to apport in Ubuntu.
https://bugs.launchpad.net/bugs/1766337

Title:
  uncaught python exception : UnicodeDecodeError: 'utf-8' codec can't
  decode byte; invalid continuation byte

Status in apport package in Ubuntu:
  Fix Released
Status in apport source package in Bionic:
  In Progress

Bug description:
  [Impact]
  Ubuntu crash reports and package installation failures may not include DpkgTerminalLog.txt if there is a UnicodeDecodeError trying to read the dpkg log file.

  [Test Case]
  Because the fix is simple and its complicated to have a dpkg log file with unicode in it we'll just test that truncating of a dpkg log file still works.

  1) Install 2vcard (so we have a recent dpkg log entry)
  2) Run update-manager
  3) Kill it with signal 11 e.g. kill -11 $PID of update-manager
  4) Run ubuntu-bug /var/crash/_usr_bin_update-manager.1000.crash
  5) Observe DpkgTerminalLog.txt only contains a log entry regarding the installation of 2vcard and matches the content in /var/log/apt/term.log.

  [Regression Potential]
  Its possible that the code to replace the DpkgTerminalLog contents of the report is wrong but the test case above ensures that it isn't.

  [Original Description]
  Hi,

  here is a python exception I got while reporting a bug:

  Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/apport/report.py", line 198, in _run_hook
      symb['add_info'](report, ui)
    File "/usr/share/apport/general-hooks/ubuntu.py", line 89, in add_info
      check_attachment_for_errors(report, attachment)
    File "/usr/share/apport/general-hooks/ubuntu.py", line 208, in check_attachment_for_errors
      trim_dpkg_log(report)
    File "/usr/share/apport/general-hooks/ubuntu.py", line 508, in trim_dpkg_log
      report['DpkgTerminalLog'] = '\n'.join([str(line.decode('utf-8')) for line in lines])
    File "/usr/share/apport/general-hooks/ubuntu.py", line 508, in <listcomp>
      report['DpkgTerminalLog'] = '\n'.join([str(line.decode('utf-8')) for line in lines])
  UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe2 in position 3070: invalid continuation byte

  It can also be seen here:

    https://launchpadlibrarian.net/367002573/HookError_ubuntu.txt

  ProblemType: BugDistroRelease: Ubuntu 18.04
  Package: apport 2.20.9-0ubuntu6
  Uname: Linux 4.16.3-041603-generic x86_64
  ApportLog:

  ApportVersion: 2.20.9-0ubuntu6
  Architecture: amd64
  CurrentDesktop: MATE
  Date: Mon Apr 23 20:13:24 2018
  PackageArchitecture: all
  ProcEnviron:
   TERM=xterm
   PATH=(custom, no user)
   XDG_RUNTIME_DIR=<set>
   LANG=en_US.UTF-8
   SHELL=/bin/bashSourcePackage: apport
  UpgradeStatus: No upgrade log present (probably fresh install)

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/apport/+bug/1766337/+subscriptions