[Bug 1370249] Re: 'ascii' codec can't decode byte 0xc3 in position 19027: ordinal not in range(128) (curtin can't handle non-ascii characters in dpkg --list output)

[Newell Jensen]

** Description changed:

  I was trying to deploy an image based on an Ubuntu Desktop 14.04 install
  using MAAS 1.7.0~beta2+bzr2967-0ubuntu1~ppa1 on Ubuntu 14.04 server.
  
  I made a "curtin-compatible" tar.gz which includes cloud-init, put it in
  place of root-tgz in my maas server, and tried to deploy a node (using
  juju bootstrap).
  
  Juju ended up complaining and timing out, and the node never fully
  completed installation. On closer look, I found curtin crashes when
  running "curthooks", specifically at some point it does dpkg --list, and
  the code that handles this (utils.subp) assumes output from its commands
  is pure ascii and so crashes when the output contains non-ascii. For
  Ubuntu Desktop, in particular, the output contains "Déjȧ Dup", which is
  what causes things to barf.
  
  I SSHd into the half-installed node and ran curtin manually like so:
  
  sudo CURTIN_STACKTRACE=1 /curtin/bin/curtin --verbose --verbose --log-
  file /media/root-rw/curtin-install.log install
  --config=configs/config-000.cfg
  http://10.10.10.1/MAAS/static/images/ubuntu/amd64/generic/trusty/release
  /root-tgz
  
  It ended in this:
  
  Running command ['chroot', '/tmp/tmpQ_dk0_/target', 'dpkg-query', '--list'] with allowed return codes [0] (shell=False, capture=True)
  Traceback (most recent call last):
    File "/curtin/curtin/commands/main.py", line 96, in main
      sys.exit(args.func(args))
    File "/curtin/curtin/commands/curthooks.py", line 443, in curthooks
      apply_debconf_selections(cfg, target)
    File "/curtin/curtin/commands/curthooks.py", line 219, in apply_debconf_selections
      pkgs_installed = get_installed_packages(target)
    File "/curtin/curtin/commands/curthooks.py", line 259, in get_installed_packages
      (out, _err) = util.subp(cmd, capture=True)
    File "/curtin/curtin/util.py", line 58, in subp
      out = out.decode()
  UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 19027: ordinal not in range(128)
  'ascii' codec can't decode byte 0xc3 in position 19027: ordinal not in range(128)
  Traceback (most recent call last):
    File "/curtin/curtin/commands/main.py", line 96, in main
      sys.exit(args.func(args))
    File "/curtin/curtin/commands/install.py", line 267, in cmd_install
      stage.run()
    File "/curtin/curtin/commands/install.py", line 104, in run
      util.subp(cmd, shell=shell, env=self.env)
    File "/curtin/curtin/util.py", line 68, in subp
      cmd=args)
  ProcessExecutionError: Unexpected error while running command.
  Command: ['curtin', 'curthooks']
  Exit code: 3
  Reason: -
  Stdout: ''
  Stderr: ''
  Unexpected error while running command.
  Command: ['curtin', 'curthooks']
  Exit code: 3
  Reason: -
  Stdout: ''
  Stderr: ''
  
  I'm attaching the full log as well.
+ 
+ 
+ SRU curtin
+ ==========
+ 
+ [Impact]
+ 
+ curtin can't handle non-ascii characters in dpkg --list output as
+ mentioned in the original bug report descripition.  This bug doesn't
+ allow curtin to continue and terminates the installation.
+ 
+ This fix is needed so that we can make subprocess calls with output that
+ contains non-ascii characters.
+ 
+ The patch supplied by the bug reporter enables non-ascii character
+ subprocess output decoding for curtin.
+ 
+ 
+ [Test Case]
+ 
+ To reproduce the bug, follow the same steps as the bug report
+ description mentiones.  Namely, making sure that dpkg --list will
+ contain output with non-ascii characters.
+ 
+ 
+ [Regression Potential] 
+ 
+ utf-8 is backwards compatible with ascii, so the likely hood of
+ regressions is small.

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to curtin in Ubuntu.
https://bugs.launchpad.net/bugs/1370249

Title:
  'ascii' codec can't decode byte 0xc3 in position 19027: ordinal not in
  range(128) (curtin can't handle non-ascii characters in dpkg --list
  output)

To manage notifications about this bug go to:
https://bugs.launchpad.net/curtin/+bug/1370249/+subscriptions