[Bug 1822886] Re: universe missing after bionic->cosmic do-release-upgrade

[Brian Murray]

** Description changed:

+ [Impact]
+ If there is a network hiccup at the right moment it's possible that some sources.list entries will be disabled when a release upgrade is calculated and then people could upgrade with surprising results e.g. universe being missing.
+ 
+ [Test Case]
+ This is somewhat convoluted as we need to create the network error only when a specific line is is being tested by the dist-upgrader.
+ 
+ 1) Use a system on xenial or bionic and ensure prompt is set appropriately (lts for xenial, normal for bionic)
+ 2) Edit your /etc/apt/sources.list file so that universe is a separate line from main, restricted, and multiverse
+ 3) Run do-release-upgrade
+ 4) Cancel the release upgrade
+ 5) cd to /tmp/ubuntu-release-upgrader-$tmpname
+ 6) edit DistUpgradeController.py with the attached patch which will cause the url check for the universe line to fail
+ 7) run 'sudo ./$release-upgrading-to'
+ 8) wait until you receive the message regarding quantities of packages to install, upgrade, etc
+ 9) look at /etc/apt/sources.list file and notice that the lines with universe are disabled
+ 
+ With the release-upgrader from -proposed you'll need to follow the same
+ steps and you'll notice that the lines with universe are not disabled.
+ 
+ [Regression Potential]
+ Because we are checking each unique uri in sources.list only one time its possible that if the first check fails then the upgrade will fail to be calculated. That is better than upgrading and having components disabled though.
+ 
  I took a bionic system and stepped through bionic->cosmic->disco via do-
  release-upgrade.
  
  Upon finishing I noticed that universe was "gone" and all packages from
  universe were removed from the system. Based on the files on my system,
  it looks like it happened during the bionic->cosmic upgrade. You'll see
  in the sources.list that cosmic-updates universe pocket is that but not
  cosmic universe.
  
- ProblemType: Bug
- DistroRelease: Ubuntu 19.04
+ ProblemType: BugDistroRelease: Ubuntu 19.04
  Package: ubuntu-release-upgrader-core 1:19.04.13
  ProcVersionSignature: Ubuntu 5.0.0-8.9-generic 5.0.1
  Uname: Linux 5.0.0-8-generic x86_64
  NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair
  ApportVersion: 2.20.10-0ubuntu23
  Architecture: amd64
  CrashDB: ubuntu
  CurrentDesktop: ubuntu:GNOME
  Date: Tue Apr  2 16:04:34 2019
  InstallationDate: Installed on 2018-07-24 (252 days ago)
  InstallationMedia: Ubuntu 18.04.1 LTS "Bionic Beaver" - Release amd64 (20180724)
  PackageArchitecture: all
  ProcEnviron:
-  TERM=xterm
-  PATH=(custom, no user)
-  XDG_RUNTIME_DIR=<set>
-  LANG=en_US.UTF-8
-  SHELL=/bin/bash
- SourcePackage: ubuntu-release-upgrader
+  TERM=xterm
+  PATH=(custom, no user)
+  XDG_RUNTIME_DIR=<set>
+  LANG=en_US.UTF-8
+  SHELL=/bin/bashSourcePackage: ubuntu-release-upgrader
  Symptom: ubuntu-release-upgrader
  UpgradeStatus: Upgraded to disco on 2019-04-02 (0 days ago)
  VarLogDistupgradeXorgFixuplog:
-  INFO:root:/usr/bin/do-release-upgrade running
-  INFO:root:No xorg.conf, exiting
+  INFO:root:/usr/bin/do-release-upgrade running
+  INFO:root:No xorg.conf, exiting
  mtime.conffile..etc.update-manager.release-upgrades: 2019-04-02T08:18:06.798258

** Patch added: "changes to DistUpgradeController for testing"
   https://bugs.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/+bug/1822886/+attachment/5254979/+files/bug-1822886-test-case.patch

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to ubuntu-release-upgrader in
Ubuntu.
https://bugs.launchpad.net/bugs/1822886

Title:
  universe missing after bionic->cosmic do-release-upgrade

Status in ubuntu-release-upgrader package in Ubuntu:
  Fix Released
Status in ubuntu-release-upgrader source package in Bionic:
  Triaged
Status in ubuntu-release-upgrader source package in Cosmic:
  Triaged

Bug description:
  [Impact]
  If there is a network hiccup at the right moment it's possible that some sources.list entries will be disabled when a release upgrade is calculated and then people could upgrade with surprising results e.g. universe being missing.

  [Test Case]
  This is somewhat convoluted as we need to create the network error only when a specific line is is being tested by the dist-upgrader.

  1) Use a system on xenial or bionic and ensure prompt is set appropriately (lts for xenial, normal for bionic)
  2) Edit your /etc/apt/sources.list file so that universe is a separate line from main, restricted, and multiverse
  3) Run do-release-upgrade
  4) Cancel the release upgrade
  5) cd to /tmp/ubuntu-release-upgrader-$tmpname
  6) edit DistUpgradeController.py with the attached patch which will cause the url check for the universe line to fail
  7) run 'sudo ./$release-upgrading-to'
  8) wait until you receive the message regarding quantities of packages to install, upgrade, etc
  9) look at /etc/apt/sources.list file and notice that the lines with universe are disabled

  With the release-upgrader from -proposed you'll need to follow the
  same steps and you'll notice that the lines with universe are not
  disabled.

  [Regression Potential]
  Because we are checking each unique uri in sources.list only one time its possible that if the first check fails then the upgrade will fail to be calculated. That is better than upgrading and having components disabled though.

  I took a bionic system and stepped through bionic->cosmic->disco via
  do-release-upgrade.

  Upon finishing I noticed that universe was "gone" and all packages
  from universe were removed from the system. Based on the files on my
  system, it looks like it happened during the bionic->cosmic upgrade.
  You'll see in the sources.list that cosmic-updates universe pocket is
  that but not cosmic universe.

  ProblemType: BugDistroRelease: Ubuntu 19.04
  Package: ubuntu-release-upgrader-core 1:19.04.13
  ProcVersionSignature: Ubuntu 5.0.0-8.9-generic 5.0.1
  Uname: Linux 5.0.0-8-generic x86_64
  NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair
  ApportVersion: 2.20.10-0ubuntu23
  Architecture: amd64
  CrashDB: ubuntu
  CurrentDesktop: ubuntu:GNOME
  Date: Tue Apr  2 16:04:34 2019
  InstallationDate: Installed on 2018-07-24 (252 days ago)
  InstallationMedia: Ubuntu 18.04.1 LTS "Bionic Beaver" - Release amd64 (20180724)
  PackageArchitecture: all
  ProcEnviron:
   TERM=xterm
   PATH=(custom, no user)
   XDG_RUNTIME_DIR=<set>
   LANG=en_US.UTF-8
   SHELL=/bin/bashSourcePackage: ubuntu-release-upgrader
  Symptom: ubuntu-release-upgrader
  UpgradeStatus: Upgraded to disco on 2019-04-02 (0 days ago)
  VarLogDistupgradeXorgFixuplog:
   INFO:root:/usr/bin/do-release-upgrade running
   INFO:root:No xorg.conf, exiting
  mtime.conffile..etc.update-manager.release-upgrades: 2019-04-02T08:18:06.798258

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/+bug/1822886/+subscriptions