[Bug 1893717] Re: Add Ubuntu Advantage service apt urls to valid mirrors
Chad Smith
1893717 at bugs.launchpad.net
Fri Oct 9 22:22:19 UTC 2020
** Description changed:
=== Begin SRU Template ===
[Impact]
Customers with Ubuntu Advantage services enabled on Trusty/Xenial/Bionic find APT config files disabled with comments after running `do-release-upgrade`.
This requires either:
1. manual intervention to uncomment and correct the apt suite for any enabled Ubuntu Advantage service stored in /etc/apt/sources.list.d/ubuntu-*.list.
OR
2. Providing AllowThirdParty=yes configuration override during do-
release-upgrade to force upgrades of all third party ppa apt
configuration urls.
Adding these supported commercial URLs to mirror.cfg allows these urls to be
treated as valid Ubuntu-proper apt URLs and would automatically update the
apt config files for any enabled Ubuntu Advantage offering without
manual intervention.
[Test Case]
# test procedure
for release in trusty xenial bionic; do
1. lxc launch daily image for the given $release enabling ubuntu-advantage-daily PPA (to allow for upgrading to viable ubuntu-advantage-tools deb
2. attach the machine to ua with a token and enable any available apt-based
services
3. Download the <upgrade-release>-proposed tarball to locally perform a "do-release-upgrade"
4. unzip that proposed tarball
- 5. Add ubuntu-advantage-daily PPA to mirrors.cfg # to avoid disabling on upgrade
- 6. and run ./${upgraderelease} upgrade on the CLI
+ 5. Add ubuntu-advantage-daily PPA to mirrors.cfg # to avoid disabling on upgrade
+ 6. and run ./${upgraderelease} upgrade on the CLI
7. check contents of /etc/apt/sources.list.d/ubuntu-*.list to ensure all
UA-related apt URLs are still available
8. apt-cache policy to check that permissions to said APT repositories are live
done
# test script
```
+
#!/bin/bash
+
+ #
+ # SRU Verification ubuntu-release-upgrader + ubuntu=advantage-tools
+ # Test procedure:
+ # - launch container Trusty, Xenial or Bionic
+ # - Attach container to UA subscription (which activates a number of commerical PPAs
+ # - download and run -proposed ubuntu-release-upgrader tool for upgrade release
+ # - Assert successful upgrade
+ # - Confirm valid mirrors not disabled
+ # - Confirm third party non-commercial PPA URLs still disabled
+ # - Confirm third party UA commercial URLs still disabled
+ # (due to expected feature gap)
+ # - Confirm UA status reports esm-infra still disabled (known feature gap)
+
set -ex
- UA_TOKEN=<REDACTED>
+ UA_TOKEN=$1
+ if [ -z "$1" ]; then
+ echo "Usage: $0 <contractTOKEN>"
+ exit 1
+ fi
+ #apt_sources:
+ # - source: "deb http://ppa.launchpad.net/ua-client/proposed/ubuntu trusty main"
cat > test-uru.yaml <<EOF
#cloud-config
+ package_update: true
package_upgrade: true
apt:
- sources:
- ua.list:
- source: deb http://ppa.launchpad.net/canonical-server/ua-client-daily/ubuntu \$RELEASE main
- keyid: 94E187AD53A59D1847E4880F8A295C4FB8B190B7
+ sources:
+ ua.proposed:
+ source: deb http://ppa.launchpad.net/ua-client/proposed/ubuntu \$RELEASE main
+ keyid: 6E34E7116C0BC933
EOF
+
+ # ua.list:
+ # source: deb http://ppa.launchpad.net/canonical-server/ua-client-daily/ubuntu \$RELEASE main
+ # keyid: 94E187AD53A59D1847E4880F8A295C4FB8B190B7
+
+ cat > checkaptpolicy.sh <<EOF
+ #!/bin/bash
+ set -x
+ RELEASE=\`lsb_release -sc\`
+ echo -n "Current release: $RELEASE"
+ echo "Assert no disabled valid mirrors in /eta/apt/sources.list"
+ ! grep disable /etc/apt/sources.list || echo "FAILURE: found disabled valid mirror urls"
+ echo "Checking commercial Ubuntu Advantage PPAs apt policy and config"
+ apt-cache policy | grep esm.ubuntu.com
+ for file in \`ls /etc/apt/sources.list.d/ubuntu-*.list\`; do
+ echo "--- file: \${file}"
+ cat \${file}
+ done
+ EOF
+ chmod 755 checkaptpolicy.sh
declare -A NEXTDIST=( [bionic]=focal [xenial]=bionic [trusty]=xenial )
for release in bionic; do
- vm=test-sru-$release
- echo "--- Launch cloud-init with ${release}-proposed enabled"
- lxc launch ubuntu-daily:${release} ${vm} -c user.user-data="$(cat proposed.yaml)"
- upgraderelease=${NEXTDIST[$release]}
- echo "--- Wait for cloud-init to finish"
- if [ "${release}" = "trusty" ]; then
- while [ "N 2" != "$(lxc exec ${vm} -- runlevel)" ]; do
- echo "waiting on runlevel 2"
- sleep 5
- done
- else
- lxc exec ${vm} -- cloud-init status --wait --long
- fi
- echo "--- Attach Ubuntu-Advantage and enable services"
- lxc exec ${vm} -- ua attach ${UA_TOKEN}
- lxc exec ${vm} -- ua status
- lxc exec ${vm} -- wget http://archive.ubuntu.com/ubuntu/dists/${upgraderelease}-proposed/main/dist-upgrader-all/current/${upgraderelease}.tar.gz
- lxc exec ${vm} -- tar xzvf ${upgraderelease}.tar.gz
- echo "--- perform do-release-upgrade to next release"
- lxc file pull ${vm}/root/mirrors.cfg .
- echo "http://ppa.launchpad.net/canonical-server/ua-client-daily/ubuntu" >> mirrors.cfg
- lxc file push mirrors.cfg ${vm}/root/
- lxc exec ${vm} -- ./${upgraderelease} --datadir=/root --frontend=DistUpgradeViewNonInteractive
- echo "--- Validate UA APT sources after upgrade"
- lxc exec ${vm} -- apt-cache policy | grep esm.ubuntu.com
- for file in `ls /etc/apt/sources.list.d/ubuntu-*.list`; do
- echo "--- file: ${file}"
- cat /etc/apt/sources.list.d/${file}
- done
- lxc exec ${vm} -- ua status
+ vm=test-sru-$release
+ echo "--- Launch cloud-init with ${release}-proposed enabled"
+ lxc launch ubuntu-daily:${release} ${vm} -c user.user-data="$(cat test-uru.yaml)"
+ upgraderelease=${NEXTDIST[$release]}
+ echo "--- Wait for cloud-init to finish"
+ if [ "${release}" = "trusty" ]; then
+ while [ "N 2" != "$(lxc exec ${vm} -- runlevel)" ]; do
+ echo "waiting on runlevel 2"
+ sleep 5
+ done
+ else
+ lxc exec ${vm} -- cloud-init status --wait --long
+ fi
+ echo "--- Attach Ubuntu-Advantage and enable services"
+ lxc exec ${vm} -- ua attach ${UA_TOKEN}
+ lxc exec ${vm} -- ua status | tee ua-status.orig
+ lxc file push checkaptpolicy.sh ${vm}/
+ lxc exec ${vm} -- /checkaptpolicy.sh > policy.orig
+ lxc exec ${vm} -- wget http://archive.ubuntu.com/ubuntu/dists/$upgraderelease-proposed/main/dist-upgrader-all/current/$upgraderelease.tar.gz
+ lxc exec ${vm} -- tar xzvf $upgraderelease.tar.gz
+ echo "--- Add proposed PPA to valid mirrors to exercise ua-tools do-release-upgrade"
+ lxc file pull ${vm}/root/mirrors.cfg .
+ sed -i 's/stable/proposed/' mirrors.cfg
+ lxc file push mirrors.cfg ${vm}/root/
+ lxc exec ${vm} -- /root/$upgraderelease --datadir=/root --frontend DistUpgradeViewNonInteractive
+ echo "--- Validate UA APT sources after upgrade"
+ lxc exec ${vm} -- /checkaptpolicy.sh > policy.upgrade
+ lxc exec test-sru-bionic grep disable /etc/apt/sources.list && "FAILURE: valid mirrors got disabled" || echo "SUCCESS: no valid mirrors disabled"
+ echo "--- Ensure UA status reports ESM disabled due to pending RT"
+ lxc exec ${vm} -- sudo ua status | egrep 'esm-infra.*disabled' || echo "FAILURE: unexpected enabled esm-infra"
+ echo "--- Expect disable reason to be no Release file in esm PPA"
+ lxc exec test-sru-bionic grep disable /var/log/dist-upgrade/main.log || echo "FAILURE: didn't find disabled update logs for esm"
+ echo "--- Expect disabled esm-infra in diffs from original and upgrade status"
+ diff -urN ua-status.orig ua-status.upgrade
done
```
[Regression Potential]
None; No automatic upgrade support has been previously offered across LTS upgrade paths for ubuntu-advantage services on Ubuntu Trusty or later.
Anyone performing a do-release-upgrade would have had to manually update
apt config files after the fact.
=== Original Description ===
Ensure Apt mirror URLs supported by Ubuntu Advantage services are included as valid mirrors instead of being treated as third party repositories and getting disabled by do-release-upgrade.
The following APT mirror URLs are supported for current and/or imminent
Ubuntu Advantage apt-based services on Xenial and later:
https://esm.ubuntu.com/ubuntu/
https://esm.ubuntu.com/apps/ubuntu/
https://esm.ubuntu.com/cc/ubuntu/
https://esm.ubuntu.com/infra/ubuntu/
https://esm.ubuntu.com/fips/ubuntu/
https://esm.ubuntu.com/fips-updates/ubuntu/
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to ubuntu-release-upgrader in
Ubuntu.
https://bugs.launchpad.net/bugs/1893717
Title:
Add Ubuntu Advantage service apt urls to valid mirrors
Status in ubuntu-release-upgrader package in Ubuntu:
Fix Released
Status in ubuntu-release-upgrader source package in Xenial:
Fix Committed
Status in ubuntu-release-upgrader source package in Bionic:
Fix Committed
Status in ubuntu-release-upgrader source package in Focal:
Fix Committed
Status in ubuntu-release-upgrader source package in Groovy:
Fix Released
Bug description:
=== Begin SRU Template ===
[Impact]
Customers with Ubuntu Advantage services enabled on Trusty/Xenial/Bionic find APT config files disabled with comments after running `do-release-upgrade`.
This requires either:
1. manual intervention to uncomment and correct the apt suite for any enabled Ubuntu Advantage service stored in /etc/apt/sources.list.d/ubuntu-*.list.
OR
2. Providing AllowThirdParty=yes configuration override during do-
release-upgrade to force upgrades of all third party ppa apt
configuration urls.
Adding these supported commercial URLs to mirror.cfg allows these urls to be
treated as valid Ubuntu-proper apt URLs and would automatically update the
apt config files for any enabled Ubuntu Advantage offering without
manual intervention.
[Test Case]
# test procedure
for release in trusty xenial bionic; do
1. lxc launch daily image for the given $release enabling ubuntu-advantage-daily PPA (to allow for upgrading to viable ubuntu-advantage-tools deb
2. attach the machine to ua with a token and enable any available apt-based
services
3. Download the <upgrade-release>-proposed tarball to locally perform a "do-release-upgrade"
4. unzip that proposed tarball
5. Add ubuntu-advantage-daily PPA to mirrors.cfg # to avoid disabling on upgrade
6. and run ./${upgraderelease} upgrade on the CLI
7. check contents of /etc/apt/sources.list.d/ubuntu-*.list to ensure all
UA-related apt URLs are still available
8. apt-cache policy to check that permissions to said APT repositories are live
done
# test script
```
#!/bin/bash
#
# SRU Verification ubuntu-release-upgrader + ubuntu=advantage-tools
# Test procedure:
# - launch container Trusty, Xenial or Bionic
# - Attach container to UA subscription (which activates a number of commerical PPAs
# - download and run -proposed ubuntu-release-upgrader tool for upgrade release
# - Assert successful upgrade
# - Confirm valid mirrors not disabled
# - Confirm third party non-commercial PPA URLs still disabled
# - Confirm third party UA commercial URLs still disabled
# (due to expected feature gap)
# - Confirm UA status reports esm-infra still disabled (known feature gap)
set -ex
UA_TOKEN=$1
if [ -z "$1" ]; then
echo "Usage: $0 <contractTOKEN>"
exit 1
fi
#apt_sources:
# - source: "deb http://ppa.launchpad.net/ua-client/proposed/ubuntu trusty main"
cat > test-uru.yaml <<EOF
#cloud-config
package_update: true
package_upgrade: true
apt:
sources:
ua.proposed:
source: deb http://ppa.launchpad.net/ua-client/proposed/ubuntu \$RELEASE main
keyid: 6E34E7116C0BC933
EOF
# ua.list:
# source: deb http://ppa.launchpad.net/canonical-server/ua-client-daily/ubuntu \$RELEASE main
# keyid: 94E187AD53A59D1847E4880F8A295C4FB8B190B7
cat > checkaptpolicy.sh <<EOF
#!/bin/bash
set -x
RELEASE=\`lsb_release -sc\`
echo -n "Current release: $RELEASE"
echo "Assert no disabled valid mirrors in /eta/apt/sources.list"
! grep disable /etc/apt/sources.list || echo "FAILURE: found disabled valid mirror urls"
echo "Checking commercial Ubuntu Advantage PPAs apt policy and config"
apt-cache policy | grep esm.ubuntu.com
for file in \`ls /etc/apt/sources.list.d/ubuntu-*.list\`; do
echo "--- file: \${file}"
cat \${file}
done
EOF
chmod 755 checkaptpolicy.sh
declare -A NEXTDIST=( [bionic]=focal [xenial]=bionic [trusty]=xenial )
for release in bionic; do
vm=test-sru-$release
echo "--- Launch cloud-init with ${release}-proposed enabled"
lxc launch ubuntu-daily:${release} ${vm} -c user.user-data="$(cat test-uru.yaml)"
upgraderelease=${NEXTDIST[$release]}
echo "--- Wait for cloud-init to finish"
if [ "${release}" = "trusty" ]; then
while [ "N 2" != "$(lxc exec ${vm} -- runlevel)" ]; do
echo "waiting on runlevel 2"
sleep 5
done
else
lxc exec ${vm} -- cloud-init status --wait --long
fi
echo "--- Attach Ubuntu-Advantage and enable services"
lxc exec ${vm} -- ua attach ${UA_TOKEN}
lxc exec ${vm} -- ua status | tee ua-status.orig
lxc file push checkaptpolicy.sh ${vm}/
lxc exec ${vm} -- /checkaptpolicy.sh > policy.orig
lxc exec ${vm} -- wget http://archive.ubuntu.com/ubuntu/dists/$upgraderelease-proposed/main/dist-upgrader-all/current/$upgraderelease.tar.gz
lxc exec ${vm} -- tar xzvf $upgraderelease.tar.gz
echo "--- Add proposed PPA to valid mirrors to exercise ua-tools do-release-upgrade"
lxc file pull ${vm}/root/mirrors.cfg .
sed -i 's/stable/proposed/' mirrors.cfg
lxc file push mirrors.cfg ${vm}/root/
lxc exec ${vm} -- /root/$upgraderelease --datadir=/root --frontend DistUpgradeViewNonInteractive
echo "--- Validate UA APT sources after upgrade"
lxc exec ${vm} -- /checkaptpolicy.sh > policy.upgrade
lxc exec test-sru-bionic grep disable /etc/apt/sources.list && "FAILURE: valid mirrors got disabled" || echo "SUCCESS: no valid mirrors disabled"
echo "--- Ensure UA status reports ESM disabled due to pending RT"
lxc exec ${vm} -- sudo ua status | egrep 'esm-infra.*disabled' || echo "FAILURE: unexpected enabled esm-infra"
echo "--- Expect disable reason to be no Release file in esm PPA"
lxc exec test-sru-bionic grep disable /var/log/dist-upgrade/main.log || echo "FAILURE: didn't find disabled update logs for esm"
echo "--- Expect disabled esm-infra in diffs from original and upgrade status"
diff -urN ua-status.orig ua-status.upgrade
done
```
[Regression Potential]
None; No automatic upgrade support has been previously offered across LTS upgrade paths for ubuntu-advantage services on Ubuntu Trusty or later.
Anyone performing a do-release-upgrade would have had to manually
update apt config files after the fact.
=== Original Description ===
Ensure Apt mirror URLs supported by Ubuntu Advantage services are included as valid mirrors instead of being treated as third party repositories and getting disabled by do-release-upgrade.
The following APT mirror URLs are supported for current and/or
imminent Ubuntu Advantage apt-based services on Xenial and later:
https://esm.ubuntu.com/ubuntu/
https://esm.ubuntu.com/apps/ubuntu/
https://esm.ubuntu.com/cc/ubuntu/
https://esm.ubuntu.com/infra/ubuntu/
https://esm.ubuntu.com/fips/ubuntu/
https://esm.ubuntu.com/fips-updates/ubuntu/
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/+bug/1893717/+subscriptions
More information about the foundations-bugs
mailing list