[Bug 2100564] Re: lxd-installer shim fails to install with snapstore error
Denis Lalaj
2100564 at bugs.launchpad.net
Fri Apr 4 18:57:35 UTC 2025
Thank you for the review points @Robie.
Re 1. The proposal for the retry on the installer was brought up with
the Store team and the consensus was that given the use in our pipelines
this change should be fine from the server POV. I will request someone
to formally confirm/sign-off for your reference.
Re 2. This was something I should have considered carefully. With some testing on my end by reproducing the issue through what you suggested in 3 (redirecting api.snapcraft.io in /etc/hosts to simulate the unavailability of the service), I get the following observation:
- The current state of lxd-installer without any retry fails to install the snap after 32.817s.
- With the proposed change, this goes up to 1m43.264s due to the retries and sleeps.
I can comment that on our pipelines, the timeout set for the action that triggers the installation is 'generous' enough at 240s so as to not require further work. The use of this lxd-installer elsewhere however is likely bound by a different timeout. I am of the opinion that such users would be willing to accommodate the retries at the possible benefit of allowing a successful installation where the cost is about 1 minute overhead. Could you suggest what you would prefer to see to make this point as little opinion-based as possible?
Re 3. Thank you for the suggestion, I will update the bug description as
a redirect in /etc/hosts is a much closer reproducer for the API
unavailability.
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to lxd-installer in Ubuntu.
https://bugs.launchpad.net/bugs/2100564
Title:
lxd-installer shim fails to install with snapstore error
Status in lxd-installer package in Ubuntu:
Fix Released
Status in lxd-installer source package in Focal:
New
Status in lxd-installer source package in Jammy:
New
Status in lxd-installer source package in Noble:
Incomplete
Status in lxd-installer source package in Oracular:
Incomplete
Status in lxd-installer source package in Plucky:
Fix Released
Bug description:
[ Impact ]
This has been affecting minimal and base pipelines for cloud images
for both oracular and noble and blocking publication for these images.
The oracular and noble images fail lxd related tests because lxd
installation cannot complete with the following error
`ConnectionResetError: [Errno 104] Connection reset by peer`. This
error is intermittent.
The proposed upload allows retries of the installation to mitigate
availability issues from the snapstore.
[ Test Plan ]
To reproduce the bug you can launch a container from either oracular
or noble like so: `lxc init ubuntu-<SUITE>-daily:<SUITE> test && lxc
start test && lxc exec test bash` where SUITE=oracular or noble. Since
this error is intermittent, this reproducer is also the same.
Upon connecting to the container you can run `lxd init --auto
--storage-backend dir` which results into the error
`ConnectionResetError: [Errno 104] Connection reset by peer`
intermittently. From journalctl lxd logs, the following appears:
`error: cannot install "lxd": Post
"https://api.snapcraft.io/v2/snaps/refresh":`
The package with the proposed changes will allow the retry loop to
attempt installing lxd in the oracular/noble containers in cases where
connection to api.snapcraft.io is not achieved in the first attempt.
[ Where problems could occur ]
The possible effect of this change is posting to snapcraft more
frequently which could increase the traffic to
`https://api.snapcraft.io`. The number of retries is however limited
and the installation failure intermittent so the additional retries
should not have a significant impact on Snapcraft. The package itself
does not have any changes as the retry targets only the installer.
[ Other Info ]
The same change in lxd-installer has been accepted for plucky and is
has been in build pipelines for our plucky images for a few weeks now;
these images do not fail on the lxd-installer anymore.
[Original Description]
Description: Ubuntu Plucky Puffin (development branch) (*LXD container)
Release: 25.04
Image serial: 20250217
Source package: https://launchpad.net/ubuntu/plucky/+package/lxd-installer
Package version:
```
$ apt-cache policy lxd-installer
lxd-installer:
Installed: 10
Candidate: 10
Version table:
*** 10 100
100 /var/lib/dpkg/status
```
Expected behaviour: `lxd init --auto --storage-backend dir` command
would trigger then `lxd-installer` shim and`lxd` would be installed in
the container.
What happened instead:
```
10:19:41 2025-02-17 17:19:41,454 [INFO] test_framework.tests.lxd_start_stop: initializing lxd
10:19:41 Installing LXD snap, please be patient.
10:19:41 Traceback (most recent call last):
10:19:41 File "<string>", line 1, in <module>
10:19:41 import socket; s=socket.socket(socket.AF_UNIX); s.connect("/run/lxd-installer.socket"); s.send(b"x"); s.recv(1)
10:21:12 ~~~~~~^^^
10:21:12 ConnectionResetError: [Errno 104] Connection reset by peer
```
The command that triggers the shim is `lxd init --auto --storage-
backend dir`
Reproducer:
This failure is intermittent, and I have had a hard time getting a reliable reproducer for it. The best that I have got is:
```
lxc init ubuntu-minimal-daily:plucky test #20250217 serial was pulled
lxc start test
lxc exec test bash
<inside the shell>
lxd init --auto --storage-backend dir # confirm with 'y'
```
The error should appear almost instantly. If it does not, stop &
delete the instance and repeat the above.
*LXD team suggested reproducer:
```
$ while lxc launch ubuntu-minimal-daily:plucky test && lxc exec --force-noninteractive test -- lxc --version && lxc delete -f test; do sleep 1; done
```
Logs:
`journalctl --grep lxd`:
```
root at genuine-satyr:~# journalctl --grep lxd
Feb 25 23:35:52 genuine-satyr useradd[324]: add 'ubuntu' to group 'lxd'
Feb 25 23:35:52 genuine-satyr useradd[324]: add 'ubuntu' to shadow group 'lxd'
Feb 25 23:35:53 genuine-satyr systemd[1]: Starting lxd-installer.socket - Helper to install lxd snap on demand...
Feb 25 23:35:53 genuine-satyr systemd[1]: Listening on lxd-installer.socket - Helper to install lxd snap on demand.
Feb 25 23:35:56 genuine-satyr cloud-init[262]: Cloud-init v. 25.1~3geb1965a4-0ubuntu1 finished at Tue, 25 Feb 2025 23:35:56 +0000. Datasource DataSourceLXD. Up 5.51 seconds
Feb 25 23:36:00 genuine-satyr systemd[1]: Created slice system-lxd\x2dinstaller.slice - Slice /system/lxd-installer.
Feb 25 23:36:00 genuine-satyr systemd[1]: Started lxd-installer at 0-503-0.service - Helper to install lxd snap on demand (PID 503/UID 0).
Feb 25 23:36:00 genuine-satyr lxd-installer-service[504]: + [ lxd-installer-service = lxd-installer-service ]
Feb 25 23:36:00 genuine-satyr lxd-installer-service[511]: + lxd_channel
Feb 25 23:36:00 genuine-satyr lxd-installer-service[504]: + snap install lxd --channel=5.21/stable/ubuntu-25.04
Feb 25 23:36:00 genuine-satyr snapd[348]: api_snaps.go:467: Installing snap "lxd" revision unset
Feb 25 23:36:00 genuine-satyr lxd-installer-service[512]: error: cannot install "lxd": Post "https://api.snapcraft.io/v2/snaps/refresh":
Feb 25 23:36:00 genuine-satyr systemd[1]: lxd-installer at 0-503-0.service: Main process exited, code=exited, status=1/FAILURE
Feb 25 23:36:00 genuine-satyr systemd[1]: lxd-installer at 0-503-0.service: Failed with result 'exit-code'.
```
`journalctl` around failure:
```
Feb 25 23:40:12 test systemd[1]: Created slice system-lxd\x2dinstaller.slice - Slice /system/lxd-installer.
Feb 25 23:40:12 test systemd[1]: Started lxd-installer at 0-508-0.service - Helper to install lxd snap on demand (PID 508/UID 0).
Feb 25 23:40:12 test lxd-installer-service[509]: + [ lxd-installer-service = lxd-installer-service ]
Feb 25 23:40:12 test lxd-installer-service[509]: + snap wait system seed.loaded
Feb 25 23:40:12 test lxd-installer-service[516]: + lxd_channel
Feb 25 23:40:12 test lxd-installer-service[516]: + track=
Feb 25 23:40:12 test lxd-installer-service[516]: + [ -r /etc/os-release ]
Feb 25 23:40:12 test lxd-installer-service[516]: + . /etc/os-release
Feb 25 23:40:12 test lxd-installer-service[516]: + PRETTY_NAME=Ubuntu Plucky Puffin (development branch)
Feb 25 23:40:12 test lxd-installer-service[516]: + NAME=Ubuntu
Feb 25 23:40:12 test lxd-installer-service[516]: + VERSION_ID=25.04
Feb 25 23:40:12 test lxd-installer-service[516]: + VERSION=25.04 (Plucky Puffin)
Feb 25 23:40:12 test lxd-installer-service[516]: + VERSION_CODENAME=plucky
Feb 25 23:40:12 test lxd-installer-service[516]: + ID=ubuntu
Feb 25 23:40:12 test lxd-installer-service[516]: + ID_LIKE=debian
Feb 25 23:40:12 test lxd-installer-service[516]: + HOME_URL=https://www.ubuntu.com/
Feb 25 23:40:12 test lxd-installer-service[516]: + SUPPORT_URL=https://help.ubuntu.com/
Feb 25 23:40:12 test lxd-installer-service[516]: + BUG_REPORT_URL=https://bugs.launchpad.net/ubuntu/
Feb 25 23:40:12 test lxd-installer-service[516]: + PRIVACY_POLICY_URL=https://www.ubuntu.com/legal/terms-and-policies/privacy-policy
Feb 25 23:40:12 test lxd-installer-service[516]: + UBUNTU_CODENAME=plucky
Feb 25 23:40:12 test lxd-installer-service[516]: + LOGO=ubuntu-logo
Feb 25 23:40:12 test lxd-installer-service[516]: + track=5.21
Feb 25 23:40:12 test lxd-installer-service[516]: + [ -n 5.21 ]
Feb 25 23:40:12 test lxd-installer-service[516]: + [ -n 25.04 ]
Feb 25 23:40:12 test lxd-installer-service[516]: + echo 5.21/stable/ubuntu-25.04
Feb 25 23:40:12 test lxd-installer-service[509]: + CHANNEL=5.21/stable/ubuntu-25.04
Feb 25 23:40:12 test lxd-installer-service[509]: + [ -z 5.21/stable/ubuntu-25.04 ]
Feb 25 23:40:12 test lxd-installer-service[509]: + snap install lxd --channel=5.21/stable/ubuntu-25.04
Feb 25 23:40:12 test snapd[345]: api_snaps.go:467: Installing snap "lxd" revision unset
Feb 25 23:40:12 test snapd[345]: store_download.go:142: no host system xdelta3 available to use deltas
Feb 25 23:40:12 test lxd-installer-service[517]: error: cannot install "lxd": Post "https://api.snapcraft.io/v2/snaps/refresh":
Feb 25 23:40:12 test lxd-installer-service[517]: context canceled
Feb 25 23:40:12 test systemd[1]: lxd-installer at 0-508-0.service: Main process exited, code=exited, status=1/FAILURE
Feb 25 23:40:12 test systemd[1]: lxd-installer at 0-508-0.service: Failed with result 'exit-code'.
Feb 25 23:40:14 test snapd[345]: overlord.go:518: Released state lock file
Feb 25 23:40:14 test snapd[345]: daemon stop requested to wait for socket activation
Feb 25 23:40:14 test systemd[1]: snapd.service: Deactivated successfully.
Feb 25 23:40:14 test systemd[1]: snapd.service: Consumed 1.014s CPU time, 37.6M memory peak.
```
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/lxd-installer/+bug/2100564/+subscriptions
More information about the foundations-bugs
mailing list