[Bug 2100564] Re: lxd-installer shim fails to install with snapstore error
Robie Basak
2100564 at bugs.launchpad.net
Wed Apr 23 13:17:49 UTC 2025
Thank you for the further analysis! This is very useful.
> The use of this lxd-installer elsewhere however is likely bound by a
different timeout. I am of the opinion that such users would be willing
to accommodate the retries at the possible benefit of allowing a
successful installation where the cost is about 1 minute overhead. Could
you suggest what you would prefer to see to make this point as little
opinion-based as possible?
I'm not sure it's possible to make this any less subjective. My feeling
is that it's regrettable to have to make this change at all, but I lean
towards it being OK on balance, given the numbers from your analysis.
*However*:
> Snapd itself already retries requests when the Store is unavailable or
errors. Snapd folks have identified a bug (tracked internally) occurring
when certain timing conditions are met on systems with no snaps
installed and a snap install is attempted when snapd goes into standby.
With this revelation, it sounds like we can fix the root cause in snapd
instead, and then behaviour would not change in any other scenario,
unlike this change (that we now better understand is only a workaround)
that does have to adjust the effective timeout.
With the knowledge that we now have, wouldn't this be preferable?
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to lxd-installer in Ubuntu.
https://bugs.launchpad.net/bugs/2100564
Title:
lxd-installer shim fails to install with snapstore error
Status in lxd-installer package in Ubuntu:
Fix Released
Status in lxd-installer source package in Focal:
New
Status in lxd-installer source package in Jammy:
New
Status in lxd-installer source package in Noble:
Incomplete
Status in lxd-installer source package in Oracular:
Incomplete
Status in lxd-installer source package in Plucky:
Fix Released
Bug description:
[ Impact ]
This has been affecting minimal and base pipelines for cloud images
for both oracular and noble and blocking publication for these images.
The oracular and noble images fail lxd related tests because lxd
installation cannot complete with the following error
`ConnectionResetError: [Errno 104] Connection reset by peer`. This
error is intermittent.
The proposed upload allows retries of the installation to mitigate
installation failures that stem from snapd.
[ Test Plan ]
To reproduce the bug you can launch a container from either oracular
or noble like so: `lxc init ubuntu-minimal-daily:<SUITE> test && lxc
start test && lxc exec test bash` where
SUITE=oracular/noble/jammy/focal.
The error is intermittent and appears to be tied to snapd timing when
going to standby and the boot time of the target instance being
tested. To reproduce locally, on the shell connecting to the container
launched above, you can run the following: `systemctl start snapd ;
sleep 5 ; lxd init --auto` which produces the following
`ConnectionResetError: [Errno 104] Connection reset by peer` error.
The package with the proposed change allows the retry loop to attempt
installation again after the time window that causes the installation
failure so this will mitigate the installation errors in our
pipelines.
[ Where problems could occur ]
The possible effect of this change is posting to snapcraft more
frequently which could increase the traffic to
`https://api.snapcraft.io`. A discussion with Store follows with a
confirmation that this possible increase in the number of requests is
fine from the Store POV.
[ Other Info ]
The same change in lxd-installer has been accepted for plucky and is
has been in build pipelines for our plucky images for a few weeks now;
these images do not fail on the lxd-installer anymore.
[Original Description]
Description: Ubuntu Plucky Puffin (development branch) (*LXD container)
Release: 25.04
Image serial: 20250217
Source package: https://launchpad.net/ubuntu/plucky/+package/lxd-installer
Package version:
```
$ apt-cache policy lxd-installer
lxd-installer:
Installed: 10
Candidate: 10
Version table:
*** 10 100
100 /var/lib/dpkg/status
```
Expected behaviour: `lxd init --auto --storage-backend dir` command
would trigger then `lxd-installer` shim and`lxd` would be installed in
the container.
What happened instead:
```
10:19:41 2025-02-17 17:19:41,454 [INFO] test_framework.tests.lxd_start_stop: initializing lxd
10:19:41 Installing LXD snap, please be patient.
10:19:41 Traceback (most recent call last):
10:19:41 File "<string>", line 1, in <module>
10:19:41 import socket; s=socket.socket(socket.AF_UNIX); s.connect("/run/lxd-installer.socket"); s.send(b"x"); s.recv(1)
10:21:12 ~~~~~~^^^
10:21:12 ConnectionResetError: [Errno 104] Connection reset by peer
```
The command that triggers the shim is `lxd init --auto --storage-
backend dir`
Reproducer:
This failure is intermittent, and I have had a hard time getting a reliable reproducer for it. The best that I have got is:
```
lxc init ubuntu-minimal-daily:plucky test #20250217 serial was pulled
lxc start test
lxc exec test bash
<inside the shell>
lxd init --auto --storage-backend dir # confirm with 'y'
```
The error should appear almost instantly. If it does not, stop &
delete the instance and repeat the above.
*LXD team suggested reproducer:
```
$ while lxc launch ubuntu-minimal-daily:plucky test && lxc exec --force-noninteractive test -- lxc --version && lxc delete -f test; do sleep 1; done
```
Logs:
`journalctl --grep lxd`:
```
root at genuine-satyr:~# journalctl --grep lxd
Feb 25 23:35:52 genuine-satyr useradd[324]: add 'ubuntu' to group 'lxd'
Feb 25 23:35:52 genuine-satyr useradd[324]: add 'ubuntu' to shadow group 'lxd'
Feb 25 23:35:53 genuine-satyr systemd[1]: Starting lxd-installer.socket - Helper to install lxd snap on demand...
Feb 25 23:35:53 genuine-satyr systemd[1]: Listening on lxd-installer.socket - Helper to install lxd snap on demand.
Feb 25 23:35:56 genuine-satyr cloud-init[262]: Cloud-init v. 25.1~3geb1965a4-0ubuntu1 finished at Tue, 25 Feb 2025 23:35:56 +0000. Datasource DataSourceLXD. Up 5.51 seconds
Feb 25 23:36:00 genuine-satyr systemd[1]: Created slice system-lxd\x2dinstaller.slice - Slice /system/lxd-installer.
Feb 25 23:36:00 genuine-satyr systemd[1]: Started lxd-installer at 0-503-0.service - Helper to install lxd snap on demand (PID 503/UID 0).
Feb 25 23:36:00 genuine-satyr lxd-installer-service[504]: + [ lxd-installer-service = lxd-installer-service ]
Feb 25 23:36:00 genuine-satyr lxd-installer-service[511]: + lxd_channel
Feb 25 23:36:00 genuine-satyr lxd-installer-service[504]: + snap install lxd --channel=5.21/stable/ubuntu-25.04
Feb 25 23:36:00 genuine-satyr snapd[348]: api_snaps.go:467: Installing snap "lxd" revision unset
Feb 25 23:36:00 genuine-satyr lxd-installer-service[512]: error: cannot install "lxd": Post "https://api.snapcraft.io/v2/snaps/refresh":
Feb 25 23:36:00 genuine-satyr systemd[1]: lxd-installer at 0-503-0.service: Main process exited, code=exited, status=1/FAILURE
Feb 25 23:36:00 genuine-satyr systemd[1]: lxd-installer at 0-503-0.service: Failed with result 'exit-code'.
```
`journalctl` around failure:
```
Feb 25 23:40:12 test systemd[1]: Created slice system-lxd\x2dinstaller.slice - Slice /system/lxd-installer.
Feb 25 23:40:12 test systemd[1]: Started lxd-installer at 0-508-0.service - Helper to install lxd snap on demand (PID 508/UID 0).
Feb 25 23:40:12 test lxd-installer-service[509]: + [ lxd-installer-service = lxd-installer-service ]
Feb 25 23:40:12 test lxd-installer-service[509]: + snap wait system seed.loaded
Feb 25 23:40:12 test lxd-installer-service[516]: + lxd_channel
Feb 25 23:40:12 test lxd-installer-service[516]: + track=
Feb 25 23:40:12 test lxd-installer-service[516]: + [ -r /etc/os-release ]
Feb 25 23:40:12 test lxd-installer-service[516]: + . /etc/os-release
Feb 25 23:40:12 test lxd-installer-service[516]: + PRETTY_NAME=Ubuntu Plucky Puffin (development branch)
Feb 25 23:40:12 test lxd-installer-service[516]: + NAME=Ubuntu
Feb 25 23:40:12 test lxd-installer-service[516]: + VERSION_ID=25.04
Feb 25 23:40:12 test lxd-installer-service[516]: + VERSION=25.04 (Plucky Puffin)
Feb 25 23:40:12 test lxd-installer-service[516]: + VERSION_CODENAME=plucky
Feb 25 23:40:12 test lxd-installer-service[516]: + ID=ubuntu
Feb 25 23:40:12 test lxd-installer-service[516]: + ID_LIKE=debian
Feb 25 23:40:12 test lxd-installer-service[516]: + HOME_URL=https://www.ubuntu.com/
Feb 25 23:40:12 test lxd-installer-service[516]: + SUPPORT_URL=https://help.ubuntu.com/
Feb 25 23:40:12 test lxd-installer-service[516]: + BUG_REPORT_URL=https://bugs.launchpad.net/ubuntu/
Feb 25 23:40:12 test lxd-installer-service[516]: + PRIVACY_POLICY_URL=https://www.ubuntu.com/legal/terms-and-policies/privacy-policy
Feb 25 23:40:12 test lxd-installer-service[516]: + UBUNTU_CODENAME=plucky
Feb 25 23:40:12 test lxd-installer-service[516]: + LOGO=ubuntu-logo
Feb 25 23:40:12 test lxd-installer-service[516]: + track=5.21
Feb 25 23:40:12 test lxd-installer-service[516]: + [ -n 5.21 ]
Feb 25 23:40:12 test lxd-installer-service[516]: + [ -n 25.04 ]
Feb 25 23:40:12 test lxd-installer-service[516]: + echo 5.21/stable/ubuntu-25.04
Feb 25 23:40:12 test lxd-installer-service[509]: + CHANNEL=5.21/stable/ubuntu-25.04
Feb 25 23:40:12 test lxd-installer-service[509]: + [ -z 5.21/stable/ubuntu-25.04 ]
Feb 25 23:40:12 test lxd-installer-service[509]: + snap install lxd --channel=5.21/stable/ubuntu-25.04
Feb 25 23:40:12 test snapd[345]: api_snaps.go:467: Installing snap "lxd" revision unset
Feb 25 23:40:12 test snapd[345]: store_download.go:142: no host system xdelta3 available to use deltas
Feb 25 23:40:12 test lxd-installer-service[517]: error: cannot install "lxd": Post "https://api.snapcraft.io/v2/snaps/refresh":
Feb 25 23:40:12 test lxd-installer-service[517]: context canceled
Feb 25 23:40:12 test systemd[1]: lxd-installer at 0-508-0.service: Main process exited, code=exited, status=1/FAILURE
Feb 25 23:40:12 test systemd[1]: lxd-installer at 0-508-0.service: Failed with result 'exit-code'.
Feb 25 23:40:14 test snapd[345]: overlord.go:518: Released state lock file
Feb 25 23:40:14 test snapd[345]: daemon stop requested to wait for socket activation
Feb 25 23:40:14 test systemd[1]: snapd.service: Deactivated successfully.
Feb 25 23:40:14 test systemd[1]: snapd.service: Consumed 1.014s CPU time, 37.6M memory peak.
```
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/lxd-installer/+bug/2100564/+subscriptions
More information about the foundations-bugs
mailing list