[Bug 2109495] Re: unattended-upgrades can (and regularly does) pause mid-update indefinitely with no error or feedback

David Buckley 2109495 at bugs.launchpad.net
Tue Jun 10 15:40:10 UTC 2025 

Occurred again.

This time, I got it right and got a trace on everything:

Output is for "thread apply all bt" in gdb for each pid (other than
systemd) in "pstree -sp 42017".

It looks like it's definitely stuck inside a network call.

PROCESS 42017

USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root       42017  0.0  0.0   2800  1956 ?        Ss   Jun02   0:00 /bin/sh /usr/lib/apt/apt.systemd.daily update


[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Download failed: Invalid argument.  Continuing without source file ./posix/../sysdeps/unix/sysv/linux/wait4.c.
0x00007d95a7f107a7 in __GI___wait4 (pid=-1, stat_loc=0x7ffce02c6f6c, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30

Thread 1 (Thread 0x7d95a81ec740 (LWP 42017) "apt.systemd.dai"):
#0  0x00007d95a7f107a7 in __GI___wait4 (pid=-1, stat_loc=0x7ffce02c6f6c, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
#1  0x00005ec3c029621d in ?? ()
#2  0x00005ec3c02964a9 in ?? ()
#3  0x00005ec3c02a25e9 in ?? ()
#4  0x00005ec3c02991b0 in ?? ()
#5  0x00005ec3c02991b0 in ?? ()
#6  0x00005ec3c0299173 in ?? ()
#7  0x00005ec3c02991b0 in ?? ()
#8  0x00005ec3c02a163d in ?? ()
#9  0x00005ec3c0294d7f in ?? ()
#10 0x00007d95a7e2a1ca in __libc_start_call_main (main=main at entry=0x5ec3c0294b20, argc=argc at entry=3, argv=argv at entry=0x7ffce02c7678) at ../sysdeps/nptl/libc_start_call_main.h:58
#11 0x00007d95a7e2a28b in __libc_start_main_impl (main=0x5ec3c0294b20, argc=3, argv=0x7ffce02c7678, init=<optimised out>, fini=<optimised out>, rtld_fini=<optimised out>, stack_end=0x7ffce02c7668) at ../csu/libc-start.c:360
#12 0x00005ec3c0294f35 in ?? ()
[Inferior 1 (process 42017) detached]

PROCESS 42022

USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root       42022  0.0  0.0   2800  1972 ?        S    Jun02   0:00 /bin/sh /usr/lib/apt/apt.systemd.daily lock_is_held update


[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Download failed: Invalid argument.  Continuing without source file ./posix/../sysdeps/unix/sysv/linux/wait4.c.
0x00007a64ed3107a7 in __GI___wait4 (pid=-1, stat_loc=0x7ffd573d4fcc, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30

Thread 1 (Thread 0x7a64ed602740 (LWP 42022) "apt.systemd.dai"):
#0  0x00007a64ed3107a7 in __GI___wait4 (pid=-1, stat_loc=0x7ffd573d4fcc, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
#1  0x00005e13a9a5421d in ?? ()
#2  0x00005e13a9a544a9 in ?? ()
#3  0x00005e13a9a605e9 in ?? ()
#4  0x00005e13a9a571b0 in ?? ()
#5  0x00005e13a9a5da49 in ?? ()
#6  0x00005e13a9a5dddf in ?? ()
#7  0x00005e13a9a6074b in ?? ()
#8  0x00005e13a9a571b0 in ?? ()
#9  0x00005e13a9a5736e in ?? ()
#10 0x00005e13a9a571b0 in ?? ()
#11 0x00005e13a9a571b0 in ?? ()
#12 0x00005e13a9a57173 in ?? ()
#13 0x00005e13a9a57173 in ?? ()
#14 0x00005e13a9a57173 in ?? ()
#15 0x00005e13a9a571b0 in ?? ()
#16 0x00005e13a9a5f63d in ?? ()
#17 0x00005e13a9a52d7f in ?? ()
#18 0x00007a64ed22a1ca in __libc_start_call_main (main=main at entry=0x5e13a9a52b20, argc=argc at entry=4, argv=argv at entry=0x7ffd573d5bc8) at ../sysdeps/nptl/libc_start_call_main.h:58
#19 0x00007a64ed22a28b in __libc_start_main_impl (main=0x5e13a9a52b20, argc=4, argv=0x7ffd573d5bc8, init=<optimised out>, fini=<optimised out>, rtld_fini=<optimised out>, stack_end=0x7ffd573d5bb8) at ../csu/libc-start.c:360
#20 0x00005e13a9a52f35 in ?? ()
[Inferior 1 (process 42022) detached]

PROCESS 42067

USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root       42067  0.0  0.0  21364 11324 ?        S    Jun02   0:00 apt-get -qq -y update


[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Download failed: Invalid argument.  Continuing without source file ./misc/../sysdeps/unix/sysv/linux/select.c.
0x000074010cb26c3e in __GI___select (nfds=12, readfds=0x7ffc979403f0, writefds=0x7ffc97940370, exceptfds=0x0, timeout=0x7ffc97940300) at ../sysdeps/unix/sysv/linux/select.c:69

Thread 1 (Thread 0x74010d08c880 (LWP 42067) "apt-get"):
#0  0x000074010cb26c3e in __GI___select (nfds=12, readfds=0x7ffc979403f0, writefds=0x7ffc97940370, exceptfds=0x0, timeout=0x7ffc97940300) at ../sysdeps/unix/sysv/linux/select.c:69
#1  0x000074010d233f27 in pkgAcquire::Run(int) () from /lib/x86_64-linux-gnu/libapt-pkg.so.6.0
#2  0x000074010d320a55 in AcquireUpdate(pkgAcquire&, int, bool, bool) () from /lib/x86_64-linux-gnu/libapt-pkg.so.6.0
#3  0x000074010d320e76 in ListUpdate(pkgAcquireStatus&, pkgSourceList&, int) () from /lib/x86_64-linux-gnu/libapt-pkg.so.6.0
#4  0x000074010d3d8385 in DoUpdate() () from /lib/x86_64-linux-gnu/libapt-private.so.0.0
#5  0x000074010d26626f in CommandLine::DispatchArg(CommandLine::Dispatch const*, bool) () from /lib/x86_64-linux-gnu/libapt-pkg.so.6.0
#6  0x000074010d3a5396 in DispatchCommandLine(CommandLine&, std::vector<CommandLine::Dispatch, std::allocator<CommandLine::Dispatch> > const&) () from /lib/x86_64-linux-gnu/libapt-private.so.0.0
#7  0x000058456934a13e in ?? ()
#8  0x000074010ca2a1ca in __libc_start_call_main (main=main at entry=0x58456934a0a0, argc=argc at entry=4, argv=argv at entry=0x7ffc97940e78) at ../sysdeps/nptl/libc_start_call_main.h:58
#9  0x000074010ca2a28b in __libc_start_main_impl (main=0x58456934a0a0, argc=4, argv=0x7ffc97940e78, init=<optimised out>, fini=<optimised out>, rtld_fini=<optimised out>, stack_end=0x7ffc97940e68) at ../csu/libc-start.c:360
#10 0x000058456934a1d5 in ?? ()
[Inferior 1 (process 42067) detached]

PROCESS 42117

USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
_apt       42117  0.0  0.0  18292  7184 ?        S    Jun02   0:00 /usr/lib/apt/methods/gpgv


[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Download failed: Invalid argument.  Continuing without source file ./misc/../sysdeps/unix/sysv/linux/select.c.
0x0000798dc8926c3e in __GI___select (nfds=1, readfds=0x7ffc549f01e0, writefds=0x0, exceptfds=0x0, timeout=0x0) at ../sysdeps/unix/sysv/linux/select.c:69

Thread 1 (Thread 0x798dc8e81880 (LWP 42117) "gpgv"):
#0  0x0000798dc8926c3e in __GI___select (nfds=1, readfds=0x7ffc549f01e0, writefds=0x0, exceptfds=0x0, timeout=0x0) at ../sysdeps/unix/sysv/linux/select.c:69
#1  0x0000798dc8fae38c in WaitFd(int, bool, unsigned long) () from /lib/x86_64-linux-gnu/libapt-pkg.so.6.0
#2  0x0000798dc8f630f3 in pkgAcqMethod::Run(bool) () from /lib/x86_64-linux-gnu/libapt-pkg.so.6.0
#3  0x0000578bf5ecc9fe in ?? ()
#4  0x0000798dc882a1ca in __libc_start_call_main (main=main at entry=0x578bf5ecc8b0, argc=argc at entry=1, argv=argv at entry=0x7ffc549f06f8) at ../sysdeps/nptl/libc_start_call_main.h:58
#5  0x0000798dc882a28b in __libc_start_main_impl (main=0x578bf5ecc8b0, argc=1, argv=0x7ffc549f06f8, init=<optimised out>, fini=<optimised out>, rtld_fini=<optimised out>, stack_end=0x7ffc549f06e8) at ../csu/libc-start.c:360
#6  0x0000578bf5eccb85 in ?? ()
[Inferior 1 (process 42117) detached]

PROCESS 42077

USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
_apt       42077  0.0  0.0  25460 10960 ?        S    Jun02   0:00 /usr/lib/apt/methods/http


[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Download failed: Invalid argument.  Continuing without source file ./misc/../sysdeps/unix/sysv/linux/select.c.
0x000074c196326c3e in __GI___select (nfds=1, readfds=0x7ffe58e6c540, writefds=0x0, exceptfds=0x0, timeout=0x0) at ../sysdeps/unix/sysv/linux/select.c:69

Thread 1 (Thread 0x74c19687fa80 (LWP 42077) "http"):
#0  0x000074c196326c3e in __GI___select (nfds=1, readfds=0x7ffe58e6c540, writefds=0x0, exceptfds=0x0, timeout=0x0) at ../sysdeps/unix/sysv/linux/select.c:69
#1  0x000074c196d6a38c in WaitFd(int, bool, unsigned long) () from /lib/x86_64-linux-gnu/libapt-pkg.so.6.0
#2  0x00005a376b2f5c97 in ?? ()
#3  0x000074c19622a1ca in __libc_start_call_main (main=main at entry=0x5a376b2f5880, argc=argc at entry=1, argv=argv at entry=0x7ffe58e6cd08) at ../sysdeps/nptl/libc_start_call_main.h:58
#4  0x000074c19622a28b in __libc_start_main_impl (main=0x5a376b2f5880, argc=1, argv=0x7ffe58e6cd08, init=<optimised out>, fini=<optimised out>, rtld_fini=<optimised out>, stack_end=0x7ffe58e6ccf8) at ../csu/libc-start.c:360
#5  0x00005a376b2f8485 in ?? ()
[Inferior 1 (process 42077) detached]

PROCESS 42078

USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
_apt       42078  0.0  0.0  25460 10964 ?        S    Jun02   0:00 /usr/lib/apt/methods/http


[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Download failed: Invalid argument.  Continuing without source file ./misc/../sysdeps/unix/sysv/linux/select.c.
0x0000762446d26c3e in __GI___select (nfds=1, readfds=0x7ffcf2cd0750, writefds=0x0, exceptfds=0x0, timeout=0x0) at ../sysdeps/unix/sysv/linux/select.c:69

Thread 1 (Thread 0x7624465afa80 (LWP 42078) "http"):
#0  0x0000762446d26c3e in __GI___select (nfds=1, readfds=0x7ffcf2cd0750, writefds=0x0, exceptfds=0x0, timeout=0x0) at ../sysdeps/unix/sysv/linux/select.c:69
#1  0x00007624476e938c in WaitFd(int, bool, unsigned long) () from /lib/x86_64-linux-gnu/libapt-pkg.so.6.0
#2  0x000055a03c457c97 in ?? ()
#3  0x0000762446c2a1ca in __libc_start_call_main (main=main at entry=0x55a03c457880, argc=argc at entry=1, argv=argv at entry=0x7ffcf2cd0f18) at ../sysdeps/nptl/libc_start_call_main.h:58
#4  0x0000762446c2a28b in __libc_start_main_impl (main=0x55a03c457880, argc=1, argv=0x7ffcf2cd0f18, init=<optimised out>, fini=<optimised out>, rtld_fini=<optimised out>, stack_end=0x7ffcf2cd0f08) at ../csu/libc-start.c:360
#5  0x000055a03c45a485 in ?? ()
[Inferior 1 (process 42078) detached]

PROCESS 42079

USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
_apt       42079  0.0  0.1  29496 16348 ?        S    Jun02   0:00 /usr/lib/apt/methods/https


[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Download failed: Invalid argument.  Continuing without source file ./misc/../sysdeps/unix/sysv/linux/select.c.
0x0000706503d26c3e in __GI___select (nfds=1, readfds=0x7ffd2b268600, writefds=0x0, exceptfds=0x0, timeout=0x0) at ../sysdeps/unix/sysv/linux/select.c:69

Thread 1 (Thread 0x706503e19a80 (LWP 42079) "https"):
#0  0x0000706503d26c3e in __GI___select (nfds=1, readfds=0x7ffd2b268600, writefds=0x0, exceptfds=0x0, timeout=0x0) at ../sysdeps/unix/sysv/linux/select.c:69
#1  0x000070650476138c in WaitFd(int, bool, unsigned long) () from /lib/x86_64-linux-gnu/libapt-pkg.so.6.0
#2  0x000059cb8a464c97 in ?? ()
#3  0x0000706503c2a1ca in __libc_start_call_main (main=main at entry=0x59cb8a464880, argc=argc at entry=1, argv=argv at entry=0x7ffd2b268dc8) at ../sysdeps/nptl/libc_start_call_main.h:58
#4  0x0000706503c2a28b in __libc_start_main_impl (main=0x59cb8a464880, argc=1, argv=0x7ffd2b268dc8, init=<optimised out>, fini=<optimised out>, rtld_fini=<optimised out>, stack_end=0x7ffd2b268db8) at ../csu/libc-start.c:360
#5  0x000059cb8a467485 in ?? ()
[Inferior 1 (process 42079) detached]

PROCESS 42782

USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
_apt       42782  0.0  0.0  26792 15120 ?        S    Jun02   0:00 /usr/lib/apt/methods/store


[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Download failed: Invalid argument.  Continuing without source file ./misc/../sysdeps/unix/sysv/linux/select.c.
0x00007cfffa926c3e in __GI___select (nfds=1, readfds=0x7ffdc639fa00, writefds=0x0, exceptfds=0x0, timeout=0x0) at ../sysdeps/unix/sysv/linux/select.c:69

Thread 1 (Thread 0x7cfffaa32880 (LWP 42782) "store"):
#0  0x00007cfffa926c3e in __GI___select (nfds=1, readfds=0x7ffdc639fa00, writefds=0x0, exceptfds=0x0, timeout=0x0) at ../sysdeps/unix/sysv/linux/select.c:69
#1  0x00007cfffb15d38c in WaitFd(int, bool, unsigned long) () from /lib/x86_64-linux-gnu/libapt-pkg.so.6.0
#2  0x00007cfffb1120f3 in pkgAcqMethod::Run(bool) () from /lib/x86_64-linux-gnu/libapt-pkg.so.6.0
#3  0x000057952d30a296 in ?? ()
#4  0x00007cfffa82a1ca in __libc_start_call_main (main=main at entry=0x57952d30a010, argc=argc at entry=1, argv=argv at entry=0x7ffdc639ff58) at ../sysdeps/nptl/libc_start_call_main.h:58
#5  0x00007cfffa82a28b in __libc_start_main_impl (main=0x57952d30a010, argc=1, argv=0x7ffdc639ff58, init=<optimised out>, fini=<optimised out>, rtld_fini=<optimised out>, stack_end=0x7ffdc639ff48) at ../csu/libc-start.c:360
#6  0x000057952d30a705 in ?? ()
[Inferior 1 (process 42782) detached]

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to apt in Ubuntu.
https://bugs.launchpad.net/bugs/2109495

Title:
  unattended-upgrades can (and regularly does) pause mid-update
  indefinitely with no error or feedback

Status in apt package in Ubuntu:
  New

Bug description:
  unattended-upgrades can pause, probably forever, mid-upgrade. This
  prevents any further upgrades until the service is restarted.

  The direct cause is probably inside apt rather than the upgrade
  process itself, but the upgrade process has apparently no defence
  against apt hanging, either. An interactive user would see their
  command hang forever and would likely interrupt it, so the issue in
  apt is not a security problem (probably -- if it were missed, I guess
  it would hold the lock and prevent upgrades, too).

  I've observed this on both Debian and Ubuntu, on systems which suspend
  often, so I assume the issue is an infinite timeout somewhere in the
  networking code being hit because of a suspend operation occurring
  mid-update. It's hard to know how common it is, as it is not directly
  visible (see below).

  In this instance, the observed state was like this:

  $ ps aux|grep apt
  root      158770  0.0  0.0   2800  1536 ?        Ss   Mar24   0:00 /bin/sh /usr/lib/apt/apt.systemd.daily update
  root      158812  0.0  0.0   2800  1584 ?        S    Mar24   0:00 /bin/sh /usr/lib/apt/apt.systemd.daily lock_is_held update
  root      158857  0.0  0.0  22232  9876 ?        S    Mar24   2:00 apt-get -qq -y update
  _apt      158867  0.0  0.0  25440  7980 ?        S    Mar24   0:00 /usr/lib/apt/methods/http
  _apt      158868  0.0  0.0  25440  8072 ?        S    Mar24   0:00 /usr/lib/apt/methods/http
  _apt      158869  0.0  0.0  29468 12396 ?        S    Mar24   0:00 /usr/lib/apt/methods/https
  _apt      158870  0.0  0.0  29408 12424 ?        S    Mar24   0:00 /usr/lib/apt/methods/https
  _apt      159069  0.0  0.0  18280  5564 ?        S    Mar24   0:00 /usr/lib/apt/methods/gpgv
  _apt      159622  0.0  0.0  26792  6456 ?        S    Mar24   0:00 /usr/lib/apt/methods/store

  Note the date: I'd not had updates performed for over a month. I use
  the machine daily, but mostly suspend it overnight, so it's had
  multiple days of awake time since then, too. I recognise
  "apt/methods/http", so I think this is the common failure mode.

  There's no clues in the log files (they appear to most recently show a
  normal no-upgrades cycle on the 22nd).

  There are no errors reported to me as a user. Note that though I'm not
  on the default desktop environment, so it's possible the error
  reporting is broken due to me not starting a service which would have
  reported it, I do get popups telling me to restart applications when
  updates do run. The only feedback I have from the system is the
  suspicious absence of these requests to restart, and inability to run
  apt manually if I try. Error in this case:

  $ sudo apt update && sudo apt full-upgrade
  Reading package lists... Done
  E: Could not get lock /var/lib/apt/lists/lock. It is held by process 158857 (apt-get)
  N: Be aware that removing the lock file is not a solution and may break your system.
  E: Unable to lock directory /var/lib/apt/lists/

  Killing some part of the tree usually wakes it back up again. I
  usually find the issue because I want to change packages or force an
  upgrade pre-reboot; this also means "fixing" the upgrade task prevents
  what I want to do as it the immediately starts doing upgrades, so I
  usually just stop the service fully and start again either manually or
  after reboot.

  * Probably the service ought to set a robust, real time, timeout on the entire upgrade process, or at least on the networking part of it. Even a 24h timeout would be sufficient to mitigate most of the damage.
  * Having a timeout in the apt code would probably be useful, too.
  * Having some feedback from unattended-upgrades when it has failed to run due to lock contention many times in a row would be good, too.

  ProblemType: Bug
  DistroRelease: Ubuntu 24.04
  Package: unattended-upgrades 2.9.1+nmu4ubuntu1
  ProcVersionSignature: Ubuntu 6.8.0-52.53-generic 6.8.12
  Uname: Linux 6.8.0-52-generic x86_64
  NonfreeKernelModules: zfs
  ApportVersion: 2.28.1-0ubuntu3.5
  Architecture: amd64
  CasperMD5CheckResult: pass
  Date: Mon Apr 28 11:00:20 2025
  InstallationDate: Installed on 2025-01-17 (101 days ago)
  InstallationMedia: Ubuntu 24.04.1 LTS "Noble Numbat" - Release amd64 (20240827.1)
  PackageArchitecture: all
  RebootRequiredPkgs: Error: path contained symlinks.
  SourcePackage: unattended-upgrades
  UpgradeStatus: No upgrade log present (probably fresh install)

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/apt/+bug/2109495/+subscriptions

More information about the foundations-bugs mailing list