[Bug 2083489] [NEW] Upgrading dbus tear down network
Giuseppe Petralia
2083489 at bugs.launchpad.net
Wed Oct 2 08:39:20 UTC 2024
Public bug reported:
While running a dist-ugprade on Jammy machines part of a Ceph cluster,
dbus got updated
```
2024-09-25 04:35:25 status triggers-pending dbus:amd64 1.12.20-2ubuntu4.1
2024-09-25 04:35:36 status triggers-pending dbus:amd64 1.12.20-2ubuntu4.1
2024-09-25 04:37:28 trigproc dbus:amd64 1.12.20-2ubuntu4.1 <none>
2024-09-25 04:37:28 status half-configured dbus:amd64 1.12.20-2ubuntu4.1
2024-09-25 04:37:28 status installed dbus:amd64 1.12.20-2ubuntu4.1
```
This I believe trigger a systemd-networkd restart
```
Sep 25 04:36:35 ********* systemd[1]: Started Network Configuration.
```
Which causes interfaces to go down and up.
```
Sep 25 04:36:35 ********* systemd-networkd[605770]: bond0: netdev exists, using existing without changing its parameters
Sep 25 04:36:35 ********* systemd-networkd[605770]: bond1: netdev exists, using existing without changing its parameters
Sep 25 04:36:35 ********* systemd-networkd[605770]: br-bond0-3486: netdev exists, using existing without changing its parameters
Sep 25 04:36:35 ********* systemd-networkd[605770]: br-bond1: netdev exists, using existing without changing its parameters
Sep 25 04:36:35 ********* systemd-networkd[605770]: br-enp68s0f0: netdev exists, using existing without changing its parameters
Sep 25 04:36:37 ********* systemd-networkd[605770]: enp129s0f1np1: Link DOWN
Sep 25 04:36:37 ********* systemd-networkd[605770]: enp129s0f1np1: Lost carrier
Sep 25 04:36:37 ********* systemd-networkd[605770]: enp129s0f0np0: Link DOWN
Sep 25 04:36:37 ********* systemd-networkd[605770]: enp129s0f0np0: Lost carrier
Sep 25 04:36:37 ********* systemd-networkd[605770]: bond1: Lost carrier
Sep 25 04:36:37 ********* systemd-networkd[605770]: enp161s0f1np1: Link DOWN
Sep 25 04:36:37 ********* systemd-networkd[605770]: enp161s0f1np1: Lost carrier
Sep 25 04:36:37 ********* systemd-networkd[605770]: enp161s0f0np0: Link DOWN
Sep 25 04:36:37 ********* systemd-networkd[605770]: enp161s0f0np0: Lost carrier
Sep 25 04:36:37 ********* systemd-networkd[605770]: bond0: Lost carrier
Sep 25 04:36:37 ********* systemd-networkd[605770]: bond0.3484: Lost carrier
Sep 25 04:36:37 ********* systemd-networkd[605770]: bond0.3486: Lost carrier
Sep 25 04:36:41 ********* systemd-networkd[605770]: enp129s0f1np1: Link UP
Sep 25 04:36:42 ********* systemd-networkd[605770]: enp129s0f1np1: Gained carrier
Sep 25 04:36:42 ********* systemd-networkd[605770]: enp129s0f0np0: Link UP
Sep 25 04:36:42 ********* systemd-networkd[605770]: enp129s0f0np0: Gained carrier
Sep 25 04:36:42 ********* systemd-networkd[605770]: enp161s0f1np1: Link UP
Sep 25 04:36:42 ********* systemd-networkd[605770]: enp161s0f0np0: Link UP
Sep 25 04:36:42 ********* systemd-networkd[605770]: enp161s0f1np1: Gained carrier
Sep 25 04:36:42 ********* systemd-networkd[605770]: enp161s0f0np0: Gained carrier
Sep 25 04:36:42 ********* systemd-networkd[605770]: bond0: Gained carrier
Sep 25 04:36:42 ********* systemd-networkd[605770]: bond0.3484: Gained carrier
Sep 25 04:36:42 ********* systemd-networkd[605770]: bond0.3486: Gained carrier
Sep 25 04:36:42 ********* systemd-networkd[605770]: bond1: Gained carrier
```
```
Sep 25 04:36:35 ********* kernel: [18085928.795785] bond1: (slave enp129s0f1np1): link status definitely down, disabling slave
Sep 25 04:36:36 ********* kernel: [18085929.535826] bond1: (slave enp129s0f0np0): link status definitely down, disabling slave
Sep 25 04:36:37 ********* kernel: [18085929.919702] bond0: (slave enp161s0f1np1): link status definitely down, disabling slave
Sep 25 04:36:37 ********* kernel: [18085930.339709] bond0: (slave enp161s0f0np0): link status definitely down, disabling slave
Sep 25 04:36:41 ********* kernel: [18085934.812301] bond0: (slave enp161s0f1np1): link status definitely up, 25000 Mbps full duplex
Sep 25 04:36:41 ********* kernel: [18085934.812968] bond0: (slave enp161s0f0np0): link status definitely up, 25000 Mbps full duplex
Sep 25 04:36:42 ********* kernel: [18085934.828048] bond1: (slave enp129s0f1np1): link status definitely up, 100000 Mbps full duplex
Sep 25 04:36:42 ********* kernel: [18085934.828491] bond1: (slave enp129s0f0np0): link status definitely up, 100000 Mbps full duplex
```
This caused a big outage on the Ceph cluster where many placement groups
went unavailable.
It will be good if the dbus upgrade can avoid to restart the dbus
service, to avoid the consequent impact to networking, and leave that to
the operator to do it at its best convenience.
** Affects: dbus (Ubuntu)
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to dbus in Ubuntu.
https://bugs.launchpad.net/bugs/2083489
Title:
Upgrading dbus tear down network
Status in dbus package in Ubuntu:
New
Bug description:
While running a dist-ugprade on Jammy machines part of a Ceph cluster,
dbus got updated
```
2024-09-25 04:35:25 status triggers-pending dbus:amd64 1.12.20-2ubuntu4.1
2024-09-25 04:35:36 status triggers-pending dbus:amd64 1.12.20-2ubuntu4.1
2024-09-25 04:37:28 trigproc dbus:amd64 1.12.20-2ubuntu4.1 <none>
2024-09-25 04:37:28 status half-configured dbus:amd64 1.12.20-2ubuntu4.1
2024-09-25 04:37:28 status installed dbus:amd64 1.12.20-2ubuntu4.1
```
This I believe trigger a systemd-networkd restart
```
Sep 25 04:36:35 ********* systemd[1]: Started Network Configuration.
```
Which causes interfaces to go down and up.
```
Sep 25 04:36:35 ********* systemd-networkd[605770]: bond0: netdev exists, using existing without changing its parameters
Sep 25 04:36:35 ********* systemd-networkd[605770]: bond1: netdev exists, using existing without changing its parameters
Sep 25 04:36:35 ********* systemd-networkd[605770]: br-bond0-3486: netdev exists, using existing without changing its parameters
Sep 25 04:36:35 ********* systemd-networkd[605770]: br-bond1: netdev exists, using existing without changing its parameters
Sep 25 04:36:35 ********* systemd-networkd[605770]: br-enp68s0f0: netdev exists, using existing without changing its parameters
Sep 25 04:36:37 ********* systemd-networkd[605770]: enp129s0f1np1: Link DOWN
Sep 25 04:36:37 ********* systemd-networkd[605770]: enp129s0f1np1: Lost carrier
Sep 25 04:36:37 ********* systemd-networkd[605770]: enp129s0f0np0: Link DOWN
Sep 25 04:36:37 ********* systemd-networkd[605770]: enp129s0f0np0: Lost carrier
Sep 25 04:36:37 ********* systemd-networkd[605770]: bond1: Lost carrier
Sep 25 04:36:37 ********* systemd-networkd[605770]: enp161s0f1np1: Link DOWN
Sep 25 04:36:37 ********* systemd-networkd[605770]: enp161s0f1np1: Lost carrier
Sep 25 04:36:37 ********* systemd-networkd[605770]: enp161s0f0np0: Link DOWN
Sep 25 04:36:37 ********* systemd-networkd[605770]: enp161s0f0np0: Lost carrier
Sep 25 04:36:37 ********* systemd-networkd[605770]: bond0: Lost carrier
Sep 25 04:36:37 ********* systemd-networkd[605770]: bond0.3484: Lost carrier
Sep 25 04:36:37 ********* systemd-networkd[605770]: bond0.3486: Lost carrier
Sep 25 04:36:41 ********* systemd-networkd[605770]: enp129s0f1np1: Link UP
Sep 25 04:36:42 ********* systemd-networkd[605770]: enp129s0f1np1: Gained carrier
Sep 25 04:36:42 ********* systemd-networkd[605770]: enp129s0f0np0: Link UP
Sep 25 04:36:42 ********* systemd-networkd[605770]: enp129s0f0np0: Gained carrier
Sep 25 04:36:42 ********* systemd-networkd[605770]: enp161s0f1np1: Link UP
Sep 25 04:36:42 ********* systemd-networkd[605770]: enp161s0f0np0: Link UP
Sep 25 04:36:42 ********* systemd-networkd[605770]: enp161s0f1np1: Gained carrier
Sep 25 04:36:42 ********* systemd-networkd[605770]: enp161s0f0np0: Gained carrier
Sep 25 04:36:42 ********* systemd-networkd[605770]: bond0: Gained carrier
Sep 25 04:36:42 ********* systemd-networkd[605770]: bond0.3484: Gained carrier
Sep 25 04:36:42 ********* systemd-networkd[605770]: bond0.3486: Gained carrier
Sep 25 04:36:42 ********* systemd-networkd[605770]: bond1: Gained carrier
```
```
Sep 25 04:36:35 ********* kernel: [18085928.795785] bond1: (slave enp129s0f1np1): link status definitely down, disabling slave
Sep 25 04:36:36 ********* kernel: [18085929.535826] bond1: (slave enp129s0f0np0): link status definitely down, disabling slave
Sep 25 04:36:37 ********* kernel: [18085929.919702] bond0: (slave enp161s0f1np1): link status definitely down, disabling slave
Sep 25 04:36:37 ********* kernel: [18085930.339709] bond0: (slave enp161s0f0np0): link status definitely down, disabling slave
Sep 25 04:36:41 ********* kernel: [18085934.812301] bond0: (slave enp161s0f1np1): link status definitely up, 25000 Mbps full duplex
Sep 25 04:36:41 ********* kernel: [18085934.812968] bond0: (slave enp161s0f0np0): link status definitely up, 25000 Mbps full duplex
Sep 25 04:36:42 ********* kernel: [18085934.828048] bond1: (slave enp129s0f1np1): link status definitely up, 100000 Mbps full duplex
Sep 25 04:36:42 ********* kernel: [18085934.828491] bond1: (slave enp129s0f0np0): link status definitely up, 100000 Mbps full duplex
```
This caused a big outage on the Ceph cluster where many placement
groups went unavailable.
It will be good if the dbus upgrade can avoid to restart the dbus
service, to avoid the consequent impact to networking, and leave that
to the operator to do it at its best convenience.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/dbus/+bug/2083489/+subscriptions
More information about the foundations-bugs
mailing list