[Bug 1828617] Re: Hosts randomly 'losing' disks, breaking ceph-osd service enumeration
Xav Paice
xav.paice at canonical.com
Tue May 28 22:55:54 UTC 2019
journalctl --no-pager -lu systemd-udevd.service >/tmp/1828617-1.out
Hostname obfusticated
lsblk:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 88.4M 1 loop /snap/core/6964
loop1 7:1 0 89.4M 1 loop /snap/core/6818
loop2 7:2 0 8.4M 1 loop /snap/canonical-livepatch/77
sda 8:0 0 1.8T 0 disk
├─sda1 8:1 0 476M 0 part /boot/efi
├─sda2 8:2 0 3.7G 0 part /boot
└─sda3 8:3 0 1.7T 0 part
└─bcache7 252:896 0 1.7T 0 disk /
sdb 8:16 0 1.8T 0 disk
└─bcache0 252:0 0 1.8T 0 disk
sdc 8:32 0 1.8T 0 disk
└─bcache6 252:768 0 1.8T 0 disk
└─crypt-7478edfc-f321-40a2-a105-8e8a2c8ca3f6 253:0 0 1.8T 0 crypt
└─ceph--7478edfc--f321--40a2--a105--8e8a2c8ca3f6-osd--block--7478edfc--f321--40a2--a105--8e8a2c8ca3f6 253:2 0 1.8T 0 lvm
sdd 8:48 0 1.8T 0 disk
└─bcache4 252:512 0 1.8T 0 disk
└─crypt-33de740d-bd8c-4b47-a601-3e6e634e489a 253:4 0 1.8T 0 crypt
└─ceph--33de740d--bd8c--4b47--a601--3e6e634e489a-osd--block--33de740d--bd8c--4b47--a601--3e6e634e489a 253:5 0 1.8T 0 lvm
sde 8:64 0 1.8T 0 disk
└─bcache3 252:384 0 1.8T 0 disk
└─crypt-eb5270dc-1110-420f-947e-aab7fae299c9 253:1 0 1.8T 0 crypt
└─ceph--eb5270dc--1110--420f--947e--aab7fae299c9-osd--block--eb5270dc--1110--420f--947e--aab7fae299c9 253:3 0 1.8T 0 lvm
sdf 8:80 0 1.8T 0 disk
└─bcache1 252:128 0 1.8T 0 disk
└─crypt-d38a7e91-cf06-4607-abbe-53eac89ac5ea 253:6 0 1.8T 0 crypt
└─ceph--d38a7e91--cf06--4607--abbe--53eac89ac5ea-osd--block--d38a7e91--cf06--4607--abbe--53eac89ac5ea 253:7 0 1.8T 0 lvm
sdg 8:96 0 1.8T 0 disk
└─bcache5 252:640 0 1.8T 0 disk
└─crypt-053e000a-76ed-427e-98b3-e5373e263f2d 253:8 0 1.8T 0 crypt
└─ceph--053e000a--76ed--427e--98b3--e5373e263f2d-osd--block--053e000a--76ed--427e--98b3--e5373e263f2d 253:9 0 1.8T 0 lvm
sdh 8:112 0 1.8T 0 disk
└─bcache8 252:1024 0 1.8T 0 disk
└─crypt-c2669da2-63aa-42e2-b049-cf00a478e076 253:25 0 1.8T 0 crypt
└─ceph--c2669da2--63aa--42e2--b049--cf00a478e076-osd--block--c2669da2--63aa--42e2--b049--cf00a478e076 253:28 0 1.8T 0 lvm
sdi 8:128 0 1.8T 0 disk
└─bcache2 252:256 0 1.8T 0 disk
└─crypt-12e68fcb-d2b6-459f-97f2-d3eb4e28c75e 253:26 0 1.8T 0 crypt
└─ceph--12e68fcb--d2b6--459f--97f2--d3eb4e28c75e-osd--block--12e68fcb--d2b6--459f--97f2--d3eb4e28c75e 253:27 0 1.8T 0 lvm
sdj 8:144 0 223.5G 0 disk
nvme0n1 259:1 0 1.5T 0 disk
├─nvme0n1p1 259:2 0 93.1G 0 part
│ └─bcache7 252:896 0 1.7T 0 disk /
├─nvme0n1p2 259:3 0 312.9G 0 part
│ └─crypt-5b0d2371-6d71-4685-9d06-f28678f81633 253:10 0 312.9G 0 crypt
│ ├─ceph--wal--4de27554--2d05--440e--874a--9921dfc6f47e-osd--wal--7478edfc--f321--40a2--a105--8e8a2c8ca3f6 253:11 0 1.9G 0 lvm
│ ├─ceph--wal--4de27554--2d05--440e--874a--9921dfc6f47e-osd--db--7478edfc--f321--40a2--a105--8e8a2c8ca3f6 253:12 0 36.3G 0 lvm
│ ├─ceph--wal--4de27554--2d05--440e--874a--9921dfc6f47e-osd--wal--33de740d--bd8c--4b47--a601--3e6e634e489a 253:13 0 1.9G 0 lvm
│ ├─ceph--wal--4de27554--2d05--440e--874a--9921dfc6f47e-osd--db--33de740d--bd8c--4b47--a601--3e6e634e489a 253:14 0 36.3G 0 lvm
│ ├─ceph--wal--4de27554--2d05--440e--874a--9921dfc6f47e-osd--wal--eb5270dc--1110--420f--947e--aab7fae299c9 253:15 0 1.9G 0 lvm
│ ├─ceph--wal--4de27554--2d05--440e--874a--9921dfc6f47e-osd--db--eb5270dc--1110--420f--947e--aab7fae299c9 253:16 0 36.3G 0 lvm
│ ├─ceph--wal--4de27554--2d05--440e--874a--9921dfc6f47e-osd--wal--d38a7e91--cf06--4607--abbe--53eac89ac5ea 253:17 0 1.9G 0 lvm
│ ├─ceph--wal--4de27554--2d05--440e--874a--9921dfc6f47e-osd--db--d38a7e91--cf06--4607--abbe--53eac89ac5ea 253:18 0 36.3G 0 lvm
│ ├─ceph--wal--4de27554--2d05--440e--874a--9921dfc6f47e-osd--wal--053e000a--76ed--427e--98b3--e5373e263f2d 253:19 0 1.9G 0 lvm
│ ├─ceph--wal--4de27554--2d05--440e--874a--9921dfc6f47e-osd--db--053e000a--76ed--427e--98b3--e5373e263f2d 253:20 0 36.3G 0 lvm
│ ├─ceph--wal--4de27554--2d05--440e--874a--9921dfc6f47e-osd--wal--c2669da2--63aa--42e2--b049--cf00a478e076 253:21 0 1.9G 0 lvm
│ ├─ceph--wal--4de27554--2d05--440e--874a--9921dfc6f47e-osd--db--c2669da2--63aa--42e2--b049--cf00a478e076 253:22 0 36.3G 0 lvm
│ ├─ceph--wal--4de27554--2d05--440e--874a--9921dfc6f47e-osd--wal--12e68fcb--d2b6--459f--97f2--d3eb4e28c75e 253:23 0 1.9G 0 lvm
│ └─ceph--wal--4de27554--2d05--440e--874a--9921dfc6f47e-osd--db--12e68fcb--d2b6--459f--97f2--d3eb4e28c75e 253:24 0 36.3G 0 lvm
└─nvme0n1p3 259:4 0 1.1T 0 part
├─bcache0 252:0 0 1.8T 0 disk
├─bcache1 252:128 0 1.8T 0 disk
│ └─crypt-d38a7e91-cf06-4607-abbe-53eac89ac5ea 253:6 0 1.8T 0 crypt
│ └─ceph--d38a7e91--cf06--4607--abbe--53eac89ac5ea-osd--block--d38a7e91--cf06--4607--abbe--53eac89ac5ea 253:7 0 1.8T 0 lvm
├─bcache2 252:256 0 1.8T 0 disk
│ └─crypt-12e68fcb-d2b6-459f-97f2-d3eb4e28c75e 253:26 0 1.8T 0 crypt
│ └─ceph--12e68fcb--d2b6--459f--97f2--d3eb4e28c75e-osd--block--12e68fcb--d2b6--459f--97f2--d3eb4e28c75e 253:27 0 1.8T 0 lvm
├─bcache3 252:384 0 1.8T 0 disk
│ └─crypt-eb5270dc-1110-420f-947e-aab7fae299c9 253:1 0 1.8T 0 crypt
│ └─ceph--eb5270dc--1110--420f--947e--aab7fae299c9-osd--block--eb5270dc--1110--420f--947e--aab7fae299c9 253:3 0 1.8T 0 lvm
├─bcache4 252:512 0 1.8T 0 disk
│ └─crypt-33de740d-bd8c-4b47-a601-3e6e634e489a 253:4 0 1.8T 0 crypt
│ └─ceph--33de740d--bd8c--4b47--a601--3e6e634e489a-osd--block--33de740d--bd8c--4b47--a601--3e6e634e489a 253:5 0 1.8T 0 lvm
├─bcache5 252:640 0 1.8T 0 disk
│ └─crypt-053e000a-76ed-427e-98b3-e5373e263f2d 253:8 0 1.8T 0 crypt
│ └─ceph--053e000a--76ed--427e--98b3--e5373e263f2d-osd--block--053e000a--76ed--427e--98b3--e5373e263f2d 253:9 0 1.8T 0 lvm
├─bcache6 252:768 0 1.8T 0 disk
│ └─crypt-7478edfc-f321-40a2-a105-8e8a2c8ca3f6 253:0 0 1.8T 0 crypt
│ └─ceph--7478edfc--f321--40a2--a105--8e8a2c8ca3f6-osd--block--7478edfc--f321--40a2--a105--8e8a2c8ca3f6 253:2 0 1.8T 0 lvm
└─bcache8 252:1024 0 1.8T 0 disk
└─crypt-c2669da2-63aa-42e2-b049-cf00a478e076 253:25 0 1.8T 0 crypt
└─ceph--c2669da2--63aa--42e2--b049--cf00a478e076-osd--block--c2669da2--63aa--42e2--b049--cf00a478e076 253:28 0 1.8T 0 lvm
Currently osd.11 and osd.24 are down.
** Attachment added: "1828617-1.out"
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1828617/+attachment/5267246/+files/1828617-1.out
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1828617
Title:
Hosts randomly 'losing' disks, breaking ceph-osd service enumeration
Status in systemd package in Ubuntu:
New
Bug description:
Ubuntu 18.04.2 Ceph deployment.
Ceph OSD devices utilizing LVM volumes pointing to udev-based physical devices.
LVM module is supposed to create PVs from devices using the links in /dev/disk/by-dname/ folder that are created by udev.
However on reboot it happens (not always, rather like race condition) that Ceph services cannot start, and pvdisplay doesn't show any volumes created. The folder /dev/disk/by-dname/ however has all necessary device created by the end of boot process.
The behaviour can be fixed manually by running "#/sbin/lvm pvscan
--cache --activate ay /dev/nvme0n1" command for re-activating the LVM
components and then the services can be started.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1828617/+subscriptions
More information about the foundations-bugs
mailing list