[Bug 1429327] Re: ISST-LTE: system drops to initramfs after install on multipath disk
Mauricio Faria de Oliveira
mauricfo at linux.vnet.ibm.com
Tue Mar 31 19:26:08 UTC 2015
The test-cases performed with the patch:
Install the modified multipath-tools' binary packages; the initramfs is
updated with the changes.
Reboot. The system booted successfully; good.
Notice:
- unmodified kernel cmdline
- root filesystem mounted from the 'by-uuid/multipath-<UUID>' symlink
- it points to a device-mapper virtual block device
- the symlink is unique (no other devices claim it).
$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinux-3.19.0-9-generic root=UUID=7fec6766-3166-4395-982a-555ecce92667 ro
$ df /
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/disk/by-uuid/multipath-7fec6766-3166-4395-982a-555ecce92667 31483800 2374984 27486444 8% /
$ udevadm info -q path /dev/disk/by-uuid/multipath-7fec6766-3166-4395-982a-555ecce92667
/devices/virtual/block/dm-4
$ ls -1 /run/udev/links/*by-uuid*multipath-7fec6766-3166-4395-982a-555ecce92667
b252:4
Verify the failure hook.
Reboot with 'break=top,post-multipath'.
Reduce the wait-for-root timeout for debugging purposes.
(initramfs) sed 's:\(ARCHDELAY\)=[0-9]\+:\1=10:' -i /scripts/local
(initramfs) exit
Check the expected root device:
(initramfs) cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinux-3.19.0-9-generic root=UUID=7fec6766-3166-4395-982a-555ecce92667 ro break=top,post-multipath
Initially, things are alright:
(initramfs) ls -l /dev/disk/by-uuid/multipath-7fec6766-3166-4395-982a-555ecce92667
lrwxrwxrwx 1 10 /dev/disk/by-uuid/multipath-7fec6766-3166-4395-982a-555ecce92667 -> ../../dm-3
Now, make things wrong:
(initramfs) multipath -F
(initramfs) ls -l /dev/disk/by-uuid/multipath-7fec6766-3166-4395-982a-555ecce92667
ls: /dev/disk/by-uuid/multipath-7fec6766-3166-4395-982a-555ecce92667: No such file or directory
(initramfs) exit
~10 seconds for wait-for-root timeout...
The informative message is displayed; good; the failure hook is OK.
...
WARNING: the root device is supposed to be a multipath device,
but it doesn't look like one: it doesn't have a multipath-UUID
symlink, but it does have a non-multipath (normal) UUID symlink.
...
If you don't try to fix anything and exit, the good and old 'root device not found' message appears, normally.
...
Gave up waiting for root device. Common problems:
...
ALERT! /dev/disk/by-uuid/multipath-7fec6766-3166-4395-982a-555ecce92667 does not exist. Dropping to a shell!
....
Verify the failure hook again, but now let's fix things:
(repeat steps above)
...
WARNING: the root device is supposed to be a multipath device,
but it doesn't look like one: it doesn't have a multipath-UUID
symlink, but it does have a non-multipath (normal) UUID symlink.
...
(initramfs) multipath -v0
(initramfs) exit
The initramfs continued on, and the system booted successfully:
$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinux-3.19.0-9-generic root=UUID=7fec6766-3166-4395-982a-555ecce92667 ro break=top,post-multipath
$ df /
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/disk/by-uuid/multipath-7fec6766-3166-4395-982a-555ecce92667 31483800 2375308 27486120 8% /
$ udevadm info -q path /dev/disk/by-uuid/multipath-7fec6766-3166-4395-982a-555ecce92667
/devices/virtual/block/dm-6
Finally, verify the failure hook again, using the 'TIP (not recommended)' steps.
It boots from the fake symlink (points to underlying/single-path device), so you get the usual errors about device-mapper failing to get the underlying because it's locked (root filesystem mountpoint), plus some "waiting" delays.
(repeat steps above)
...
WARNING: the root device is supposed to be a multipath device,
but it doesn't look like one: it doesn't have a multipath-UUID
symlink, but it does have a non-multipath (normal) UUID symlink.
...
(initramfs) uuid=7fec6766-3166-4395-982a-555ecce92667
(initramfs) ln -s $uuid /dev/disk/by-uuid/multipath-$uuid && exit
...
Starting LSB: multipath daemon...
[...] device-mapper: table: 252:4: multipath: error getting device
[...] device-mapper: table: 252:4: multipath: error getting device
...
A start job is running for dev-mapp...x2dpart3.device (37s / 1min 30s)
...
$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinux-3.19.0-9-generic root=UUID=7fec6766-3166-4395-982a-555ecce92667 ro break=top,post-multipath
$ df /
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/disk/by-uuid/multipath-7fec6766-3166-4395-982a-555ecce92667 31483800 2386948 27474480 8% /
$ udevadm info -q path /dev/disk/by-uuid/multipath-7fec6766-3166-4395-982a-555ecce92667
/devices/vio/71000002/host0/target0:0:1/0:0:1:0/block/sdb/sdb2
$ ls -1 /run/udev/links/*by-uuid*multipath-7fec6766-3166-4395-982a-555ecce92667
ls: cannot access /run/udev/links/*by-uuid*multipath-7fec6766-3166-4395-982a-555ecce92667: No such file or directory
(i.e., the fake symlink is expectedly not maintained by
udev, which is part of why this is not recommended :)
If you remove the UUID option from GRUB, it doesn't touch anything:
# GRUB_DISABLE_LINUX_UUID=true update-grub
...
# grep root= /boot/grub/grub.cfg
linux /boot/vmlinux-3.19.0-9-generic root=/dev/mapper/mpath1-part2 ro
linux /boot/vmlinux-3.19.0-9-generic root=/dev/mapper/mpath1-part2 ro
linux /boot/vmlinux-3.19.0-9-generic root=/dev/mapper/mpath1-part2 ro recovery nomodeset
...
# reboot
...
$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinux-3.19.0-9-generic root=/dev/mapper/mpath1-part2 ro
$ df /
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/mpath1-part2 31483800 2387108 27474320 8% /
So, I believe these tests cover all the functionality added.
Good to go?
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to debian-installer in Ubuntu.
https://bugs.launchpad.net/bugs/1429327
Title:
ISST-LTE: system drops to initramfs after install on multipath disk
Status in debian-installer package in Ubuntu:
Confirmed
Bug description:
-- Problem Description --
We try to install Ubuntu version 15.04 build 20150218 on our system which use the ibmveth for the network and bluefin (lpfc) for this disks.
The Installation completed and ask to continue and reboot, but then it drops to initramfs....
(initramfs) cat /proc/modules
usbhid 63247 0 - Live 0xd000000003520000
hid 128631 1 usbhid, Live 0xd0000000034a0000
lpfc 778399 1 - Live 0xd000000002e30000
dm_multipath 25603 0 - Live 0xd000000002cf0000
scsi_dh 10388 1 dm_multipath, Live 0xd000000002c10000
scsi_transport_fc 71871 1 lpfc, Live 0xd000000002890000
(initramfs) cat /proc/cmdline
BOOT_IMAGE=/vmlinux-3.18.0-13-generic root=/dev/mapper/creeklp1--vg-root ro splash quiet
Look like it cannot fine the vg-root to boot up.
?????????????????????????? [!!] Partition disks ???????????????????????????
? ?
? Note that all data on the disk you select will be erased, but not ?
? before you have confirmed that you really want to make the changes. ?
? ?
? Select disk to partition: ?
? ?
? Multipath mpath0 (WWID 36005076307ffc7b00000000000000717) - 16.1 ?
? Multipath mpath1 (WWID 36005076307ffc7b00000000000000718) - 16.1 ?
? Multipath mpath2 (WWID 36005076307ffc7b00000000000000719) - 16.1 ? ?
? Multipath mpath3 (WWID 36005076307ffc7b0000000000000071a) - 16.1 ? ?
? Multipath mpath4 (WWID 36005076307ffc7b0000000000000071b) - 64.4 ? ?
? SCSI1 (0,0,0) (sda) - 16.1 GB IBM 2107900 ? ?
? SCSI1 (0,5,1) (sdaa) - 16.1 GB IBM 2107900 ? ?
? SCSI1 (0,5,2) (sdab) - 16.1 GB IBM 2107900 ? ?
? SCSI1 (0,5,3) (sdac) - 16.1 GB IBM 2107900 ? ?
? SCSI1 (0,5,4) (sdad) - 64.4 GB IBM 2107900 ?
? ?
? <Go Back> ?
? ?
???????????????????????????????????????????????????????????????????????????
conflicting device node '/dev/mapper/mpath4p1' found, link to '/dev/dm-5' will not be created
conflicting device node '/dev/mapper/mpath4p2' found, link to '/dev/dm-6' will not be created
conflicting device node '/dev/mapper/mpath4p3' found, link to '/dev/dm-7' will not be created
??????????????????????????? [!] Partition disks ???????????????????????????
? ?
? You may use the whole volume group for guided partitioning, or part ?
? of it. If you use only part of it, or if you add more disks later, ?
? then you will be able to grow logical volumes later using the LVM ?
? tools, so using a smaller part of the volume group at installation ?
? time may offer more flexibility. ?
? ?
? The minimum size of the selected partitioning recipe is 596.0 MB (or ?
? 0%); please note that the packages you choose to install may require ?
? more space than this. The maximum available size is 64.2 GB. ?
? ?
? Hint: "max" can be used as a shortcut to specify the maximum size, or ?
? enter a percentage (e.g. "20%") to use that percentage of the maximum ?
? size. ?
? ?
? 64.2 GB______________________________________________________________ ?
? ?
? <Go Back> <Continue> ?
? ?
???????????????
?????????????????????????? [!!] Partition disks ??????????????????????????
? ?
? If you continue, the changes listed below will be written to the ?
? disks. Otherwise, you will be able to make further changes manually. ?
? ?
? WARNING: This will destroy all data on any partitions you have ?
? removed as well as on the partitions that are going to be formatted. ?
? ?
? The partition tables of the following devices are changed: ?
? LVM VG creeklp1-vg, LV root ?
? LVM VG creeklp1-vg, LV swap_1 ?
? Multipath mpath4 (WWID 36005076307ffc7b0000000000000071b) ?
? SCSI1 (0,5,4) (sdad) ?
? SCSI1 (0,0,4) (sde) ?
? SCSI1 (0,1,4) (sdj) ?
? SCSI1 (0,2,4) (sdo) ?
? SCSI1 (0,3,4) (sdt) ?
? ?
? <Yes> <No> ?
? ?
??????????????????????????????????????????????????????????????????????????
????????????????????? [!!] Finish the installation ??????????????????????
? ?
?? Installation complete ?
?? Installation is complete, so it is time to boot into your new system. ?
?? Make sure to remove the installation media (CD-ROM, floppies), so ?
?? that you boot into the new system rather than restarting the ?
?? installation. ?
?? ?
?? <Go Back> <Continue> ?
? ?
?????????????????????????????????????????????????????????????????????????
creeklp2 is now booting. I made a few changes to the system to accomplish this.
First I got the machine to boot by doing the following from the
initramfs:
1. modprobe scsi_dh_alua
2. multipath -v0
3. exit
Once the machine was booted, I did the following:
1. Added scsi_dh_alua to /etc/initramfs-tools/modules
2. Ran update-initramfs to update the initramfs with this change
3. Set GRUB_DISABLE_LINUX_UUID=true in /etc/default/grub
4. Rebuild grub.cfg by using grub-mkconfig
I'm still getting a lot of errors booting the system from udev due to
the fact that udev is spawning a call to multipath for each /dev/sd
device on the system.
I decided to change to not use UUID when I saw in the initramfs that
the /dev/disk/by-uuid symlink was pointing to one of the paths of the
multipath device rather than the multipath device itself.
It seems there may be some udev configuration issues here when booting
from a multipath device.
Can we get someone from Canonical to take a look at this?
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/debian-installer/+bug/1429327/+subscriptions
More information about the foundations-bugs
mailing list