[Bug 1432062] Re: Ship the default /etc/multipath.conf on multipath-tools-boot (for user_friendly_names)
Mauricio Faria de Oliveira
mauricfo at linux.vnet.ibm.com
Mon Apr 13 22:35:40 UTC 2015
Hi Mathieu,
I belive the actual fix is better than the requirement of user_friendly_names.
The patch is attached (sorry for the delay, things turned a bit :)
Test scenario:
Boot without 'splash quiet' and with 'debug break=top,pre-multipath'.
On break #1 you can change udev rules/modify user_friendly_names;
On break #2 you can read the udev debug output more clearly.
1) The change to kpartx.udev (replace dmsetup_env with equivalent udev rules)
fixes this call to kpartx_id, which ends up mis-detecting multipath as raid.
Before:
------
[ 0.755972] systemd-udevd[94]: IMPORT 'kpartx_id 252 0 mpath-0QEMU QEMU HARDDISK serial_one' /lib/udev/rules.d/95-kpartx.rules:21
[ 0.756185] systemd-udevd[180]: starting 'kpartx_id 252 0 mpath-0QEMU QEMU HARDDISK serial_one'
[ 0.757407] systemd-udevd[94]: 'kpartx_id 252 0 mpath-0QEMU QEMU HARDDISK serial_one'(err) 'device-mapper: deps ioctl on failed: No such device or address'
[ 0.757526] systemd-udevd[94]: 'kpartx_id 252 0 mpath-0QEMU QEMU HARDDISK serial_one'(err) ''
[ 0.757602] systemd-udevd[94]: 'kpartx_id 252 0 mpath-0QEMU QEMU HARDDISK serial_one'(err) 'Command failed'
[ 0.757850] systemd-udevd[94]: 'kpartx_id 252 0 mpath-0QEMU QEMU HARDDISK serial_one'(out) 'DM_TYPE=raid'
[ 0.757927] systemd-udevd[94]: 'kpartx_id 252 0 mpath-0QEMU QEMU HARDDISK serial_one' [180] exit with return code 0
[ 0.758017] systemd-udevd[94]: LINK 'disk/by-id/raid-mpath0' /lib/udev/rules.d/95-kpartx.rules:27
(initramfs) ls -l /dev/disk/by-id/ | sort
...
lrwxrwxrwx 1 10 raid-mpath0 -> ../../dm-0
lrwxrwxrwx 1 10 raid-mpath0-part1-part1 -> ../../dm-1
lrwxrwxrwx 1 10 raid-mpath0-part2-part2 -> ../../dm-2
lrwxrwxrwx 1 10 raid-mpath0-part3-part3 -> ../../dm-3
...
After:
-----
[ 16.020662] systemd-udevd[112]: IMPORT '/sbin/dmsetup info -j 252 -m 0 -c --nameprefixes --noheadings --rows -o attr' /lib/udev/rules.d/95-kpartx.rules:18
...
[ 16.021746] systemd-udevd[205]: starting '/sbin/dmsetup info -j 252 -m 0 -c --nameprefixes --noheadings --rows -o attr'
[ 16.023274] systemd-udevd[112]: '/sbin/dmsetup info -j 252 -m 0 -c --nameprefixes --noheadings --rows -o attr'(out) 'DM_ATTR='L--w''
[ 16.023554] systemd-udevd[112]: '/sbin/dmsetup info -j 252 -m 0 -c --nameprefixes --noheadings --rows -o attr' [205] exit with return code 0
...
[ 16.035336] systemd-udevd[210]: starting 'kpartx_id 252 0 mpath-0QEMU\x20\x20\x20\x20QEMU\x20HARDDISK\x20\x20\x20serial_one'
[ 16.037665] systemd-udevd[111]: 'kpartx_id 252 0 mpath-0QEMU\x20\x20\x20\x20QEMU\x20HARDDISK\x20\x20\x20serial_one'(out) 'DM_TYPE=scsi'
[ 16.037764] systemd-udevd[111]: 'kpartx_id 252 0 mpath-0QEMU\x20\x20\x20\x20QEMU\x20HARDDISK\x20\x20\x20serial_one' [210] exit with return code 0
[ 16.037872] systemd-udevd[111]: LINK 'disk/by-id/scsi-mpath0' /lib/udev/rules.d/95-kpartx.rules:29
(initramfs) ls -l /dev/disk/by-id/ | sort
...
lrwxrwxrwx 1 10 scsi-mpath0 -> ../../dm-0
lrwxrwxrwx 1 10 scsi-mpath0-part1 -> ../../dm-1
lrwxrwxrwx 1 10 scsi-mpath0-part2 -> ../../dm-2
lrwxrwxrwx 1 10 scsi-mpath0-part3 -> ../../dm-3
...
2) The change to libmultipath/defaults.h (scsi_id --replace-whitespace) fixes
the name used in the call to dmsetup --exec 'kpartx -a' if the disk identifier
has spaces in it and user_friendly_names is not enabled
This* is the case when system is not installed w/ multipath enabled and
after installed with multipath-tools-boot (which is realistic).
Before:
-------
[ 20.649487] systemd-udevd[214]: starting '/sbin/dmsetup ls --target multipath --exec '/sbin/kpartx -a -p -part' -j 252 -m 0'
[ 20.651196] systemd-udevd[111]: '/sbin/dmsetup ls --target multipath --exec '/sbin/kpartx -a -p -part' -j 252 -m 0'(out) 'failed to stat() /dev/mapper/0QEMU QEMU HARDDISK serial_one'
[ 20.651489] systemd-udevd[111]: '/sbin/dmsetup ls --target multipath --exec '/sbin/kpartx -a -p -part' -j 252 -m 0' [214] exit with return code 0
Obviously, no partitions on multipath disks show up (kpartx failed):
(initramfs) ls -l /dev/mapper/
crw------- 1 10, 236 control
lrwxrwxrwx 1 7 0QEMU\x20\x20\x20\x20QEMU\x20HARDDISK\x20\x20\x20serial_one -> ../dm-0
brw-rw---- 1 252, 0 0QEMU QEMU HARDDISK serial_one
And the system won't boot, because device-mapper is still locking the
underlying device:
Loading, please wait...
starting version 219
<...>
Begin: Discovering multipaths ... done.
[ 0.666073] device-mapper: table: 252:1: linear: Invalid argument count
[ 0.666158] device-mapper: ioctl: error adding target to table
device-mapper: reload ioctl on 0QEMU\x20\x20\x20\x20QEMU\x20HARDDISK\x20\x20\x20serial_one-part1 failed: Invalid argument
create/reload failed on 0QEMU QEMU HARDDISK serial_one-part1
[ 0.667907] device-mapper: table: 252:1: linear: Invalid argument count
[ 0.667972] device-mapper: ioctl: error adding target to table
device-mapper: reload ioctl on 0QEMU\x20\x20\x20\x20QEMU\x20HARDDISK\x20\x20\x20serial_one-part2 failed: Invalid argument
create/reload failed on 0QEMU QEMU HARDDISK serial_one-part2
[ 0.669568] device-mapper: table: 252:1: linear: Invalid argument count
[ 0.669632] device-mapper: ioctl: error adding target to table
device-mapper: reload ioctl on 0QEMU\x20\x20\x20\x20QEMU\x20HARDDISK\x20\x20\x20serial_one-part3 failed: Invalid argument
create/reload failed on 0QEMU QEMU HARDDISK serial_one-part3
done.
Begin: Running /scripts/local-premount ... done.
mount: mounting /dev/disk/by-uuid/396d068f-e872-4204-9cc0-927822d2d84f on /root failed: Device or resource busy
Begin: Running /scripts/local-bottom ... done.
done.
Begin: Running /scripts/init-bottom ... mount: mounting /dev on /root/dev failed: No such file or directory
done.
mount: mounting /sys on /root/sys failed: No such file or directory
mount: mounting /proc on /root/proc failed: No such file or directory
Target filesystem doesn't have requested /sbin/init.
No init found. Try passing init= bootarg.
<...>
(initramfs)
Test/work-around:
(initramfs) mount /dev/disk/by-uuid/396d068f-e872-4204-9cc0-927822d2d84f /root
mount: mounting /dev/disk/by-uuid/396d068f-e872-4204-9cc0-927822d2d84f on /root failed: Device or resource busy
(initramfs) dmsetup table
0QEMU QEMU HARDDISK serial_one: 0 67108864 multipath 0 0 1 1 round-robin 0 1 1 8:0 1000
(initramfs) dmsetup remove_all
(initramfs) mount /dev/disk/by-uuid/396d068f-e872-4204-9cc0-927822d2d84f /root
[ 151.208898] EXT4-fs (sda2): couldn't mount as ext3 due to feature incompatibilities
[ 151.209281] EXT4-fs (sda2): couldn't mount as ext2 due to feature incompatibilities
[ 151.217284] EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: (null)
(it worked.)
After:
-----
[ 38.242167] systemd-udevd[218]: starting '/sbin/dmsetup ls --target multipath --exec '/sbin/kpartx -a -p -part' -j 252 -m 0'
<...>
[ 38.272317] systemd-udevd[118]: handling device node '/dev/dm-1', devnum=b252:1, mode=0600, uid=0, gid=0
<...>
[ 38.289123] systemd-udevd[119]: handling device node '/dev/dm-2', devnum=b252:2, mode=0600, uid=0, gid=0
<...>
[ 38.290023] systemd-udevd[120]: handling device node '/dev/dm-3', devnum=b252:3, mode=0600, uid=0, gid=0
<...>
[ 38.286179] systemd-udevd[115]: '/sbin/dmsetup ls --target multipath --exec '/sbin/kpartx -a -p -part' -j 252 -m 0' [218] exit with return code 0
(initramfs) ls -l /dev/mapper/
crw------- 1 10, 236 control
lrwxrwxrwx 1 7 0QEMU_QEMU_HARDDISK_serial_one -> ../dm-0
brw------- 1 252, 1 0QEMU_QEMU_HARDDISK_serial_one-part1
brw------- 1 252, 2 0QEMU_QEMU_HARDDISK_serial_one-part2
brw------- 1 252, 3 0QEMU_QEMU_HARDDISK_serial_one-part3
And the system booted successfully.
For reference, some affected platforms/disks, and the values
without/with scsi-id --replace-whitespace:
POWER8 non-virtualized (usually has IPR controllers, AFAICT):
# /lib/udev/scsi_id --whitelisted --device=/dev/sda
1IBM IPR-5 5D9D4A00000002A0
# /lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/sda
1IBM_IPR-5_5D9D4A00000002A0
KVM guest:
# /lib/udev/scsi_id --whitelisted --device=/dev/sda
0QEMU QEMU HARDDISK serial_one
# /lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/sda
0QEMU_QEMU_HARDDISK_serial_one
PowerVM LPAR:
# /lib/udev/scsi_id --whitelisted --device=/dev/sda
SAIX VDASD 00f6773800004c000000013458a93c13.41
# /lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/sda
SAIX_VDASD_00f6773800004c000000013458a93c13.41
** Patch added: "multipath-tools_whitespacedIDs.debdiff"
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1432062/+attachment/4374518/+files/multipath-tools_whitespacedIDs.debdiff
** Summary changed:
- Ship the default /etc/multipath.conf on multipath-tools-boot (for user_friendly_names)
+ multipath-tools-boot: support booting without user_friendly_names on devices with spaces in identifiers
** Tags added: patch
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to multipath-tools in Ubuntu.
https://bugs.launchpad.net/bugs/1432062
Title:
multipath-tools-boot: support booting without user_friendly_names on
devices with spaces in identifiers
Status in multipath-tools package in Ubuntu:
Triaged
Bug description:
If a system is not installed w/ multipath support (i.e., no disk-detect/multipath/enable=true), the /etc/multipath.conf file is not installed.
If an user later installs multipath-tools-boot, it will enable the udev rules for multipath support.
Those rules don't handle disk devices w/ spaces on their names/uuids/models very well..
That's because of udev's SYMLINK command using spaces to separate
multiple links, and the kernel sysfs/dm informing \x20 instead, which
is not correctly interpreted by some commands, resulting in file not
found errors, for example.
Thus, the system fails to boot.
There's no problem, however, if user_friendly_names is enabled in
multipath.conf (which is enabled in the default multipath.conf from
the installer, if it has multipath enabled).
Notice it's an acceptable case to install w/out multipath support, and
enable it later for booting.
Disk devices w/ spaces in naming is not common over SAN/storage systems, but that happens often for conventional disks; for example:
- IBM IPR ( IBM IPR-0 5DB6F40000000080 )
- IBM VDASD ( AIX VDASD 00c96f0700004c000000014bb8e713f0.14 )
- QEMU HARDDISK ( QEMU QEMU HARDDISK <serial> )
So, please, is it possible to ship the default multipath.conf (e.g.,
from installer) w/ multipath-tools-boot?
For users not to their systems failing to boot after installing
multipath-tools-boot manually, after a non-multipath install.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1432062/+subscriptions
More information about the foundations-bugs
mailing list