[Bug 1565817] Comment bridged from LTC Bugzilla
bugproxy
bugproxy at us.ibm.com
Mon Apr 4 15:59:46 UTC 2016
------- Comment From mauricfo at br.ibm.com 2016-04-04 11:42 EDT-------
Details about the 'property blacklisting' problem.
---
Debugging this in the initramfs (w/ break=top,pre-multipath,post-
multipath).
This LVM PV should be on a multipath device, but is on an individual
path.
(initramfs) lvm pvdisplay
...
Found duplicate PV SNTOUh7JC3dzwS0BfzNf14bqpw1u58Uh: using /dev/sdao3 not /dev/sdl3
Using duplicate PV /dev/sdao3 without holders, ignoring /dev/sdl3
--- Physical volume ---
PV Name /dev/sdao3
...
The udev rule which removes the partition nodes from invididual paths is
(56-dm-mpath-lvm.rules):
PROGRAM=="/sbin/multipath -c $devnode", \
ENV{DM_MULTIPATH_DEVICE_PATH}="1", \
ENV{ID_FS_TYPE}="mpath_member", \
RUN+="/usr/bin/partx -d --nr 1-1024 $devnode", \
GOTO="end_mpath"
This indicates 'multipath -c' stated the path was not a valid path,
but running it manually tells otherwise:
(initramfs) multipath -c /dev/sdao
/dev/sdao is a valid multipath device path
(initramfs) multipath -c /dev/sdl
/dev/sdl is a valid multipath device path
By increasing verbosity to 'multipath -v3' and logging the debug output of udev,
we find the 'property blacklist' is behind it:
starting '/sbin/multipath -c -v3 /dev/sdao'
...
'/sbin/multipath -c -v3 /dev/sdao'(out) 'Mar 30 14:48:46 | sdao: blacklisted, udev property missing'
'/sbin/multipath -c -v3 /dev/sdao'(out) 'Mar 30 14:48:46 | /dev/sdao: failed to get wwid'
...
Process '/sbin/multipath -c -v3 /dev/sdao' failed with exit code 1.
Comparing the 'multipath -v3 -c' output with running it manually afterward,
the properties are indeed present:
(initramfs) multipath -c -v3 /dev/sdao
...
Mar 30 16:21:17 | sdao: udev property ID_WWN whitelisted
Mar 30 16:21:17 | scope limited to 36005076307ffd1540000000000000115
/dev/sdao is a valid multipath device path
(initramfs) udevadm info -q property /dev/sdao | grep ^ID_WWN
ID_WWN=0x6005076307ffd1540000000000000115
ID_WWN_WITH_EXTENSION=0x6005076307ffd1540000000000000115
So, it turns out the proberties are not yet available during the udev
rule execution of the PROGRAM='multipath -c <path>' rule.
It happens because the properties are only written to the udev database
after the udev rules processing, and before the 'RUN' rules are
executed.
For example, this is the comparison of the 'udevadm info' command on PROGRAM and RUN rules.
The latter can read a lot more of udev properties.
(initramfs) cat /udev.sda.udevadm-info.program
P: /devices/vio/71000002/host0/target0:0:0/0:0:0:0/block/sda
N: sda
E: DEVNAME=/dev/sda
E: DEVPATH=/devices/vio/71000002/host0/target0:0:0/0:0:0:0/block/sda
E: DEVTYPE=disk
E: MAJOR=8
E: MINOR=0
E: SUBSYSTEM=block
(initramfs) cat /udev.sda.udevadm-info.run
P: /devices/vio/71000002/host0/target0:0:0/0:0:0:0/block/sda
N: sda
S: disk/by-id/scsi-0QEMU_QEMU_HARDDISK_xenialroot
E: DEVLINKS=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_xenialroot
E: DEVNAME=/dev/sda
E: DEVPATH=/devices/vio/71000002/host0/target0:0:0/0:0:0:0/block/sda
E: DEVTYPE=disk
E: DM_MULTIPATH_DEVICE_PATH=1
E: ID_BUS=scsi
E: ID_FS_TYPE=mpath_member
E: ID_MODEL=QEMU_HARDDISK
E: ID_MODEL_ENC=QEMU\x20HARDDISK\x20\x20\x20
...
E: SCSI_IDENT_SERIAL=xenialroot
...
E: SCSI_VENDOR=QEMU
E: SCSI_VENDOR_ENC=QEMU\x20\x20\x20\x20
E: SUBSYSTEM=block
E: USEC_INITIALIZED=9582006
This is some of call chain in the systemd/src/udev/ and-friends dirs:
worker_spawn()
/* PROGRAM */
udev_event_execute_rules()
/* case TK_M_PROGRAM: { */
udev_event_spawn()
spawn_exec(event, cmd, argv, udev_device_get_properties_envp(event->dev),
udev_device_update_db
device_update_db
ORDERED_HASHMAP_FOREACH_KEY(value, property, device->properties_db, i)
fprintf(f, "E:%s=%s\n", property, value);
/* RUN */
udev_event_execute_run()
udev_event_spawn()
spawn_exec(event, cmd, argv, udev_device_get_properties_envp(event->dev),
This 2 upstream commits (present with the merge) are relevant to this issue:
- [1] multipath: implement option '-u' for uevents
- [2] multipath: check for running daemon when called with '-u'
Basically, '-u' addresses exactly this issue (commit message):
multipath: implement option '-u' for uevents
When calling 'multipath' from an uevent udev will not have
all information for that device, as it's being written into
the database _after_ the event has been processed.
This patch implements an option '-u' which uses the information
from the program environment when checking the device.
So, let's change the udev rule from 'multipath -c' to 'multipath -u'.
This requires starting multipathd before udev rules are processed
(-u checks for multipathd running) - so init-top starts multipathd now.
In order to confirm it's working, it's a bit tricky - 'multipath -u -v3'
logs to the system logger rather than stdio, so I modified the multipath
initramfs hook to copy the rsyslogd stuff into the initramfs, and
manually started it on break=top.
@ /usr/share/initramfs/hooks/multipath
mkdir -p $DESTDIR/var/log
mkdir -p $DESTDIR/usr/sbin
copy_exec /usr/sbin/rsyslogd /usr/sbin
cp -rp /usr/lib/rsyslog $DESTDIR/usr/lib
cp -rp /etc/rsyslog* $DESTDIR/etc
...
(initramfs) rsyslogd
Verifying the 'multipath -u -v3' messages on /var/log/syslog,
we can confirm the udev properties are now correctly found,
and the paths are considered valid:
(initramfs) grep 'multipath: sda:' /var/log/syslog
...
Apr 4 15:37:59 (none) multipath: sda: udev property SCSI_IDENT_LUN_VENDOR whitelisted
(initramfs) grep 'multipath: sdb:' /var/log/syslog
...
Apr 4 15:37:59 (none) multipath: sdb: udev property SCSI_IDENT_LUN_VENDOR whitelisted
(initramfs) grep 'multipath: sdc:' /var/log/syslog
...
Apr 4 15:37:59 (none) multipath: sdc: udev property SCSI_IDENT_LUN_VENDOR whitelisted
And the partition nodes for sda and sdb (paths of a multipath device) are removed,
and those for sdc (not from a multipath device) are not -- which match 'multipath -c':
(initramfs) ls -1 /dev/sd* | sort
/dev/sda
/dev/sdb
/dev/sdc
/dev/sdc1
/dev/sdc2
/dev/sdc3
(initramfs) multipath -c /dev/sda
/dev/sda is a valid multipath device path
(initramfs) multipath -c /dev/sdb
/dev/sdb is a valid multipath device path
(initramfs) multipath -c /dev/sdc
/dev/sdc is not a valid multipath device path
(initramfs) multipath -l
mpatha (0QEMU_QEMU_HARDDISK_xenialroot) dm-0 QEMU,QEMU HARDDISK
size=32G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=0 status=active
| `- 0:0:0:0 sda 8:0 active undef running
`-+- policy='round-robin 0' prio=0 status=enabled
`- 0:0:1:0 sdb 8:16 active undef running
(initramfs)
Links:
[1] http://git.opensvc.com/gitweb.cgi?p=multipath-tools/.git;a=commit;h=98cbca1773eec5d740bb0c441b27383c75a30449
[2] http://git.opensvc.com/gitweb.cgi?p=multipath-tools/.git;a=commit;h=77febddfdce8a7d746227c09066e5f507b53d2fb
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to multipath-tools in Ubuntu.
https://bugs.launchpad.net/bugs/1565817
Title:
multipath: LVM physical volumes on individual paths
Status in multipath-tools package in Ubuntu:
New
Bug description:
== Comment: #0 - Mauricio Faria De Oliveira - 2016-04-04 10:02:55 ==
---Problem Description---
multipath: LVM physical volumes on individual paths
Contact Information = Mauricio Faria de Oliveira <mauricfo at linux.vnet.ibm.com> / Gabriel Krisman <krisman at linux.vnet.ibm.com>
This bug reports/patches 2 regressions introduced with multipath-tools 0.5.0+git1.656f8865-5ubuntu1 (see LP 1551952).
1) The udev rule which removes partition nodes from individual paths
is not working.
This happens due to the introduction of the 'udev property blacklisting' feature (which blacklists devices if they don't have certain udev properties defined).
Because 'multipath -c' is run at udev rules PROGRAM-rule time, it can't read the required udev properties (defined by sg3-utils udev rules), because they are not yet written to the udev database -- this only happens after the rules finish processing (they would be available at RUN-rule time).
The solution is to use 'multipath -u', which reads them directly from the udev program environment (and not from the not-yet-updated database) afaik, and makes things work at PROGRAM-rule time.
This required a change between the local-top/init-top initramfs scripts.
Since 'multipath -u' checks if multipathd is started, it moved into init-top, so it's started before udev rules are processed.
While in there, the local-top script is now redundant with other initramfs scripts, and can be removed. See the changelog for details/explanation.
2) The find_multipath options is no longer enabled by default.
It's non-default upstream, and with the merge (when the downstream patch was removed), the change/setting was lost.
A new patch is introduced which explicitly enables it by the default.
I'll add more details for the regression #1 shortly.
Patch attached.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1565817/+subscriptions
More information about the foundations-bugs
mailing list