[Bug 1628723] Re: Trusty: multipathd SIGSEGV on path addition or removal
Christopher Unkel
1628723 at bugs.launchpad.net
Mon Oct 3 22:30:02 UTC 2016
I've attached two test scripts that configure the local machine to serve
as an iSCSI target and use then use those as targets to attach and
detach.
The best way to observe the issue is to use the first version of the
script with a single target and single iteration of the test loop, while
running multipathd under valgrind, i.e.:
service multipath-tools stop
valgrind multipathd -d &
bug1628723 1 1
When I do this, I observe the following in valgrind output:
==31667== Invalid read of size 1
==31667== at 0x4C2E4E1: __GI_strncpy (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31667== by 0x56FD3C9: sysfs_get_tgt_nodename (discovery.c:257)
==31667== by 0x56FDF98: scsi_sysfs_pathinfo (discovery.c:494)
==31667== by 0x56FEA01: sysfs_pathinfo (discovery.c:696)
==31667== by 0x56FF174: pathinfo (discovery.c:830)
==31667== by 0x56FC775: store_pathinfo (discovery.c:57)
==31667== by 0x40504E: uev_add_path (main.c:386)
==31667== by 0x4062EC: uev_trigger (main.c:777)
==31667== by 0x5713938: service_uevq (uevent.c:118)
==31667== by 0x5713B47: uevent_dispatch (uevent.c:167)
==31667== by 0x406435: uevqloop (main.c:814)
==31667== by 0x4E3F183: start_thread (pthread_create.c:312)
==31667== Address 0x7176790 is 0 bytes inside a block of size 37 free'd
==31667== at 0x4C2BDEC: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31667== by 0x54D7A1B: ??? (in /lib/x86_64-linux-gnu/libudev.so.1.3.5)
==31667== by 0x54D7ADD: ??? (in /lib/x86_64-linux-gnu/libudev.so.1.3.5)
==31667== by 0x54D84B6: udev_device_unref (in /lib/x86_64-linux-gnu/libudev.so.1.3.5)
==31667== by 0x56FD3AA: sysfs_get_tgt_nodename (discovery.c:255)
==31667== by 0x56FDF98: scsi_sysfs_pathinfo (discovery.c:494)
==31667== by 0x56FEA01: sysfs_pathinfo (discovery.c:696)
==31667== by 0x56FF174: pathinfo (discovery.c:830)
==31667== by 0x56FC775: store_pathinfo (discovery.c:57)
==31667== by 0x40504E: uev_add_path (main.c:386)
==31667== by 0x4062EC: uev_trigger (main.c:777)
==31667== by 0x5713938: service_uevq (uevent.c:118)
Which is due to the issue resolved in commit cb0f7127ba90ab5e8e71fc534a0a16cdbe96a88f,
and later:
==31667== Invalid read of size 8
==31667== at 0x56FC388: find_path_by_dev (structs.c:322)
==31667== by 0x404FBF: uev_add_path (main.c:376)
==31667== by 0x4062EC: uev_trigger (main.c:777)
==31667== by 0x5713938: service_uevq (uevent.c:118)
==31667== by 0x5713B47: uevent_dispatch (uevent.c:167)
==31667== by 0x406435: uevqloop (main.c:814)
==31667== by 0x4E3F183: start_thread (pthread_create.c:312)
==31667== by 0x5A2B37C: clone (clone.S:111)
==31667== Address 0x72508a0 is 0 bytes inside a block of size 40 free'd
==31667== at 0x4C2CE8E: realloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.
so)
==31667== by 0x56F4162: vector_del_slot (vector.c:99)
==31667== by 0x571EF72: verify_paths (structs_vec.c:477)
==31667== by 0x405397: ev_add_path (main.c:446)
==31667== by 0x4050C7: uev_add_path (main.c:398)
==31667== by 0x4062EC: uev_trigger (main.c:777)
==31667== by 0x5713938: service_uevq (uevent.c:118)
==31667== by 0x5713B47: uevent_dispatch (uevent.c:167)
==31667== by 0x406435: uevqloop (main.c:814)
==31667== by 0x4E3F183: start_thread (pthread_create.c:312)
==31667== by 0x5A2B37C: clone (clone.S:111)
==31667==
==31667== Invalid free() / delete / delete[] / realloc()
==31667== at 0x4C2CE8E: realloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.
so)
==31667== by 0x56F3F2B: vector_alloc_slot (vector.c:43)
==31667== by 0x56FC05F: store_path (structs.c:225)
==31667== by 0x56FC793: store_pathinfo (discovery.c:62)
==31667== by 0x40504E: uev_add_path (main.c:386)
==31667== by 0x4062EC: uev_trigger (main.c:777)
==31667== by 0x5713938: service_uevq (uevent.c:118)
==31667== by 0x5713B47: uevent_dispatch (uevent.c:167)
==31667== by 0x406435: uevqloop (main.c:814)
==31667== by 0x4E3F183: start_thread (pthread_create.c:312)
==31667== by 0x5A2B37C: clone (clone.S:111)
==31667== Address 0x72508a0 is 0 bytes inside a block of size 40 free'd
==31667== at 0x4C2CE8E: realloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31667== by 0x56F4162: vector_del_slot (vector.c:99)
==31667== by 0x571EF72: verify_paths (structs_vec.c:477)
==31667== by 0x405397: ev_add_path (main.c:446)
==31667== by 0x4050C7: uev_add_path (main.c:398)
==31667== by 0x4062EC: uev_trigger (main.c:777)
==31667== by 0x5713938: service_uevq (uevent.c:118)
==31667== by 0x5713B47: uevent_dispatch (uevent.c:167)
==31667== by 0x406435: uevqloop (main.c:814)
==31667== by 0x4E3F183: start_thread (pthread_create.c:312)
==31667== by 0x5A2B37C: clone (clone.S:111)
==31667==
Which are due to the issue resolved in commit
828d2fbaab304d1ec7db2f563a59eaf2c7a516ea.
As is often the case with use-after-free errors, these only sometimes
result in a a SIGSEGV. The error report from valgrind is perfectly
reproducible. I can intermittently reproduce the SIGSEGV with these
scripts when left running in a loop. Restarting multipathd during while
the test script is running sometimes aids in causing the crash.
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to multipath-tools in Ubuntu.
https://bugs.launchpad.net/bugs/1628723
Title:
Trusty: multipathd SIGSEGV on path addition or removal
Status in multipath-tools package in Ubuntu:
Incomplete
Bug description:
In a system test that involves the repeated addition and removal of iSCSI
targets that form multipath devices, I am observing multipathd exiting with
SIGSEGV.
The issue is reproducible on Trusty with multipath-tools 0.4.9-3ubuntu7.13
as well as when built from source for 0.4.9-3ubuntu7.14.
The following is a typical backtrace from a resulting core file:
Core was generated by `/sbin/multipathd'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 malloc_consolidate (av=av at entry=0x7fe0bc000020) at malloc.c:4151
4151 malloc.c: No such file or directory.
(gdb) bt
#0 malloc_consolidate (av=av at entry=0x7fe0bc000020) at malloc.c:4151
#1 0x00007fe0c6f82ce8 in _int_malloc (av=0x7fe0bc000020, bytes=16384) at malloc.c:3423
#2 0x00007fe0c6f856c0 in __GI___libc_malloc (bytes=16384) at malloc.c:2891
#3 0x00007fe0c79924d7 in dm_task_run () from /lib/x86_64-linux-gnu/libdevmapper.so.1.02.1
#4 0x00007fe0c72d7e58 in dm_map_present (str=0x7fe0bc5a8730 "mpath10p1") at devmapper.c:304
#5 0x0000000000404a77 in ev_add_map (dev=0x7fe0c0019a53 "dm-13", alias=0x7fe0bc5a8730 "mpath10p1", vecs=0x22da100) at main.c:256
#6 0x0000000000404a3c in uev_add_map (uev=0x7fe0c00199d0, vecs=0x22da100) at main.c:243
#7 0x00000000004061ed in uev_trigger (uev=0x7fe0c00199d0, trigger_data=0x22da100) at main.c:755
#8 0x00007fe0c72f6939 in service_uevq (tmpq=0x7fe0c7f8fde0) at uevent.c:118
#9 0x00007fe0c72f6b48 in uevent_dispatch (uev_trigger=0x406130 <uev_trigger>, trigger_data=0x22da100) at uevent.c:167
#10 0x0000000000406436 in uevqloop (ap=0x22da100) at main.c:814
#11 0x00007fe0c7bac184 in start_thread (arg=0x7fe0c7f90700) at pthread_create.c:312
#12 0x00007fe0c6ffd37d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
After debugging with valgrind/memcheck, I have traced the errors reported by
valgrind down to two use-after-free issues that have been resolved in the
upstream multipath-tools but are not included in multipath-tools
0.4.9-3ubuntu7.14.
The first was in commit 828d2fbaab304d1ec7db2f563a59eaf2c7a516ea, which
resolves a bug in which the result value of realloc is assigned to the wrong
place, resulting in continued use of now-freed original block.
The second was in commit cb0f7127ba90ab5e8e71fc534a0a16cdbe96a88f, which
resolves a bug in which a result value from udev_device_get_sysattr_value is
used after the underlying struct udev_device has been released with
udev_unref_device. This also results in a use-after-free.
After applying these patches, running my system stress test no longer results
in SIGSEGV or errors detected by valgrind/memcheck.
I suggest that these two commits be backported.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1628723/+subscriptions
More information about the foundations-bugs
mailing list