[Bug 1628723] Re: Trusty: multipathd SIGSEGV on path addition or removal

[Christopher Unkel]

Thanks for the reply.

As regards 1535989, as I understand that fix is included in
0.4.9-3ubuntu7.14 and I am able to reproduce the issue in that build.

When it comes to reproducing the issue, currently I have something that
cannot easily be replicated outside my test environment.  Basically my
environment has a number (specifically 4) of iSCSI targets serving the
same drives available on the network.  This test is a loop that randomly
either does an iSCSI login to all the paths available for a drive that
is available, or does an iSCSI logout to all paths on the device.  In
turn multipathd processes the path additions or removals triggering the
issue.

I think that the issue in commit
cb0f7127ba90ab5e8e71fc534a0a16cdbe96a88f is triggered by multipathd
processing a path addition to any iSCSI device.  I don't think a SIGSEGV
is reproducible from this issue alone, but the valgrind/memcheck report
of free-after-use is.

I think that the issue in commit
828d2fbaab304d1ec7db2f563a59eaf2c7a516ea would be triggered by
mulitpathd processing any path removal from a multipath with multiple
paths and is not about iSCSI in particular.  In the iSCSI context the
path removal is a consequence of an iSCSI logout rather than a physical
reconfiguration.  I think this is the issue causing the SIGSEGV in
practice.

Let me see if I can isolate a test script than can be used standalone
and ideally be incorporated into any regression suite you have.  It may
take a couple of business days.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to multipath-tools in Ubuntu.
https://bugs.launchpad.net/bugs/1628723

Title:
  Trusty: multipathd SIGSEGV on path addition or removal

Status in multipath-tools package in Ubuntu:
  Incomplete

Bug description:
  In a system test that involves the repeated addition and removal of iSCSI
  targets that form multipath devices, I am observing multipathd exiting with
  SIGSEGV.

  The issue is reproducible on Trusty with multipath-tools 0.4.9-3ubuntu7.13
  as well as when built from source for 0.4.9-3ubuntu7.14.

  The following is a typical backtrace from a resulting core file:

  Core was generated by `/sbin/multipathd'.
  Program terminated with signal SIGSEGV, Segmentation fault.
  #0  malloc_consolidate (av=av at entry=0x7fe0bc000020) at malloc.c:4151
  4151	malloc.c: No such file or directory.
  (gdb) bt
  #0  malloc_consolidate (av=av at entry=0x7fe0bc000020) at malloc.c:4151
  #1  0x00007fe0c6f82ce8 in _int_malloc (av=0x7fe0bc000020, bytes=16384) at malloc.c:3423
  #2  0x00007fe0c6f856c0 in __GI___libc_malloc (bytes=16384) at malloc.c:2891
  #3  0x00007fe0c79924d7 in dm_task_run () from /lib/x86_64-linux-gnu/libdevmapper.so.1.02.1
  #4  0x00007fe0c72d7e58 in dm_map_present (str=0x7fe0bc5a8730 "mpath10p1") at devmapper.c:304
  #5  0x0000000000404a77 in ev_add_map (dev=0x7fe0c0019a53 "dm-13", alias=0x7fe0bc5a8730 "mpath10p1", vecs=0x22da100) at main.c:256
  #6  0x0000000000404a3c in uev_add_map (uev=0x7fe0c00199d0, vecs=0x22da100) at main.c:243
  #7  0x00000000004061ed in uev_trigger (uev=0x7fe0c00199d0, trigger_data=0x22da100) at main.c:755
  #8  0x00007fe0c72f6939 in service_uevq (tmpq=0x7fe0c7f8fde0) at uevent.c:118
  #9  0x00007fe0c72f6b48 in uevent_dispatch (uev_trigger=0x406130 <uev_trigger>, trigger_data=0x22da100) at uevent.c:167
  #10 0x0000000000406436 in uevqloop (ap=0x22da100) at main.c:814
  #11 0x00007fe0c7bac184 in start_thread (arg=0x7fe0c7f90700) at pthread_create.c:312
  #12 0x00007fe0c6ffd37d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

  After debugging with valgrind/memcheck, I have traced the errors reported by
  valgrind down to two use-after-free issues that have been resolved in the
  upstream multipath-tools but are not included in multipath-tools
  0.4.9-3ubuntu7.14.

  The first was in commit 828d2fbaab304d1ec7db2f563a59eaf2c7a516ea, which
  resolves a bug in which the result value of realloc is assigned to the wrong
  place, resulting in continued use of now-freed original block.

  The second was in commit cb0f7127ba90ab5e8e71fc534a0a16cdbe96a88f, which
  resolves a bug in which a result value from udev_device_get_sysattr_value is
  used after the underlying struct udev_device has been released with
  udev_unref_device.  This also results in a use-after-free.

  After applying these patches, running my system stress test no longer results
  in SIGSEGV or errors detected by valgrind/memcheck.

  I suggest that these two commits be backported.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1628723/+subscriptions