[Bug 1628723] Re: Trusty: multipathd SIGSEGV on path addition or removal

[ChristianEhrhardt]

Thank you so much already for the debugging and identifying the patches!

To continue integrating and testing and finally SRUing the patches one needs to be able to reproduce the test. So far you just said "repeated addition and removal of iSCSI
targets".
Would you have a series of commands at hand from your testing that one can use to recreate this?

** Tags added: server-next

** Changed in: multipath-tools (Ubuntu)
       Status: New => Triaged

** Changed in: multipath-tools (Ubuntu)
       Status: Triaged => Incomplete

** Changed in: multipath-tools (Ubuntu)
   Importance: Undecided => High

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to multipath-tools in Ubuntu.
https://bugs.launchpad.net/bugs/1628723

Title:
  Trusty: multipathd SIGSEGV on path addition or removal

Status in multipath-tools package in Ubuntu:
  Incomplete

Bug description:
  In a system test that involves the repeated addition and removal of iSCSI
  targets that form multipath devices, I am observing multipathd exiting with
  SIGSEGV.

  The issue is reproducible on Trusty with multipath-tools 0.4.9-3ubuntu7.13
  as well as when built from source for 0.4.9-3ubuntu7.14.

  The following is a typical backtrace from a resulting core file:

  Core was generated by `/sbin/multipathd'.
  Program terminated with signal SIGSEGV, Segmentation fault.
  #0  malloc_consolidate (av=av at entry=0x7fe0bc000020) at malloc.c:4151
  4151	malloc.c: No such file or directory.
  (gdb) bt
  #0  malloc_consolidate (av=av at entry=0x7fe0bc000020) at malloc.c:4151
  #1  0x00007fe0c6f82ce8 in _int_malloc (av=0x7fe0bc000020, bytes=16384) at malloc.c:3423
  #2  0x00007fe0c6f856c0 in __GI___libc_malloc (bytes=16384) at malloc.c:2891
  #3  0x00007fe0c79924d7 in dm_task_run () from /lib/x86_64-linux-gnu/libdevmapper.so.1.02.1
  #4  0x00007fe0c72d7e58 in dm_map_present (str=0x7fe0bc5a8730 "mpath10p1") at devmapper.c:304
  #5  0x0000000000404a77 in ev_add_map (dev=0x7fe0c0019a53 "dm-13", alias=0x7fe0bc5a8730 "mpath10p1", vecs=0x22da100) at main.c:256
  #6  0x0000000000404a3c in uev_add_map (uev=0x7fe0c00199d0, vecs=0x22da100) at main.c:243
  #7  0x00000000004061ed in uev_trigger (uev=0x7fe0c00199d0, trigger_data=0x22da100) at main.c:755
  #8  0x00007fe0c72f6939 in service_uevq (tmpq=0x7fe0c7f8fde0) at uevent.c:118
  #9  0x00007fe0c72f6b48 in uevent_dispatch (uev_trigger=0x406130 <uev_trigger>, trigger_data=0x22da100) at uevent.c:167
  #10 0x0000000000406436 in uevqloop (ap=0x22da100) at main.c:814
  #11 0x00007fe0c7bac184 in start_thread (arg=0x7fe0c7f90700) at pthread_create.c:312
  #12 0x00007fe0c6ffd37d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

  After debugging with valgrind/memcheck, I have traced the errors reported by
  valgrind down to two use-after-free issues that have been resolved in the
  upstream multipath-tools but are not included in multipath-tools
  0.4.9-3ubuntu7.14.

  The first was in commit 828d2fbaab304d1ec7db2f563a59eaf2c7a516ea, which
  resolves a bug in which the result value of realloc is assigned to the wrong
  place, resulting in continued use of now-freed original block.

  The second was in commit cb0f7127ba90ab5e8e71fc534a0a16cdbe96a88f, which
  resolves a bug in which a result value from udev_device_get_sysattr_value is
  used after the underlying struct udev_device has been released with
  udev_unref_device.  This also results in a use-after-free.

  After applying these patches, running my system stress test no longer results
  in SIGSEGV or errors detected by valgrind/memcheck.

  I suggest that these two commits be backported.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1628723/+subscriptions