[Bug 1623311] [NEW] instance file system will become read only after disabling a FC port

[KC]

Public bug reported:

1. This bug is triggered will qualifying Canonical OpenStack(Mitaka) with EMC XtremIO;
2. The Ubuntu server deployed through MAAS is Ubuntu 14.04.5 LTS;
3. multipath-tools 0.4.9-3ubuntu7.13 is used;
4. Below multipath.conf is used:
defaults {
        verbosity 2
        wwids_file /etc/multipath/wwids
        user_friendly_names no
        flush_on_last_del yes
}
blacklist {
        devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
        devnode "^hd[a-z]"
        devnode "^dcssblk[0-9]*"
        devnode "^cciss!c[0-9]d[0-9]*"
        devnode "^nvme.*"
        device {
                vendor "DGC"
                product "LUNZ"
        }
        device {
                vendor "EMC"
                product "LUNZ"
        }
        device {
                vendor "IBM"
                product "S/390.*"
        }
        device {
                vendor "IBM"
                product "S/390.*"
        }
        device {
                vendor "STK"
                product "Universal Xport"
        }
}
blacklist_exceptions {
}
devices {
        device {
                vendor XtremIO
                product XtremApp
                path_selector "queue-length 0"
                rr_min_io_rq 1
                path_grouping_policy multibus
                path_checker tur
                failback immediate
                fast_io_fail_tmo 15
        }
... other array's device definition ...
}
multipaths {
}

5. XtremIO array paths can be managed successfully:
ubuntu at CA31:~$ sudo multipath -ll
3514f0c5e4c0002aa dm-3 XtremIO ,XtremApp
size=5.0G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 13:0:7:1   sdq   65:0   active ready  running
|-+- policy='round-robin 0' prio=1 status=enabled
| `- 13:0:6:1   sdr   65:16  active ready  running
|-+- policy='round-robin 0' prio=1 status=enabled
| `- 12:0:6:1   sdt   65:48  active ready  running
`-+- policy='round-robin 0' prio=1 status=enabled
  `- 12:0:7:1   sds   65:32  active ready  running

6. Reproduce steps:
a. Launch a instance(cirros-0.3.4-x86_64-disk.img) with the help of OpenStack dashboard;
b. Attach the volume seen in step 5 to the instance;
c. Format and mount the volume in the instance, then start IO;
d. Disable a FC port(2 x paths will be impacted) at the switch side;
e. After a while(30 ~ 60 seconds), the file system based on the volume in the instance will become read only;
f. Restart the instance will clear the problem;
g. Based on the symptom, it seems multipath does not fail over IO of the instance to alive paths in time.

7. Logs from the host and instance are attached to this bug.

** Affects: multipath-tools (Ubuntu)
     Importance: Undecided
         Status: New

** Attachment added: "multipath-tools.bug.tar.gz"
   https://bugs.launchpad.net/bugs/1623311/+attachment/4740527/+files/multipath-tools.bug.tar.gz

** Summary changed:

- instance file system will become ready only after disabling a FC port
+ instance file system will become read only after disabling a FC port

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to multipath-tools in Ubuntu.
https://bugs.launchpad.net/bugs/1623311

Title:
  instance file system will become read only after disabling a FC port

Status in multipath-tools package in Ubuntu:
  New

Bug description:
  1. This bug is triggered will qualifying Canonical OpenStack(Mitaka) with EMC XtremIO;
  2. The Ubuntu server deployed through MAAS is Ubuntu 14.04.5 LTS;
  3. multipath-tools 0.4.9-3ubuntu7.13 is used;
  4. Below multipath.conf is used:
  defaults {
          verbosity 2
          wwids_file /etc/multipath/wwids
          user_friendly_names no
          flush_on_last_del yes
  }
  blacklist {
          devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
          devnode "^hd[a-z]"
          devnode "^dcssblk[0-9]*"
          devnode "^cciss!c[0-9]d[0-9]*"
          devnode "^nvme.*"
          device {
                  vendor "DGC"
                  product "LUNZ"
          }
          device {
                  vendor "EMC"
                  product "LUNZ"
          }
          device {
                  vendor "IBM"
                  product "S/390.*"
          }
          device {
                  vendor "IBM"
                  product "S/390.*"
          }
          device {
                  vendor "STK"
                  product "Universal Xport"
          }
  }
  blacklist_exceptions {
  }
  devices {
          device {
                  vendor XtremIO
                  product XtremApp
                  path_selector "queue-length 0"
                  rr_min_io_rq 1
                  path_grouping_policy multibus
                  path_checker tur
                  failback immediate
                  fast_io_fail_tmo 15
          }
  ... other array's device definition ...
  }
  multipaths {
  }

  5. XtremIO array paths can be managed successfully:
  ubuntu at CA31:~$ sudo multipath -ll
  3514f0c5e4c0002aa dm-3 XtremIO ,XtremApp
  size=5.0G features='0' hwhandler='0' wp=rw
  |-+- policy='round-robin 0' prio=1 status=active
  | `- 13:0:7:1   sdq   65:0   active ready  running
  |-+- policy='round-robin 0' prio=1 status=enabled
  | `- 13:0:6:1   sdr   65:16  active ready  running
  |-+- policy='round-robin 0' prio=1 status=enabled
  | `- 12:0:6:1   sdt   65:48  active ready  running
  `-+- policy='round-robin 0' prio=1 status=enabled
    `- 12:0:7:1   sds   65:32  active ready  running

  6. Reproduce steps:
  a. Launch a instance(cirros-0.3.4-x86_64-disk.img) with the help of OpenStack dashboard;
  b. Attach the volume seen in step 5 to the instance;
  c. Format and mount the volume in the instance, then start IO;
  d. Disable a FC port(2 x paths will be impacted) at the switch side;
  e. After a while(30 ~ 60 seconds), the file system based on the volume in the instance will become read only;
  f. Restart the instance will clear the problem;
  g. Based on the symptom, it seems multipath does not fail over IO of the instance to alive paths in time.

  7. Logs from the host and instance are attached to this bug.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1623311/+subscriptions