[Bug 1647067] Re: Dangling UDEV links after removing FC LUNs

[wondra]

apport information

** Tags added: apport-collected trusty

** Description changed:

  We're using Q-Logic QLE2562 Fibre Channel adapters (qla2xxx driver)
  against a HPE 3PAR 7400c storage array in an OpenStack environment. The
  OpenStack 3PAR driver manages volume attachments from the array to the
  servers. There is 4 path multipath to every volume.
  
  As the LUNs are removed, sometimes udev does not remove all links,
  particularly in /run/udev/links and /dev/disk/by-path. The symptoms are
  multiple records in one by-path directory under /run/udev/links, broken
  links to no longer attached luns in dev/disk/by-path and links between
  wrong LUNs and scsi devices there.
  
  OpenStack relies on these links. When another volume is attached using a
  LUN that has these leftover links and it happens that it is the first of
  the 4 paths, OpenStack incorrectly identifies the volume and attaches
  the same volume to multiple instances, leading to data loss.
  
  What could be causing this behavior?
  
  Ubuntu version 14.04 
  Linux version Ubuntu 4.4.0-47.68~14.04.1-generic 4.4.24
  udev 204-5ubuntu20.19
+ --- 
+ ApportVersion: 2.14.1-0ubuntu3.21
+ Architecture: amd64
+ CustomUdevRuleFiles: 20-3par-unmap.rulez
+ DistroRelease: Ubuntu 14.04
+ InstallationDate: Installed on 2015-10-01 (429 days ago)
+ InstallationMedia: Ubuntu-Server 14.04.3 LTS "Trusty Tahr" - Beta amd64 (20150805)
+ MachineType: HP ProLiant DL380 Gen9
+ Package: udev 204-5ubuntu20.19
+ PackageArchitecture: amd64
+ ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-47-generic root=/dev/mapper/hostname--vg-root ro
+ ProcVersionSignature: Ubuntu 4.4.0-47.68~14.04.1-generic 4.4.24
+ Tags:  trusty
+ UdevLog: Error: [Errno 2] No such file or directory: '/var/log/udev'
+ Uname: Linux 4.4.0-47-generic x86_64
+ UpgradeStatus: No upgrade log present (probably fresh install)
+ UserGroups:
+  
+ _MarkForUpload: True
+ dmi.bios.date: 11/03/2014
+ dmi.bios.vendor: HP
+ dmi.bios.version: P89
+ dmi.chassis.type: 23
+ dmi.chassis.vendor: HP
+ dmi.modalias: dmi:bvnHP:bvrP89:bd11/03/2014:svnHP:pnProLiantDL380Gen9:pvr:cvnHP:ct23:cvr:
+ dmi.product.name: ProLiant DL380 Gen9
+ dmi.sys.vendor: HP

** Attachment added: "BootDmesg.txt"
   https://bugs.launchpad.net/bugs/1647067/+attachment/4786936/+files/BootDmesg.txt

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to udev in Ubuntu.
https://bugs.launchpad.net/bugs/1647067

Title:
  Dangling UDEV links after removing FC LUNs

Status in udev package in Ubuntu:
  Confirmed

Bug description:
  We're using Q-Logic QLE2562 Fibre Channel adapters (qla2xxx driver)
  against a HPE 3PAR 7400c storage array in an OpenStack environment.
  The OpenStack 3PAR driver manages volume attachments from the array to
  the servers. There is 4 path multipath to every volume.

  As the LUNs are removed, sometimes udev does not remove all links,
  particularly in /run/udev/links and /dev/disk/by-path. The symptoms
  are multiple records in one by-path directory under /run/udev/links,
  broken links to no longer attached luns in dev/disk/by-path and links
  between wrong LUNs and scsi devices there.

  OpenStack relies on these links. When another volume is attached using
  a LUN that has these leftover links and it happens that it is the
  first of the 4 paths, OpenStack incorrectly identifies the volume and
  attaches the same volume to multiple instances, leading to data loss.

  What could be causing this behavior?

  Ubuntu version 14.04 
  Linux version Ubuntu 4.4.0-47.68~14.04.1-generic 4.4.24
  udev 204-5ubuntu20.19
  --- 
  ApportVersion: 2.14.1-0ubuntu3.21
  Architecture: amd64
  CustomUdevRuleFiles: 20-3par-unmap.rulez
  DistroRelease: Ubuntu 14.04
  InstallationDate: Installed on 2015-10-01 (429 days ago)
  InstallationMedia: Ubuntu-Server 14.04.3 LTS "Trusty Tahr" - Beta amd64 (20150805)
  MachineType: HP ProLiant DL380 Gen9
  Package: udev 204-5ubuntu20.19
  PackageArchitecture: amd64
  ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-47-generic root=/dev/mapper/hostname--vg-root ro
  ProcVersionSignature: Ubuntu 4.4.0-47.68~14.04.1-generic 4.4.24
  Tags:  trusty
  UdevLog: Error: [Errno 2] No such file or directory: '/var/log/udev'
  Uname: Linux 4.4.0-47-generic x86_64
  UpgradeStatus: No upgrade log present (probably fresh install)
  UserGroups:
   
  _MarkForUpload: True
  dmi.bios.date: 11/03/2014
  dmi.bios.vendor: HP
  dmi.bios.version: P89
  dmi.chassis.type: 23
  dmi.chassis.vendor: HP
  dmi.modalias: dmi:bvnHP:bvrP89:bd11/03/2014:svnHP:pnProLiantDL380Gen9:pvr:cvnHP:ct23:cvr:
  dmi.product.name: ProLiant DL380 Gen9
  dmi.sys.vendor: HP

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/udev/+bug/1647067/+subscriptions