[Bug 1874270] Re: NVMe/FC connections fail to reestablish after controller is reset

Jennifer Duong 1874270 at bugs.launchpad.net
Thu Apr 8 19:52:52 UTC 2021 

Dan, I've attached /var/log/syslog and journalctl logs of a recreate
after installing nvme-cli_1.9-1ubuntu0.1+bug1874270v20210408b2_amd64 and
rebooting the host.

Apr  8 14:44:00 ICTM1608S01H1 root: JD: Resetting controller B
Apr  8 14:44:09 ICTM1608S01H1 kernel: [  196.190003] lpfc 0000:af:00.1: 5:(0):6172 NVME rescanned DID x3d3800 port_state x2
Apr  8 14:44:09 ICTM1608S01H1 kernel: [  196.190082] nvme nvme2: NVME-FC{2}: controller connectivity lost. Awaiting Reconnect
Apr  8 14:44:09 ICTM1608S01H1 kernel: [  196.190176] lpfc 0000:18:00.1: 1:(0):6172 NVME rescanned DID x3d3800 port_state x2
Apr  8 14:44:09 ICTM1608S01H1 kernel: [  196.190268] nvme nvme6: NVME-FC{6}: controller connectivity lost. Awaiting Reconnect
Apr  8 14:44:09 ICTM1608S01H1 kernel: [  196.211805] nvme nvme2: NVME-FC{2}: io failed due to lldd error 6
Apr  8 14:44:09 ICTM1608S01H1 systemd[1]: Started NVMf auto-connect scan upon nvme discovery controller Events.
Apr  8 14:44:09 ICTM1608S01H1 systemd[1]: nvmf-connect@\x2d\x2ddevice\x3dnone\x20\x2d\x2dtransport\x3dfc\x20\x2d\x2dtraddr\x3dnn\x2d0x200200a098d8580e:pn\x2d0x202300a098d8580e\x20\x2d\x2dtrsvcid\x3dnone\x20\x2d\x2dhost\x2dtraddr\x3dnn\x2d0x20000090fadcc5ce:pn\x2d0x10000090fadcc5ce.service: Succeeded.
Apr  8 14:44:09 ICTM1608S01H1 systemd-udevd[2827]: filp(639:nvmf-connect@\x2d\x2ddevice\x3dnone\x20\x2d\x2dtransport\x3dfc\x20\x2d\x2dtraddr\x3dnn\x2d0x200200a098d8580e:pn\x2d0x202300a098d8580e\x20\x2d\x2dtrsvcid\x3dnone\x20\x2d\x2dhost\x2dtraddr\x3dnn\x2d0x20000090fadcc5ce:pn\x2d0x10000090fadcc5ce.service): Failed to process device, ignoring: File name too long
Apr  8 14:44:09 ICTM1608S01H1 systemd-udevd[2828]: dentry(639:nvmf-connect@\x2d\x2ddevice\x3dnone\x20\x2d\x2dtransport\x3dfc\x20\x2d\x2dtraddr\x3dnn\x2d0x200200a098d8580e:pn\x2d0x202300a098d8580e\x20\x2d\x2dtrsvcid\x3dnone\x20\x2d\x2dhost\x2dtraddr\x3dnn\x2d0x20000090fadcc5ce:pn\x2d0x10000090fadcc5ce.service): Failed to process device, ignoring: File name too long
Apr  8 14:44:09 ICTM1608S01H1 systemd-udevd[2827]: pid(639:nvmf-connect@\x2d\x2ddevice\x3dnone\x20\x2d\x2dtransport\x3dfc\x20\x2d\x2dtraddr\x3dnn\x2d0x200200a098d8580e:pn\x2d0x202300a098d8580e\x20\x2d\x2dtrsvcid\x3dnone\x20\x2d\x2dhost\x2dtraddr\x3dnn\x2d0x20000090fadcc5ce:pn\x2d0x10000090fadcc5ce.service): Failed to process device, ignoring: File name too long
Apr  8 14:44:09 ICTM1608S01H1 systemd-udevd[2828]: inode_cache(639:nvmf-connect@\x2d\x2ddevice\x3dnone\x20\x2d\x2dtransport\x3dfc\x20\x2d\x2dtraddr\x3dnn\x2d0x200200a098d8580e:pn\x2d0x202300a098d8580e\x20\x2d\x2dtrsvcid\x3dnone\x20\x2d\x2dhost\x2dtraddr\x3dnn\x2d0x20000090fadcc5ce:pn\x2d0x10000090fadcc5ce.service): Failed to process device, ignoring: File name too long
Apr  8 14:44:09 ICTM1608S01H1 systemd-udevd[2829]: kmalloc-rcl-512(639:nvmf-connect@\x2d\x2ddevice\x3dnone\x20\x2d\x2dtransport\x3dfc\x20\x2d\x2dtraddr\x3dnn\x2d0x200200a098d8580e:pn\x2d0x202300a098d8580e\x20\x2d\x2dtrsvcid\x3dnone\x20\x2d\x2dhost\x2dtraddr\x3dnn\x2d0x20000090fadcc5ce:pn\x2d0x10000090fadcc5ce.service): Failed to process device, ignoring: File name too long
Apr  8 14:44:09 ICTM1608S01H1 systemd-udevd[2829]: PING(639:nvmf-connect@\x2d\x2ddevice\x3dnone\x20\x2d\x2dtransport\x3dfc\x20\x2d\x2dtraddr\x3dnn\x2d0x200200a098d8580e:pn\x2d0x202300a098d8580e\x20\x2d\x2dtrsvcid\x3dnone\x20\x2d\x2dhost\x2dtraddr\x3dnn\x2d0x20000090fadcc5ce:pn\x2d0x10000090fadcc5ce.service): Failed to process device, ignoring: File name too long
Apr  8 14:44:09 ICTM1608S01H1 systemd-udevd[2828]: skbuff_head_cache(639:nvmf-connect@\x2d\x2ddevice\x3dnone\x20\x2d\x2dtransport\x3dfc\x20\x2d\x2dtraddr\x3dnn\x2d0x200200a098d8580e:pn\x2d0x202300a098d8580e\x20\x2d\x2dtrsvcid\x3dnone\x20\x2d\x2dhost\x2dtraddr\x3dnn\x2d0x20000090fadcc5ce:pn\x2d0x10000090fadcc5ce.service): Failed to process device, ignoring: File name too long
Apr  8 14:44:09 ICTM1608S01H1 systemd-udevd[2827]: kmalloc-1k(639:nvmf-connect@\x2d\x2ddevice\x3dnone\x20\x2d\x2dtransport\x3dfc\x20\x2d\x2dtraddr\x3dnn\x2d0x200200a098d8580e:pn\x2d0x202300a098d8580e\x20\x2d\x2dtrsvcid\x3dnone\x20\x2d\x2dhost\x2dtraddr\x3dnn\x2d0x20000090fadcc5ce:pn\x2d0x10000090fadcc5ce.service): Failed to process device, ignoring: File name too long
Apr  8 14:44:09 ICTM1608S01H1 systemd-udevd[2830]: sock_inode_cache(639:nvmf-connect@\x2d\x2ddevice\x3dnone\x20\x2d\x2dtransport\x3dfc\x20\x2d\x2dtraddr\x3dnn\x2d0x200200a098d8580e:pn\x2d0x202300a098d8580e\x20\x2d\x2dtrsvcid\x3dnone\x20\x2d\x2dhost\x2dtraddr\x3dnn\x2d0x20000090fadcc5ce:pn\x2d0x10000090fadcc5ce.service): Failed to process device, ignoring: File name too long
Apr  8 14:44:09 ICTM1608S01H1 systemd-udevd[2829]: kmalloc-64(639:nvmf-connect@\x2d\x2ddevice\x3dnone\x20\x2d\x2dtransport\x3dfc\x20\x2d\x2dtraddr\x3dnn\x2d0x200200a098d8580e:pn\x2d0x202300a098d8580e\x20\x2d\x2dtrsvcid\x3dnone\x20\x2d\x2dhost\x2dtraddr\x3dnn\x2d0x20000090fadcc5ce:pn\x2d0x10000090fadcc5ce.service): Failed to process device, ignoring: File name too long
Apr  8 14:44:09 ICTM1608S01H1 systemd-udevd[2830]: mm_struct(639:nvmf-connect@\x2d\x2ddevice\x3dnone\x20\x2d\x2dtransport\x3dfc\x20\x2d\x2dtraddr\x3dnn\x2d0x200200a098d8580e:pn\x2d0x202300a098d8580e\x20\x2d\x2dtrsvcid\x3dnone\x20\x2d\x2dhost\x2dtraddr\x3dnn\x2d0x20000090fadcc5ce:pn\x2d0x10000090fadcc5ce.service): Failed to process device, ignoring: File name too long
Apr  8 14:44:09 ICTM1608S01H1 systemd-udevd[2828]: vm_area_struct(639:nvmf-connect@\x2d\x2ddevice\x3dnone\x20\x2d\x2dtransport\x3dfc\x20\x2d\x2dtraddr\x3dnn\x2d0x200200a098d8580e:pn\x2d0x202300a098d8580e\x20\x2d\x2dtrsvcid\x3dnone\x20\x2d\x2dhost\x2dtraddr\x3dnn\x2d0x20000090fadcc5ce:pn\x2d0x10000090fadcc5ce.service): Failed to process device, ignoring: File name too long
Apr  8 14:44:09 ICTM1608S01H1 systemd-udevd[2832]: cred_jar(639:nvmf-connect@\x2d\x2ddevice\x3dnone\x20\x2d\x2dtransport\x3dfc\x20\x2d\x2dtraddr\x3dnn\x2d0x200200a098d8580e:pn\x2d0x202300a098d8580e\x20\x2d\x2dtrsvcid\x3dnone\x20\x2d\x2dhost\x2dtraddr\x3dnn\x2d0x20000090fadcc5ce:pn\x2d0x10000090fadcc5ce.service): Failed to process device, ignoring: File name too long
Apr  8 14:44:09 ICTM1608S01H1 systemd-udevd[2827]: anon_vma_chain(639:nvmf-connect@\x2d\x2ddevice\x3dnone\x20\x2d\x2dtransport\x3dfc\x20\x2d\x2dtraddr\x3dnn\x2d0x200200a098d8580e:pn\x2d0x202300a098d8580e\x20\x2d\x2dtrsvcid\x3dnone\x20\x2d\x2dhost\x2dtraddr\x3dnn\x2d0x20000090fadcc5ce:pn\x2d0x10000090fadcc5ce.service): Failed to process device, ignoring: File name too long
Apr  8 14:44:09 ICTM1608S01H1 systemd-udevd[2833]: anon_vma(639:nvmf-connect@\x2d\x2ddevice\x3dnone\x20\x2d\x2dtransport\x3dfc\x20\x2d\x2dtraddr\x3dnn\x2d0x200200a098d8580e:pn\x2d0x202300a098d8580e\x20\x2d\x2dtrsvcid\x3dnone\x20\x2d\x2dhost\x2dtraddr\x3dnn\x2d0x20000090fadcc5ce:pn\x2d0x10000090fadcc5ce.service): Failed to process device, ignoring: File name too long
Apr  8 14:44:09 ICTM1608S01H1 kernel: [  196.235193] nvme nvme6: NVME-FC{6}: io failed due to lldd error 6

** Attachment added: "1.9-1ubuntu0.1+bug1874270v20210408b2-logs.zip"
   https://bugs.launchpad.net/ubuntu/+source/nvme-cli/+bug/1874270/+attachment/5485701/+files/1.9-1ubuntu0.1+bug1874270v20210408b2-logs.zip

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to nvme-cli in Ubuntu.
https://bugs.launchpad.net/bugs/1874270

Title:
  NVMe/FC connections fail to reestablish after controller is reset

Status in nvme-cli package in Ubuntu:
  Triaged

Bug description:
  My FC host can't seem to reestablish NVMe/FC connections after
  resetting one of my E-Series controllers. this is with Ubuntu 20.04
  kernel-5.4.0-25-generic nvme-cli 1.9-1. I'm seeing this on my fabric-
  attached and direct-connect systems. These are the HBAs I'm running
  with:

  Emulex LPe16002B-M6 FV12.4.243.11 DV12.6.0.4 HN:ICTM1610S01H1 OS:Linux
  Emulex LPe16002B-M6 FV12.4.243.11 DV12.6.0.4 HN:ICTM1610S01H1 OS:Linux
  Emulex LPe32002-M2 FV12.4.243.17 DV12.6.0.4 HN:ICTM1610S01H1 OS:Linux
  Emulex LPe32002-M2 FV12.4.243.17 DV12.6.0.4 HN:ICTM1610S01H1 OS:Linux
  Emulex LPe35002-M2 FV12.4.243.23 DV12.6.0.4 HN:ICTM1610S01H1 OS:Linux
  Emulex LPe35002-M2 FV12.4.243.23 DV12.6.0.4 HN:ICTM1610S01H1 OS:Linux

  QLE2742 FW:v8.08.231 DVR:v10.01.00.19-k
  QLE2742 FW:v8.08.231 DVR:v10.01.00.19-k
  QLE2692 FW:v8.08.231 DVR:v10.01.00.19-k
  QLE2692 FW:v8.08.231 DVR:v10.01.00.19-k

  ProblemType: Bug
  DistroRelease: Ubuntu 20.04
  Package: nvme-cli 1.9-1
  ProcVersionSignature: Ubuntu 5.4.0-25.29-generic 5.4.30
  Uname: Linux 5.4.0-25-generic x86_64
  ApportVersion: 2.20.11-0ubuntu27
  Architecture: amd64
  CasperMD5CheckResult: skip
  Date: Wed Apr 22 09:26:00 2020
  InstallationDate: Installed on 2020-04-13 (8 days ago)
  InstallationMedia: Ubuntu-Server 20.04 LTS "Focal Fossa" - Alpha amd64 (20200124)
  ProcEnviron:
   TERM=xterm
   PATH=(custom, no user)
   XDG_RUNTIME_DIR=<set>
   LANG=en_US.UTF-8
   SHELL=/bin/bash
  SourcePackage: nvme-cli
  UpgradeStatus: No upgrade log present (probably fresh install)
  modified.conffile..etc.nvme.hostnqn: ictm1610s01h1-hostnqn
  mtime.conffile..etc.nvme.hostnqn: 2020-04-14T16:02:14.512816

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/nvme-cli/+bug/1874270/+subscriptions

More information about the foundations-bugs mailing list