[Bug 1874270] Re: NVMe/FC connections fail to reestablish after controller is reset

Jennifer Duong 1874270 at bugs.launchpad.net
Thu Apr 8 17:10:26 UTC 2021 

Dan, I've attached /var/log/syslog and journalctl logs of a recreate
after installing nvme-cli_1.9-1ubuntu0.1+bug1874270v20210408b1_amd64 and
rebooting the host. It looks like connect-all didn't recognize the
"matching" flag.

Apr  8 11:48:45 ICTM1608S01H1 root: JD: Resetting controller B
Apr  8 11:49:39 ICTM1608S01H1 kernel: [  545.652088] lpfc 0000:af:00.1: 5:(0):6172 NVME rescanned DID x3d3800 port_state x2
Apr  8 11:49:39 ICTM1608S01H1 kernel: [  545.652166] nvme nvme2: NVME-FC{2}: controller connectivity lost. Awaiting Reconnect
Apr  8 11:49:39 ICTM1608S01H1 kernel: [  545.652203] lpfc 0000:18:00.1: 1:(0):6172 NVME rescanned DID x3d3800 port_state x2
Apr  8 11:49:39 ICTM1608S01H1 kernel: [  545.652276] nvme nvme6: NVME-FC{6}: controller connectivity lost. Awaiting Reconnect
Apr  8 11:49:39 ICTM1608S01H1 kernel: [  545.673853] nvme nvme2: NVME-FC{2}: io failed due to lldd error 6
Apr  8 11:49:39 ICTM1608S01H1 systemd[1]: Started NVMf auto-connect scan upon nvme discovery controller Events.
Apr  8 11:49:39 ICTM1608S01H1 nvme[7329]: connect-all: unrecognized option '--matching'
Apr  8 11:49:39 ICTM1608S01H1 nvme[7329]: Discover NVMeoF subsystems and connect to them  [  --transport=<LIST>, -t <LIST> ]    --- transport type
Apr  8 11:49:39 ICTM1608S01H1 nvme[7329]:   [  --traddr=<LIST>, -a <LIST> ]       --- transport address
Apr  8 11:49:39 ICTM1608S01H1 nvme[7329]:   [  --trsvcid=<LIST>, -s <LIST> ]      --- transport service id (e.g. IP
Apr  8 11:49:39 ICTM1608S01H1 nvme[7329]:                                             port)
Apr  8 11:49:39 ICTM1608S01H1 nvme[7329]:   [  --host-traddr=<LIST>, -w <LIST> ]  --- host traddr (e.g. FC WWN's)
Apr  8 11:49:39 ICTM1608S01H1 nvme[7329]:   [  --hostnqn=<LIST>, -q <LIST> ]      --- user-defined hostnqn (if default
Apr  8 11:49:39 ICTM1608S01H1 nvme[7329]:                                             not used)
Apr  8 11:49:39 ICTM1608S01H1 nvme[7329]:   [  --hostid=<LIST>, -I <LIST> ]       --- user-defined hostid (if default
Apr  8 11:49:39 ICTM1608S01H1 nvme[7329]:                                             not used)
Apr  8 11:49:39 ICTM1608S01H1 nvme[7329]:   [  --raw=<LIST>, -r <LIST> ]          --- raw output file
Apr  8 11:49:39 ICTM1608S01H1 nvme[7329]:   [  --device=<LIST>, -d <LIST> ]       --- use existing discovery controller
Apr  8 11:49:39 ICTM1608S01H1 nvme[7329]:                                             device
Apr  8 11:49:39 ICTM1608S01H1 nvme[7329]:   [  --keep-alive-tmo=<LIST>, -k <LIST> ] --- keep alive timeout period in
Apr  8 11:49:39 ICTM1608S01H1 nvme[7329]:                                             seconds
Apr  8 11:49:39 ICTM1608S01H1 nvme[7329]:   [  --reconnect-delay=<LIST>, -c <LIST> ] --- reconnect timeout period in
Apr  8 11:49:39 ICTM1608S01H1 nvme[7329]:                                             seconds
Apr  8 11:49:39 ICTM1608S01H1 nvme[7329]:   [  --ctrl-loss-tmo=<LIST>, -l <LIST> ] --- controller loss timeout period in
Apr  8 11:49:39 ICTM1608S01H1 nvme[7329]:                                             seconds
Apr  8 11:49:39 ICTM1608S01H1 nvme[7329]:   [  --hdr_digest, -g ]                 --- enable transport protocol header
Apr  8 11:49:39 ICTM1608S01H1 nvme[7329]:                                             digest (TCP transport)
Apr  8 11:49:39 ICTM1608S01H1 nvme[7329]:   [  --data_digest, -G ]                --- enable transport protocol data
Apr  8 11:49:39 ICTM1608S01H1 nvme[7329]:                                             digest (TCP transport)
Apr  8 11:49:39 ICTM1608S01H1 nvme[7329]:   [  --nr-io-queues=<LIST>, -i <LIST> ] --- number of io queues to use
Apr  8 11:49:39 ICTM1608S01H1 nvme[7329]:                                             (default is core count)
Apr  8 11:49:39 ICTM1608S01H1 nvme[7329]:   [  --nr-write-queues=<LIST>, -W <LIST> ] --- number of write queues to use
Apr  8 11:49:39 ICTM1608S01H1 nvme[7329]:                                             (default 0)
Apr  8 11:49:39 ICTM1608S01H1 nvme[7329]:   [  --nr-poll-queues=<LIST>, -P <LIST> ] --- number of poll queues to use
Apr  8 11:49:39 ICTM1608S01H1 nvme[7329]:                                             (default 0)
Apr  8 11:49:39 ICTM1608S01H1 nvme[7329]:   [  --queue-size=<LIST>, -Q <LIST> ]   --- number of io queue elements to
Apr  8 11:49:39 ICTM1608S01H1 nvme[7329]:                                             use (default 128)
Apr  8 11:49:39 ICTM1608S01H1 nvme[7329]:   [  --persistent, -p ]                 --- persistent discovery connection
Apr  8 11:49:39 ICTM1608S01H1 nvme[7329]:   [  --quiet, -Q ]                      --- suppress already connected errors

** Attachment added: "nvme-cli-1.9-1ubuntu0.1+bug1874270v20210408b1-logs.zip"
   https://bugs.launchpad.net/ubuntu/+source/nvme-cli/+bug/1874270/+attachment/5485673/+files/nvme-cli-1.9-1ubuntu0.1+bug1874270v20210408b1-logs.zip

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to nvme-cli in Ubuntu.
https://bugs.launchpad.net/bugs/1874270

Title:
  NVMe/FC connections fail to reestablish after controller is reset

Status in nvme-cli package in Ubuntu:
  Incomplete

Bug description:
  My FC host can't seem to reestablish NVMe/FC connections after
  resetting one of my E-Series controllers. this is with Ubuntu 20.04
  kernel-5.4.0-25-generic nvme-cli 1.9-1. I'm seeing this on my fabric-
  attached and direct-connect systems. These are the HBAs I'm running
  with:

  Emulex LPe16002B-M6 FV12.4.243.11 DV12.6.0.4 HN:ICTM1610S01H1 OS:Linux
  Emulex LPe16002B-M6 FV12.4.243.11 DV12.6.0.4 HN:ICTM1610S01H1 OS:Linux
  Emulex LPe32002-M2 FV12.4.243.17 DV12.6.0.4 HN:ICTM1610S01H1 OS:Linux
  Emulex LPe32002-M2 FV12.4.243.17 DV12.6.0.4 HN:ICTM1610S01H1 OS:Linux
  Emulex LPe35002-M2 FV12.4.243.23 DV12.6.0.4 HN:ICTM1610S01H1 OS:Linux
  Emulex LPe35002-M2 FV12.4.243.23 DV12.6.0.4 HN:ICTM1610S01H1 OS:Linux

  QLE2742 FW:v8.08.231 DVR:v10.01.00.19-k
  QLE2742 FW:v8.08.231 DVR:v10.01.00.19-k
  QLE2692 FW:v8.08.231 DVR:v10.01.00.19-k
  QLE2692 FW:v8.08.231 DVR:v10.01.00.19-k

  ProblemType: Bug
  DistroRelease: Ubuntu 20.04
  Package: nvme-cli 1.9-1
  ProcVersionSignature: Ubuntu 5.4.0-25.29-generic 5.4.30
  Uname: Linux 5.4.0-25-generic x86_64
  ApportVersion: 2.20.11-0ubuntu27
  Architecture: amd64
  CasperMD5CheckResult: skip
  Date: Wed Apr 22 09:26:00 2020
  InstallationDate: Installed on 2020-04-13 (8 days ago)
  InstallationMedia: Ubuntu-Server 20.04 LTS "Focal Fossa" - Alpha amd64 (20200124)
  ProcEnviron:
   TERM=xterm
   PATH=(custom, no user)
   XDG_RUNTIME_DIR=<set>
   LANG=en_US.UTF-8
   SHELL=/bin/bash
  SourcePackage: nvme-cli
  UpgradeStatus: No upgrade log present (probably fresh install)
  modified.conffile..etc.nvme.hostnqn: ictm1610s01h1-hostnqn
  mtime.conffile..etc.nvme.hostnqn: 2020-04-14T16:02:14.512816

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/nvme-cli/+bug/1874270/+subscriptions

More information about the foundations-bugs mailing list