[Bug 1305343] [NEW] tgt connection issues can make it impossible to delete targets

[Vish Ishaya]

Public bug reported:

Under heavy load, openiscsi will sometimes report connection errors when
using tgt:

Apr  8 22:02:44 node003-cont001 iscsid: Connection2:0 to [target:
iqn.2010-10.org.openstack:volume-b32d317e-4063-4e4f-963e-b0fdc29198f9,
portal: 172.21.1.1,3260] through [iface: default] is operational now

Apr  8 22:10:53 node003-cont001 iscsid: Kernel reported iSCSI connection 2:0 error (1021 - ISCSI_ERR_SCSI_EH_SESSION_RST: Session was dropped as a result of SCSI error recovery) state (3)
Apr  8 22:10:56 node003-cont001 iscsid: connection2:0 is operational after recovery (1 attempts)
Apr  8 22:18:47 node003-cont001 iscsid: Kernel reported iSCSI connection 2:0 error (1021 - ISCSI_ERR_SCSI_EH_SESSION_RST: Session was dropped as a result of SCSI error recovery) state (3)
Apr  8 22:18:50 node003-cont001 iscsid: connection2:0 is operational after recovery (1 attempts)

Even though openiscsi has disconnected:

Apr  8 22:34:04 node003-cont001 iscsid: Connection2:0 to [target:
iqn.2010-10.org.openstack:volume-b32d317e-4063-4e4f-963e-b0fdc29198f9,
portal: 172.21.1.1,3260] through [iface: default] is shutdown.

tgt still shows the old connections:

$ sudo tgtadm --op show --mode target
Target 5: iqn.2010-10.org.openstack:volume-b32d317e-4063-4e4f-963e-b0fdc29198f9
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
        I_T nexus: 5
            Initiator: iqn.1993-08.org.debian:01:3844c047b3a9 alias: node003-cont001
            Connection: 0
                IP Address: 172.21.1.3
        I_T nexus: 6
            Initiator: iqn.1993-08.org.debian:01:3844c047b3a9 alias: node003-cont001
            Connection: 0
                IP Address: 172.21.1.3

When these duplicate connections are there, the volume cannot be
deleted:

$ sudo tgt-admin -v --force --delete iqn.2010-10.org.openstack:volume-b32d317e-4063-4e4f-963e-b0fdc29198f9
# Removing target: iqn.2010-10.org.openstack:volume-b32d317e-4063-4e4f-963e-b0fdc29198f9
tgtadm -C 0 --op delete --mode conn --tid 5 --sid 6 --cid 0
tgtadm -C 0 --op delete --mode conn --tid 5 --sid 5 --cid 0
tgtadm -C 0 --mode target --op delete --tid=5
tgtadm: this target is still active
Command:
	tgtadm -C 0 --mode target --op delete --tid=5
exited with code: 22.

The following errors show up in syslog illustrating the force delete
failing:

Apr  9 22:31:50 node001-cont001 tgtd: conn_close_admin(237) close 6 0
Apr  9 22:31:50 node001-cont001 tgtd: tgt_event_modify(241) Cannot find event 21
Apr  9 22:31:50 node001-cont001 tgtd: iscsi_event_modify(558) tgt_event_modify failed
Apr  9 22:31:50 node001-cont001 tgtd: conn_close_admin(237) close 5 0
Apr  9 22:31:50 node001-cont001 tgtd: tgt_event_modify(241) Cannot find event 19
Apr  9 22:31:50 node001-cont001 tgtd: iscsi_event_modify(558) tgt_event_modify failed
Apr  9 22:31:50 node001-cont001 tgtd: tgt_target_destroy(2211) target 5 still has it nexus

And the connections are still there:

$ sudo tgt-admin -v --show
tgtadm -C 0 --op show --mode target
Target 5: iqn.2010-10.org.openstack:volume-b32d317e-4063-4e4f-963e-b0fdc29198f9
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
        I_T nexus: 5
            Initiator: iqn.1993-08.org.debian:01:3844c047b3a9 alias: node003-cont001
            Connection: 0
                IP Address: 172.21.1.3
        I_T nexus: 6
            Initiator: iqn.1993-08.org.debian:01:3844c047b3a9 alias: node003-cont001
            Connection: 0
                IP Address: 172.21.1.3

** Affects: tgt (Ubuntu)
     Importance: Undecided
         Status: New

** Summary changed:

- tgt connection issues can make it impossible to delete volumes
+ tgt connection issues can make it impossible to delete targets

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to tgt in Ubuntu.
https://bugs.launchpad.net/bugs/1305343

Title:
  tgt connection issues can make it impossible to delete targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/tgt/+bug/1305343/+subscriptions