[Bug 1987663] Re: cinder-volume: "Failed to re-export volume, setting to ERROR" with "tgtadm: failed to send request hdr to tgt daemon, Transport endpoint is not connected" on service startup

James Page 1987663 at bugs.launchpad.net
Mon Jul 8 13:04:17 UTC 2024 

This bug was fixed in the package cinder - 2:20.3.1-0ubuntu1.4~cloud0
---------------

 cinder (2:20.3.1-0ubuntu1.4~cloud0) focal; urgency=medium
 .
   * SECURITY UPDATE for Ubuntu Cloud Archive. backport to focal.
 .
 cinder (2:20.3.1-0ubuntu1.4) jammy-security; urgency=medium
 .
   * SECURITY UPDATE: Arbitrary file access via custom QCOW2 external data
     (LP: #2059809)
     - debian/patches/CVE-2024-32498.patch: check for external qcow2 data
       file.
     - debian/control: added qemu-utils to Build-Depends so qemu-img is
       available for new tests.
     - CVE-2024-32498
 .
 cinder (2:20.3.1-0ubuntu1.2) jammy; urgency=medium
 .
   [ Jorge Merlino ]
   * Increase size of volume image metadata values to 65535 bytes
     (LP: #1988942)
 .
   [ Heather Lemon ]
   * Start cinder-volume.service after tgt.service started (LP: #1987663)
     - d/cinder-volume.service.conf: drop-in with 'After=' and 'Wants='
       ('Wants=' is not generated by pkgos-gen-systemd-unit currently).
     - d/cinder-volume.install: ship the systemd service drop-in file.
 .
   [ Seyeong Kim ]
   * HPE3PAR: Failing to clone a volume having children (LP: #1994521):
     - d/p/0001-HPE-3PAR-Fix-umanaged-volumes-snapshots-missing.patch
     - d/p/0002-3PAR-Error-out-if-vol-cannot-be-converted-to-base.patch
     - api 4.0.17 is added as it is in the middle of the main patch
       (4.0.18)
 .
 cinder (2:20.3.1-0ubuntu1.1) jammy; urgency=medium
 .
   * Revert driver assisted volume retype (LP: #2019190):
     - d/p/0001-Revert-Driver-assisted-migration-on-retype-when-it-s.patch
 .
 cinder (2:20.3.1-0ubuntu1) jammy; urgency=medium
 .
   * New stable point release for OpenStack Yoga (LP: #2037332).
 .
 cinder (2:20.3.0-0ubuntu1) jammy; urgency=medium
 .
   * New stable point release for OpenStack Yoga (LP: #2025503).
   * d/p/CVE-2023-2088.patch: Dropped. Fixed in point release.
 .
 cinder (2:20.2.0-0ubuntu1.1) jammy-security; urgency=medium
 .
   * SECURITY UPDATE: Unauthorized File Access (LP: #2021980)
     - debian/patches/CVE-2023-2088.patch: Reject unsafe delete
       attachment calls.
     - CVE-2023-2088
 .
 cinder (2:20.2.0-0ubuntu1) jammy; urgency=medium
 .
   * New stable point release for OpenStack Yoga (LP: #2019759).
   * d/p/lp1945500.patch: Dropped. Fixed in stable point release.
 .
 cinder (2:20.1.0-0ubuntu2.2) jammy-security; urgency=medium
 .
   * SECURITY REGRESSION: Regressions in other projects (LP: #2020111)
     - debian/patches/series: Do not apply CVE-2023-2088.patch until
       patches are ready for all upstream OpenStack projects.
     - CVE-2023-2088
 .
 cinder (2:20.1.0-0ubuntu2.1) jammy-security; urgency=medium
 .
   * SECURITY UPDATE: Unauthorized File Access
     - debian/patches/CVE-2023-2088.patch: Reject unsafe delete
       attachment calls.
     - CVE-2023-2088
 .
 cinder (2:20.1.0-0ubuntu2) jammy; urgency=medium
 .
   * d/p/lp1945500.patch: Filter reserved image properties (LP: #1945500).
 .
 cinder (2:20.1.0-0ubuntu1) jammy; urgency=medium
 .
   * New stable point release for OpenStack Yoga (LP: #2004030).
 .
 cinder (2:20.0.1-0ubuntu1) jammy; urgency=medium
 .
   * d/gbp.conf: Create stable/yoga branch.
   * New stable point release for OpenStack Yoga (LP: #1985084).
 .
 cinder (2:20.0.0-0ubuntu1) jammy; urgency=medium
 .
   * d/watch: Scope to 20.x.
   * New upstream release for OpenStack Yoga.
   * d/control: Align (Build-)Depends with upstream.
 .
 cinder (2:19.0.0+git2022030310.b49fb59a6-0ubuntu2) jammy; urgency=medium
 .
   * d/p/fix-qos-computation.patch: Cherry-pick from upstream review to
     fix TypeError exception when generating QOS feature name (LP: #1948507).
 .
 cinder (2:19.0.0+git2022030310.b49fb59a6-0ubuntu1) jammy; urgency=medium
 .
   * New upstream snapshot for OpenStack Yoga.
 .
 cinder (2:19.0.0+git2022011215.23494a6d6-0ubuntu1) jammy; urgency=medium
 .
   * New upstream snapshot for OpenStack Yoga.
   * d/control, d/rules: Bump debhelper compat to 13.
 .
 cinder (2:19.0.0+git2021120811.e5ef39604-0ubuntu2) jammy; urgency=medium
 .
   * d/t/control: Add allow-stderr restriction to prevent autopkgtest failure
     when SQLAlchemy issues a warning.
 .
 cinder (2:19.0.0+git2021120811.e5ef39604-0ubuntu1) jammy; urgency=medium
 .
   * New upstream snapshot for OpenStack Yoga.
   * d/control: Align (Build-)Depends with upstream.
 .
 cinder (2:19.0.0-0ubuntu2) impish; urgency=medium
 .
   * d/py3dist-overrides: Add SQLAlchemy to ensure d/control is not overridden.
   * d/control: Align (Build-)Depends with upstream.
 .
 cinder (2:19.0.0-0ubuntu1) impish; urgency=medium
 .
   * d/watch: Scope to 19.x.
   * New upstream release for OpenStack Xena.
 .
 cinder (2:19.0.0~b1+git2021091409.768b8996b-0ubuntu1) impish; urgency=medium
 .
   * New upstream snapshot for OpenStack Xena.
 .
 cinder (2:18.0.0+git2021072116.81f2aaeea-0ubuntu1) impish; urgency=medium
 .
   * New upstream snapshot for OpenStack Xena.
   * d/control: Align (Build-)Depends with upstream.
 .
 cinder (2:18.0.0+git2021061414.d5f0e5187-0ubuntu1) impish; urgency=medium
 .
   * New upstream snapshot for OpenStack Xena.
   * d/control: Align (Build-)Depends with upstream.
 .
 cinder (2:18.0.0-0ubuntu3) hirsute; urgency=medium
 .
   * d/p/skip-victoria-failures.patch: Restored and rebased. This is still
     necessary for Launchpad builds.
 .
 cinder (2:18.0.0-0ubuntu2) hirsute; urgency=medium
 .
   * d/p/skip-victoria-failures.patch: Dropped. Fixed upstream.
   * d/p/add-mock-psutil-in-quobyte-tests.patch: Dropped. Fixed upstream.
 .
 cinder (2:18.0.0-0ubuntu1) hirsute; urgency=medium
 .
   * New upstream release for OpenStack Wallaby.
 .
 cinder (2:18.0.0~b1-0ubuntu2) hirsute; urgency=medium
 .
   * d/py3dist-overrides: Add boto3 which is a Suggests.
 .
 cinder (2:18.0.0~b1-0ubuntu1) hirsute; urgency=medium
 .
   * d/watch: Track 18.x series.
   * New upstream milestone for OpenStack Wallaby.
   * d/control: Align (Build-)Depends with upstream.
   * d/p/skip-moto-tests.patch: Skip test dependency that is not yet
     packaged in Ubuntu and was added late in cycle.
   * d/p/patch-botocore-exceptions.patch: Account for changes to botocore
     vendored exceptions.
 .
 cinder (2:17.0.1+git2021012507.d26092348-0ubuntu3) hirsute; urgency=medium
 .
   * d/*: Remove tgt in favor of targetcli-fb.
 .
 cinder (2:17.0.1+git2021012507.d26092348-0ubuntu2) hirsute; urgency=medium
 .
   * d/p/add-mock-psutil-in-quobyte-tests.patch: Add a mock of psutil
     disk_partitions to fix failing unit test (LP: #1913607).
 .
 cinder (2:17.0.1+git2021012507.d26092348-0ubuntu1) hirsute; urgency=medium
 .
   * New upstream snapshot for OpenStack Wallaby.
 .
 cinder (2:17.0.1+git2021010614.a9c922ab7-0ubuntu1) hirsute; urgency=medium
 .
   * New upstream snapshot for OpenStack Wallaby.
   * d/control: Align (Build-)Depends with upstream.
 .
 cinder (2:17.0.1+git2020120911.d3ffa90ba-0ubuntu1) hirsute; urgency=medium
 .
   * New upstream snapshot for OpenStack Wallaby.
   * d/control: Align (Build-)Depends with upstream.
 .
 cinder (2:17.0.0-0ubuntu1) groovy; urgency=medium
 .
   * New upstream release for OpenStack Victoria.
 .
 cinder (2:17.0.0~rc2-0ubuntu1) groovy; urgency=medium
 .
   * d/control: Update VCS paths for move to lp:~ubuntu-openstack-dev.
   * d/watch: Track 17.x series.
   * New upstream release candidate for OpenStack Victoria.
   * d/control: Align (Build-)Depends with upstream.
 .
 cinder (2:17.0.0~b3~git2020091007.afcaf0b9d-0ubuntu3) groovy; urgency=medium
 .
   * d/py3dist-overrides: Add python3-zstd to py3dist-overrides.
 .
 cinder (2:17.0.0~b3~git2020091007.afcaf0b9d-0ubuntu2) groovy; urgency=medium
 .
   * d/p/skip-victoria-failures.patch: Restored to skip failing unit tests.
 .
 cinder (2:17.0.0~b3~git2020091007.afcaf0b9d-0ubuntu1) groovy; urgency=medium
 .
   * d/control: Remove Breaks/Replaces that are older than Focal (LP: #1878419).
   * New upstream snapshot for OpenStack Victoria.
   * d/control: Align (Build-)Depends with upstream.
   * d/p/*: Removed. Changes landed upstream and tests fixed.
   * d/control: Add new python3-zstd package to depends.
 .
 cinder (2:17.0.0~b2~git2020073012.2124f39f9-0ubuntu1) groovy; urgency=medium
 .
   * New upstream snapshot for OpenStack Victoria.
   * d/p/*: Refreshed.
 .
 cinder (2:17.0.0~b1~git2020062409.85fcf1057-0ubuntu1) groovy; urgency=medium
 .
   * SECURITY UPDATE: Dell EMC ScaleIO/VxFlex OS Backend Credentials Exposure
     (LP: #1823200)
     - Remove VxFlex OS credentials from connection_properties. Passwords are
       now stored in separate file and are retrieved during each attach/detach
       operation. Cinder is patched in 16.1.0 stable point release.
     - d/control: Align (Build-)Depends with min version of python3-os-brick
       required to fix credential exposure.
     - CVE-2020-10755
   * New upstream snapshot for OpenStack Victoria.
   * d/control: Align (Build-)Depends with upstream.
   * d/p/py38skip.patch: Dropped. No longer needed.
   * d/p/skip-victoria-failures.patch: Rebased and updated with upstream bug.


** Changed in: cloud-archive/yoga
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to cinder in Ubuntu.
https://bugs.launchpad.net/bugs/1987663

Title:
  cinder-volume: "Failed to re-export volume, setting to ERROR" with
  "tgtadm: failed to send request hdr to tgt daemon, Transport endpoint
  is not connected" on service startup

Status in Ubuntu Cloud Archive:
  Fix Released
Status in Ubuntu Cloud Archive antelope series:
  Triaged
Status in Ubuntu Cloud Archive bobcat series:
  Fix Released
Status in Ubuntu Cloud Archive caracal series:
  Fix Released
Status in Ubuntu Cloud Archive ussuri series:
  Triaged
Status in Ubuntu Cloud Archive victoria series:
  Won't Fix
Status in Ubuntu Cloud Archive wallaby series:
  Won't Fix
Status in Ubuntu Cloud Archive xena series:
  Won't Fix
Status in Ubuntu Cloud Archive yoga series:
  Fix Released
Status in Ubuntu Cloud Archive zed series:
  Won't Fix
Status in cinder package in Ubuntu:
  Fix Released
Status in cinder source package in Bionic:
  Won't Fix
Status in cinder source package in Focal:
  Fix Released
Status in cinder source package in Jammy:
  Fix Released
Status in cinder source package in Lunar:
  Won't Fix
Status in cinder source package in Mantic:
  Fix Released
Status in cinder source package in Noble:
  Fix Released

Bug description:
  [Impact]

   * The cinder-volume service might fail to re-export volumes
     in-use on startup if tgt.service isn't fully started yet.

   * This affects the 'lvm' driver with 'tgtadm' target helper
     (which runs 'tgtadm' commands that need the service ready).

   * Snippets from /var/log/cinder/cinder-volume.log:

     Failed to re-export volume, setting to ERROR.
     ...
     Command: tgtadm --lld iscsi --op show --mode target
     ...
     Stderr: 'tgtadm: failed to send request hdr to tgt daemon,
     Transport endpoint is not connected\n'

   * This issue is more common in openstack compute nodes
     with networking (ovs/ovn) that takes long to startup,
     which might delay the startup of tgt.service _after_
     cinder-volume.service.

  [Test Steps]

   * Steps to reproduce are detailed in comment #3.
     Summary:

   * Install mysql, rabbitmq-server, keystone, and cinder
     (controller and storage nodes; backup node unneeded).

   * Configure cinder-volume (storage node) for LVM backend
     and tgtadm iSCSI helper (tgt.service).

   * Create a cinder volume, and configure it as 'in-use'.

   * Simulate a start delay on tgt.service with a drop-in.

   * Restart services: cinder-volume.service tgt.service

   * Check sequence of service startup.

   * Check status of the cinder volume:
     'in-use' (expected) or 'error' (bug).

   * Check log file /var/log/cinder/cinder-volume.log for
     'tgtadm: failed to send request hdr to tgt daemon'.

  [Regression Potential]

   * The fix introduces systemd unit 'After=' and 'Wants='
     properties for tgt.service in cinder-volume.service,
     thus might delay the boot process (multi-user.target).

       $ systemctl show cinder-volume.service | grep WantedBy=
       WantedBy=multi-user.target

   * However, the boot process already waits on tgt.service
     anyway, thus the difference (if any) should not be big,
     and would provide more correct behavior.

       $ systemctl show tgt.service | grep WantedBy=
       WantedBy=multi-user.target

   * If tgt.service is not present (tgt package not installed)
     _no errors_ occur, as both 'After=' and 'Wants=' are weak
     ordering/dependency properties (man 5 systemd.unit).

  [Other Info]

   * The fix uses a systemd service drop-in snippet because
     the service unit is generated by openstack-pkg-tools
     (pkgos-gen-systemd-unit) based on the 'init' service,
     and it only emits 'Wants=' for network-online.target.

   * Changing that in openstack-pkg-tools changes behavior
     in stable releases, and only manifest at build time,
     for many openstack packages that have no issues now.

   * We'll continue to pursue the general improvement in
     Debian, so it comes into Ubuntu development release,
     but for the Ubuntu stable releases, this should do.

  [Original Bug Description]

  Real-world example in comment #2.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1987663/+subscriptions

More information about the Ubuntu-openstack-bugs mailing list