[Bug 2020139] Re: [SRU] Image fetch speed from Glance to Cinder volume service can be slow and capped by client CPU

[Rodrigo Barbieri]

** Also affects: cloud-archive/yoga
   Importance: Undecided
       Status: New

** Also affects: cloud-archive/caracal
   Importance: Undecided
       Status: New

** Also affects: cloud-archive/bobcat
   Importance: Undecided
       Status: New

** Changed in: cloud-archive/bobcat
       Status: New => Fix Released

** Changed in: cloud-archive/caracal
       Status: New => Fix Released

** Changed in: cloud-archive
       Status: New => Fix Released

** Changed in: cloud-archive/yoga
       Status: New => In Progress

** Changed in: cloud-archive/yoga
     Assignee: (unassigned) => Rodrigo Barbieri (rodrigo-barbieri2010)

** Changed in: python-glanceclient (Ubuntu)
       Status: New => Fix Released

** Changed in: python-openstacksdk (Ubuntu)
       Status: New => Fix Released

** Also affects: python-glanceclient (Ubuntu Jammy)
   Importance: Undecided
       Status: New

** Also affects: python-openstacksdk (Ubuntu Jammy)
   Importance: Undecided
       Status: New

** Also affects: python-glanceclient (Ubuntu Noble)
   Importance: Undecided
       Status: New

** Also affects: python-openstacksdk (Ubuntu Noble)
   Importance: Undecided
       Status: New

** Changed in: python-glanceclient (Ubuntu Jammy)
       Status: New => In Progress

** Changed in: python-glanceclient (Ubuntu Jammy)
     Assignee: (unassigned) => Rodrigo Barbieri (rodrigo-barbieri2010)

** Changed in: python-glanceclient (Ubuntu Noble)
       Status: New => Fix Released

** Changed in: python-openstacksdk (Ubuntu Noble)
       Status: New => Fix Released

** Changed in: python-openstacksdk (Ubuntu Jammy)
       Status: New => In Progress

** Changed in: python-openstacksdk (Ubuntu Jammy)
     Assignee: (unassigned) => Rodrigo Barbieri (rodrigo-barbieri2010)

** Description changed:

+ *********** SRU TEMPLATE AT THE BOTTOM **************
+ 
  When Cinder volume service creates a volume from an image, it will
  download an image from Glance to a cinder-volume unit (to be precise,
  when CoW is not available by the backend between Glance and Cinder).
  
  The image fetch speed is not super fast as of today:
  
  2023-05-17 02:52:28.275 158391 DEBUG cinder.image.image_utils [req-e4234307-ae7c-4220-a10a-15760679ac01 a32b8e6ed20f484eba8ddae8f7b6eb1e 08dd29cd8676470c8e0b73d2667f8cf0 - - -] Image fetch details: dest /var/lib/cinder/conversion/image_fetch_741953c3-08f7-43b6-b423-2be938ac8d02_bc1hvgd7cinder at cinder-ceph, size 1907.35 MB, duration 27.80 sec fetch /usr/lib/python3/dist-packages/cinder/image/image_utils.py:637
  2023-05-17 02:52:28.275 158391 INFO cinder.image.image_utils [req-e4234307-ae7c-4220-a10a-15760679ac01 a32b8e6ed20f484eba8ddae8f7b6eb1e 08dd29cd8676470c8e0b73d2667f8cf0 - - -] Image download 1907.35 MB at 68.61 MB/s
  
  -> ~ 549 Mbps
  
  By running a profiler, it looks like serious_integrity_iter
  (glanceclient/common/utils.py) is consuming a lot of CPU cycles on the
  client side (in this case the cinder-volume unit).
  
  Image caching can overcome this kind of challenges but this report is for when we need to download images (new images, updated images, etc.).
  https://docs.openstack.org/cinder/latest/admin/image-volume-cache.html
+ 
+ 
+ ===============
+ SRU DESCRIPTION
+ ===============
+ 
+ [Impact]
+ 
+ TBD
+ 
+ [Test case]
+ 
+ TBD
+ 
+ [Where problems could occur]
+ 
+ TBD
+ 
+ [Other Info]
+ 
+ TBD

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to python-glanceclient in Ubuntu.
https://bugs.launchpad.net/bugs/2020139

Title:
  [SRU] Image fetch speed from Glance to Cinder volume service can be
  slow and capped by client CPU

Status in Ubuntu Cloud Archive:
  Fix Released
Status in Ubuntu Cloud Archive bobcat series:
  Fix Released
Status in Ubuntu Cloud Archive caracal series:
  Fix Released
Status in Ubuntu Cloud Archive yoga series:
  In Progress
Status in Glance Client:
  Fix Committed
Status in python-glanceclient package in Ubuntu:
  Fix Released
Status in python-openstacksdk package in Ubuntu:
  Fix Released
Status in python-glanceclient source package in Jammy:
  In Progress
Status in python-openstacksdk source package in Jammy:
  In Progress
Status in python-glanceclient source package in Noble:
  Fix Released
Status in python-openstacksdk source package in Noble:
  Fix Released

Bug description:
  *********** SRU TEMPLATE AT THE BOTTOM **************

  When Cinder volume service creates a volume from an image, it will
  download an image from Glance to a cinder-volume unit (to be precise,
  when CoW is not available by the backend between Glance and Cinder).

  The image fetch speed is not super fast as of today:

  2023-05-17 02:52:28.275 158391 DEBUG cinder.image.image_utils [req-e4234307-ae7c-4220-a10a-15760679ac01 a32b8e6ed20f484eba8ddae8f7b6eb1e 08dd29cd8676470c8e0b73d2667f8cf0 - - -] Image fetch details: dest /var/lib/cinder/conversion/image_fetch_741953c3-08f7-43b6-b423-2be938ac8d02_bc1hvgd7cinder at cinder-ceph, size 1907.35 MB, duration 27.80 sec fetch /usr/lib/python3/dist-packages/cinder/image/image_utils.py:637
  2023-05-17 02:52:28.275 158391 INFO cinder.image.image_utils [req-e4234307-ae7c-4220-a10a-15760679ac01 a32b8e6ed20f484eba8ddae8f7b6eb1e 08dd29cd8676470c8e0b73d2667f8cf0 - - -] Image download 1907.35 MB at 68.61 MB/s

  -> ~ 549 Mbps

  By running a profiler, it looks like serious_integrity_iter
  (glanceclient/common/utils.py) is consuming a lot of CPU cycles on the
  client side (in this case the cinder-volume unit).

  Image caching can overcome this kind of challenges but this report is for when we need to download images (new images, updated images, etc.).
  https://docs.openstack.org/cinder/latest/admin/image-volume-cache.html

  
  ===============
  SRU DESCRIPTION
  ===============

  [Impact]

  TBD

  [Test case]

  TBD

  [Where problems could occur]

  TBD

  [Other Info]

  TBD

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/2020139/+subscriptions