[Bug 2020139] Re: [SRU] Image fetch speed from Glance to Cinder volume service can be slow and capped by client CPU
Rodrigo Barbieri
2020139 at bugs.launchpad.net
Wed Jun 25 19:01:02 UTC 2025
** Description changed:
*********** SRU TEMPLATE AT THE BOTTOM **************
When Cinder volume service creates a volume from an image, it will
download an image from Glance to a cinder-volume unit (to be precise,
when CoW is not available by the backend between Glance and Cinder).
The image fetch speed is not super fast as of today:
2023-05-17 02:52:28.275 158391 DEBUG cinder.image.image_utils [req-e4234307-ae7c-4220-a10a-15760679ac01 a32b8e6ed20f484eba8ddae8f7b6eb1e 08dd29cd8676470c8e0b73d2667f8cf0 - - -] Image fetch details: dest /var/lib/cinder/conversion/image_fetch_741953c3-08f7-43b6-b423-2be938ac8d02_bc1hvgd7cinder at cinder-ceph, size 1907.35 MB, duration 27.80 sec fetch /usr/lib/python3/dist-packages/cinder/image/image_utils.py:637
2023-05-17 02:52:28.275 158391 INFO cinder.image.image_utils [req-e4234307-ae7c-4220-a10a-15760679ac01 a32b8e6ed20f484eba8ddae8f7b6eb1e 08dd29cd8676470c8e0b73d2667f8cf0 - - -] Image download 1907.35 MB at 68.61 MB/s
-> ~ 549 Mbps
By running a profiler, it looks like serious_integrity_iter
(glanceclient/common/utils.py) is consuming a lot of CPU cycles on the
client side (in this case the cinder-volume unit).
Image caching can overcome this kind of challenges but this report is for when we need to download images (new images, updated images, etc.).
https://docs.openstack.org/cinder/latest/admin/image-volume-cache.html
===============
SRU DESCRIPTION
===============
[Impact]
This issue causes images to be download much slower than they should,
causing VM creation to potentially time out if the image is too big and
takes too long. The fix proposed greatly improves performance and
prevents timeouts in most cases where it would timeout without the fix.
[Test case]
+ Steps for python-glanceclient:
+ ------------------------------
+
1. Deploy OpenStack with Cinder and Glance
2. Upload a resonable large image, like jammy ~650mb is enough.
3. Create a volume from the image
openstack volume create --size 3 --image jammy v1
4. Check the logs for messages of time to fetch/download the image:
... cinder.image.image_utils ... Image fetch details: dest
/var/lib/cinder/conversion/image_fetch_ed429779-0d11-45cb-8ee5-fee7c2003d86_rubea933juju-056b6e-jy3-0 at LVM-
default, size 645.16 MB, duration 3.59 sec fetch ...
... cinder.image.image_utils ... Image download 645.16 MB at 179.58 MB/s
5. Repeat steps (3) and (4) 3 times just to make sure it is consistent
and to have an average (avoid underlying infra fluctuations)
6. Install fix and restart all cinder services (such as apache2, cinder-
scheduler and cinder-volume)
7. Repeat steps (3), (4) and (5), and make sure the new duration and
download rate is at least about 50% faster.
+ Steps for python-openstacksdk:
+ ------------------------------
+
+ 1. Deploy OpenStack with Glance
+
+ 2. Upload a resonable large image, like jammy ~650mb is enough.
+
+ 3. Install python3-openstackclient in your client machine
+
+ sudo apt install python3-openstackclient
+
+ 4. Download the image (repeat about 5 times to have an average)
+
+ time openstack image save jammy --file /dev/null
+
+ 5. Install fixed package (python3-openstacksdk)
+
+ 6. Download the image again (repeat about 5 times to have an average).
+ This time there should be a significant decrease in time
+
[Where problems could occur]
If there are issues with the fix, it could impact the ability to
download images at all, therefore impacting the ability to create
volumes from images, and create VMs. Reverting the package is enough to
undo the changes and restore previous functionality.
[Other Info]
Both python-glanceclient and openstacksdk fixes listed below are
necessary.
https://review.opendev.org/c/openstack/openstacksdk/+/883461
https://review.opendev.org/c/openstack/python-glanceclient/+/924060
** Attachment added: "lp2020139_focal_yoga_verification.txt"
https://bugs.launchpad.net/ubuntu/jammy/+source/python-openstacksdk/+bug/2020139/+attachment/5886141/+files/lp2020139_focal_yoga_verification.txt
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to python-glanceclient in Ubuntu.
https://bugs.launchpad.net/bugs/2020139
Title:
[SRU] Image fetch speed from Glance to Cinder volume service can be
slow and capped by client CPU
Status in Ubuntu Cloud Archive:
Fix Released
Status in Ubuntu Cloud Archive bobcat series:
Fix Released
Status in Ubuntu Cloud Archive caracal series:
Fix Released
Status in Ubuntu Cloud Archive yoga series:
Fix Committed
Status in Glance Client:
Fix Committed
Status in python-glanceclient package in Ubuntu:
Fix Released
Status in python-openstacksdk package in Ubuntu:
Fix Released
Status in python-glanceclient source package in Jammy:
Fix Committed
Status in python-openstacksdk source package in Jammy:
Fix Committed
Status in python-glanceclient source package in Noble:
Fix Released
Status in python-openstacksdk source package in Noble:
Fix Released
Bug description:
*********** SRU TEMPLATE AT THE BOTTOM **************
When Cinder volume service creates a volume from an image, it will
download an image from Glance to a cinder-volume unit (to be precise,
when CoW is not available by the backend between Glance and Cinder).
The image fetch speed is not super fast as of today:
2023-05-17 02:52:28.275 158391 DEBUG cinder.image.image_utils [req-e4234307-ae7c-4220-a10a-15760679ac01 a32b8e6ed20f484eba8ddae8f7b6eb1e 08dd29cd8676470c8e0b73d2667f8cf0 - - -] Image fetch details: dest /var/lib/cinder/conversion/image_fetch_741953c3-08f7-43b6-b423-2be938ac8d02_bc1hvgd7cinder at cinder-ceph, size 1907.35 MB, duration 27.80 sec fetch /usr/lib/python3/dist-packages/cinder/image/image_utils.py:637
2023-05-17 02:52:28.275 158391 INFO cinder.image.image_utils [req-e4234307-ae7c-4220-a10a-15760679ac01 a32b8e6ed20f484eba8ddae8f7b6eb1e 08dd29cd8676470c8e0b73d2667f8cf0 - - -] Image download 1907.35 MB at 68.61 MB/s
-> ~ 549 Mbps
By running a profiler, it looks like serious_integrity_iter
(glanceclient/common/utils.py) is consuming a lot of CPU cycles on the
client side (in this case the cinder-volume unit).
Image caching can overcome this kind of challenges but this report is for when we need to download images (new images, updated images, etc.).
https://docs.openstack.org/cinder/latest/admin/image-volume-cache.html
===============
SRU DESCRIPTION
===============
[Impact]
This issue causes images to be download much slower than they should,
causing VM creation to potentially time out if the image is too big
and takes too long. The fix proposed greatly improves performance and
prevents timeouts in most cases where it would timeout without the
fix.
[Test case]
Steps for python-glanceclient:
------------------------------
1. Deploy OpenStack with Cinder and Glance
2. Upload a resonable large image, like jammy ~650mb is enough.
3. Create a volume from the image
openstack volume create --size 3 --image jammy v1
4. Check the logs for messages of time to fetch/download the image:
... cinder.image.image_utils ... Image fetch details: dest
/var/lib/cinder/conversion/image_fetch_ed429779-0d11-45cb-8ee5-fee7c2003d86_rubea933juju-056b6e-jy3-0 at LVM-
default, size 645.16 MB, duration 3.59 sec fetch ...
... cinder.image.image_utils ... Image download 645.16 MB at 179.58
MB/s
5. Repeat steps (3) and (4) 3 times just to make sure it is consistent
and to have an average (avoid underlying infra fluctuations)
6. Install fix and restart all cinder services (such as apache2,
cinder-scheduler and cinder-volume)
7. Repeat steps (3), (4) and (5), and make sure the new duration and
download rate is at least about 50% faster.
Steps for python-openstacksdk:
------------------------------
1. Deploy OpenStack with Glance
2. Upload a resonable large image, like jammy ~650mb is enough.
3. Install python3-openstackclient in your client machine
sudo apt install python3-openstackclient
4. Download the image (repeat about 5 times to have an average)
time openstack image save jammy --file /dev/null
5. Install fixed package (python3-openstacksdk)
6. Download the image again (repeat about 5 times to have an average).
This time there should be a significant decrease in time
[Where problems could occur]
If there are issues with the fix, it could impact the ability to
download images at all, therefore impacting the ability to create
volumes from images, and create VMs. Reverting the package is enough
to undo the changes and restore previous functionality.
[Other Info]
Both python-glanceclient and openstacksdk fixes listed below are
necessary.
https://review.opendev.org/c/openstack/openstacksdk/+/883461
https://review.opendev.org/c/openstack/python-glanceclient/+/924060
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/2020139/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list