[Bug 1934849] Re: s3 backend takes time exponentially

Sat Sep 20 16:23:47 UTC 2025

For `openstack` command instead of the direct glance command,  it switched the backend from the glanceclient to OpenStackSDK starting at Ussuri.
https://review.opendev.org/c/openstack/python-openstackclient/+/650374

Whenever `openstack image create --file` is used, it should report the
image size in the request properly and the glance-api server side should
switch to the multipart upload mode when the image is bigger than 100MB
(s3_store_large_object_size). That means the upload should be split into
100MB chunks so it shouldn't take exponential time but it should be
linear instead.

This is with a 2GB image.

$ time openstack image create \
    --file my-image.img \
    --disk-format raw --container-format bare \
    --progress \
    "my-image"

real    0m17.204s
user    0m3.859s
sys     0m2.388s

$ time glance image-create \
    --store s3 \
    --file my-image.img --name "my-image" \
    --disk-format raw --container-format bare \
    --progress

real    3m51.790s
user    0m1.962s
sys     0m1.393s

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1934849

Title:
  s3 backend takes time exponentially

Status in Ubuntu Cloud Archive:
  Fix Released
Status in Ubuntu Cloud Archive ussuri series:
  Fix Released
Status in Ubuntu Cloud Archive victoria series:
  Fix Released
Status in Ubuntu Cloud Archive wallaby series:
  Fix Released
Status in glance_store:
  Fix Released
Status in python-glance-store package in Ubuntu:
  Fix Released
Status in python-glance-store source package in Focal:
  Fix Released
Status in python-glance-store source package in Hirsute:
  Fix Released

Bug description:
  [Impact]

  Glance with S3 backend cannot accept image uploads in a realistic time
  frame. For example, an 1GB image upload takes ~60 minutes although other
  backends such as swift can complete it with 10 seconds.

  [Test Plan]

  1. Deploy a partial OpenStack with multiple Glance backends including S3
    (zaza test bundles can be used with "ceph" which will set up "rbd", "swift", and "s3" backends - https://opendev.org/openstack/charm-glance/src/branch/master/tests/tests.yaml)
  2. Upload multiple images with variety of sizes
  3. confirm the duration of uploading images are shorter in general after applying the updated package
    (expected duration of 1GB is from ~60 minutes to 1-3 minutes)

  for backend in ceph swift s3; do
      echo "[$backend]"
      for i in {0,3,5,9,10,128,512,1024}; do
          dd if=/dev/zero of=my-image.img bs=1M count=$i oflag=sync
          echo "${i}MiB"
          time glance image-create \
              --store $backend \
              --file my-image.img --name "my-image-${backend}-${i}MiB" \
              --disk-format raw --container-format bare \
              --progress
      done
  done

  [Where problems could occur]

  Since we bump WRITE_CHUNKSIZE from 64KiB to 5MiB, there might be a case where image uploads fail if the size of the image is less than WRITE_CHUNKSIZE. Or there might be an unexpected latency in the worst case scenario. We will try to address the concerns by testing multiple images uploads with multiple sizes including some corner cases as follows:
  - 0 - zero
  - 3MiB - less than the new WRITE_CHUNKSIZE(5MiB)
  - 5MiB - exactly same as the new WRITE_CHUNKSIZE(5MiB)
  - 9MiB - bigger than new WRITE_CHUNKSIZE(5MiB) but less than twice
  - 10MiB - exactly twice as the new WRITE_CHUNKSIZE(5MiB)
  - 128MiB, 512MiB, 1024MiB - some large images

  ====

  I have a test Ceph cluster as an object storage with both Swift and S3
  protocols enabled for Glance (Ussuri). When I use Swift backend with
  Glance, an image upload completes quickly enough. But with S3 backend
  Glance, it takes much more time to upload an image and it seems to rise
  exponentially.

  It's worth noting that when uploading an image with S3 backend, a single
  core is consumed 100% by glance-api process.

  [swift]
  8MB   -  2.4s
  16MB  -  2.8s
  32MB  -  2.6s
  64MB  -  2.7s
  128MB -  3.1s
  ...
  512MB -  5.9s

  [s3]
  8MB   -  2.2s
  16MB  -  2.9s
  32MB  -  5.5s
  64MB  - 16.3s
  128MB - 54.9s
  ...
  512MB - 14m26s

  Btw, downloading of 512MB image with S3 backend can complete with less
  than 10 seconds.

  $ time openstack image save --file downloaded.img 917c5424-4350-4bc5-98ca-66d40e101843
  real    0m5.673s

  $ du -h downloaded.img
  512M    downloaded.img

  [/etc/glance/glance-api.conf]

  enabled_backends = local:file, ceph:rbd, swift:swift, s3:s3

  [swift]
  auth_version = 3
  auth_address = http://192.168.151.131:5000/v3
  ...
  container = glance
  large_object_size = 5120
  large_object_chunk_size = 200

  [s3]
  s3_store_host = http://192.168.151.137:80/
  ...
  s3_store_bucket = zaza-glance-s3-test
  s3_store_large_object_size = 5120
  s3_store_large_object_chunk_size = 200

  ProblemType: Bug
  DistroRelease: Ubuntu 20.04
  ProblemType: BugDistroRelease: Ubuntu 20.04
  Package: python3-glance-store 2.0.0-0ubuntu1
  ProcVersionSignature: Ubuntu 5.4.0-77.86-generic 5.4.119
  Uname: Linux 5.4.0-77-generic x86_64
  ApportVersion: 2.20.11-0ubuntu27.18
  Architecture: amd64
  CasperMD5CheckResult: skip
  Date: Wed Jul  7 04:46:05 2021
  PackageArchitecture: all
  ProcEnviron:
   TERM=screen-256color
   PATH=(custom, no user)
   LANG=C.UTF-8
   SHELL=/bin/bash
  SourcePackage: python-glance-store
   SHELL=/bin/bashSourcePackage: python-glance-store
  UpgradeStatus: No upgrade log present (probably fresh install)

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1934849/+subscriptions