[Bug 1812672] Re: impossible to create nova instances after upgrading to rocky

[Takashi Kajinami]

This prolbme was mostly marked resolved but there is no explanation
about actual action item in oslo.cache.... I'd close this now but please
feel free to reopen it in case you know any legit problem in oslo.cache
itself.

** Changed in: oslo.cache
       Status: Confirmed => Invalid

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1812672

Title:
  impossible to create nova instances after upgrading to rocky

Status in OpenStack Nova Cloud Controller Charm:
  Fix Released
Status in Ubuntu Cloud Archive:
  Fix Released
Status in oslo.cache:
  Invalid
Status in nova package in Ubuntu:
  Fix Released
Status in python-oslo.cache package in Ubuntu:
  Fix Released

Bug description:
  Hi,

  I'm using bionic with 18.11 charms. I recently upgraded openstack from
  queens to rocky. After that, I was unable to create nova instances -
  they were stuck in BUILD state, without any error in nova-cloud-
  controller, neutron-api or keystone logs.

  While investigating, I noticed that "openstack compute service list"
  was empty, and this was generating an error in nova-api-os-compute.log
  (see below for the traceback).

  My investigations lead to
  https://github.com/openstack/oslo.cache/blob/master/oslo_cache/_memcache_pool.py#L48
  being the problem. If I comment out this line, then "openstack service
  list" works fine, and I can create instances without problem. I
  however don't know the consequences in the long run.

  Please advise.

  Thanks !

  
  nova-api-os-compute.log traceback when running "openstack service list" :

  2019-01-21 10:45:43.729 87283 ERROR nova.context Traceback (most recent call last):
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/oslo_cache/_memcache_pool.py", line 163, in _get
  2019-01-21 10:45:43.729 87283 ERROR nova.context     conn = self.queue.pop().connection
  2019-01-21 10:45:43.729 87283 ERROR nova.context IndexError: pop from an empty deque
  2019-01-21 10:45:43.729 87283 ERROR nova.context
  2019-01-21 10:45:43.729 87283 ERROR nova.context During handling of the above exception, another exception occurred:
  2019-01-21 10:45:43.729 87283 ERROR nova.context
  2019-01-21 10:45:43.729 87283 ERROR nova.context Traceback (most recent call last):
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/nova/context.py", line 438, in gather_result
  2019-01-21 10:45:43.729 87283 ERROR nova.context     result = fn(cctxt, *args, **kwargs)
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper
  2019-01-21 10:45:43.729 87283 ERROR nova.context     result = fn(cls, context, *args, **kwargs)
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/nova/objects/service.py", line 601, in get_all
  2019-01-21 10:45:43.729 87283 ERROR nova.context     context, db_services)
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/nova/availability_zones.py", line 88, in set_availability_zones
  2019-01-21 10:45:43.729 87283 ERROR nova.context     service['host'], az)
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/nova/availability_zones.py", line 108, in update_host_availability_zone_cache
  2019-01-21 10:45:43.729 87283 ERROR nova.context     cache.delete(cache_key)
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/nova/cache_utils.py", line 122, in delete
  2019-01-21 10:45:43.729 87283 ERROR nova.context     return self.region.delete(key)
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/dogpile/cache/region.py", line 1002, in delete
  2019-01-21 10:45:43.729 87283 ERROR nova.context     self.backend.delete(key)
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/dogpile/cache/backends/memcached.py", line 188, in delete
  2019-01-21 10:45:43.729 87283 ERROR nova.context     self.client.delete(key)
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/oslo_cache/backends/memcache_pool.py", line 31, in _run_method
  2019-01-21 10:45:43.729 87283 ERROR nova.context     with self.client_pool.acquire() as client:
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
  2019-01-21 10:45:43.729 87283 ERROR nova.context     return next(self.gen)
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/oslo_cache/_memcache_pool.py", line 127, in acquire
  2019-01-21 10:45:43.729 87283 ERROR nova.context     conn = self.get(timeout=self._connection_get_timeout)
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/eventlet/queue.py", line 295, in get
  2019-01-21 10:45:43.729 87283 ERROR nova.context     return self._get()
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/oslo_cache/_memcache_pool.py", line 214, in _get
  2019-01-21 10:45:43.729 87283 ERROR nova.context     conn = ConnectionPool._get(self)
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/oslo_cache/_memcache_pool.py", line 165, in _get
  2019-01-21 10:45:43.729 87283 ERROR nova.context     conn = self._create_connection()
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/oslo_cache/_memcache_pool.py", line 206, in _create_connection
  2019-01-21 10:45:43.729 87283 ERROR nova.context     return _MemcacheClient(self.urls, **self._arguments)
  2019-01-21 10:45:43.729 87283 ERROR nova.context TypeError: object() takes no parameters
  2019-01-21 10:45:43.729 87283 ERROR nova.context

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1812672/+subscriptions