[Bug 1799406] Re: [SRU] Alarms fail on Rocky

Ben Nemec openstack at nemebean.com
Wed Nov 28 18:31:30 UTC 2018 

I don't think this is invalid on oslo.i18n. It sounds like there's a
problem with the Message class that caused this. The Aodh fix is only a
workaround for that.

There have been discussions around removing lazy translation completely,
in which case this would go away. If that doesn't happen, we should look
into how to fix this.

** Changed in: oslo.i18n
       Status: Invalid => Confirmed

** Changed in: oslo.i18n
   Importance: Undecided => Medium

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1799406

Title:
  [SRU] Alarms fail on Rocky

Status in Aodh:
  Fix Released
Status in OpenStack AODH Charm:
  Invalid
Status in Ubuntu Cloud Archive:
  Fix Released
Status in Ubuntu Cloud Archive rocky series:
  Fix Released
Status in oslo.i18n:
  Confirmed
Status in aodh package in Ubuntu:
  Fix Released
Status in aodh source package in Cosmic:
  Fix Released

Bug description:
  [Impact]
  In a rocky deployment aodh is failing to raise alarms due to what seems like a locale issue. If I purge packages:

  apt-get purge aodh-api aodh-evaluator aodh-expirer aodh-notifier aodh-
  listener python3-aodh libapache2-mod-wsgi-py3 python-apt python-aodh
  python-memcache

  Then disable the AodhCharmRocky class and rerun install so that it
  pulls in the py2 packages then things start working.

  The error from /var/log/aodh/aodh-listener.log:

  http://paste.ubuntu.com/p/FCft2GGnfN/

  ERROR oslo_db.sqlalchemy.exc_filters [-] DB exception wrapped.: AttributeError: 'list' object has no attribute 'lower'
  ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last):
  ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
  ERROR oslo_db.sqlalchemy.exc_filters     context)
  ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 508, in do_execute
  ERROR oslo_db.sqlalchemy.exc_filters     cursor.execute(statement, parameters)
  ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 163, in execute
  ERROR oslo_db.sqlalchemy.exc_filters     query = self.mogrify(query, args)
  ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 142, in mogrify
  ERROR oslo_db.sqlalchemy.exc_filters     query = query % self._escape_args(args, conn)
  ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 122, in _escape_args
  ERROR oslo_db.sqlalchemy.exc_filters     return dict((key, conn.literal(val)) for (key, val) in args.items())
  ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 122, in <genexpr>
  ERROR oslo_db.sqlalchemy.exc_filters     return dict((key, conn.literal(val)) for (key, val) in args.items())
  ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 819, in literal
  ERROR oslo_db.sqlalchemy.exc_filters     return self.escape(obj, self.encoders)
  ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 806, in escape
  ERROR oslo_db.sqlalchemy.exc_filters     return "'" + self.escape_string(obj) + "'"
  ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 825, in escape_string
  ERROR oslo_db.sqlalchemy.exc_filters     return converters.escape_string(s)
  ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python3/dist-packages/pymysql/converters.py", line 73, in _escape_unicode
  ERROR oslo_db.sqlalchemy.exc_filters     return value.translate(_escape_table)
  ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python3/dist-packages/oslo_i18n/_message.py", line 85, in translate
  ERROR oslo_db.sqlalchemy.exc_filters     self.has_plural_form)
  ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python3/dist-packages/oslo_i18n/_message.py", line 117, in _translate_msgid
  ERROR oslo_db.sqlalchemy.exc_filters     fallback=True)
  ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python3.6/gettext.py", line 525, in translation
  ERROR oslo_db.sqlalchemy.exc_filters     mofiles = find(domain, localedir, languages, all=True)
  ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python3.6/gettext.py", line 482, in find
  ERROR oslo_db.sqlalchemy.exc_filters     for nelang in _expand_lang(lang):
  ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python3.6/gettext.py", line 206, in _expand_lang
  ERROR oslo_db.sqlalchemy.exc_filters     loc = locale.normalize(loc)
  ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python3.6/locale.py", line 397, in normalize
  ERROR oslo_db.sqlalchemy.exc_filters     code = localename.lower()
  ERROR oslo_db.sqlalchemy.exc_filters AttributeError: 'list' object has no attribute 'lower'
  ERROR oslo_db.sqlalchemy.exc_filters
  ERROR aodh.evaluator [-] alarm state update failed: oslo_db.exception.DBError: 'list' object has no attribute 'lower'
  ERROR aodh.evaluator Traceback (most recent call last):
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
  ERROR aodh.evaluator     context)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 508, in do_execute
  ERROR aodh.evaluator     cursor.execute(statement, parameters)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 163, in execute
  ERROR aodh.evaluator     query = self.mogrify(query, args)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 142, in mogrify
  ERROR aodh.evaluator     query = query % self._escape_args(args, conn)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 122, in _escape_args
  ERROR aodh.evaluator     return dict((key, conn.literal(val)) for (key, val) in args.items())
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 122, in <genexpr>
  ERROR aodh.evaluator     return dict((key, conn.literal(val)) for (key, val) in args.items())
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 819, in literal
  ERROR aodh.evaluator     return self.escape(obj, self.encoders)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 806, in escape
  ERROR aodh.evaluator     return "'" + self.escape_string(obj) + "'"
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 825, in escape_string
  ERROR aodh.evaluator     return converters.escape_string(s)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/pymysql/converters.py", line 73, in _escape_unicode
  ERROR aodh.evaluator     return value.translate(_escape_table)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/oslo_i18n/_message.py", line 85, in translate
  ERROR aodh.evaluator     self.has_plural_form)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/oslo_i18n/_message.py", line 117, in _translate_msgid
  ERROR aodh.evaluator     fallback=True)
  ERROR aodh.evaluator   File "/usr/lib/python3.6/gettext.py", line 525, in translation
  ERROR aodh.evaluator     mofiles = find(domain, localedir, languages, all=True)
  ERROR aodh.evaluator   File "/usr/lib/python3.6/gettext.py", line 482, in find
  ERROR aodh.evaluator     for nelang in _expand_lang(lang):
  ERROR aodh.evaluator   File "/usr/lib/python3.6/gettext.py", line 206, in _expand_lang
  ERROR aodh.evaluator     loc = locale.normalize(loc)
  ERROR aodh.evaluator   File "/usr/lib/python3.6/locale.py", line 397, in normalize
  ERROR aodh.evaluator     code = localename.lower()
  ERROR aodh.evaluator AttributeError: 'list' object has no attribute 'lower'
  ERROR aodh.evaluator
  ERROR aodh.evaluator The above exception was the direct cause of the following exception:
  ERROR aodh.evaluator
  ERROR aodh.evaluator Traceback (most recent call last):
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/aodh/evaluator/__init__.py", line 128, in _refresh
  ERROR aodh.evaluator     self._storage_conn.update_alarm(alarm)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/aodh/storage/impl_sqlalchemy.py", line 273, in update_alarm
  ERROR aodh.evaluator     alarm.as_dict())
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/sqlalchemy/orm/query.py", line 3426, in update
  ERROR aodh.evaluator     update_op.exec_()
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/sqlalchemy/orm/persistence.py", line 1326, in exec_
  ERROR aodh.evaluator     self._do_exec()
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/sqlalchemy/orm/persistence.py", line 1493, in _do_exec
  ERROR aodh.evaluator     self._execute_stmt(update_stmt)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/sqlalchemy/orm/persistence.py", line 1333, in _execute_stmt
  ERROR aodh.evaluator     mapper=self.mapper)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/sqlalchemy/orm/session.py", line 1176, in execute
  ERROR aodh.evaluator     bind, close_with_result=True).execute(clause, params or {})
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 948, in execute
  ERROR aodh.evaluator     return meth(self, multiparams, params)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
  ERROR aodh.evaluator     return connection._execute_clauseelement(self, multiparams, params)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
  ERROR aodh.evaluator     compiled_sql, distilled_params
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
  ERROR aodh.evaluator     context)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1409, in _handle_dbapi_exception
  ERROR aodh.evaluator     util.raise_from_cause(newraise, exc_info)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
  ERROR aodh.evaluator     reraise(type(exception), exception, tb=exc_tb, cause=cause)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 186, in reraise
  ERROR aodh.evaluator     raise value.with_traceback(tb)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
  ERROR aodh.evaluator     context)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 508, in do_execute
  ERROR aodh.evaluator     cursor.execute(statement, parameters)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 163, in execute
  ERROR aodh.evaluator     query = self.mogrify(query, args)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 142, in mogrify
  ERROR aodh.evaluator     query = query % self._escape_args(args, conn)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 122, in _escape_args
  ERROR aodh.evaluator     return dict((key, conn.literal(val)) for (key, val) in args.items())
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 122, in <genexpr>
  ERROR aodh.evaluator     return dict((key, conn.literal(val)) for (key, val) in args.items())
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 819, in literal
  ERROR aodh.evaluator     return self.escape(obj, self.encoders)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 806, in escape
  ERROR aodh.evaluator     return "'" + self.escape_string(obj) + "'"
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 825, in escape_string
  ERROR aodh.evaluator     return converters.escape_string(s)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/pymysql/converters.py", line 73, in _escape_unicode
  ERROR aodh.evaluator     return value.translate(_escape_table)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/oslo_i18n/_message.py", line 85, in translate
  ERROR aodh.evaluator     self.has_plural_form)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/oslo_i18n/_message.py", line 117, in _translate_msgid
  ERROR aodh.evaluator     fallback=True)
  ERROR aodh.evaluator   File "/usr/lib/python3.6/gettext.py", line 525, in translation
  ERROR aodh.evaluator     mofiles = find(domain, localedir, languages, all=True)
  ERROR aodh.evaluator   File "/usr/lib/python3.6/gettext.py", line 482, in find
  ERROR aodh.evaluator     for nelang in _expand_lang(lang):
  ERROR aodh.evaluator   File "/usr/lib/python3.6/gettext.py", line 206, in _expand_lang
  ERROR aodh.evaluator     loc = locale.normalize(loc)
  ERROR aodh.evaluator   File "/usr/lib/python3.6/locale.py", line 397, in normalize
  ERROR aodh.evaluator     code = localename.lower()
  ERROR aodh.evaluator oslo_db.exception.DBError: 'list' object has no attribute 'lower'
  ERROR aodh.evaluator

  [Test Case]
  1) Deploy openstack with aodh (we're deploying with the development versions of the openstack charms which deploy py3 payloads)
  2) Once deployed/configured create an alarm with:
  openstack alarm create \
    --type event \
    --name instance_off \
    --description 'Instance powered OFF' \
    --event-type "compute.instance.power_off.*" \
    --enable True \
    --query "traits.instance_id=string::INSTANCE_ID" \
    --alarm-action 'log://' \
    --ok-action 'log://' \
    --insufficient-data-action 'log://'
  3) create a server instance
  4) power server instance off to trigger alarm
  5) ensure alarm is successfully raised with no traceback (see /var/log/aodh/aodh-listener.log)

  [Regression Potential]
  The fix is a minimal fix that should result in low regression potential.

To manage notifications about this bug go to:
https://bugs.launchpad.net/aodh/+bug/1799406/+subscriptions

More information about the Ubuntu-openstack-bugs mailing list