[Bug 2108994] Re: apscheduler retrieving decimal.Decimal via sqlalchemy

Thu Jan 8 01:18:49 UTC 2026

** Description changed:

  [Impact]

  Affected Jammy, Noble ( Yoga, Caracal ). for Focal, SQLAlchemy version
  is too low.

  apscheduler’s get_next_run_time() function expects the value to be of type float.
  However, SQLAlchemy’s scalar() may return a decimal.Decimal instead.
  In that case, apscheduler raises a TypeError.

  The issue is addressed by the following patch in Watcher, which explicitly handles the type conversion.
  This fix was merged starting with the 2024.2 release, so releases prior to 2024.1 do not include it.

  I reproduced it with Noble, Jammy, ( keep testing more )

  From e24cf31b608e92d9efe377662fba63887860c0c9 Mon Sep 17 00:00:00 2001
  From: Takashi Kajinami <kajinamit at oss.nttdata.com>
  Date: Wed, 8 May 2024 15:24:52 +0900
  Subject: [PATCH] SQLAlchemy 2.0: Omnibus fixes patch

  This was originally five patches, but they are all needed to pass
  any of the test jobs now, so they have been squashed into one:

  Co-Authored-By: Dan Smith (dms at danplanet.com)

  First:

  The autoload argument was removed[1] in SQLAlchemy and only
  the autoload_with argument should be passed.

  The autoload argument is set according to the autoload_with argument
  automatically even in SQLAlchemy 1.x[2] so is not at all needed.

  [1] https://github.com/sqlalchemy/sqlalchemy/commit/c932123bacad9bf047d160b85e3f95d396c513ae
  [2] https://github.com/sqlalchemy/sqlalchemy/commit/ad8f921e969b6f735dc8e08d882c961dde78f2b1

  Second:

  Remove _warn_on_bytestring for newer SA, AFAICT, this flag has been
  removed from SQLAlchemy and that is why watcher-db-manage fails to
  initialize the DB for me on jammy. This migration was passing the
  default value (=False) anyway, so I assume this is the right "fix".

  Third:

  Fix joinedload passing string attribute names

  Fourth:

  Fix engine.select pattern to use begin() per the migration guide.

  Fifth:

  Override the apscheduler get_next_run_time() which appears to be
  trivially not compatible with SQLAlchemy 2.0 because of a return type
  from scalar().

  Change-Id: I000e5e78f97f82ed4ea64d42f1c38354c3252e08

  [Test Case]
- 1. deploy openstack env with watcher. ( based on juju )
- 2. juju ssh watcher/0 -- sudo systemctl restart watcher-decision-engine
+ 1. deploy env with watcher. ( based on juju ) [1](bundle)
+ 2. 
+  - source novarc
+  - openstack optimize audit list
+  - openstack optimize audit create   -t CONTINUOUS   --interval 60   --goal dummy   --strategy dummy   --name reproduction-audit
+ 3. juju ssh watcher/0 -- sudo systemctl restart watcher-decision-engine
  4. You may see error msg like below.
  - Dec 16 06:15:45 node-15 watcher-decision-engine[88884]: TypeError: 'decimal.Decimal' object cannot be interpreted as an integer

  [Where problems could occur]
  The patch modifies SQLAlchemy database interactions.
  (1) removes _warn_on_bytestring parameter.
  (2) changes joinedload() from string keys to model attributes.
  (3) replaces engine.execute() with engine.begin() context manager.
  (4) adds float() coercion for get_next_run_time().

  All follow official SQLAlchemy 2.0 migration patterns.
  Regressions are unlikely as these are well-established patterns, but any issues would appear as database operation failures in the watcher-decision-engine scheduler.

  [Others]

  =original description=

  Kolla-Ansible: 2023.2
  Ubuntu 22.04.5 LTS

  I have a fresh installation in a three-node testing environment with
  Watcher enabled (deployed via Kolla Ansible).

  When creating a continuous audit, the watcher_engine container logs the
  following traceback:

  Exception in thread APScheduler:
  Traceback (most recent call last):
    File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
      self.run()
    File "/usr/lib/python3.10/threading.py", line 953, in run
      self._target(*self._args, **self._kwargs)
    File "/var/lib/kolla/venv/lib/python3.10/site-packages/apscheduler/schedulers/blocking.py", line 32, in _main_loop
      wait_seconds = self._process_jobs()
    File "/var/lib/kolla/venv/lib/python3.10/site-packages/apscheduler/schedulers/base.py", line 1005, in _process_jobs
      jobstore_next_run_time = jobstore.get_next_run_time()
    File "/var/lib/kolla/venv/lib/python3.10/site-packages/apscheduler/jobstores/sqlalchemy.py", line 86, in get_next_run_time
      return utc_timestamp_to_datetime(next_run_time)
    File "/var/lib/kolla/venv/lib/python3.10/site-packages/apscheduler/util.py", line 179, in utc_timestamp_to_datetime
      return datetime.fromtimestamp(timestamp, utc)
  TypeError: 'decimal.Decimal' object cannot be interpreted as an integer

  Source:
  https://github.com/agronholm/apscheduler/blob/be291699755e58ff398f90b5e71bff1e163df1db/apscheduler/jobstores/sqlalchemy.py#L85

  For some reason, this line retrieves a decimal.Decimal, even though
  there’s no explicit configuration for float columns to default to
  asdecimal=False.

  This behavior aligns with the note by danms in the following commit:
  https://opendev.org/openstack/watcher/commit/d6f169197efc5b4f6c8a2e6bc38177b0641ca05c

  But danms assumes SQLAlchemy 2.0 stuff, which definetly isn't in my
  case.

  I also tested minimal Python scripts within the container to verify
  whether the issue is related to the versions of APScheduler (3.10.1) or
  SQLAlchemy (1.4.41), but I was unable to reproduce the problem. In those
  tests, float values were correctly returned from float-type columns.
+ 
+ [1]
+ ❯ cat bundle-caracal
+ applications:
+   gnocchi:
+     charm: gnocchi
+     channel: 2024.1/stable
+     revision: 245
+     base: ubuntu at 22.04/stable
+     num_units: 1
+     to:
+     - "5"
+     constraints: arch=amd64
+   keystone:
+     charm: keystone
+     channel: 2024.1/stable
+     revision: 753
+     base: ubuntu at 22.04/stable
+     num_units: 1
+     to:
+     - "1"
+     constraints: arch=amd64
+   mysql:
+     charm: mysql-innodb-cluster
+     channel: 8.0/stable
+     revision: 159
+     base: ubuntu at 20.04/stable
+     num_units: 3
+     to:
+     - "2"
+     - "3"
+     - "4"
+     constraints: arch=amd64
+   rabbitmq-server:
+     charm: rabbitmq-server
+     channel: 3.9/stable
+     revision: 246
+     base: ubuntu at 20.04/stable
+     num_units: 1
+     to:
+     - "6"
+     constraints: arch=amd64
+   watcher:
+     charm: watcher
+     channel: 2024.1/stable
+     revision: 34
+     base: ubuntu at 22.04/stable
+     num_units: 1
+     to:
+     - "7"
+     options:
+       datasources: gnocchi
+       planner-config: '{"weights":"change_node_power_state:9,change_nova_service_state:50,migrate:30,nop:70,resize:20,sleep:40,turn_host_to_acpi_s3_state:10,volume_migrate:60","parallelization":"change_node_power_state:2,change_nova_service_state:1,migrate:2,nop:1,resize:2,sleep:1,turn_host_to_acpi_s3_state:2,volume_migrate:2"}'
+     constraints: arch=amd64
+ machines:
+   "1":
+     constraints: arch=amd64
+     base: ubuntu at 22.04/stable
+   "2":
+     constraints: arch=amd64
+     base: ubuntu at 20.04/stable
+   "3":
+     constraints: arch=amd64
+     base: ubuntu at 20.04/stable
+   "4":
+     constraints: arch=amd64
+     base: ubuntu at 20.04/stable
+   "5":
+     constraints: arch=amd64
+     base: ubuntu at 22.04/stable
+   "6":
+     constraints: arch=amd64
+     base: ubuntu at 20.04/stable
+   "7":
+     constraints: arch=amd64
+     base: ubuntu at 22.04/stable
+ relations:
+ - - keystone:identity-service
+   - watcher:identity-service
+ - - keystone:shared-db
+   - mysql:shared-db
+ - - mysql:shared-db
+   - watcher:shared-db
+ - - rabbitmq-server:amqp
+   - gnocchi:amqp
+ - - rabbitmq-server:amqp
+   - watcher:amqp
+ - - gnocchi:identity-service
+   - keystone:identity-service
+ - - gnocchi:shared-db
+   - mysql:shared-db

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/2108994

Title:
  apscheduler retrieving decimal.Decimal via sqlalchemy

Status in Ubuntu Cloud Archive:
  Fix Released
Status in Ubuntu Cloud Archive antelope series:
  New
Status in Ubuntu Cloud Archive bobcat series:
  New
Status in Ubuntu Cloud Archive caracal series:
  In Progress
Status in Ubuntu Cloud Archive dalmatian series:
  Fix Released
Status in Ubuntu Cloud Archive epoxy series:
  Fix Released
Status in Ubuntu Cloud Archive flamingo series:
  Fix Released
Status in Ubuntu Cloud Archive gazpacho series:
  Fix Released
Status in Ubuntu Cloud Archive ussuri series:
  New
Status in Ubuntu Cloud Archive yoga series:
  New
Status in watcher package in Ubuntu:
  Fix Released
Status in watcher source package in Focal:
  Invalid
Status in watcher source package in Jammy:
  New
Status in watcher source package in Noble:
  In Progress
Status in watcher source package in Plucky:
  Fix Released
Status in watcher source package in Questing:
  Fix Released
Status in watcher source package in Resolute:
  Fix Released

Bug description:
  [Impact]