[Bug 1832075] [NEW] [19.04][Queens -> Rocky] python3-pymysql is not installed before use

[Dmitrii Shcherbakov]

Public bug reported:

Ran an upgrade from bionic-queens to bionic-rocky and encountered the
following:

/var/log/apt/term.log

https://paste.ubuntu.com/p/n6WZJbDy6p/

  File "/usr/lib/python3/dist-packages/django/db/backends/mysql/base.py", line 25, in <module>
    import pymysql
ModuleNotFoundError: No module named 'pymysql'
dpkg: error processing package openstack-dashboard

The problem is that python3-django only "suggests" that python3-pymysql
is installed and does not depend on it while

root at juju-f2e601-4-lxd-2:~# dpkg -l | grep pymysql
ii  python-pymysql                   0.8.0-1                           all          Pure-Python MySQL driver - Python 2.x

root at juju-f2e601-4-lxd-2:~# dpkg -l | grep -P 'python[3]?-django\s'
ii  python-django                    1:1.11.11-1ubuntu1.3              all          High-level Python web development framework (Python 2 version)
ii  python3-django                   1:1.11.11-1ubuntu1.3              all          High-level Python web development framework (Python 3 version)

# python3-django does not depend on pymysql - only suggests it

```
apt-rdepends -f Depends,PreDepends,Suggests python3-django | grep pymysql
Reading package lists... Done
Building dependency tree
Reading state information... Done

python3-pymysql
python-pymysql-doc
```

While during openstack-dashboard package setup there is a code path
leading to an import of pymysql. See below the following line in the
term.log:

"Setting up openstack-dashboard (3:15.0.0-0ubuntu1~cloud0) ..."

The dashboard charm contains python3-pymysql, however, and apt looks to
be processing packages in the order they are specified (which leads to
openstack-dashboard being set up before python3-pymysql as there is no
direct dependency).

BASE_PACKAGES = [
    'haproxy',
    'memcached',
    'openstack-dashboard',
    'openstack-dashboard-ubuntu-theme',
    'python-keystoneclient',
    'python-memcache',
    'python-novaclient',
]

PY3_PACKAGES = [
    'python3-django-horizon',
    'python3-designate-dashboard',
    'python3-heat-dashboard',
    'python3-neutron-lbaas-dashboard',
    'python3-neutron-fwaas-dashboard',
    'python3-keystoneclient',
    'python3-novaclient',
    'python3-memcache',
    'python3-pymysql',
    'libapache2-mod-wsgi-py3',
]


BASE_PACKAGES go first, then PY3_PACKAGES:
# ...
def determine_packages():
    """Determine packages to install"""
    packages = deepcopy(BASE_PACKAGES)
# ...
    if release >= 'rocky':
        packages = [p for p in packages if not p.startswith('python-')]
        packages.extend(PY3_PACKAGES)

** Affects: charm-openstack-dashboard
     Importance: Undecided
         Status: New

** Affects: horizon (Ubuntu)
     Importance: Undecided
         Status: New


** Tags: cpe-onsite

** Also affects: charm-openstack-dashboard
   Importance: Undecided
       Status: New

** Description changed:

  Ran an upgrade from bionic-queens to bionic-rocky and encountered the
  following:
  
  /var/log/apt/term.log
  
  https://paste.ubuntu.com/p/n6WZJbDy6p/
  
-   File "/usr/lib/python3/dist-packages/django/db/backends/mysql/base.py", line 25, in <module>
-     import pymysql
+   File "/usr/lib/python3/dist-packages/django/db/backends/mysql/base.py", line 25, in <module>
+     import pymysql
  ModuleNotFoundError: No module named 'pymysql'
- dpkg: error processing package openstack-dashboard 
+ dpkg: error processing package openstack-dashboard
  
- 
- The problem is that python3-django only "suggests" that python3-pymysql is installed and does not depend on it while 
+ The problem is that python3-django only "suggests" that python3-pymysql
+ is installed and does not depend on it while
  
  root at juju-f2e601-4-lxd-2:~# dpkg -l | grep pymysql
  ii  python-pymysql                   0.8.0-1                           all          Pure-Python MySQL driver - Python 2.x
  
  root at juju-f2e601-4-lxd-2:~# dpkg -l | grep -P 'python[3]?-django\s'
  ii  python-django                    1:1.11.11-1ubuntu1.3              all          High-level Python web development framework (Python 2 version)
  ii  python3-django                   1:1.11.11-1ubuntu1.3              all          High-level Python web development framework (Python 3 version)
  
- 
  # python3-django does not depend on pymysql - only suggests it
  
  ```
  apt-rdepends -f Depends,PreDepends,Suggests python3-django | grep pymysql
  Reading package lists... Done
- Building dependency tree       
+ Building dependency tree
  Reading state information... Done
  
  python3-pymysql
  python-pymysql-doc
  ```
  
  While during openstack-dashboard package setup there is a code path
  leading to an import of pymysql. See below the following line in the
  term.log:
  
  "Setting up openstack-dashboard (3:15.0.0-0ubuntu1~cloud0) ..."
  
  The dashboard charm contains python3-pymysql, however, and apt looks to
  be processing packages in the order they are specified (which leads to
  openstack-dashboard being set up before python3-pymysql as there is no
  direct dependency).
  
- 
  BASE_PACKAGES = [
-     'haproxy',
-     'memcached',
-     'openstack-dashboard',
-     'openstack-dashboard-ubuntu-theme',
-     'python-keystoneclient',
-     'python-memcache',
-     'python-novaclient',
+     'haproxy',
+     'memcached',
+     'openstack-dashboard',
+     'openstack-dashboard-ubuntu-theme',
+     'python-keystoneclient',
+     'python-memcache',
+     'python-novaclient',
  ]
  
  PY3_PACKAGES = [
-     'python3-django-horizon',
-     'python3-designate-dashboard',
-     'python3-heat-dashboard',
-     'python3-neutron-lbaas-dashboard',
-     'python3-neutron-fwaas-dashboard',
-     'python3-keystoneclient',
-     'python3-novaclient',
-     'python3-memcache',
-     'python3-pymysql',
-     'libapache2-mod-wsgi-py3',
+     'python3-django-horizon',
+     'python3-designate-dashboard',
+     'python3-heat-dashboard',
+     'python3-neutron-lbaas-dashboard',
+     'python3-neutron-fwaas-dashboard',
+     'python3-keystoneclient',
+     'python3-novaclient',
+     'python3-memcache',
+     'python3-pymysql',
+     'libapache2-mod-wsgi-py3',
  ]
+ 
+ 
+ BASE_PACKAGES go first, then PY3_PACKAGES:
+ # ...
+ def determine_packages():
+     """Determine packages to install"""
+     packages = deepcopy(BASE_PACKAGES)
+ # ...
+     if release >= 'rocky':
+         packages = [p for p in packages if not p.startswith('python-')]
+         packages.extend(PY3_PACKAGES)

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to horizon in Ubuntu.
https://bugs.launchpad.net/bugs/1832075

Title:
  [19.04][Queens -> Rocky] python3-pymysql is not installed before use

Status in OpenStack openstack-dashboard charm:
  New
Status in horizon package in Ubuntu:
  New

Bug description:
  Ran an upgrade from bionic-queens to bionic-rocky and encountered the
  following:

  /var/log/apt/term.log

  https://paste.ubuntu.com/p/n6WZJbDy6p/

    File "/usr/lib/python3/dist-packages/django/db/backends/mysql/base.py", line 25, in <module>
      import pymysql
  ModuleNotFoundError: No module named 'pymysql'
  dpkg: error processing package openstack-dashboard

  The problem is that python3-django only "suggests" that
  python3-pymysql is installed and does not depend on it while

  root at juju-f2e601-4-lxd-2:~# dpkg -l | grep pymysql
  ii  python-pymysql                   0.8.0-1                           all          Pure-Python MySQL driver - Python 2.x

  root at juju-f2e601-4-lxd-2:~# dpkg -l | grep -P 'python[3]?-django\s'
  ii  python-django                    1:1.11.11-1ubuntu1.3              all          High-level Python web development framework (Python 2 version)
  ii  python3-django                   1:1.11.11-1ubuntu1.3              all          High-level Python web development framework (Python 3 version)

  # python3-django does not depend on pymysql - only suggests it

  ```
  apt-rdepends -f Depends,PreDepends,Suggests python3-django | grep pymysql
  Reading package lists... Done
  Building dependency tree
  Reading state information... Done

  python3-pymysql
  python-pymysql-doc
  ```

  While during openstack-dashboard package setup there is a code path
  leading to an import of pymysql. See below the following line in the
  term.log:

  "Setting up openstack-dashboard (3:15.0.0-0ubuntu1~cloud0) ..."

  The dashboard charm contains python3-pymysql, however, and apt looks
  to be processing packages in the order they are specified (which leads
  to openstack-dashboard being set up before python3-pymysql as there is
  no direct dependency).

  BASE_PACKAGES = [
      'haproxy',
      'memcached',
      'openstack-dashboard',
      'openstack-dashboard-ubuntu-theme',
      'python-keystoneclient',
      'python-memcache',
      'python-novaclient',
  ]

  PY3_PACKAGES = [
      'python3-django-horizon',
      'python3-designate-dashboard',
      'python3-heat-dashboard',
      'python3-neutron-lbaas-dashboard',
      'python3-neutron-fwaas-dashboard',
      'python3-keystoneclient',
      'python3-novaclient',
      'python3-memcache',
      'python3-pymysql',
      'libapache2-mod-wsgi-py3',
  ]

  
  BASE_PACKAGES go first, then PY3_PACKAGES:
  # ...
  def determine_packages():
      """Determine packages to install"""
      packages = deepcopy(BASE_PACKAGES)
  # ...
      if release >= 'rocky':
          packages = [p for p in packages if not p.startswith('python-')]
          packages.extend(PY3_PACKAGES)

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-openstack-dashboard/+bug/1832075/+subscriptions