[Bug 1421377] [NEW] "MAAS failed to start" recurs if postgresql is stopped

[Mike Pontillo]

Public bug reported:

This is using the MAAS 1.8 experimental branch.

A race condition is possible in which a persistent exception is thrown
when hitting http://<server>/MAAS, if maas-regiond starts before
postgresql. (that is, even if you start postgresql, the MAAS web
interface never recovers.)

==> Exception seen on the web page <==

MAAS failed to start

Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 783, in __bootstrap
    self.__bootstrap_inner()
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
--- <exception caught here> ---
  File "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 191, in _worker
    result = context.call(ctx, function, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext
    return func(*args,**kw)
  File "/usr/lib/python2.7/dist-packages/maasserver/webapp.py", line 83, in prepareApplication
    start_up.start_up()
  File "/usr/lib/python2.7/dist-packages/maasserver/start_up.py", line 48, in start_up
    security.get_shared_secret()
  File "/usr/lib/python2.7/dist-packages/provisioningserver/utils/twisted.py", line 113, in wrapper
    return func_in_reactor(*args, **kwargs).wait(timeout)
  File "/usr/lib/python2.7/dist-packages/crochet/_eventloop.py", line 219, in wait
    result.raiseException()
  File "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 191, in _worker
    result = context.call(ctx, function, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext
    return func(*args,**kw)
  File "/usr/lib/python2.7/dist-packages/provisioningserver/utils/twisted.py", line 148, in wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/maasserver/utils/async.py", line 164, in call_within_transaction
    return func_outside_txn(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/maasserver/utils/orm.py", line 216, in retrier
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/django/db/transaction.py", line 338, in inner
    with self:
  File "/usr/lib/python2.7/dist-packages/django/db/transaction.py", line 237, in __enter__
    if not connection.get_autocommit():
  File "/usr/lib/python2.7/dist-packages/django/db/backends/__init__.py", line 324, in get_autocommit
    self.ensure_connection()
  File "/usr/lib/python2.7/dist-packages/django/db/backends/__init__.py", line 124, in ensure_connection
    self.connect()
  File "/usr/lib/python2.7/dist-packages/django/db/utils.py", line 99, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/usr/lib/python2.7/dist-packages/django/db/backends/__init__.py", line 124, in ensure_connection
    self.connect()
  File "/usr/lib/python2.7/dist-packages/django/db/backends/__init__.py", line 112, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/usr/lib/python2.7/dist-packages/django/db/backends/postgresql_psycopg2/base.py", line 116, in get_new_connection
    return Database.connect(**conn_params)
  File "/usr/lib/python2.7/dist-packages/psycopg2/__init__.py", line 179, in connect
    connection_factory=connection_factory, async=async)
django.db.utils.OperationalError: could not connect to server: Connection refused
	Is the server running on host "localhost" (127.0.0.1) and accepting
	TCP/IP connections on port 5432?


==> Steps to Reproduce <==

$ sudo service postgresql stop
 * Stopping PostgreSQL 9.3 database server                                                                                                          [ OK ] 

$ sudo service maas-clusterd stop
maas-clusterd stop/waiting

$ sudo service maas-regiond stop
maas-regiond stop/waiting

$ sudo service maas-regiond start
maas-regiond start/running, process 10964

(then hit the MAAS web page)

(Note: this was seen upon system boot.)

==> Workaround <==

$ sudo service maas-clusterd restart

** Affects: maas (Ubuntu)
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to maas in Ubuntu.
https://bugs.launchpad.net/bugs/1421377

Title:
  "MAAS failed to start" recurs if postgresql is stopped

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/maas/+bug/1421377/+subscriptions