[Bug 1248283] Re: Juju deploy of Charm in MAAS fails because dbus fails
Robie Basak
1248283 at bugs.launchpad.net
Mon Nov 25 13:14:53 UTC 2013
MAAS on its own does not deploy a stopped dbus service in my testing; it
only happens when juju is involved. I've isolated this to "service
networking restart" done by juju using cloud-init userdata. I've updated
the description. I think a fix for juju would be straightforward: don't
do that; use more targeted "ifdown" and "ifup" as required instead. Note
that the "ifdown" should be called before changing
/etc/network/interfaces.
However, adding a dbus task, as it is arguable that dbus should not die
in this case, and certainly not silently. A "Won't Fix" resolution for
dbus dying might be acceptable, though, if it is not expected to survive
all network interfaces being deconfigured. But a log message explaining
would be nice.
** Description changed:
+ [Impact]
+
+ juju fails to deploy charms on Saucy.
+
+ [Steps to Reproduce]
+
+ On Saucy:
+
+ root at foo:~# dpkg-query -W dbus
+ dbus 1.6.12-0ubuntu10
+ root at foo:~# status dbus
+ dbus start/running, process 817
+ root at foo:~# service networking restart
+ networking stop/waiting
+ networking start/running
+ root at foo:~# status dbus
+ dbus stop/waiting
+
+ Verified on amd64. I believe this also affects armhf.
+
+ [Analysis]
+
+ Either:
+
+ 1) dbus shouldn't fail when networking is restarted; or
+ 2) juju should not restart networking in this way as a runcmd in cloud-init userdata, and then expect dbus to work.
+
+ The relevant part of juju's userdata is:
+
+ - "cat > /etc/network/eth0.config << EOF\niface eth0 inet manual\n\nauto br0\niface
+ br0 inet dhcp\n bridge_ports eth0\nEOF\n"
+ - sed -i "s/iface eth0 inet dhcp/source \/etc\/network\/eth0.config/" /etc/network/interfaces
+ - service networking restart
+
+ I assume juju is restarting networking to set up the bridge that it
+ configured; in this case, an "ifdown eth0" and a subsequent "ifup"s as
+ required should suffice. No need to restart everything (eg. lo, etc)
+ which I [rbasak] presume is causing the issue.
+
+ Note that dbus appears to silently fail; if it is going to stop, it
+ should do so with a sensible log message.
+
+ [Original Description]
+
I'm running a MAAS and Juju enviornment using 64-bit Ubuntu Server 13.10
for the MAAS node as well as all juju nodes, including the bootstrap
node.
I can't seem to deploy any charms to the nodes without the services
getting stuck in a 'pending' state. The problem seems to be that upon
charm deployment dbus for some reason is stopped, and thus the whole
process of deploying the charm gets stuck.
It is almost verbatim of the problem someone appears to have hit on
raring, too:
http://askubuntu.com/questions/364714/juju-deploy-of-charm-mysql-in-
maas-provider-failing-after-successful-bootstrap
For me, however, it's not just mysql...it's any service I try to deploy.
juju release: 1.16.2-0ubuntu1~ubuntu13.10.1~juju1
maas version: 1.4+bzr1693+dfsg-0
Ubuntu OS: 13.10, 64-bit for everything
Steps to reproduce:
1) Set up a maas server using 13.10 with all the latest updates.
2) Install juju-core from the stable ppa (1.16.2 in this case).
3) Enlist and commission some nodes.
4) download some charms locally (I'm in a semi-restricted network environment, so, it's easier for me to pull down the charms locally). For example, bzr branch lp:charms/rabbitmq-server
- 5) bootstrap the juju environment. Due to restrictions in my network I have to use "juju bootstrap --upload-tools". Firewall blocks the ability to run "juju sync-tools" beforehand.
+ 5) bootstrap the juju environment. Due to restrictions in my network I have to use "juju bootstrap --upload-tools". Firewall blocks the ability to run "juju sync-tools" beforehand.
6) Once the environment is bootstrapped, try to deploy a charm. This is what I used:
juju deploy --repository=charms/ local:rabbitmq-server --show-log.
Actual results:
- Installation of the OS goes fine, along with the cloud-init after reboot. I can also ssh into the node via juju.
- The charm, however, stays in a 'pending' state long after it is deployed on the server.
- ssh-ing into the node and looking at /var/log/juju/machine-x.log reveals that the deployment stage fails due to being unable to connect to the system bus.
- This seems to prevent the charm from ever deploying.
+ Installation of the OS goes fine, along with the cloud-init after reboot. I can also ssh into the node via juju.
+ The charm, however, stays in a 'pending' state long after it is deployed on the server.
+ ssh-ing into the node and looking at /var/log/juju/machine-x.log reveals that the deployment stage fails due to being unable to connect to the system bus.
+ This seems to prevent the charm from ever deploying.
Expected results:
- OS installs and charm deploys without issue.
+ OS installs and charm deploys without issue.
Workaround:
I can work around this by ssh-ing into the node and starting dbus by hand. I then go back and re-deploy the charm via juju using:
juju destroy-unit rabbitmq-server/0
juju destroy-service rabbitmq-server
juju deploy --to 1 --repository=charms/ local:rabbitmq-server --show-log
At that point everything starts working again. I can even deploy
another charm onto the same machine without problems once dbus is up and
going.
I'm not sure what or why dbus is failing to start and I can't tell if
dbus was running and it suddenly shut down prior to charm deployment.
I've attached a machine log from an attempt to deploy rabbitmq-server
onto a bare metal node using 13.10. The same thing happens to any charm
I try to deploy in this manner.
Let me know if you need anything else.
** Changed in: maas
Status: Incomplete => Invalid
** Also affects: dbus (Ubuntu)
Importance: Undecided
Status: New
** Tags added: midway
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to dbus in Ubuntu.
https://bugs.launchpad.net/bugs/1248283
Title:
Juju deploy of Charm in MAAS fails because dbus fails
Status in juju-core:
Triaged
Status in MAAS:
Invalid
Status in “dbus” package in Ubuntu:
New
Bug description:
[Impact]
juju fails to deploy charms on Saucy.
[Steps to Reproduce]
On Saucy:
root at foo:~# dpkg-query -W dbus
dbus 1.6.12-0ubuntu10
root at foo:~# status dbus
dbus start/running, process 817
root at foo:~# service networking restart
networking stop/waiting
networking start/running
root at foo:~# status dbus
dbus stop/waiting
Verified on amd64. I believe this also affects armhf.
[Analysis]
Either:
1) dbus shouldn't fail when networking is restarted; or
2) juju should not restart networking in this way as a runcmd in cloud-init userdata, and then expect dbus to work.
The relevant part of juju's userdata is:
- "cat > /etc/network/eth0.config << EOF\niface eth0 inet manual\n\nauto br0\niface
br0 inet dhcp\n bridge_ports eth0\nEOF\n"
- sed -i "s/iface eth0 inet dhcp/source \/etc\/network\/eth0.config/" /etc/network/interfaces
- service networking restart
I assume juju is restarting networking to set up the bridge that it
configured; in this case, an "ifdown eth0" and a subsequent "ifup"s as
required should suffice. No need to restart everything (eg. lo, etc)
which I [rbasak] presume is causing the issue.
Note that dbus appears to silently fail; if it is going to stop, it
should do so with a sensible log message.
[Original Description]
I'm running a MAAS and Juju enviornment using 64-bit Ubuntu Server
13.10 for the MAAS node as well as all juju nodes, including the
bootstrap node.
I can't seem to deploy any charms to the nodes without the services
getting stuck in a 'pending' state. The problem seems to be that
upon charm deployment dbus for some reason is stopped, and thus the
whole process of deploying the charm gets stuck.
It is almost verbatim of the problem someone appears to have hit on
raring, too:
http://askubuntu.com/questions/364714/juju-deploy-of-charm-mysql-in-
maas-provider-failing-after-successful-bootstrap
For me, however, it's not just mysql...it's any service I try to
deploy.
juju release: 1.16.2-0ubuntu1~ubuntu13.10.1~juju1
maas version: 1.4+bzr1693+dfsg-0
Ubuntu OS: 13.10, 64-bit for everything
Steps to reproduce:
1) Set up a maas server using 13.10 with all the latest updates.
2) Install juju-core from the stable ppa (1.16.2 in this case).
3) Enlist and commission some nodes.
4) download some charms locally (I'm in a semi-restricted network environment, so, it's easier for me to pull down the charms locally). For example, bzr branch lp:charms/rabbitmq-server
5) bootstrap the juju environment. Due to restrictions in my network I have to use "juju bootstrap --upload-tools". Firewall blocks the ability to run "juju sync-tools" beforehand.
6) Once the environment is bootstrapped, try to deploy a charm. This is what I used:
juju deploy --repository=charms/ local:rabbitmq-server --show-log.
Actual results:
Installation of the OS goes fine, along with the cloud-init after reboot. I can also ssh into the node via juju.
The charm, however, stays in a 'pending' state long after it is deployed on the server.
ssh-ing into the node and looking at /var/log/juju/machine-x.log reveals that the deployment stage fails due to being unable to connect to the system bus.
This seems to prevent the charm from ever deploying.
Expected results:
OS installs and charm deploys without issue.
Workaround:
I can work around this by ssh-ing into the node and starting dbus by hand. I then go back and re-deploy the charm via juju using:
juju destroy-unit rabbitmq-server/0
juju destroy-service rabbitmq-server
juju deploy --to 1 --repository=charms/ local:rabbitmq-server --show-
log
At that point everything starts working again. I can even deploy
another charm onto the same machine without problems once dbus is up
and going.
I'm not sure what or why dbus is failing to start and I can't tell if
dbus was running and it suddenly shut down prior to charm deployment.
I've attached a machine log from an attempt to deploy rabbitmq-server
onto a bare metal node using 13.10. The same thing happens to any
charm I try to deploy in this manner.
Let me know if you need anything else.
To manage notifications about this bug go to:
https://bugs.launchpad.net/juju-core/+bug/1248283/+subscriptions
More information about the foundations-bugs
mailing list