[Bug 1167512] [NEW] nova-network fails to start if bindir is not set

LaMont Jones lamont.jones at canonical.com
Wed Apr 10 18:59:07 UTC 2013 

Public bug reported:

The setup:
- grizzly rc2 upgraded to grizzly (1:2013.1-0ubuntu1~cloud0), running on precise.
- fresh install configured based on what appears to be a folsom howto

Starting nova-network results in the following in the nova-network log:

...
2013-04-10 08:59:18.788 WARNING nova.db.sqlalchemy.api [req-8094076e-9813-42c4-968e-9dcb1f640659 69e6b81d4dbd41669313d522df780566 fdd2025e9254488dbe52919938c96edf] Change will make usage less than 0 for the following resources: ['fixed_ips']
2013-04-10 08:59:19.098 ERROR nova.openstack.common.rpc.amqp [req-8094076e-9813-42c4-968e-9dcb1f640659 69e6b81d4dbd41669313d522df780566 fdd2025e9254488dbe52919938c96edf] Exception during message handling
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 430, in _process_data
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp     rval = self.proxy.dispatch(ctxt, version, method, **args)
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 133, in dispatch
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp     return getattr(proxyobj, method)(ctxt, **kwargs)
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/network/floating_ips.py", line 187, in deallocate_for_instance
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp     super(FloatingIP, self).deallocate_for_instance(context, **kwargs)
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 550, in deallocate_for_instance
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp     self.deallocate_fixed_ip(context, fixed_ip['address'], host=host)
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 243, in deallocate_fixed_ip
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp     address)
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 938, in deallocate_fixed_ip
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp     self._teardown_network_on_host(context, network)
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 1630, in _teardown_network_on_host
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp     self.driver.update_dhcp(elevated, dev, network)
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 938, in update_dhcp
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp     restart_dhcp(context, dev, network_ref)
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 242, in inner
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp     retval = f(*args, **kwargs)
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 1048, in restart_dhcp
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp     _execute(*cmd, run_as_root=True)
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 1141, in _execute
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp     return utils.execute(*cmd, **kwargs)
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 239, in execute
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp     cmd=' '.join(cmd))
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp ProcessExecutionError: Unexpected error while running command.
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp Command: sudo nova-rootwrap /etc/nova/rootwrap.conf env CONFIG_FILE=["/etc/nova/nova.conf"] NETWORK_ID=1 dnsmasq --strict-order --bind-interfaces --conf-file= --domain=novalocal --pid-file=/var/lib/nova/networks/nova-eth0.pid --listen-address=172.28.20.1 --except-interface=lo --dhcp-range=set:private,172.28.20.2,static,120s --dhcp-lease-max=1024 --dhcp-hostsfile=/var/lib/nova/networks/nova-eth0.conf --dhcp-script=/usr/lib/python2.7/dist-packages/bin/nova-dhcpbridge --leasefile-ro
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp Exit code: 3
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp Stdout: ''
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp Stderr: 'sh: 1: /usr/lib/python2.7/dist-packages/bin/nova-dhcpbridge: not found\n\ndnsmasq: cannot run lease-init script /usr/lib/python2.7/dist-packages/bin/nova-dhcpbridge: No such file or directory\n'
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp
2013-04-10 08:59:19.103 ERROR nova.openstack.common.rpc.common [req-8094076e-9813-42c4-968e-9dcb1f640659 69e6b81d4dbd41669313d522df780566 fdd2025e9254488dbe52919938c96edf] Returning exception Unexpected error while running command.
Command: sudo nova-rootwrap /etc/nova/rootwrap.conf env CONFIG_FILE=["/etc/nova/nova.conf"] NETWORK_ID=1 dnsmasq --strict-order --bind-interfaces --conf-file= --domain=novalocal --pid-file=/var/lib/nova/networks/nova-eth0.pid --listen-address=172.28.20.1 --except-interface=lo --dhcp-range=set:private,172.28.20.2,static,120s --dhcp-lease-max=1024 --dhcp-hostsfile=/var/lib/nova/networks/nova-eth0.conf --dhcp-script=/usr/lib/python2.7/dist-packages/bin/nova-dhcpbridge --leasefile-ro
Exit code: 3
Stdout: ''
Stderr: 'sh: 1: /usr/lib/python2.7/dist-packages/bin/nova-dhcpbridge: not found\n\ndnsmasq: cannot run lease-init script /usr/lib/python2.7/dist-packages/bin/nova-dhcpbridge: No such file or directory\n' to caller
2013-04-10 08:59:19.104 ERROR nova.openstack.common.rpc.common [req-8094076e-9813-42c4-968e-9dcb1f640659 69e6b81d4dbd41669313d522df780566 fdd2025e9254488dbe52919938c96edf] ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 430, in _process_data\n    rval = self.proxy.dispatch(ctxt, version, method, **args)\n', '  File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 133, in dispatch\n    return getattr(proxyobj, method)(ctxt, **kwargs)\n', '  File "/usr/lib/python2.7/dist-packages/nova/network/floating_ips.py", line 187, in deallocate_for_instance\n    super(FloatingIP, self).deallocate_for_instance(context, **kwargs)\n', '  File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 550, in deallocate_for_instance\n    self.deallocate_fixed_ip(context, fixed_ip[\'address\'], host=host)\n', '  File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 243, in deallocate_fixed_ip\n    address)\n', '  File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 938, in deallocate_fixed_ip\n    self._teardown_network_on_host(context, network)\n', '  File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 1630, in _teardown_network_on_host\n    self.driver.update_dhcp(elevated, dev, network)\n', '  File "/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 938, in update_dhcp\n    restart_dhcp(context, dev, network_ref)\n', '  File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 242, in inner\n    retval = f(*args, **kwargs)\n', '  File "/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 1048, in restart_dhcp\n    _execute(*cmd, run_as_root=True)\n', '  File "/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 1141, in _execute\n    return utils.execute(*cmd, **kwargs)\n', '  File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 239, in execute\n    cmd=\' \'.join(cmd))\n', 'ProcessExecutionError: Unexpected error while running command.\nCommand: sudo nova-rootwrap /etc/nova/rootwrap.conf env CONFIG_FILE=["/etc/nova/nova.conf"] NETWORK_ID=1 dnsmasq --strict-order --bind-interfaces --conf-file= --domain=novalocal --pid-file=/var/lib/nova/networks/nova-eth0.pid --listen-address=172.28.20.1 --except-interface=lo --dhcp-range=set:private,172.28.20.2,static,120s --dhcp-lease-max=1024 --dhcp-hostsfile=/var/lib/nova/networks/nova-eth0.conf --dhcp-script=/usr/lib/python2.7/dist-packages/bin/nova-dhcpbridge --leasefile-ro\nExit code: 3\nStdout: \'\'\nStderr: \'sh: 1: /usr/lib/python2.7/dist-packages/bin/nova-dhcpbridge: not found\\n\\ndnsmasq: cannot run lease-init script /usr/lib/python2.7/dist-packages/bin/nova-dhcpbridge: No such file or directory\\n\'\n']
2013-04-10 09:07:31.246 5003 INFO nova.service [-] Caught SIGTERM, exiting
2013-04-10 09:08:34.740 1334 INFO nova.manager [-] Skipping periodic task _periodic_update_dns because its interval is negative
2013-04-10 09:08:34.742 1334 INFO nova.network.driver [-] Loading network driver 'nova.network.linux_net'
2013-04-10 09:08:34.761 1334 AUDIT nova.service [-] Starting network node (version 2013.1)
2013-04-10 09:09:04.146 1334 CRITICAL nova [-] Unexpected error while running command.
Command: sudo nova-rootwrap /etc/nova/rootwrap.conf env CONFIG_FILE=["/etc/nova/nova.conf"] NETWORK_ID=1 dnsmasq --strict-order --bind-interfaces --conf-file= --domain=novalocal --pid-file=/var/lib/nova/networks/nova-eth0.pid --listen-address=172.28.20.1 --except-interface=lo --dhcp-range=set:private,172.28.20.2,static,120s --dhcp-lease-max=1024 --dhcp-hostsfile=/var/lib/nova/networks/nova-eth0.conf --dhcp-script=/usr/lib/python2.7/dist-packages/bin/nova-dhcpbridge --leasefile-ro
Exit code: 3
...

I initially created a symlink pointing /usr/lib/python2.7/dist-packages/bin/nova-dhcpbridge -> /usr/bin/nova-dhcpbridge
which fixed the problem, setting bindir=/usr/bin in nova.conf is a cleaner workaround.

I do not know if the default nova.conf has bindir specified, but it
strikes me that /usr/bin is a much better default bindir than
/var/lib/python2.7/dist-packages/bin...

lamont

** Affects: nova (Ubuntu)
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to nova in Ubuntu.
https://bugs.launchpad.net/bugs/1167512

Title:
  nova-network fails to start if bindir is not set

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/nova/+bug/1167512/+subscriptions

More information about the Ubuntu-server-bugs mailing list