JUJU OAI bundle deployment failure

Navin Sridharan navinsridharan at ymail.com
Fri Dec 8 00:29:58 UTC 2017 

Hi Narinder,

I'm new to open source platforms and so it took a while for me to get used to mailing lists and getting support from the community but I should definitely thank you for your help and support so far. I would like to take this opportunity to thank Heather as well for guiding me through certain issues. I guess I should have changed the subject line which I did now. The error I am facing currently is related to deployment of juju OAI ( Open Air Interface) bundle in MAAS cloud. I would really appreciate if someone can assist me in triaging this issue.Thank you so much for your time!!

Regards,
Navin S

Sent from my iPhone

> On Dec 7, 2017, at 6:19 PM, Narinder Gupta <narinder.gupta at canonical.com> wrote:
> 
> I am sorry Navin I can not spend more time on debugging your environment as I have more commitment from Canonical but please free to engage juju community on irc freenode  #juju channel they should able to help you. As juju community is very active and they can help you with your concerns.
> 
> 
> 
> Thanks and Regards,
> Narinder Gupta (PMP)                   narinder.gupta at canonical.com
> Canonical, Ltd.                    narindergupta [irc.freenode.net]
> +1.281.736.5150                            narindergupta2007[skype]
> 
> Ubuntu- Linux for human beings | www.ubuntu.com | www.canonical.com
> 
>> On Thu, Dec 7, 2017 at 2:34 PM, Navin Sridharan <navinsridharan at ymail.com> wrote:
>> Hi Heather,
>> 
>> As advised, I brought up MAAS cloud and had JUJU bootstrap successfully to that of MAAS. Once bootstrap was done, I tried deploying the JUJU OAI bundle( cs:~navid-nikaein/bundle/oai-nfv-4g-1) but landed up with the following error. It would be more than a help if you could kindly help me troubleshoot this issue . Thank you so much!!
>> 
>>  
>> ubuntu at ubuntu-ProLiant-DL380-G6:~/joid/ci$  sudo juju status --format=tabular
>> Model      Controller              Cloud/Region            Version  SLA
>> oai-model  opnfv-virtualpod1-maas  opnfv-virtualpod1-maas  2.2.6    unsupported
>> 
>> App       Version  Status       Scale  Charm     Store       Rev  OS      Notes
>> mysql              waiting        0/1  mysql     jujucharms   56  ubuntu
>> oai-enb            maintenance      1  oai-enb   jujucharms   24  ubuntu
>> oai-hss            waiting        0/1  oai-hss   jujucharms    8  ubuntu
>> oai-mme            waiting        0/1  oai-mme   jujucharms   16  ubuntu
>> oai-spgw           error          0/1  oai-spgw  jujucharms   13  ubuntu
>> 
>> Unit        Workload     Agent       Machine  Public address   Ports  Message
>> mysql/0     waiting      allocating  1/lxd/0                          waiting for machine
>> oai-enb/0*  maintenance  executing   0        192.168.122.152         (install) Building OAI eNodeB_3GPP for SDR USRP
>> oai-hss/0   waiting      allocating  1/lxd/1                          waiting for machine
>> oai-mme/0   waiting      allocating  1/lxd/2                          waiting for machine
>> oai-spgw/0  error        lost        1        192.168.122.153         hook failed: "install"
>> 
>> Machine  State    DNS              Inst id  Series  AZ       Message
>> 0        started  192.168.122.152  tsktsa   xenial  default  Deployed
>> 1        down     192.168.122.153  axekc4   xenial  default  Deployed
>> 1/lxd/0  pending                   pending  xenial           failed to start instance (failed to bridge devices: bridge activaction error: bridge activation failed: Internet Systems Consortium DHCP Client 4.3.3
>> Copyright 2004-2015 Internet Systems Consortium.
>> All rights reserved.
>> For info, please visit https://www.isc.org/software/dhcp/
>> 
>> Cannot find device "br-eth0"
>> Error getting hardware address for "br-eth0": No such device
>> 
>> If you think you have received this message due to a bug rather
>> than a configuration issue please read the section on submitting
>> bugs on either our web page at www.isc.org or in the README file
>> before submitting a bug.  These pages explain the proper
>> process and the information we find helpful for debugging..
>> 
>> exiting.
>> Bringing up bridged interfaces failed, see system logs and /etc/network/interfaces.new
>> RTNETLINK answers: File exists
>> Internet Systems Consortium DHCP Client 4.3.3
>> Copyright 2004-2015 Internet Systems Consortium.
>> All rights reserved.
>> For info, please visit https://www.isc.org/software/dhcp/
>> 
>> Cannot find device "br-eth0"
>> Error getting hardware address for "br-eth0": No such device
>> 
>> If you think you have received this message due to a bug rather
>> than a configuration issue please read the section on submitting
>> bugs on either our web page at www.isc.org or in the README file
>> before submitting a bug.  These pages explain the proper
>> process and the information we find helpful for debugging..
>> 
>> exiting.
>> ), retrying in 10s (10 more attempts)
>> 1/lxd/1  pending    pending  xenial
>> 1/lxd/2  pending    pending  xenial
>> 
>> Relation provider  Requirer       Interface  Type
>> mysql:cluster      mysql:cluster  mysql-ha   peer
>> mysql:db           oai-hss:db     mysql      regular
>> oai-hss:hss        oai-mme:hss    S6a-hss    regular
>> oai-mme:mme        oai-enb:mme    S1-C       regular
>> oai-spgw:spgw      oai-mme:spgw   S11        regular
>> 
>> Regards,
>> Navin S
>> 
>> 
>> On Monday 27 November 2017, 11:44:55 PM GMT-5, Navin Sridharan <navinsridharan at ymail.com> wrote:
>> 
>> 
>> Hi Heather,
>> 
>> Following is the CLI command I used to bootstrap JUJU in which I have specified the network UUID as private:
>> 
>> juju bootstrap openstack --debug --config image-metadata-url=$SWIFT_URL/simplestreams/images --config use-floating-ip=true --config network=private
>> 
>> Even though I specify the network, it still throws below error message:
>> 
>> caused by: request (http://192.168.122.190:8774/v2.1/850ecb27752344278c8816a39e26b6cd/servers) returned unexpected status: 409; error info: {"conflictingRequest": {"message": "Multiple possible networks found, use a Network ID to be more specific.", "code": 409}}}])
>> 
>> When I boot up the instance manually in Openstack cloud, I am able to successfully boot up and I mentioned the network as "private"( PFA screenshot) . Below given is the log from nova.compute log:
>> 
>> 
>> 2017-11-28 02:19:00.839 278905 INFO oslo.privsep.daemon [req-242cfa20-8f75-4643-b2ff-447a5ed557ac 4ac237f6708e4f0688589ce4efcae6af 850ecb27752344278c8816a39e26b6cd - 7565538147a74fc3add7dc0086b6ff1a 7565538147a74fc3add7dc0086b6ff1a] Running privsep helper: ['sudo', 'nova-rootwrap', '/etc/nova/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/nova/nova.conf', '--config-file', '/etc/nova/nova-compute.conf', '--privsep_context', 'vif_plug_ovs.privsep.vif_plug', '--privsep_sock_path', '/tmp/tmp0T_zTc/privsep.sock']
>> 2017-11-28 02:19:01.734 278905 INFO oslo.privsep.daemon [req-242cfa20-8f75-4643-b2ff-447a5ed557ac 4ac237f6708e4f0688589ce4efcae6af 850ecb27752344278c8816a39e26b6cd - 7565538147a74fc3add7dc0086b6ff1a 7565538147a74fc3add7dc0086b6ff1a] Spawned new privsep daemon via rootwrap
>> 2017-11-28 02:19:01.738 278905 WARNING oslo.privsep.daemon [-] privsep log: sudo: unable to resolve host rack-vir-m2
>> 2017-11-28 02:19:01.738 278905 WARNING oslo.privsep.daemon [-] privsep log: Option "logdir" from group "DEFAULT" is deprecated. Use option "log-dir" from group "DEFAULT".
>> 2017-11-28 02:19:01.627 2076418 INFO oslo.privsep.daemon [-] privsep daemon starting
>> 2017-11-28 02:19:01.633 2076418 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0
>> 2017-11-28 02:19:01.638 2076418 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_NET_ADMIN/CAP_NET_ADMIN/none
>> 2017-11-28 02:19:01.638 2076418 INFO oslo.privsep.daemon [-] privsep daemon running as pid 2076418
>> 2017-11-28 02:19:02.828 278905 INFO os_vif [req-242cfa20-8f75-4643-b2ff-447a5ed557ac 4ac237f6708e4f0688589ce4efcae6af 850ecb27752344278c8816a39e26b6cd - 7565538147a74fc3add7dc0086b6ff1a 7565538147a74fc3add7dc0086b6ff1a] Successfully plugged vif VIFBridge(active=False,address=fa:16:3e:73:ed:ff,bridge_name='qbre78939c1-66',has_traffic_filtering=True,id=e78939c1-66d4-4139-929b-7891295e60eb,network=Network(383fd64b-4c4c-497d-809d-3bcf8ed72e1c),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tape78939c1-66')
>> 2017-11-28 02:19:11.995 278905 INFO nova.compute.manager [-] [instance: 7a732fc2-0a35-4127-9138-22f58e580b2f] VM Started (Lifecycle Event)
>> 2017-11-28 02:19:12.015 278905 INFO nova.virt.libvirt.driver [-] [instance: 7a732fc2-0a35-4127-9138-22f58e580b2f] Instance spawned successfully.
>> 2017-11-28 02:19:12.015 278905 INFO nova.compute.manager [req-242cfa20-8f75-4643-b2ff-447a5ed557ac 4ac237f6708e4f0688589ce4efcae6af 850ecb27752344278c8816a39e26b6cd - 7565538147a74fc3add7dc0086b6ff1a 7565538147a74fc3add7dc0086b6ff1a] [instance: 7a732fc2-0a35-4127-9138-22f58e580b2f] Took 11.56 seconds to spawn the instance on the hypervisor.
>> 2017-11-28 02:19:12.508 278905 INFO nova.compute.manager [req-ffc7ee12-dbc2-407b-9c44-5ad70de22585 - - - - -] [instance: 7a732fc2-0a35-4127-9138-22f58e580b2f] VM Paused (Lifecycle Event)
>> 2017-11-28 02:19:12.547 278905 INFO nova.compute.manager [req-242cfa20-8f75-4643-b2ff-447a5ed557ac 4ac237f6708e4f0688589ce4efcae6af 850ecb27752344278c8816a39e26b6cd - 7565538147a74fc3add7dc0086b6ff1a 7565538147a74fc3add7dc0086b6ff1a] [instance: 7a732fc2-0a35-4127-9138-22f58e580b2f] Took 100.40 seconds to build instance.
>> 
>> 
>> 
>> <image.png>
>> 
>> 
>> Just to let you know that I am using "nosdn" scenario and not Opencontrail. Thanks for your continued support, much appreciated!!
>> 
>> Regards,
>> Navin S
>> 
>> 
>> 
>> 
>> On Monday, 27 November 2017 7:46 PM, Heather Lanigan <heather.lanigan at canonical.com> wrote:
>> 
>> 
>> Navin,
>> 
>> The error thrown by juju is received from OpenStack when we tried to boot an instance.  OpenStack is requiring the name or uuid of the network to use because it doesn't know which network to choose.  The fix for that error is using --config network=<network> during bootstrap.
>> 
>> Were there any hints in the nova log regarding details for the No Valid Hosts failure?  When you manually booted the OpenStack instance, did you need to specify `--nic net-id=net-uuid` or `--nic net-name=network-name`?
>> 
>> Unfortunately I'm not familiar with Opencontrail SDN.  
>> 
>> -Heather
>> 
>> On Mon, Nov 27, 2017 at 7:27 PM, Navin Sridharan <navinsridharan at ymail.com> wrote:
>> Hi Heather,
>> 
>> Thank you for the response. Earlier I used Opencontrail SDN while deploying JUJU, so I am guessing the issue was to do with the port binding as the script I was using to deploy was not compatible with the OpenContrail SDN. I tried re-deploying with "nosdn", but JUJU now throws the following error message:
>> 
>> nova.SecurityGroupName{Name:" juju-0fa027ff-0141-420f-8ec9- f27860520823-4325eb8d-2171- 427b-82cb-8beada255496-0"}}, Networks:[]nova. ServerNetworks{}, AvailabilityZone:"nova", Metadata:map[string]string{" juju-is-controller":"true", "juju-model-uuid":"4325eb8d- 2171-427b-82cb-8beada255496", "juju-controller-uuid":" 0fa027ff-0141-420f-8ec9- f27860520823"}, ConfigDrive:false}
>> caused by: request (http://192.168.122.190:8774/ v2.1/ 850ecb27752344278c8816a39e26b6 cd/servers) returned unexpected status: 409; error info: {"conflictingRequest": {"message": "Multiple possible networks found, use a Network ID to be more specific.", "code": 409}}
>> 
>> Also, I am able to boot up an instance manually in the OpenStack cloud using m1.medium and the status of the instance reads "Active".  Any insights on the above error please?
>> 
>> I see my error related to this post - https://bugs.launchpad.net/ juju/+bug/1570219
>> 
>> Regards,
>> Navin S
>> 
>> 
>> 
>> 
>> On Monday, 27 November 2017 6:55 PM, Heather Lanigan <heather.lanigan at canonical.com > wrote:
>> 
>> 
>> Navin,
>> 
>> From your email, it appears that juju is finding the network specified correctly.  Is there more information leading to believe that is the problem?
>> 
>> Are you able to boot an instance within the OpenStack Cloud specifying the same network, using the m1.medium flavor?  I'd recommend checking the nova logs to see what more info can be found on the cause of the No Valid Host error.
>> 
>> -Heather
>> 
>> On Mon, Nov 27, 2017 at 5:22 PM, Navin Sridharan <navinsridharan at ymail.com> wrote:
>> Hi all,
>> 
>> I am fairly new to JUJU and am currently working on deploying OAI (Open Air Interface) on OPNFV using JOID installer. 
>> 
>> H/W used : HP- Proliant DL380G6 G6 bare metal box 
>> OS              : Ubuntu 16.04 Xenial
>> 
>> I have spun up 3 VM's using MAAS (Metal as a Service) that's been deployed in the bare metal box. JUJU has been installed in the bare metal and I am trying to bootstrap it to Openstack cloud whose control and compute nodes are hosted on these VM's. Ultimate objective is to deploy OAI model using JUJU in the Openstack cloud.
>> 
>> During bootsrapping using the below command, it throws an error "failed to bootstrap model: cannot start bootstrap instance: cannot run instance: No valid host was found."  I believe the issue is with the Openstack's network UUID that's passed on as a parameter while bootstrapping.
>>  
>> juju bootstrap openstack --debug --config image-metadata-url=$SWIFT_URL/ simplestreams/images --config use-floating-ip=true --config network=private
>>  
>> Below given is the snippet from the error log:
>>  
>>  10:36:16 DEBUG juju.provider.openstack provider.go:1022 using network id "383fd64b-4c4c-497d-809d- 3bcf8ed72e1c"
>> 10:36:20 INFO  juju.provider.openstack provider.go:1146 trying to build instance in availability zone "nova"
>> 10:36:33 INFO  juju.provider.openstack provider.go:1126 Instance "78df0280-a1f0-4993-8416- c0be69050a11" in ERROR state with fault "No valid host was found. "
>> 10:36:33 INFO  juju.provider.openstack provider.go:1127 Deleting instance "78df0280-a1f0-4993-8416- c0be69050a11" in ERROR state
>> 10:36:33 INFO  juju.provider.openstack provider.go:1155 failed to build instance in availability zone "nova"
>> 10:36:33 ERROR juju.cmd.juju. commands bootstrap.go:496 failed to bootstrap model: cannot start bootstrap instance: cannot run instance: No valid host was found.
>>  
>>  
>> 
>> From the error log I see that it uses network id "383fd64b-4c4c-497d-809d- 3bcf8ed72e1c" which is the network id of private subnet and I confirmed that from the Openstack GUI (PFA screenshot), but then fails to bootstrap. I am stuck at this point in time and any response would be considered more than a help. Thank you all!!
>> 
>> <datauri-file.jpeg>
>> 
>> 
>> Regards,
>> Navin S
>> 
>> --
>> Juju mailing list
>> Juju at lists.ubuntu.com
>> Modify settings or unsubscribe at: https://lists.ubuntu.com/ mailman/listinfo/juju
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> --
>> Juju mailing list
>> Juju at lists.ubuntu.com
>> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
>> 
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju/attachments/20171207/52e5cae6/attachment-0001.html>

More information about the Juju mailing list