juju hangs during bootstrapping

Sat Sep 30 04:03:03 UTC 2017

From: Narinder Gupta [mailto:narinder.gupta at canonical.com]
Sent: Friday, September 29, 2017 10:52 PM
To: Chen2, Dave <Dave_Chen2 at Dell.com>
Cc: juju <juju at lists.ubuntu.com>
Subject: Re: juju hangs during bootstrapping

Hi Dave,
May I know which division of Dell you are working on? As i have setup Openstack deployed t Dell multiple time with MAAS and have not seen this issue so far.

So please send me log /var/log/cloud-init-output.log which will let us know what is wrong. Also try sudo apt-get update on the bootstrap node to confirm you have external access.

In MAAS you can always add the ssh keys to land into the installed nodes though.
[Dave Chen] Hi Narinder, thanks for your response, I am from Dell ESI, could you more elaborate on how to configure to make sure ssh keys can be landed into the installed nodes? I was thinking the ssh keys has been injected into each node correctly, but no idea why it was rejected although I can ssh to deployed manually, see below pls.

“
$ ssh ubuntu at 10.20.3.254<mailto:ubuntu at 10.20.3.254>
The authenticity of host ' 10.20.3.254 (10.20.3.254)' can't be established.
ECDSA key fingerprint is SHA256:4FVm21s4dx7gc0/yDgz0+QAMGK4qWODoIqeoWtZg9RI.
Are you sure you want to continue connecting (yes/no)?
”

If I can know how user-data is written/located it might be helpful, but I cannot find those script ☹.

Thanks and Regards,

Narinder Gupta (PMP)                   narinder.gupta at canonical.com<mailto:narinder.gupta at canonical.com>

Canonical, Ltd.                    narindergupta [irc.freenode.net<http://irc.freenode.net>]

+1.281.736.5150                            narindergupta2007[skype]

Ubuntu- Linux for human beings | www.ubuntu.com<http://www.ubuntu.com> | www.canonical.com<http://www.canonical.com>

On Fri, Sep 29, 2017 at 9:42 AM, <Dave.Chen at dell.com<mailto:Dave.Chen at dell.com>> wrote:
Hi All,

I am trying to bootstrap a MAAS cloud based on juju’s official guide (https://jujucharms.com/docs/2.2/clouds-maas), everything seems correct but after the Operation System (Ubuntu 16.04 or Ubuntu14.0) has been installed, juju hangs when attempting to connect to the MAAS node, here is what I can see from the terminal,

$ juju bootstrap maas-cloud

Creating Juju controller "maas-cloud" on maas-cloud

Looking for packaged Juju agent version 2.2.4 for amd64

Launching controller instance(s) on maas-cloud...

- cka68p (arch=amd64 mem=32G cores=12)

Fetching Juju GUI 2.9.2

Waiting for address

Attempting to connect to 10.20.3.254:22<http://10.20.3.254:22> (JUJU hangs here!)

And it’s pending here forever, so I tried it again with the debug mode,

$ juju bootstrap --show-log --debug --bootstrap-series=trusty maas-cloud maas-cloud-controller

I saw some detail information like below,

Attempting to connect to 10.20.3.254:22<http://10.20.3.254:22>

19:33:11 DEBUG juju.provider.common bootstrap.go:497 connection attempt for 10.20.3.254 failed: ssh: connect to host 10.20.3.254 port 22: Connection refused

19:33:16 DEBUG juju.provider.common bootstrap.go:497 connection attempt for 10.20.3.254 failed: ssh: connect to host 10.20.3.254 port 22: Connection refused

19:33:21 DEBUG juju.provider.common bootstrap.go:497 connection attempt for 10.20.3.254 failed: ssh: connect to host 10.20.3.254 port 22: Connection refused

19:33:56 DEBUG juju.provider.common bootstrap.go:497 connection attempt for 10.20.3.254 failed: /var/lib/juju/nonce.txt does not exist

19:34:32 DEBUG juju.provider.common bootstrap.go:497 connection attempt for 10.20.3.254 failed: /var/lib/juju/nonce.txt does not exist

19:35:08 DEBUG juju.provider.common bootstrap.go:497 connection attempt for 10.20.3.254 failed: /var/lib/juju/nonce.txt does not exist

19:35:43 INFO  juju.cloudconfig userdatacfg_unix.go:410 Fetching agent: curl -sSfw 'tools from %{url_effective} downloaded: HTTP %{http_code}; time %{time_total}s; size %{size_download} bytes; speed %{speed_download} bytes/s ' --retry 10 -o $bin/tools.tar.gz <[https://streams.canonical.com/juju/tools/agent/2.2.4/juju-2.2.4-ubuntu-amd64.tgz]><https://streams.canonical.com/juju/tools/agent/2.2.4/juju-2.2.4-ubuntu-amd64.tgz%5d%3e>

I have no idea what’s going wrong since I can telnet to the node and ssh to that node is also possible, I just need type “yes” then I can login to the node,
$ ssh ubuntu at 10.20.3.254<mailto:ubuntu at 10.20.3.254>
The authenticity of host ' 10.20.3.254 (10.20.3.254)' can't be established.
ECDSA key fingerprint is SHA256:4FVm21s4dx7gc0/yDgz0+QAMGK4qWODoIqeoWtZg9RI.
Are you sure you want to continue connecting (yes/no)?

From the console of that node, I can find the controller’s public key has been injected to the node,

-----BEGIN SSH HOST KEY KEYS-------

…

-----END SSH KEY FINGERPRINTS--------
…
Cloud-init v. 0.7.9 finished at … Datasource DataSourceMAAS [http://...:5240/MAAS/metadata/]. Up 153.77 seconds.       (cloud-init hangs here!)

I googled it and found someone said it is because “authorized-keys-path” is commented out in the “environments.yaml” [1], but the juju version I am using is “2.2.4-xenial-amd64”, the MAAS version is 2.2.2,
Initially, I installed juju 1.25 and configured environments.yaml, but now I have uninstalled juju 1.25, removed all those file in $home/.juju/ and start it over again with juju 2.2.4.
I really cannot figure out why it always hangs at this step, is there any cache persisted anywhere that masked the  “authorized-keys-path” even after the uninstallation of juju1.25? or there is any step I missed with juju 2.2.4?

Where is user-data of cloud-init persisted on the filesystem? Any more detail logs I can refer to?

I feel frustration after trying several days without any progress, pls help me out, many many thanks for any inputs!

[1]  https://serverfault.com/questions/588967/juju-bootstrap-fails-connection-refused-port-22

Best Regards,
Dave Chen

Best Regards,
Dave Chen

--
Juju mailing list
Juju at lists.ubuntu.com<mailto:Juju at lists.ubuntu.com>
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju/attachments/20170930/371a9417/attachment.html>