juju hangs during bootstrapping

Sat Sep 30 03:52:47 UTC 2017

From: Andrew Wilkins [mailto:andrew.wilkins at canonical.com]
Sent: Saturday, September 30, 2017 12:23 AM
To: Chen2, Dave <Dave_Chen2 at Dell.com>; juju at lists.ubuntu.com
Subject: Re: juju hangs during bootstrapping

On Fri, Sep 29, 2017 at 10:43 AM <Dave.Chen at dell.com<mailto:Dave.Chen at dell.com>> wrote:
Hi All,

I am trying to bootstrap a MAAS cloud based on juju’s official guide (https://jujucharms.com/docs/2.2/clouds-maas), everything seems correct but after the Operation System (Ubuntu 16.04 or Ubuntu14.0) has been installed, juju hangs when attempting to connect to the MAAS node, here is what I can see from the terminal,

$ juju bootstrap maas-cloud

Creating Juju controller "maas-cloud" on maas-cloud

Looking for packaged Juju agent version 2.2.4 for amd64

Launching controller instance(s) on maas-cloud...

- cka68p (arch=amd64 mem=32G cores=12)

Fetching Juju GUI 2.9.2

Waiting for address

Attempting to connect to 10.20.3.254:22<http://10.20.3.254:22> (JUJU hangs here!)

And it’s pending here forever, so I tried it again with the debug mode,

$ juju bootstrap --show-log --debug --bootstrap-series=trusty maas-cloud maas-cloud-controller

I saw some detail information like below,

Attempting to connect to 10.20.3.254:22<http://10.20.3.254:22>

19:33:11 DEBUG juju.provider.common bootstrap.go:497 connection attempt for 10.20.3.254 failed: ssh: connect to host 10.20.3.254 port 22: Connection refused

19:33:16 DEBUG juju.provider.common bootstrap.go:497 connection attempt for 10.20.3.254 failed: ssh: connect to host 10.20.3.254 port 22: Connection refused

19:33:21 DEBUG juju.provider.common bootstrap.go:497 connection attempt for 10.20.3.254 failed: ssh: connect to host 10.20.3.254 port 22: Connection refused

19:33:56 DEBUG juju.provider.common bootstrap.go:497 connection attempt for 10.20.3.254 failed: /var/lib/juju/nonce.txt does not exist

19:34:32 DEBUG juju.provider.common bootstrap.go:497 connection attempt for 10.20.3.254 failed: /var/lib/juju/nonce.txt does not exist

19:35:08 DEBUG juju.provider.common bootstrap.go:497 connection attempt for 10.20.3.254 failed: /var/lib/juju/nonce.txt does not exist

19:35:43 INFO  juju.cloudconfig userdatacfg_unix.go:410 Fetching agent: curl -sSfw 'tools from %{url_effective} downloaded: HTTP %{http_code}; time %{time_total}s; size %{size_download} bytes; speed %{speed_download} bytes/s ' --retry 10 -o $bin/tools.tar.gz <[https://streams.canonical.com/juju/tools/agent/2.2.4/juju-2.2.4-ubuntu-amd64.tgz]><https://streams.canonical.com/juju/tools/agent/2.2.4/juju-2.2.4-ubuntu-amd64.tgz%5d%3e>

Is this the last thing logged? Try running that curl command on the machine manually. Perhaps there's an issue getting out to the internet.
[Dave] Yes, this is the last line I saw, our network topology is MAAS server can access internet (dual NIC with one NIC can access outer network), but each node that is deployed by MAAS/JUJU only get the IP from internal DHCP service, do you mean each deployed node also need access outer internet?

And… Do you have any idea about the  ssh connection refused error?

I have no idea what’s going wrong since I can telnet to the node and ssh to that node is also possible, I just need type “yes” then I can login to the node,
$ ssh ubuntu at 10.20.3.254<mailto:ubuntu at 10.20.3.254>
The authenticity of host ' 10.20.3.254 (10.20.3.254)' can't be established.
ECDSA key fingerprint is SHA256:4FVm21s4dx7gc0/yDgz0+QAMGK4qWODoIqeoWtZg9RI.
Are you sure you want to continue connecting (yes/no)?

From the console of that node, I can find the controller’s public key has been injected to the node,

-----BEGIN SSH HOST KEY KEYS-------

…

-----END SSH KEY FINGERPRINTS--------
…
Cloud-init v. 0.7.9 finished at … Datasource DataSourceMAAS [http://...:5240/MAAS/metadata/]. Up 153.77 seconds.       (cloud-init hangs here!)

I googled it and found someone said it is because “authorized-keys-path” is commented out in the “environments.yaml” [1], but the juju version I am using is “2.2.4-xenial-amd64”, the MAAS version is 2.2.2,
Initially, I installed juju 1.25 and configured environments.yaml, but now I have uninstalled juju 1.25, removed all those file in $home/.juju/ and start it over again with juju 2.2.4.
I really cannot figure out why it always hangs at this step, is there any cache persisted anywhere that masked the  “authorized-keys-path” even after the uninstallation of juju1.25? or there is any step I missed with juju 2.2.4?

Where is user-data of cloud-init persisted on the filesystem? Any more detail logs I can refer to?

I feel frustration after trying several days without any progress, pls help me out, many many thanks for any inputs!

[1]  https://serverfault.com/questions/588967/juju-bootstrap-fails-connection-refused-port-22

Best Regards,
Dave Chen

Best Regards,
Dave Chen

--
Juju mailing list
Juju at lists.ubuntu.com<mailto:Juju at lists.ubuntu.com>
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju/attachments/20170930/a9b43ce7/attachment.html>