juju hangs during bootstrapping

Sat Sep 30 11:58:11 UTC 2017

On Fri, Sep 29, 2017 at 11:55 PM <Dave.Chen at dell.com> wrote:

>
>
>
>
> *From:* Andrew Wilkins [mailto:andrew.wilkins at canonical.com]
> *Sent:* Saturday, September 30, 2017 12:23 AM
> *To:* Chen2, Dave <Dave_Chen2 at Dell.com>; juju at lists.ubuntu.com
> *Subject:* Re: juju hangs during bootstrapping
>
>
>
> On Fri, Sep 29, 2017 at 10:43 AM <Dave.Chen at dell.com> wrote:
>
> Hi All,
>
>
>
> I am trying to bootstrap a MAAS cloud based on juju’s official guide (
> https://jujucharms.com/docs/2.2/clouds-maas), everything seems correct
> but after the Operation System (Ubuntu 16.04 or Ubuntu14.0) has been
> installed, juju hangs when attempting to connect to the MAAS node, here is
> what I can see from the terminal,
>
>
>
> $ juju bootstrap maas-cloud
>
> Creating Juju controller "maas-cloud" on maas-cloud
>
> Looking for packaged Juju agent version 2.2.4 for amd64
>
> Launching controller instance(s) on maas-cloud...
>
> - cka68p (arch=amd64 mem=32G cores=12)
>
> Fetching Juju GUI 2.9.2
>
> Waiting for address
>
> Attempting to connect to 10.20.3.254:22 (JUJU hangs here!)
>
>
>
> And it’s pending here forever, so I tried it again with the debug mode,
>
> $ juju bootstrap --show-log --debug --bootstrap-series=trusty maas-cloud
> maas-cloud-controller
>
>
>
> I saw some detail information like below,
>
> Attempting to connect to 10.20.3.254:22
>
> 19:33:11 DEBUG juju.provider.common bootstrap.go:497 connection attempt
> for 10.20.3.254 failed: ssh: connect to host 10.20.3.254 port 22:
> Connection refused
>
> 19:33:16 DEBUG juju.provider.common bootstrap.go:497 connection attempt
> for 10.20.3.254 failed: ssh: connect to host 10.20.3.254 port 22:
> Connection refused
>
> 19:33:21 DEBUG juju.provider.common bootstrap.go:497 connection attempt
> for 10.20.3.254 failed: ssh: connect to host 10.20.3.254 port 22:
> Connection refused
>
> 19:33:56 DEBUG juju.provider.common bootstrap.go:497 connection attempt
> for 10.20.3.254 failed: /var/lib/juju/nonce.txt does not exist
>
> 19:34:32 DEBUG juju.provider.common bootstrap.go:497 connection attempt
> for 10.20.3.254 failed: /var/lib/juju/nonce.txt does not exist
>
> 19:35:08 DEBUG juju.provider.common bootstrap.go:497 connection attempt
> for 10.20.3.254 failed: /var/lib/juju/nonce.txt does not exist
>
> 19:35:43 INFO  juju.cloudconfig userdatacfg_unix.go:410 Fetching agent:
> curl -sSfw 'tools from %{url_effective} downloaded: HTTP %{http_code}; time
> %{time_total}s; size %{size_download} bytes; speed %{speed_download}
> bytes/s ' --retry 10 -o $bin/tools.tar.gz <[
> https://streams.canonical.com/juju/tools/agent/2.2.4/juju-2.2.4-ubuntu-amd64.tgz]>
>
>
>
> Is this the last thing logged? Try running that curl command on the
> machine manually. Perhaps there's an issue getting out to the internet.
>
> *[Dave] Yes, this is the last line I saw, our network topology is MAAS
> server can access internet (dual NIC with one NIC can access outer
> network), but each node that is deployed by MAAS/JUJU only get the IP from
> internal DHCP service, do you mean each deployed node also need access
> outer internet?*
>

The bootstrap machine requires access to the Juju agent repository, which
is on  "streams.canonical.com". It's possible to point Juju at a private
mirror to avoid the need to go to the internet.

*And… Do you have any idea about the  ssh connection refused error?*
>

That's just log spam, we should clean that up. Juju requests the MAAS
server, and then immediately starts attempting to connect via SSH. When the
server is still starting, we expect to see connection refused errors.

>
> I have no idea what’s going wrong since I can telnet to the node and ssh
> to that node is also possible, I just need type “yes” then I can login to
> the node,
>
> $ ssh ubuntu at 10.20.3.254
>
> The authenticity of host ' 10.20.3.254 (10.20.3.254)' can't be established.
>
> ECDSA key fingerprint is
> SHA256:4FVm21s4dx7gc0/yDgz0+QAMGK4qWODoIqeoWtZg9RI.
>
> Are you sure you want to continue connecting (yes/no)?
>
>
>
> From the console of that node, I can find the controller’s public key has
> been injected to the node,
>
> -----BEGIN SSH HOST KEY KEYS-------
>
> …
>
> -----END SSH KEY FINGERPRINTS--------
>
> …
>
> Cloud-init v. 0.7.9 finished at … Datasource DataSourceMAAS [http://...:5240/MAAS/metadata/].
> Up 153.77 seconds.       (cloud-init hangs here!)
>
>
>
>
>
> I googled it and found someone said it is because “authorized-keys-path”
> is commented out in the “environments.yaml” [1], but the juju version I am
> using is “2.2.4-xenial-amd64”, the MAAS version is 2.2.2,
>
> Initially, I installed juju 1.25 and configured environments.yaml, but now
> I have uninstalled juju 1.25, removed all those file in $home/.juju/ and
> start it over again with juju 2.2.4.
>
> I really cannot figure out why it always hangs at this step, is there any
> cache persisted anywhere that masked the  “authorized-keys-path” even after
> the uninstallation of juju1.25? or there is any step I missed with juju
> 2.2.4?
>
>
>
> Where is user-data of cloud-init persisted on the filesystem? Any more
> detail logs I can refer to?
>
>
>
>
>
> I feel frustration after trying several days without any progress, pls
> help me out, many many thanks for any inputs!
>
>
>
>
>
> [1]
> https://serverfault.com/questions/588967/juju-bootstrap-fails-connection-refused-port-22
>
>
>
>
>
> Best Regards,
>
> Dave Chen
>
>
>
> Best Regards,
>
> Dave Chen
>
>
>
> --
> Juju mailing list
> Juju at lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju/attachments/20170930/5dd2022c/attachment.html>