[Bug 1639202] Re: Cannot enlist/commission machines in MAAS 2.1 with usb network adapter

Thu Feb 9 02:55:03 UTC 2017

This is happening to me on an Intel NUC 5i5MYHE blade on PXE.  The error
is an iSCSI destination error apparently.  Same scenario...

Maas version:  MAAS Version 2.1.3+bzr5573-0ubuntu1 (16.04.1)

On node PXE boot console output (sorry, but some of this is cut off on
this screen, typing what I can see.  will switch to another monitor and
reproduce if this is not enough of the error):

lvmetad is not active yet, using sysinit

disk/by-path/ip-192.168.199.2:3260-iscsi-iqn.2004-05.com.ubuntu:maas:ephemeral...
...ubuntu-amd64-generic-xenial-daily-lun-1"

tgtd on the MaaS node seems to be running fine but netstat output does
not show the initiator contacting it when I search for the IP that MaaS
assigned on PXE init.

tgtd MaaS node is Raspberry Pi 3B, so not a Dell issue specifically:

root at juju-rack2:~# cat /etc/issue
Ubuntu 16.04.1 LTS \n \l

root at juju-rack2:~# uname -r
4.4.43-v7+

MAAS Version 2.1.3+bzr5573-0ubuntu1 (16.04.1)

Intel blade is also running PCIe solid state drive, not a USB device.

Fails about every 60 seconds according to maas.log:

"Feb  8 21:47:17 juju-rack2 maas.service_monitor: [error] While monitoring service 'tgt' an error was encountered: Unable to parse the active state from systemd for service 'tgt', active state reported as 'deactivating'.
Feb  8 21:48:18 juju-rack2 maas.service_monitor: [error] While monitoring service 'tgt' an error was encountered: Unable to parse the active state from systemd for service 'tgt', active state reported as 'deactivating'.
Feb  8 21:49:17 juju-rack2 maas.service_monitor: [error] While monitoring service 'tgt' an error was encountered: Unable to parse the active state from systemd for service 'tgt', active state reported as 'deactivating'."

/var/log/rackd/log output:

2017-02-08 21:32:23 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:hwe-16.04:xenial: to_add=['20170207'] to_remove=[]
2017-02-08 21:32:23 sstreams: [info] maas:v2:download/maas:boot:ubuntu:armhf:hwe-16.04:xenial: to_add=['20170207'] to_remove=[]
2017-02-08 21:32:23 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:ga-16.04-lowlatency:xenial: to_add=['20170207'] to_remove=[]
2017-02-08 21:32:23 sstreams: [info] maas:v2:download/maas:boot:grub-efi:arm64:generic:uefi: to_add=['20170125.0'] to_remove=[]
2017-02-08 21:32:23 sstreams: [info] maas:v2:download/maas:boot:grub-efi-signed:amd64:generic:uefi: to_add=['20170125.0'] to_remove=[]
2017-02-08 21:32:23 sstreams: [info] maas:v2:download/maas:boot:ubuntu:armhf:ga-16.04:xenial: to_add=['20170207'] to_remove=[]
2017-02-08 21:32:23 sstreams: [info] maas:v2:download/maas:boot:ubuntu:armhf:generic-lpae:xenial: to_add=['20170207'] to_remove=[]
2017-02-08 21:32:23 sstreams: [info] maas:v2:download/maas:boot:ubuntu:armhf:hwe-16.04-edge:xenial: to_add=['20170207'] to_remove=[]
2017-02-08 21:32:23 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:ga-16.04:xenial: to_add=['20170207'] to_remove=[]
2017-02-08 21:32:23 sstreams: [info] maas:v2:download/maas:boot:centos:amd64:generic:centos70: to_add=['20161201_01'] to_remove=[]
2017-02-08 21:32:23 sstreams: [info] maas:v2:download/maas:boot:grub-ieee1275:ppc64el:generic:open-firmware: to_add=['20170125.0'] to_remove=[]
2017-02-08 21:32:23 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:hwe-16.04-edge:xenial: to_add=['20170207'] to_remove=[]
2017-02-08 21:32:23 sstreams: [info] maas:v2:download/maas:boot:pxelinux:i386:generic:pxe: to_add=['20160930.0'] to_remove=[]
2017-02-08 21:32:23 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:hwe-16.04-lowlatency:xenial: to_add=['20170207'] to_remove=[]
2017-02-08 21:32:23 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:hwe-16.04-lowlatency-edge:xenial: to_add=['20170207'] to_remove=[]
2017-02-08 21:32:23 provisioningserver.rackdservices.image_download_service: [critical] Downloading images failed.

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 434, in errback
    self._startRunCallbacks(fail)
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 501, in _startRunCallbacks
    self._runCallbacks()
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 588, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1184, in gotResult
    _inlineCallbacks(r, g, deferred)
--- <exception caught here> ---
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1126, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/lib/python3/dist-packages/provisioningserver/rackdservices/image_download_service.py", line 128, in maybe_start_download
    yield self._start_download()
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1126, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/lib/python3/dist-packages/provisioningserver/rackdservices/image_download_service.py", line 115, in _start_download
    get_proxy_url("https"))
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1126, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/lib/python3/dist-packages/provisioningserver/rpc/boot_images.py", line 145, in _import_boot_images
    imported = yield deferToThread(_run_import, sources, **proxies)
  File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 246, in inContext
    result = inContext.theWork()
  File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 262, in <lambda>
    inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
  File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 81, in callWithContext
    return func(*args,**kw)
  File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 225, in wrapper
    result = func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/provisioningserver/rpc/boot_images.py", line 119, in _run_import
    imported = boot_resources.import_images(sources)
  File "/usr/lib/python3/dist-packages/provisioningserver/import_images/boot_resources.py", line 278, in import_images
    update_iscsi_targets(os.path.join(storage, 'current'))
  File "/usr/lib/python3/dist-packages/provisioningserver/import_images/boot_resources.py", line 238, in update_iscsi_targets
    update_targets_conf(snapshot_path)
  File "/usr/lib/python3/dist-packages/provisioningserver/import_images/boot_resources.py", line 198, in update_targets_conf
    service_monitor.ensureService("tgt").wait(30)
  File "/usr/lib/python3/dist-packages/crochet/_eventloop.py", line 231, in wait
    result.raiseException()
  File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 368, in raiseException
    raise self.value.with_traceback(self.tb)
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1126, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/lib/python3/dist-packages/provisioningserver/utils/service_monitor.py", line 535, in _ensureService
    state = yield self.getServiceState(service.name, now=True)
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1126, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/lib/python3/dist-packages/provisioningserver/utils/service_monitor.py", line 251, in getServiceState
    active_state, process_state = yield self._loadServiceState(service)
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks
    result = g.send(result)
  File "/usr/lib/python3/dist-packages/provisioningserver/utils/service_monitor.py", line 478, in _loadSystemDServiceState
    service.service_name, active_state))
provisioningserver.utils.service_monitor.ServiceParsingError: Unable to parse the active state from systemd for service 'tgt', active state reported as 'deactivating'.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to initramfs-tools in Ubuntu.
https://bugs.launchpad.net/bugs/1639202

Title:
  Cannot enlist/commission machines in MAAS 2.1 with usb network adapter

Status in MAAS:
  Invalid
Status in maas-images:
  Invalid
Status in initramfs-tools package in Ubuntu:
  Invalid
Status in linux package in Ubuntu:
  Invalid

Bug description:
  After upgrade from MAAS 2.0 to MAAS 2.1:

  Cannot enlist/commission client machines via network boot (or pre-
  staging machine) when using usb-c network adapter D59GG (e.g.
  Precision 5510). This was working ok with MAAS 2.0.

  Using the Xenial 16.04 base image for enlist/commission (no minimum
  kernel set).

  Enlist/Commission/Deploy works fine with other laptop and desktop
  models which have a built in NIC.

  From an already deployed Precision 5510 system (16.04.1) I can see the
  following module is loaded when the usb-c adapter is connected (and
  working):

  $ lsmod
  usbnet	45056 1 cdc_ether

  Errors received on client during enlisting with MAAS:

  no /run/net-bootif.conf
  lvmetad is not activated
  ...invalid path for logical volume.
  gave up waiting for root device
  common problems: boot args (cat /proc/cmdline)
  check rootdelay = (did system wait long enough)
  check root = (did the system wait for the right device?)
  missing modules (cat /proc/modules)
  ALERT!  /dev/disk/by-path/ip-<ipaddress>:3260-iscsi-iqn,2004.05.com.ubuntu:maas:ephemeral-ubuntu-amd64-generic-xenial-daily-lun-1 does not exist. Dropping to shell!
  usbcore: registered new interface driver usbhid
  usbhid: USB HID core driver

  - cat /proc/cmdline shows mac address that of usb-c network adapter.
  - cat /proc/modules includes “usbnet 45056 cdc_ether, Live 0xffffffffc009c000”
  - cat /proc/modules | grep usb … includes “ usbnet 45056 and usbhid 49152”

  I've tried enlisting with older boot image and different kernel versions (14.04 and 16.04 with ga-16.04, hwe-16.04 set) but get the same problem.

  It seems like the usb-c network adapter isn't loading properly or
  maybe just not quickly enough?

  Please let me know if you require any more info. I can provide info
  from /var/log/maas/* and dpkg -l '*maas*'|cat if need be.

To manage notifications about this bug go to:
https://bugs.launchpad.net/maas/+bug/1639202/+subscriptions