Issues with DHCP and IPMI

Thu Nov 16 03:54:04 UTC 2017

Hi Андрей,

Thanks for your feedback, see the answers below.

1.

a.

> Sep 26 18:11:42 maas maas.api: [info] comic-salmon: Enlisted new
machine
> Sep 26 18:11:43 maas maas.node: [info] BMC could not save
extracted IP address ‘172.Х.Х.Х': 'get() returned more than one
StaticIPAddress -- it returned 2!'

https://github.com/maas/maas/blob/2.3.0-rc1/src/maasserver/api/machines
.py#L1030
def create_machine(request):
    """Service an http request to create a machine.
...
        maaslog.info("%s: Enlisted new machine", machine.hostname)
        return machine

b.

https://github.com/maas/maas/blob/2.3.0-rc1/src/maasserver/models/bmc.p
y#L256-L259

                    with transaction.atomic():
                        subnet =
Subnet.objects.get_best_subnet_for_ip(new_ip)
                        (self.ip_address,
                         _) = StaticIPAddress.objects.get_or_create(
                            ip=new_ip,
                            defaults={
                                'alloc_type': IPADDRESS_TYPE.STICKY,
                                'subnet': subnet,
                            })
                except Exception as error:
                    maaslog.info(
                        "BMC could not save extracted IP "
                        "address '%s': '%s'", new_ip, error)

c.

"get() returned more than one" - django-level message https://github.co
m/django/django/blob/1.8.7/django/db/models/query.py#L336-L339 on
get_or_create -> get code path

    def get(self, *args, **kwargs):
        """
        Performs the query and returns a single object matching the
given
        keyword arguments.
        """
...

        raise self.model.MultipleObjectsReturned(
            "get() returned more than one %s -- it returned %s!" %
...

So, when a new power address entry is to be created there are already
two entries present matching the criteria.

I wonder if this could happen due to an IPv4 conflict with hosts
residing in the same L2 domain. In any case this should be
investigated a bit more.

Could you please file a bug with the info you have?

https://bugs.launchpad.net/maas/+filebug

Also, what kind of subnet mode did you have configured? Managed or
unmanaged?

https://docs.ubuntu.com/maas/2.3/en/intro-concepts#ip-ranges

=====

2.

As far as I know, MAAS does not have any auto-detection capabilities
with regards to power address changes (i.e. MAC tracking followed by
auto-updates to the DB). Maybe it's worthwhile to add that to MAAS for
power addresses or make an explicit setting - I would create a bug on
launchpad and it may receive a feature request status.

It does monitor the network though for the purposes for auto-discovery 
https://docs.ubuntu.com/maas/2.3/en/installconfig-network-dev-discovery

If we go step-by-step:

* PXE boot stage - MAAS' dhcpd supplies a server TFTP address in a DHCP
header and then serves a bootloader via TFTP to a client;
* bootloader loads the kernel and initramfs into memory and hands
control to an init script;
* With MAAS 2.3 a squashfs of an ephemeral image will be pulled via
HTTP (instead of iSCSI as before) and made a new rootfs;
* cloud-init will run enlistment scripts based on metadata obtained
from MAAS;
* a machine will end up in MAAS as *new*.

This is the behavior you are getting.

Code-wise this is expected as you powered on the node yourself - not
through MAAS so the metadata fetched at the cloud-init stage is for
enlistment, not managed boot. Downloaded images themselves are not
modifed to contain metadata - it is retrieved instead by talking to
regiond:

In the regiond.log you may see:

2017-11-16 08:06:24 regiond: [info] 127.0.0.1 GET
/MAAS/metadata/latest/by-id/krnscf/?op=get_preseed HTTP/1.1 --> 200 OK
(referrer: -; agent: Cloud-Init/0.7.9)

Following the code-path

https://github.com/maas/maas/blob/2.3.0-rc1/src/metadataserver/api.py#L
1067-L1075
    def get_preseed(self, request, version=None, system_id=None):
        """Render and return a preseed script for the given node."""
        return HttpResponse(get_preseed(node),
content_type="text/plain")

https://github.com/maas/maas/blob/2.3.0-rc1/src/maasserver/preseed.py#L
553-L566

def get_preseed_type_for(node):
    """Returns the preseed type for the node.
    If the node is in a commissioning like status then the
commissioning
    preseed will be used. Otherwise the node will use the curtin
installer.
    """
    is_commissioning_preseed = (
        node.status in COMMISSIONING_LIKE_STATUSES or
        node.get_boot_purpose() == 'poweroff' # <---- this
        )
    if is_commissioning_preseed:
        return PRESEED_TYPE.COMMISSIONING
    else:
        return PRESEED_TYPE.CURTIN  # <---- this

a boot purpose is determined in node.get_boot_purpose

https://github.com/maas/maas/blob/2.3.0-rc1/src/maasserver/models/node.
py#L3380-L3418
    def get_boot_purpose(self):
        """
        Return a suitable "purpose" for this boot, e.g. "install".
...
        else:
            return "poweroff"

which is poweroff in this case (see the "..." part).

https://github.com/maas/maas/blob/2.3.0-rc1/contrib/preseeds_v2/enlist

https://github.com/maas/maas/blob/2.3.0-rc1/contrib/preseeds_v2/enlist_
userdata

So this behavior is expected as all management besides initial
enlistment is expected to happen via MAAS.

Albeit this doc haven't been modified for some time it is still mostly
valid - it describes the above steps as well but for commissioning
instead of enlistment:
https://github.com/maas/maas/blob/master/docs/development/notes/anatomy
-of-recommissioning-in-maas-2.0.rst

3. 

The code is still there in an enlistment preseed and comissioning
script.

https://github.com/maas/maas/blame/2.3.0-rc1/contrib/preseeds_v2/enlist
_userdata#L21

   # If IPMI network settings have been configured statically, you can
   # make them DHCP. If 'true', the IPMI network source will be changed
   # to DHCP.
   IPMI_CHANGE_STATIC_TO_DHCP="false"

https://github.com/maas/maas/blame/2.3.0-rc1/src/metadataserver/user_data/templates/commissioning.template#L15
Not sure about the button - I don't see any references to the code
actually setting this from the UI.

You can always set this to true for enlistment:

ubuntu at maas:~$ grep -RiP static_to_dhcp /etc/maas/preseeds/
/etc/maas/preseeds/enlist_userdata:   IPMI_CHANGE_STATIC_TO_DHCP="false
"
/etc/maas/preseeds/enlist_userdata:   if $IPMI_CHANGE_STATIC_TO_DHCP;
then

I agree that a button would be nice to have.

4. If it's not mentioned in 2.3.x milestones it's best to file this as
a bug. If you include a version and logs (in case you still have them)
that should help the MAAS team to narrow this down.

https://launchpad.net/maas/+milestones

https://launchpad.net/maas/+milestone/2.3.0rc1

https://launchpad.net/maas/+milestone/2.3.x

Best Regards,
Dmitrii Shcherbakov

Field Software Engineer
IRC (freenode): Dmitrii-Sh