[Bug 1677668] Re: no GARPs during ephemeral boot

[ChristianEhrhardt]

First of all thanks Jay for the great in depth extra-insight!

As Sam added, Cloud-Init can't do a lot here since it doesn't get a config to this stage.
The only thing it could consider to do is unconditionally always trigger a GARP.

But that would be:
1. offending the cloud-init design which is that it is supposed to do what it is told (not more not less)
2. add the dependency to a tool like arping for the image so that cloud-init can issue the GARP

So one step back - implementation in dhclient as-is as script:
1. this would still have to be unconditionally, but at least be bound to dhcp
2. it would have the same need for arping (or similar) in the image
  2.1 an alternative to the dependency would be to re-implement, but redundancy always is worse

I wonder if the following would be an option:
You already get your IP address, so you get your dhcp reply; Just the environment doesn't realize you moved. IIRC dhcp can transport extra options, so to get rid of the "unconditional" thing. Could dhcp grow a feature to understand a "dhcp-please-garp" option? Handling this option could be done in a dhclient script then. And on the other side the MAAS dhcp server could present this option.

That at least would make it
1. conditionally only where requested
2. maas has the control since it is the dhcp
3. not affect environments where things are not providing this option (for SRU-ability)

Making the cloud-init a wishlist item, since it seems a feature request
more than a bug there and also as outlined above not really fixable
there. Instead adding a dhclient task.

All of this has to consider, will more of systemd-networkd replace
dhclient? If so considerations have to be made for that or it will
regress as soon as things are switched.

** Also affects: isc-dhcp (Ubuntu)
   Importance: Undecided
       Status: New

** Changed in: cloud-init (Ubuntu)
       Status: New => Incomplete

** Changed in: cloud-init (Ubuntu)
   Importance: Undecided => Wishlist

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to isc-dhcp in Ubuntu.
https://bugs.launchpad.net/bugs/1677668

Title:
  no GARPs during ephemeral boot

Status in cloud-init package in Ubuntu:
  Incomplete
Status in isc-dhcp package in Ubuntu:
  New

Bug description:
  Deploys time out with an error on the console that says,

  "Can not apply stage final, no datasource found! Likely bad things to
  come!"

  How to duplicate:
  MAAS Version 2.1.3+bzr5573-0ubuntu1 (16.04.1)
  1) Rack Controller and Region Controller in different VLANs
  2) Use Cisco ASA as the router with "ARP Inspection" enabled
  3) Clear the router ARP cache
  4) Deploy 2 maas machines with interfaces set to "Static assign"
  5) Observe deploys successfully
  6) Release both machines and swap IP's.
  7) Redeploy the same 2 machines
  8) Observe deploy failure with the machine consoles stuck in the "ubuntu login" screen with "Can not apply stage final, no datasource Found! Likely bad things to come!"
   
  The root cause is that during ephemeral PXE booting, no GARPs are sent, which in our environment will cause our router (Cisco ASA) to hold on to ARP table entries until it expires (default= 4 hours).  Then combined with ASA feature "ARP Inspection" will drop packets from a MaaS machine using the previously used IP from a different MaaS machine.

  The ephemeral boot image ephemeral-ubuntu-amd64-ga-16.04-xenial-daily.

  Running tcpdump on the Rack Controller, showed no GARPs from the
  deploying MaaS machine.  If there were GARPs sent, then the router
  would refresh its ARP cache thus avoiding the ARP Inspection dropping.

  [Excerpt from Cisco ASA]
  http://www.cisco.com/c/en/us/td/docs/security/asa/asa94/config-guides/cli/general/asa-94-general-config/basic-arp-mac.pdf
  When you enable ARP inspection, the ASA compares the MAC address, IP address, and source interface in
  all ARP packets to static entries in the ARP table, and takes the following actions:
  • If the IP address, MAC address, and source interface match an ARP entry, the packet is passed through.
  • If there is a mismatch between the MAC address, the IP address, or the interface, then the ASA drops
  the packet.
  • If the ARP packet does not match any entries in the static ARP table, then you can set the ASA to either
  forward the packet out all interfaces (flood), or to drop the packet.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1677668/+subscriptions