[Bug 2101860] Re: Different networking behavior with 2nd IP on 2nd NIC

[James Falcon]

Also adding netplan and systemd here as the rendered networkd config
doesn't change, and the netplan apply call is racy.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/2101860

Title:
  Different networking behavior with 2nd IP on 2nd NIC

Status in cloud-init package in Ubuntu:
  New
Status in netplan.io package in Ubuntu:
  New
Status in systemd package in Ubuntu:
  New

Bug description:
  This integration test is currently failing on plucky:
  https://github.com/canonical/cloud-init/blob/a136a979dd5a5084f71f5def5abf6d80d7cc263a/tests/integration_tests/test_networking.py#L313

  This test will launch an ec2 instance, attach a 2nd NIC, and then
  assign two (total) IP addresses to that NIC. We then check that all 3
  IP addresses can `nc` to the SSH port with the public IPs. This works
  consistently on Oracular and below, but it does not work on Plucky.
  The secondary IP on the secondary NIC can no longer connect. If I
  compare the generated netplan configurations between the two instances
  along with the generated networkd configurations, they are identical
  (minus different IP addresses). I can even downgrade netplan on plucky
  to the oracular version, but it still doesn't work on plucky. I have
  also manually verified that nc does work on the internal secondary IP
  address.

  Edit: A lot of the comments here is me dumping my debug journey, but
  the main problem seems to be that a policy based routing rule is often
  missing from the rule db. Calling `netplan apply` (with no changes to
  the configuration) will sometimes fix it, but most of the time calling
  `netplan apply` leaves it without the necessary rule.

  cloud-init's rendered Netplan configuration on plucky:
  # cat /etc/netplan/50-cloud-init.yaml
  network:
    version: 2
    ethernets:
      ens5:
        match:
          macaddress: "06:f1:2f:0b:5f:7d"
        dhcp4: true
        dhcp4-overrides:
          route-metric: 100
        dhcp6: true
        dhcp6-overrides:
          route-metric: 100
        set-name: "ens5"
      ens6:
        match:
          macaddress: "06:f0:7b:f6:b6:5f"
        addresses:
        - "192.168.15.230/20"
        dhcp4: true
        dhcp4-overrides:
          use-routes: true
          route-metric: 200
        dhcp6: true
        dhcp6-overrides:
          use-routes: true
          route-metric: 200
        set-name: "ens6"
        routes:
        - table: 101
          to: "0.0.0.0/0"
          via: "192.168.0.1"
        - scope: "link"
          table: 101
          to: "192.168.0.0/20"
        - scope: "link"
          table: 101
          to: "2600:1f16:67f:f200:0:0:0:0/64"
        routing-policy:
        - table: 101
          from: "192.168.12.6"
        - table: 101
          from: "192.168.15.230"
        - table: 101
          from: "2600:1f16:67f:f200:25ac:a1c8:2d0d:ae87"

  cloud-init's rendered Netplan configuration on oracular:
  root at ip-192-168-1-227:/home/ubuntu# cat /etc/netplan/50-cloud-init.yaml
  network:
    version: 2
    ethernets:
      ens5:
        match:
          macaddress: "06:34:a4:59:1c:8f"
        dhcp4: true
        dhcp4-overrides:
          route-metric: 100
        dhcp6: true
        dhcp6-overrides:
          route-metric: 100
        set-name: "ens5"
      ens6:
        match:
          macaddress: "06:48:2a:91:51:79"
        addresses:
        - "192.168.2.219/20"
        dhcp4: true
        dhcp4-overrides:
          use-routes: true
          route-metric: 200
        dhcp6: true
        dhcp6-overrides:
          use-routes: true
          route-metric: 200
        set-name: "ens6"
        routes:
        - table: 101
          to: "0.0.0.0/0"
          via: "192.168.0.1"
        - scope: "link"
          table: 101
          to: "192.168.0.0/20"
        - scope: "link"
          table: 101
          to: "2600:1f16:67f:f200:0:0:0:0/64"
        routing-policy:
        - table: 101
          from: "192.168.12.51"
        - table: 101
          from: "192.168.2.219"
        - table: 101
          from: "2600:1f16:67f:f200:9225:d84d:9f5e:462d"

  plucky networkd configurations:
  root at ip-192-168-11-242:/home/ubuntu# cat /run/systemd/network/10-netplan-ens5.link
  [Match]
  PermanentMACAddress=06:f1:2f:0b:5f:7d

  [Link]
  Name=ens5
  WakeOnLan=off
  root at ip-192-168-11-242:/home/ubuntu# cat /run/systemd/network/10-netplan-ens5.network
  [Match]
  PermanentMACAddress=06:f1:2f:0b:5f:7d
  Name=ens5

  [Network]
  DHCP=yes
  LinkLocalAddressing=ipv6

  [DHCP]
  RouteMetric=100
  UseMTU=true
  root at ip-192-168-11-242:/home/ubuntu# cat /run/systemd/network/10-netplan-ens6.link
  [Match]
  PermanentMACAddress=06:f0:7b:f6:b6:5f

  [Link]
  Name=ens6
  WakeOnLan=off
  root at ip-192-168-11-242:/home/ubuntu# cat /run/systemd/network/10-netplan-ens6.network
  [Match]
  PermanentMACAddress=06:f0:7b:f6:b6:5f
  Name=ens6

  [Network]
  DHCP=yes
  LinkLocalAddressing=ipv6
  Address=192.168.15.230/20

  [Route]
  Destination=0.0.0.0/0
  Gateway=192.168.0.1
  Table=101

  [Route]
  Destination=192.168.0.0/20
  Scope=link
  Table=101

  [Route]
  Destination=2600:1f16:67f:f200:0:0:0:0/64
  Scope=link
  Table=101

  [RoutingPolicyRule]
  From=192.168.12.6
  Table=101

  [RoutingPolicyRule]
  From=192.168.15.230
  Table=101

  [RoutingPolicyRule]
  From=2600:1f16:67f:f200:25ac:a1c8:2d0d:ae87
  Table=101

  [DHCP]
  RouteMetric=200
  UseMTU=true

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/2101860/+subscriptions