[Bug 2083695] Re: netplan does not fully generate routes for PBR

[Alberto Contreras]

*** This bug is a duplicate of bug 2078009 ***
    https://bugs.launchpad.net/bugs/2078009

Sweet, many thanks Danilo.

As this problem is another instance of
https://bugs.launchpad.net/netplan/+bug/2078009, I am going to mark it
as a duplicate of it.

** This bug has been marked a duplicate of bug 2078009
   "netplan apply" not applying after cloud-init hotplug

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to netplan.io in Ubuntu.
Matching subscriptions: foundations-bugs
https://bugs.launchpad.net/bugs/2083695

Title:
  netplan does not fully generate routes for PBR

Status in netplan.io package in Ubuntu:
  New
Status in netplan.io source package in Oracular:
  New

Bug description:
  Cloud-init's hotplug / policy-base routing feature for multi-nic
  instances on EC2 is broken, due to the following issue, I believe:

  Steps to reproduce:

  1. Launch an Oracular vm on EC2.
  2. Add a NIC with an IPv4 floating ip associated to it.

  The issue:

  The cloud-init's hotplug functionality generates and applies the
  following netplan configuration in /etc/netplan/50-cloud-init.yaml:

  ```
  network:
    version: 2
    ethernets:
   ens5:
     match:
       macaddress: "06:1c:fb:9b:d3:e5"
     dhcp4: true
     dhcp4-overrides:
       route-metric: 100
     dhcp6: true
     dhcp6-overrides:
       route-metric: 100
     set-name: "ens5"
   ens6:
     match:
       macaddress: "06:cb:49:de:56:51"
     dhcp4: true
     dhcp4-overrides:
       use-routes: true
       route-metric: 200
     dhcp6: false
     set-name: "ens6"
     routes:
     - table: 101
       to: "0.0.0.0/0"
       via: "192.168.0.1"
     - scope: "link"
       table: 101
       to: "192.168.0.0/20"
     routing-policy:
     - table: 101
       from: "192.168.12.94"
  ```

  But, the routes added are:

  ```
  $ ip route
  default via 192.168.0.1 dev ens5 proto dhcp src 192.168.0.64 metric 1002 mtu 9001
  default via 192.168.0.1 dev ens6 proto dhcp src 192.168.12.94 metric 1003 mtu 9001
  192.168.0.0/20 dev ens5 proto dhcp scope link src 192.168.0.64 metric 1002 mtu 9001
  192.168.0.0/20 dev ens6 proto dhcp scope link src 192.168.12.94 metric 1003 mtu 9001
  192.168.0.1 dev ens5 proto dhcp scope link src 192.168.0.64 metric 100
  192.168.0.2 dev ens5 proto dhcp scope link src 192.168.0.64 metric 100

  $ ip a
  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
   link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
   inet 127.0.0.1/8 scope host lo
      valid_lft forever preferred_lft forever
   inet6 ::1/128 scope host noprefixroute
      valid_lft forever preferred_lft forever
  2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
   link/ether 06:1c:fb:9b:d3:e5 brd ff:ff:ff:ff:ff:ff
   altname enp0s5
   inet 192.168.0.64/20 brd 192.168.15.255 scope global dynamic ens5
      valid_lft 3513sec preferred_lft 3063sec
   inet6 2a05:d011:311:a00:c57a:7c3e:723a:8efc/128 scope global dynamic noprefixroute
      valid_lft 415sec preferred_lft 105sec
   inet6 fe80::41c:fbff:fe9b:d3e5/64 scope link proto kernel_ll
      valid_lft forever preferred_lft forever
  3: ens6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
   link/ether 06:cb:49:de:56:51 brd ff:ff:ff:ff:ff:ff
   altname enp0s6
   inet 192.168.12.94/20 brd 192.168.15.255 scope global dynamic noprefixroute ens6
      valid_lft 3514sec preferred_lft 3064sec
   inet6 fe80::4cb:49ff:fede:5651/64 scope link proto kernel_ll
      valid_lft forever preferred_lft forever
  ```

  We can see that the route table lacks

  192.168.0.1 dev ens6 proto dhcp scope link src 192.168.12.94 metric 200
  192.168.0.2 dev ens6 proto dhcp scope link src 192.168.12.94 metric 200

  and that the metrics are not aligned.

  With this network configuration, the instance is not reachable from
  outside using the floating ip associated with ens6.

  If we do the same in Ubuntu Noble, we get the following routing table
  from an equivalent netplan config and the instance is reachable via
  ens6:

  ```
  default via 192.168.0.1 dev ens5 proto dhcp src 192.168.9.45 metric 100
  default via 192.168.0.1 dev ens6 proto dhcp src 192.168.12.179 metric 200
  192.168.0.0/20 dev ens5 proto kernel scope link src 192.168.9.45 metric 100
  192.168.0.0/20 dev ens6 proto kernel scope link src 192.168.12.179 metric 200
  192.168.0.1 dev ens5 proto dhcp scope link src 192.168.9.45 metric 100
  192.168.0.1 dev ens6 proto dhcp scope link src 192.168.12.179 metric 200
  192.168.0.2 dev ens5 proto dhcp scope link src 192.168.9.45 metric 100
  192.168.0.2 dev ens6 proto dhcp scope link src 192.168.12.179 metric 200
  ```

  I have tried downgrading cloud-init to 24.4~2g2e4c39b7-0ubuntu1 in
  oracular, and I get the same behavior consistently. This discards the
  issue being introduced in cloud-init 24.3 or 23.3.1.

  This is covered by cloud-init CI and the last passing execution was at
  Sep 1, 2024.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/netplan.io/+bug/2083695/+subscriptions