[Bug 2025071] [NEW] Order in which files are parsed and interfaces dependencies

[Danilo Egea Gondolfo]

Public bug reported:

When parsing a file, if netplan finds an interface A that depends on an
interface B (for example a bond that depends on the interfaces found in
the "interfaces" list), it will look for B in the table of parsed
definitions. If it can't find it there, B will be added to the
missing_id table and the parser will keep looking for it while the file
is parsed.

When it finishes parsing all the file and there are still missing IDs,
it will fail with "interface B is not defined".

Consider the example below:

file 10-vlan.yaml:
---
network:
  renderer: NetworkManager
  vlans:
    vlan100:
      id: 100
      link: "eth0"
---

file 99-zzz.yaml:
---
network:
  renderer: NetworkManager
  ethernets:
    eth0: {}
---

In this case, the VLAN "vlan100" depends on the definition of "eth0". "eth0" is defined in a separate YAML.
But because we sort the files in ASCIIbetical order, the file "10-vlan.yaml" will be parsed first and "99-zzz.yaml" will be parsed after that.

The netplan's parser controls the missing IDs on a per-file basis and
resets it before parsing the next file. In the end, if "eth0" is not
already in the parsed definitions table OR in the file that's being
parsed at the moment, it will not bother looking for it in the next
files and will fail with:

/etc/netplan/10-vlan.yaml:6:13: Error in network definition: vlan100: interface 'eth0' is not defined
      link: "eth0"
            ^

Renaming the file "99-zzz.yaml" to "01-zzz.yaml" is enough to get
netplan working for this example.

Although, when we create netplan YAML configuration through our
integration with Network Manager, we don't have control of the file
names, they will be called 90-NM-<connection UUID>.yaml. So the order in
which YAML will parsed is unpredictable. But it's not a big problem
right now because we create standalone YAML from Network Manager, the
things that would create a dependency issue are injected in the
passthrough section of the YAML so we don't bother tying them together
in the netplan state.

But if we want to start creating better netplan YAML from NM, we'll need
to address this issue.

One solution for that is to make the missing_ids table survive across
different files and keep looking for the missing interfaces until we
don't have any more files to look at. Only then we can say the interface
B is nowhere to be found in the netplan configuration.


Possible solution

The missing IDs handling could be done automatically if the caller is
using the "netplan_parser_load_yaml_hierarchy()" API. So the logic
implement in "process_document()" could be moved there. This would make
the parser use the missing IDs data for every file it's parsing.

One problem with that approach is that we destroy the YAML document
after parsing it in "_netplan_parser_load_single_file()", and the
missing ID object has a reference to the YAML node where the ID was
referenced.

If the consumer is calling "netplan_parser_load_yaml()" for files
individually, they will be responsible for checking if the missing_ids
table has any item when they finish parsing their files.


Reasons for fixing it

Netplan users will not have to bother anymore about where they should
defined their interfaces.

It will enable us to do a better job when loading NM keyfiles into
Netplan's state. Due to this issue, we can't link interface inside
netplan, because if we do that, and because of the order netplan will
parse NM-generated files with their random names, eventually it will
fail to find dependencies.

** Affects: netplan.io (Ubuntu)
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to netplan.io in Ubuntu.
Matching subscriptions: foundations-bugs
https://bugs.launchpad.net/bugs/2025071

Title:
  Order in which files are parsed and interfaces dependencies

Status in netplan.io package in Ubuntu:
  New

Bug description:
  When parsing a file, if netplan finds an interface A that depends on
  an interface B (for example a bond that depends on the interfaces
  found in the "interfaces" list), it will look for B in the table of
  parsed definitions. If it can't find it there, B will be added to the
  missing_id table and the parser will keep looking for it while the
  file is parsed.

  When it finishes parsing all the file and there are still missing IDs,
  it will fail with "interface B is not defined".

  Consider the example below:

  file 10-vlan.yaml:
  ---
  network:
    renderer: NetworkManager
    vlans:
      vlan100:
        id: 100
        link: "eth0"
  ---

  file 99-zzz.yaml:
  ---
  network:
    renderer: NetworkManager
    ethernets:
      eth0: {}
  ---

  In this case, the VLAN "vlan100" depends on the definition of "eth0". "eth0" is defined in a separate YAML.
  But because we sort the files in ASCIIbetical order, the file "10-vlan.yaml" will be parsed first and "99-zzz.yaml" will be parsed after that.

  The netplan's parser controls the missing IDs on a per-file basis and
  resets it before parsing the next file. In the end, if "eth0" is not
  already in the parsed definitions table OR in the file that's being
  parsed at the moment, it will not bother looking for it in the next
  files and will fail with:

  /etc/netplan/10-vlan.yaml:6:13: Error in network definition: vlan100: interface 'eth0' is not defined
        link: "eth0"
              ^

  Renaming the file "99-zzz.yaml" to "01-zzz.yaml" is enough to get
  netplan working for this example.

  Although, when we create netplan YAML configuration through our
  integration with Network Manager, we don't have control of the file
  names, they will be called 90-NM-<connection UUID>.yaml. So the order
  in which YAML will parsed is unpredictable. But it's not a big problem
  right now because we create standalone YAML from Network Manager, the
  things that would create a dependency issue are injected in the
  passthrough section of the YAML so we don't bother tying them together
  in the netplan state.

  But if we want to start creating better netplan YAML from NM, we'll
  need to address this issue.

  One solution for that is to make the missing_ids table survive across
  different files and keep looking for the missing interfaces until we
  don't have any more files to look at. Only then we can say the
  interface B is nowhere to be found in the netplan configuration.

  
  Possible solution

  The missing IDs handling could be done automatically if the caller is
  using the "netplan_parser_load_yaml_hierarchy()" API. So the logic
  implement in "process_document()" could be moved there. This would
  make the parser use the missing IDs data for every file it's parsing.

  One problem with that approach is that we destroy the YAML document
  after parsing it in "_netplan_parser_load_single_file()", and the
  missing ID object has a reference to the YAML node where the ID was
  referenced.

  If the consumer is calling "netplan_parser_load_yaml()" for files
  individually, they will be responsible for checking if the missing_ids
  table has any item when they finish parsing their files.

  
  Reasons for fixing it

  Netplan users will not have to bother anymore about where they should
  defined their interfaces.

  It will enable us to do a better job when loading NM keyfiles into
  Netplan's state. Due to this issue, we can't link interface inside
  netplan, because if we do that, and because of the order netplan will
  parse NM-generated files with their random names, eventually it will
  fail to find dependencies.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/netplan.io/+bug/2025071/+subscriptions