[Bug 1939881] Re: OVN DNS interception leads to OOM killing the local ovn-controller process

[Frode Nordahl]

Workaround would be:
1) Disable the Neutron ML2 DNS feature so that it does not populate the OVN NB DB with DNS records.
2) Empty the OVN NB DNS table for any records pertaining the affected Logical Switch (LS).

Lack of DNS records for a LS will make ovn-northd disable the feature
completely by not installing the flows at all ref: [0][1][2]

0: https://github.com/ovn-org/ovn/blob/db15cf29a1f9857b55389f424c5d747406550cb7/northd/northd.c#L5736-L5750
1: https://github.com/ovn-org/ovn/blob/db15cf29a1f9857b55389f424c5d747406550cb7/northd/northd.c#L6771-L6779
2: https://github.com/ovn-org/ovn/blob/db15cf29a1f9857b55389f424c5d747406550cb7/northd/northd.c#L8225-L8251

** Also affects: ovn (Ubuntu)
   Importance: Undecided
       Status: New

** Changed in: networking-ovn
       Status: New => Invalid

** Changed in: charm-ovn-chassis
       Status: New => Invalid

** Changed in: charm-ovn-central
       Status: New => Invalid

** Changed in: ovn (Ubuntu)
       Status: New => Triaged

** Changed in: ovn (Ubuntu)
   Importance: Undecided => High

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ovn in Ubuntu.
https://bugs.launchpad.net/bugs/1939881

Title:
  OVN DNS interception leads to OOM killing the local ovn-controller
  process

Status in charm-ovn-central:
  Invalid
Status in charm-ovn-chassis:
  Invalid
Status in networking-ovn:
  Invalid
Status in ovn package in Ubuntu:
  Triaged

Bug description:
  
  Whilst undertaking network performance testing on an Instance with virtio interfaces, we had a tester configured to generate packets that looked like DNS requests (i.e. they were UDP and to port 53). Whilst this  was obviously not the best idea, it did highlight an issue with the DNS interception feature.

  The Instance we were testing was configured with IP forwarding between
  two interfaces, so it's not clear in which direction the interception
  took place on. One might assume it was on egress from the Instance
  rather than on ingress.

  These packets were intercepted by OVS/OVN as detailed in "Ingress
  Table 18 DNS Lookup" and "Ingress Table 19 DNS Responses" of the ovn-
  northd manpage
  (http://manpages.ubuntu.com/manpages/hirsute/en/man8/ovn-
  northd.8.html). Note other versions of the manpage may number the
  sections differently.

  However the packets were not valid DNS queries (they were padded with
  an 'X' character) and so caused the local ovn-controller process to
  buffer them until the point that it consumed all free RAM on the host
  and was killed. Presumably the buffering was due to the time take to
  (fail to) process the DNS request and the relatively high PPS due to
  this being a network performance test.

  Whilst this was an entirely synthetic test, and we could disable the
  OVN DNS interception feature, the behaviour may not be reasonable as
  there could be valid use cases that trigger this behaviour (e.g. an
  Instance running a heavily used recursive resolver).

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-ovn-central/+bug/1939881/+subscriptions