Hi Frode,
I went through the captures I have and effectively, if we filter out the
packets where the srcport != 53 or dstport != 6081 , we end up only with
valid dns traffic.
I believe this explains what we are seeing, and the other findings you
did regards the missing flows, we can say with certainty that the DNS
issues we are seeing are only a symptom of instances losing North-South
traffic. I'm closing the bug as Invalid.
$ for PCAP in *.pcap; do echo -e "$PCAP\t$(tshark -r $PCAP | grep -i "unknown" | wc -l)\t$(tshark -r $PCAP 'udp.dstport ne 6081' | grep -i unknown | wc -l)\t$(tshark -r $PCAP 'udp.srcport ne 53' | grep -i unknown | wc -l)";done
dns-port-machine-0.pcap 90 0 0
dns-port-machine-10.pcap 104 0 0
dns-port-machine-11.pcap 26 0 0
dns-port-machine-12.pcap 0 0 0
dns-port-machine-13.pcap 0 0 0
dns-port-machine-2.pcap 6 0 0
dns-port-machine-3.pcap 72 0 0
dns-port-machine-4.pcap 0 0 0
dns-port-machine-5.pcap 59 0 0
dns-port-machine-6.pcap 68 0 0
dns-port-machine-7.pcap 46 0 0
dns-port-machine-8.pcap 104 0 0
dns-port-machine-9.pcap 32 0 0
** Changed in: ovn (Ubuntu)
Status: Triaged => Invalid
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ovn in Ubuntu.
https://bugs.launchpad.net/bugs/1959847
Title:
Unknown operation (12) 0x0240 Unknown (7680)[Malformed Packet]
Status in ovn package in Ubuntu:
Invalid
Bug description:
There have been DNS issues while running CI jobs for the openstack
charms, when I captured the dns traffic in one of the nova-compute
units[0][1], it can be seen how certain queries are malformed:
~ $ tshark -r dns-port-53.pcap | grep -i malformed | tail -n 2
15403 3602.706021 10.245.160.32 → 10.245.160.114 DNS 126 Unknown operation (12) 0x0240 Unknown (7680)[Malformed Packet]
15404 3602.706023 10.245.160.32 → 10.245.160.114 DNS 126 Unknown operation (12) 0x0240 Unknown (7680)[Malformed Packet]
~ $ tshark -r dns-port-53.pcap | grep -i malformed | wc -l
408
~ $ tshark -r dns-port-53.pcap | wc -l
15728
Another symptom found is within the VM's systemd-resolved, it logs
queries timing out https://pastebin.ubuntu.com/p/pJnd9sprpF/ [2]
Packages installed:
# dpkg -l ovn-common
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-==========================-============-=================================
ii ovn-common 20.03.2-0ubuntu0.20.04.2.0 amd64 OVN common components
# apt policy ovn-common
ovn-common:
Installed: 20.03.2-0ubuntu0.20.04.2.0
Candidate: 21.12.0-0ubuntu1.0~20.04.0
Version table:
21.12.0-0ubuntu1.0~20.04.0 500
500 http://ppa.launchpad.net/fnordahl/serverstack/ubuntu focal/main amd64 Packages
*** 20.03.2-0ubuntu0.20.04.2.0 500
500 http://ppa.launchpad.net/fnordahl/lp1857026/ubuntu focal/main amd64 Packages
100 /var/lib/dpkg/status
20.03.2-0ubuntu0.20.04.2 500
500 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
20.03.0-0ubuntu1 500
500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages
[0] tcpdump -i any -ln port 53
[1] https://private-fileshare.canonical.com/~freyes/dns-port-53.pcap
[2] note: this is a different run from the tcpdump capture, so they could be different root causes
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/1959847/+subscriptions