[Bug 1967834] [NEW] openstack port-mirroring mirrors traffic even when the port is not the destined port for the ingress traffic

Tue Apr 5 07:43:00 UTC 2022

Public bug reported:

With Openstack Pike release, Observed that traffic is being mirrored
from one of the access ports even though the port is not the actual
destination for the traffic.

Have a configuration like this.

Two VMs VM1 and VM2 having trunk port and sub-ports that carry VLAN
tagged traffic and another VM VM3 having an access port. Mirror is
created with VM3 port as source interface for this mirror. When ping is
initiated from VM1 destined to VM2 (Sub-ports on subnet 40.40.2.0/24),
we see that the traffic also passes through the tap interface of VM3
access port attached to the integration bridge (br-int) and since mirror
is created, the same is traffic is mirrored to the remote tunnel end-
point.

VM1 - 40.40.2.189
VM2 - 40.40.2.204
VM3 - 40.40.2.104

Mirror created with VM3's port as source interface

ovs-vsctl add-port br-int vxlan0 -- set Interface vxlan0 type=vxlan options:local_ip=10.115.57
.203 options:remote_ip=30.1.1.16 options:key=0 option:dst_port=4789 -- --id=@p get port vxlan0 -- --id=@m create mirror name=mirror0 output-port=@p -- set bridge br-int mir
rors=@m

ovs-vsctl -- --id=@p1 get Port vhu35512a63-ea -- add Mirror mirror0 select-dst-port @p1
ovs-vsctl -- --id=@p1 get Port vhu35512a63-ea -- add Mirror mirror0 select-src-port @p1

Traffic is ping from 40.40.2.189 --> 40.40.2.204 (VM1 - VM2).

Traffic also passes through the VM3 connected tap interface on
integration bridge (vhu35512a63-ea) and since a mirror is created here,
the same traffic is being mirrored to remote destination.

Possible root cause found is:

In this scenario, We observed that on the integration bridge there are three ports carrying the same VLAN tag.

Port "spi-e0212fcc-b7" <== patch port corresponding to bridge “tbr-683e3538-0” (IP: 40.40.2.204)
      tag: 3
     Interface "spi-e0212fcc-b7"
       type: patch
       options: {peer="spt-e0212fcc-b7"}

     Port "vhu35512a63-ea". <== VM access port corresponding to VM having IP 40.40.2.104
       tag: 3
      Interface "vhu35512a63-ea"
      type: dpdkvhostuserclient
      options: {vhost-server-path="/var/run/openvswitch/vhu35512a63-ea"}

    Port "spi-2013cf5d-dd".  <== patch port corresponding to bridge “tbr-fb95e589-3” (IP: 40.40.2.189)
        tag: 3
        Interface "spi-2013cf5d-dd"
        type: patch
        options: {peer="spt-2013cf5d-dd"}

Now say if we try to send any traffic between the above two patch ports. We can see that the traffic will also be punted to the VM access port. (Ping from 40.40.2.189 > 40.40.2.204)

Tcpdump on vhu35512a63-ea

IP 40.40.2.189 > 40.40.2.204: ICMP echo request, id 12125, seq 171, length 64
IP 40.40.2.189 > 40.40.2.204: ICMP echo request, id 12125, seq 172, length 64

And the same traffic is being mirrored.

PS: Bridge tbr-fb95e589-3 and tbr-683e3538-0 created when the trunk
ports are created.

All the ports have default security group attached.

ALLOW IPv6 1-65535/tcp to ::/0
ALLOW IPv4 ip_proto=253 from 0.0.0.0/0
ALLOW IPv6 1-65535/tcp from ::/0
ALLOW IPv6 ipv6-icmp from ::/0
ALLOW IPv4 ip_proto=47 from 0.0.0.0/0
ALLOW IPv6 to ::/0
ALLOW IPv4 ip_proto=253 to 0.0.0.0/0
ALLOW IPv4 1-65535/tcp from 0.0.0.0/0
ALLOW IPv4 1-65535/udp to 0.0.0.0/0
ALLOW IPv6 from default
ALLOW IPv4 ip_proto=47 to 0.0.0.0/0
ALLOW IPv4 1-65535/tcp to 0.0.0.0/0
ALLOW IPv4 to 0.0.0.0/0
ALLOW IPv4 icmp from 0.0.0.0/0
ALLOW IPv4 from default
ALLOW IPv4 1-65535/udp from 0.0.0.0/0
ALLOW IPv4 icmp to 0.0.0.0/0

Is this expected behaviour. Why different ports have same VLAN tag on
the integration bridge (br-int).

** Affects: cloud-archive
     Importance: Undecided
         Status: New

** Tags: openvswitch

** Summary changed:

- openstack port-mirroring mirrors traffic even when the port is not part of the mirror
+ openstack port-mirroring mirrors traffic even when the port is not the destined port for the ingress traffic

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1967834

Title:
  openstack port-mirroring mirrors traffic even when the port is not the
  destined port for the ingress traffic

Status in Ubuntu Cloud Archive:
  New

Bug description:
  With Openstack Pike release, Observed that traffic is being mirrored
  from one of the access ports even though the port is not the actual
  destination for the traffic.

  Have a configuration like this.

  Two VMs VM1 and VM2 having trunk port and sub-ports that carry VLAN
  tagged traffic and another VM VM3 having an access port. Mirror is
  created with VM3 port as source interface for this mirror. When ping
  is initiated from VM1 destined to VM2 (Sub-ports on subnet
  40.40.2.0/24), we see that the traffic also passes through the tap
  interface of VM3 access port attached to the integration bridge (br-
  int) and since mirror is created, the same is traffic is mirrored to
  the remote tunnel end-point.

  VM1 - 40.40.2.189
  VM2 - 40.40.2.204
  VM3 - 40.40.2.104

  Mirror created with VM3's port as source interface

  ovs-vsctl add-port br-int vxlan0 -- set Interface vxlan0 type=vxlan options:local_ip=10.115.57
  .203 options:remote_ip=30.1.1.16 options:key=0 option:dst_port=4789 -- --id=@p get port vxlan0 -- --id=@m create mirror name=mirror0 output-port=@p -- set bridge br-int mir
  rors=@m

  ovs-vsctl -- --id=@p1 get Port vhu35512a63-ea -- add Mirror mirror0 select-dst-port @p1
  ovs-vsctl -- --id=@p1 get Port vhu35512a63-ea -- add Mirror mirror0 select-src-port @p1

  Traffic is ping from 40.40.2.189 --> 40.40.2.204 (VM1 - VM2). 

  Traffic also passes through the VM3 connected tap interface on
  integration bridge (vhu35512a63-ea) and since a mirror is created
  here, the same traffic is being mirrored to remote destination.

  Possible root cause found is:

  In this scenario, We observed that on the integration bridge there are three ports carrying the same VLAN tag.

  Port "spi-e0212fcc-b7" <== patch port corresponding to bridge “tbr-683e3538-0” (IP: 40.40.2.204)
        tag: 3
       Interface "spi-e0212fcc-b7"
         type: patch
         options: {peer="spt-e0212fcc-b7"}

       Port "vhu35512a63-ea". <== VM access port corresponding to VM having IP 40.40.2.104
         tag: 3
        Interface "vhu35512a63-ea"
        type: dpdkvhostuserclient
        options: {vhost-server-path="/var/run/openvswitch/vhu35512a63-ea"}

      Port "spi-2013cf5d-dd".  <== patch port corresponding to bridge “tbr-fb95e589-3” (IP: 40.40.2.189)
          tag: 3
          Interface "spi-2013cf5d-dd"
          type: patch
          options: {peer="spt-2013cf5d-dd"}

  Now say if we try to send any traffic between the above two patch ports. We can see that the traffic will also be punted to the VM access port. (Ping from 40.40.2.189 > 40.40.2.204)

  Tcpdump on vhu35512a63-ea

  IP 40.40.2.189 > 40.40.2.204: ICMP echo request, id 12125, seq 171, length 64
  IP 40.40.2.189 > 40.40.2.204: ICMP echo request, id 12125, seq 172, length 64

  And the same traffic is being mirrored.

  PS: Bridge tbr-fb95e589-3 and tbr-683e3538-0 created when the trunk
  ports are created.

  All the ports have default security group attached.

  ALLOW IPv6 1-65535/tcp to ::/0
  ALLOW IPv4 ip_proto=253 from 0.0.0.0/0
  ALLOW IPv6 1-65535/tcp from ::/0
  ALLOW IPv6 ipv6-icmp from ::/0
  ALLOW IPv4 ip_proto=47 from 0.0.0.0/0
  ALLOW IPv6 to ::/0
  ALLOW IPv4 ip_proto=253 to 0.0.0.0/0
  ALLOW IPv4 1-65535/tcp from 0.0.0.0/0
  ALLOW IPv4 1-65535/udp to 0.0.0.0/0
  ALLOW IPv6 from default
  ALLOW IPv4 ip_proto=47 to 0.0.0.0/0
  ALLOW IPv4 1-65535/tcp to 0.0.0.0/0
  ALLOW IPv4 to 0.0.0.0/0
  ALLOW IPv4 icmp from 0.0.0.0/0
  ALLOW IPv4 from default
  ALLOW IPv4 1-65535/udp from 0.0.0.0/0
  ALLOW IPv4 icmp to 0.0.0.0/0

  Is this expected behaviour. Why different ports have same VLAN tag on
  the integration bridge (br-int).

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1967834/+subscriptions