[Bug 894608] Re: Implementing a bridge slows 10G network
Serge Hallyn
894608 at bugs.launchpad.net
Tue Nov 29 04:39:08 UTC 2011
Thanks, Mike. I tried to reproduce this with two lucid machines. With both
not using bridges, I got:
------------------------------------------------------------
Client connecting to 10.55.55.5, TCP port 5001
TCP window size: 166 KByte (default)
------------------------------------------------------------
[ 5] local 10.55.55.5 port 34214 connected with 10.55.55.5 port 5001
[ 4] local 10.55.55.5 port 5001 connected with 10.55.55.5 port 34214
[ ID] Interval Transfer Bandwidth
[ 5] 0.0-10.0 sec 16.3 GBytes 14.0 Gbits/sec
[ 4] 0.0-10.0 sec 16.3 GBytes 14.0 Gbits/sec
root at mabolo:~# iperf -c 10.55.55.5 -d
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 10.55.55.5, TCP port 5001
TCP window size: 199 KByte (default)
------------------------------------------------------------
[ 5] local 10.55.55.5 port 34215 connected with 10.55.55.5 port 5001
[ 4] local 10.55.55.5 port 5001 connected with 10.55.55.5 port 34215
[ ID] Interval Transfer Bandwidth
[ 5] 0.0-10.0 sec 16.4 GBytes 14.1 Gbits/sec
[ 4] 0.0-10.0 sec 16.4 GBytes 14.1 Gbits/sec
With one having a bridge, I got:
------------------------------------------------------------
Client connecting to 10.55.55.5, TCP port 5001
TCP window size: 49.7 KByte (default)
------------------------------------------------------------
[ 3] local 10.55.55.5 port 34216 connected with 10.55.55.5 port 5001
[ 5] local 10.55.55.5 port 5001 connected with 10.55.55.5 port 34216
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 23.0 GBytes 19.7 Gbits/sec
[ 5] 0.0-10.0 sec 23.0 GBytes 19.7 Gbits/sec
root at mabolo:~# iperf -c 10.55.55.5 -d
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 10.55.55.5, TCP port 5001
TCP window size: 49.7 KByte (default)
------------------------------------------------------------
[ 4] local 10.55.55.5 port 34217 connected with 10.55.55.5 port 5001
[ 5] local 10.55.55.5 port 5001 connected with 10.55.55.5 port 34217
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 23.2 GBytes 20.0 Gbits/sec
[ 5] 0.0-10.0 sec 23.2 GBytes 20.0 Gbits/sec
However, looking back at your results, look at the TCP window sizes. They
are similar to mine in all but the degenerate case, where it is 1.25Mb.
Could you show the values of the following files:
/proc/sys/net/ipv4/tcp_rmem
/proc/sys/net/ipv4/tcp_wmem
on both hosts?
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to bridge-utils in Ubuntu.
https://bugs.launchpad.net/bugs/894608
Title:
Implementing a bridge slows 10G network
Status in “bridge-utils” package in Ubuntu:
Confirmed
Status in “linux” package in Ubuntu:
Confirmed
Bug description:
Installing bridge-utils and then setting up a bridge in
/etc/netwrok/interfaces decimates my netwrok speed. Uninstalling the
package and doing a reboot restores the speed.
This bug was originally posted here:
https://bugs.launchpad.net/qemu/+bug/861141 but I now realise it is
bridge-utils specific.
Hardware:
I have two identical machines (DELL poweredge R815), each with Broadcom NetXtreme II BCM57711 10-Gigabit PCIe connected via a 10G switch.
Each machine runs 4 socket 12 core AMD Opteron(tm) Processor 6174 and 128GB of ram
Software:
Both machines ar running ubuntu server 10.04 lts Kernels are:
Linux whitlam 2.6.32-28-server #55-Ubuntu SMP Mon Jan 10 23:57:16 UTC 2011 x86_64 GNU/Linux
Linux fraser 2.6.32-28-server #55-Ubuntu SMP Mon Jan 10 23:57:16 UTC 2011 x86_64 GNU/Linux
I have installed iperf on both machines.
The tests here involve running whitlam as the iperf server and fraser as the client.
On whitlam I run:
bioadmin at whitlam:~# iperf -sm
On fraser I run:
bioadmin at fraser:~# iperf -c whitlam -d
before installation I get:
bioadmin at whitlam:~# iperf -sm
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 5] local 10.168.48.14 port 5001 connected with 10.168.48.13 port 37960
------------------------------------------------------------
Client connecting to 10.168.48.13, TCP port 5001
TCP window size: 110 KByte (default)
------------------------------------------------------------
[ 6] local 10.168.48.14 port 49627 connected with 10.168.48.13 port 5001
Waiting for server threads to complete. Interrupt again to force quit.
[ 5] 0.0-10.0 sec 10.8 GBytes 9.23 Gbits/sec
[ 5] MSS size 1448 bytes (MTU 1500 bytes, ethernet)
[ 6] 0.0-10.0 sec 10.6 GBytes 9.11 Gbits/sec
[ 6] MSS size 1448 bytes (MTU 1500 bytes, ethernet)
Which is nice.
Next I run:
sudo aptitude -y install bridge-utils
and then I add a bridge to the interfaces file like thus:
FROM:
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto eth4
iface eth4 inet static
address xx.xx.xx.xx
netmask 255.255.255.0
network xx.xx.xx.0
broadcast xx.xx.xx.255
gateway xx.xx.xx.1
# dns-* options are implemented by the resolvconf package, if installed
dns-nameservers blah blah
TO:
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
#auto eth4
#iface eth4 inet manual
auto br0
iface br0 inet static
address xx.xx.xx.xx
netmask 255.255.255.0
network xx.xx.xx.0
broadcast xx.xx.xx.255
gateway xx.xx.xx.1
bridge_ports eth4
bridge_stp off
# dns-* options are implemented by the resolvconf package, if installed
dns-nameservers blah blah
And restart networking
sudo /etc/init.d/netwroking restart
Which results in:
bioadmin at whitlam:~# iperf -sm
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 4] local 10.168.48.14 port 5001 connected with 10.168.48.13 port 56405
------------------------------------------------------------
Client connecting to 10.168.48.13, TCP port 5001
TCP window size: 1.25 MByte (default)
------------------------------------------------------------
[ 6] local 10.168.48.14 port 57001 connected with 10.168.48.13 port 5001
Waiting for server threads to complete. Interrupt again to force quit.
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 10.5 GBytes 9.00 Gbits/sec
[ 4] MSS size 1448 bytes (MTU 1500 bytes, ethernet)
[ 6] 0.0-10.0 sec 2.51 GBytes 2.15 Gbits/sec
[ 6] MSS size 1448 bytes (MTU 1500 bytes, ethernet)
Note that the connection from the machine with bridge-utils installed
(fraser) to the other machine (whitlam) is not affected. Only the
connection from whitlam to fraser.
Both machines are fresh installs with nfs-common, iperf installed.
Fraser also has bridge-utils
Any help is very much appreciated!
---
AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access /dev/snd/: No such file or directory
AplayDevices: Error: [Errno 2] No such file or directory
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
DistroRelease: Ubuntu 10.04
InstallationMedia: Ubuntu-Server 10.04.2 LTS "Lucid Lynx" - Release amd64 (20110211.1)
MachineType: Dell Inc. PowerEdge R815
Package: linux (not installed)
PackageArchitecture: amd64
PciMultimedia:
ProcCmdLine: BOOT_IMAGE=/vmlinuz-2.6.32-35-server root=UUID=cfbcc41d-7968-457b-971a-6b7d5d56e6d2 ro quiet
ProcEnviron:
LANG=en_US.UTF-8
SHELL=/bin/bash
ProcVersionSignature: Ubuntu 2.6.32-35.78-server 2.6.32.46+drm33.20
Regression: No
Reproducible: Yes
Tags: lucid lucid networking needs-upstream-testing
Uname: Linux 2.6.32-35-server x86_64
UserGroups: adm admin cdrom dialout lpadmin plugdev sambashare
dmi.bios.date: 08/02/2010
dmi.bios.vendor: Dell Inc.
dmi.bios.version: 1.2.1
dmi.board.name: 04Y8PT
dmi.board.vendor: Dell Inc.
dmi.board.version: A00
dmi.chassis.type: 23
dmi.chassis.vendor: Dell Inc.
dmi.modalias: dmi:bvnDellInc.:bvr1.2.1:bd08/02/2010:svnDellInc.:pnPowerEdgeR815:pvr:rvnDellInc.:rn04Y8PT:rvrA00:cvnDellInc.:ct23:cvr:
dmi.product.name: PowerEdge R815
dmi.sys.vendor: Dell Inc.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/bridge-utils/+bug/894608/+subscriptions
More information about the foundations-bugs
mailing list