[Bug 2003027] Re: fq_codel qdisc compiled but not enabled on 22.10

Fri Aug 30 18:39:40 UTC 2024

BQL ended up in the virtio-net driver in a recent kernel release and showed
significant benefits with fq_codel on the test suite they used.

At one level I don´t mind this fix not making the default next release, in
that perhaps we could get some informed testing of before/after
particularly with things like wireguard, or udp based applications. It
would be really great to see it also come out for older releases.

I am not in a position to write a regression test though, if you were
asking me.

Thanks for all your efforts in getting this out there.

On Wed, Aug 28, 2024 at 7:56 AM Robie Basak <2003027 at bugs.launchpad.net>
wrote:

> Accepting for 24.04. It is unfortunate to have to change behaviour in
> this way, but this seems preferable than to switch to the reported-worse
> qdisc just for 24.04, and also preferable to do sooner rather than later
> in 24.04's lifetime, given that the .1 release is due this week.
>
> However, if this is important enough to make a change in a stable
> release, and given that it regressed accidentally in Ubuntu, then it's
> important enough to have a regression test for. Consider this a slap on
> the wrist for not having included a test already. Given the above it's
> not appropriate to block pending a test being added, but it should be
> considered part of the necessary work to fix this properly regardless.
> Please get that done to prevent this from regressing again in the
> future. I filed bug 2078101 to track this.
>
> I appreciate that it's tricky to write a test that might need a VM to
> pick up kernel defaults to fully validate this, but we could at least
> have an autopkgtest that ensures that /etc/sysctl.d/ contains the
> expected net.core.default_qdisc setting and not any duplicate settings.
> You could also include checks for anything else that is Ubuntu-specific
> in there.
>
> ** Changed in: procps (Ubuntu Noble)
>        Status: In Progress => Fix Committed
>
> ** Tags added: verification-needed verification-needed-noble
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/2003027
>
> Title:
>   fq_codel qdisc compiled but not enabled on 22.10
>
> Status in Ubuntu:
>   Fix Released
> Status in procps package in Ubuntu:
>   Fix Released
> Status in The Noble Numbat:
>   Confirmed
> Status in procps source package in Noble:
>   Fix Committed
> Status in The Oracular Oriole:
>   Fix Released
> Status in procps source package in Oracular:
>   Fix Released
>
> Bug description:
>   [ Impact ]
>   Starting with Ubuntu 22.10, the default qdisc regressed to pfifo_fast.
> This is due to a change we merged from Debian, removing systemd's
> /usr/lib/50-default.conf file. pfifo_fast does not prevent bufferbloat
> issues, severely degrading networking performance on Ubuntu systems.
>
>   [ Test Plan ]
>   On an Ubuntu VM, check that net.core.default_qdisc is set to 'fq_codel'.
> `tc qdisc show` should also confirm fq_codel is being used.
>
>   halves at noble-vm:~$ sysctl net.core.default_qdisc
>   net.core.default_qdisc = fq_codel
>
>   halves at noble-vm:~$ tc qdisc show
>   qdisc noqueue 0: dev lo root refcnt 2
>   qdisc mq 0: dev enp5s0 root
>   qdisc fq_codel 0: dev enp5s0 parent :1 limit 10240p flows 1024 quantum
> 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
>
>   [ Where Problems Could Occur ]
>   Users relying on the default pfifo_fast qdisc could see different
> networking behavior. Although fq_codel should be an across-the-board
> improvement over pfifo_fast, we could monitor network latency stats and
> throughput to make sure no weird regressions arise from the qdisc scheduler.
>
>   [ Other Info ]
>   Instead of adding a sysctl.conf file, Debian has decided to change their
> default kernel configs to use fq_codel. This has the disadvantage of being
> "hidden" from end users, instead of being a queryable setting in e.g.
> /etc/sysctl.d/, and all kernel documentation still refers to pfifo_fast as
> being the default.
>
>   Systemd's /usr/lib/50-default.conf file includes a lot of other
>   settings that are either overruled elsewhere (e.g. from procps) or are
>   entirely missing from our stable releases. Introducing a diff from
>   Debian to ship this file only for the net.core.default_qdisc changes
>   seems excessive, so having a dedicated place under procps where other
>   similar toggles are already present seems like the more appropriate
>   solution.
>
>   --
>   [ Original Description ]
>   fq_codel has been the default in ubuntu since #1436945
>
>   It is also now the default in debian. Somewhere between 22.04 and
>   22.10 though, it stopped being enabled by systemd? (it used to be
>   enabled there via a sysctl), and thus 22.10 (I tried both ubuntu
>   server and ubuntu studio) doesn't have it on. I rather *immediately*
>   noticed this huge regression (I run my ethernet at 100Mbit, and thus
>   immediately saw 400+ms latency) - and I hope it can be fixed in the
>   next release?
>
>   it can also be made the default via compiling it into the kernel as a
>   kernel default.
>
>   Aggh! death to fifos!
>
>   root at localhost:~# sysctl -a | grep qdisc
>   net.core.default_qdisc = pfifo_fast
>
>   disc pfifo_fast 0: dev eth0 parent :1 bands 3 priomap 1 2 2 2 1 2 0 0 1
> 1 1 1 1 1 1 1
>    Sent 5250 bytes 53 pkt (dropped 0, overlimits 0 requeues 0)
>    backlog 0b 0p requeues 0
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+bug/2003027/+subscriptions
>
>

-- 
Artists/Musician Campout Aug 9-11
https://www.eventbrite.com/e/healing-arts-event-tickets-928910826287
Dave Täht CSO, LibreQos

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to procps in Ubuntu.
https://bugs.launchpad.net/bugs/2003027

Title:
  fq_codel qdisc compiled but not enabled on 22.10

Status in Ubuntu:
  Fix Released
Status in procps package in Ubuntu:
  Fix Released
Status in The Noble Numbat:
  Confirmed
Status in procps source package in Noble:
  Fix Committed
Status in The Oracular Oriole:
  Fix Released
Status in procps source package in Oracular:
  Fix Released

Bug description:
  [ Impact ]
  Starting with Ubuntu 22.10, the default qdisc regressed to pfifo_fast. This is due to a change we merged from Debian, removing systemd's /usr/lib/50-default.conf file. pfifo_fast does not prevent bufferbloat issues, severely degrading networking performance on Ubuntu systems.

  [ Test Plan ]
  On an Ubuntu VM, check that net.core.default_qdisc is set to 'fq_codel'. `tc qdisc show` should also confirm fq_codel is being used.

  halves at noble-vm:~$ sysctl net.core.default_qdisc
  net.core.default_qdisc = fq_codel

  halves at noble-vm:~$ tc qdisc show
  qdisc noqueue 0: dev lo root refcnt 2
  qdisc mq 0: dev enp5s0 root
  qdisc fq_codel 0: dev enp5s0 parent :1 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64

  [ Where Problems Could Occur ]
  Users relying on the default pfifo_fast qdisc could see different networking behavior. Although fq_codel should be an across-the-board improvement over pfifo_fast, we could monitor network latency stats and throughput to make sure no weird regressions arise from the qdisc scheduler.

  [ Other Info ]
  Instead of adding a sysctl.conf file, Debian has decided to change their default kernel configs to use fq_codel. This has the disadvantage of being "hidden" from end users, instead of being a queryable setting in e.g. /etc/sysctl.d/, and all kernel documentation still refers to pfifo_fast as being the default.

  Systemd's /usr/lib/50-default.conf file includes a lot of other
  settings that are either overruled elsewhere (e.g. from procps) or are
  entirely missing from our stable releases. Introducing a diff from
  Debian to ship this file only for the net.core.default_qdisc changes
  seems excessive, so having a dedicated place under procps where other
  similar toggles are already present seems like the more appropriate
  solution.

  --
  [ Original Description ]
  fq_codel has been the default in ubuntu since #1436945

  It is also now the default in debian. Somewhere between 22.04 and
  22.10 though, it stopped being enabled by systemd? (it used to be
  enabled there via a sysctl), and thus 22.10 (I tried both ubuntu
  server and ubuntu studio) doesn't have it on. I rather *immediately*
  noticed this huge regression (I run my ethernet at 100Mbit, and thus
  immediately saw 400+ms latency) - and I hope it can be fixed in the
  next release?

  it can also be made the default via compiling it into the kernel as a
  kernel default.

  Aggh! death to fifos!

  root at localhost:~# sysctl -a | grep qdisc
  net.core.default_qdisc = pfifo_fast

  disc pfifo_fast 0: dev eth0 parent :1 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
   Sent 5250 bytes 53 pkt (dropped 0, overlimits 0 requeues 0)
   backlog 0b 0p requeues 0

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+bug/2003027/+subscriptions