[Bug 2028810] Re: rsync 3.1.3 performance regression

[Lena Voytek]

Re-verified with documentation:

$ lxc launch ubuntu:focal test-rsync-receiver
$ lxc exec test-rsync-receiver bash
# apt update && apt dist-upgrade -y
# apt install openssh-server rsync -y

# passwd ubuntu
New password:
Retype new password:
passwd: password updated successfully

# sed -i 's/PasswordAuthentication no/PasswordAuthentication yes/' /etc/ssh/sshd_config
# systemctl restart sshd

# exit

- Check ip of receiver with lxc list
$ lxc list

+---------------------+---------+----------------------+-----------------------------------------------+-----------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+---------------------+---------+----------------------+-----------------------------------------------+-----------+-----------+
| test-rsync-receiver | RUNNING | 10.190.23.243 (eth0) | -------------------------------------- (eth0) | CONTAINER | 0 |
+---------------------+---------+----------------------+-----------------------------------------------+-----------+-----------+

$ lxc launch ubuntu:focal test-rsync-sender
$ lxc exec test-rsync-sender bash

# cat <<EOF >/etc/apt/sources.list.d/ubuntu-$(lsb_release -cs)-proposed.list
# Enable Ubuntu proposed archive
deb http://archive.ubuntu.com/ubuntu/ $(lsb_release -cs)-proposed restricted main multiverse universe
EOF

# apt update && apt dist-upgrade -y

# apt install rsync -y

# rsync --help

...
 -s, --protect-args          no space-splitting; only wildcard special-chars
     --trust-sender          trust the remote sender's file list
     --address=ADDRESS       bind address for outgoing socket to daemon
...

# man rsync

...
       --trust-sender
              Disable the extra validation of the file list from a remote sender (this safety feature was added to address the performance downgrade after
              fixing CVE 2022-29154).  This should only be done if you trust the sender to not try to do something malicious, which should be the case  if
              they're running a stock rsync.

              Normally when pulling files from a remote rsync, the
client runs 2 extra validation checks:

              o      Verify that additional arg items didn't get added
at the top of the transfer.

              o      Verify that none of the items in the file list
should have been excluded.

              Note that various options can turn off one or both of
these checks if the option interferes with the validation.  For
instance:

              o      Using a per-directory filter file reads filter
rules that only the server knows about, so the filter checking is
disabled.

              o      Using the --old-args option allows the sender to
manipulate the requested args, so the arg checking is disabled.

              o      Reading the files-from list from the server side
means that the client doesn't know the arg list, so the arg checking is
disabled.

              o      Using --read-batch disables both checks since the
batch file's contents will have been verified when it was created.

              This option may help an under-powered client server if the extra pattern matching is slowing things down on a huge transfer.  It can also be
              used to work around a bug in the verification logic, possibly after using the --list-only option combined with --trust-sender to  look  over
              the full file list.
...

# dd if=/dev/urandom of=randomfile.bin bs=1M count=1000

# rsync -av randomfile.bin ubuntu at 10.190.23.243:~/file1.bin
ubuntu at 10.190.23.243's password:
sending incremental file list
randomfile.bin

sent 1,048,832,093 bytes  received 35 bytes  99,888,774.10 bytes/sec
total size is 1,048,576,000  speedup is 1.00

# rsync -av --trust-sender randomfile.bin ubuntu at 10.190.23.243:~/file2.bin
ubuntu at 10.190.23.243's password:
sending incremental file list
randomfile.bin

sent 1,048,832,093 bytes  received 35 bytes  139,844,283.73 bytes/sec
total size is 1,048,576,000  speedup is 1.00


** Tags removed: verification-needed verification-needed-focal
** Tags added: verification-done verification-done-focal

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to rsync in Ubuntu.
https://bugs.launchpad.net/bugs/2028810

Title:
  rsync 3.1.3 performance regression

Status in rsync package in Ubuntu:
  Fix Released
Status in rsync source package in Focal:
  Fix Committed

Bug description:
  [Impact]

  Recent necessary security fixes to rsync have caused a slow down in
  transfer speeds due to additional authentication. In more recent
  versions of rsync this can be mitigated when the environment is
  trusted with the --trust-sender flag.

  In order to accomidate this use case, the flag should be backported to
  focal too.

  [Test Plan]

  $ lxc launch ubuntu:focal test-rsync-receiver
  $ lxc exec test-rsync-receiver bash
  # apt update && apt dist-upgrade -y
  # apt install openssh-server rsync -y
  # passwd ubuntu
  - set password for user
  # exit

  - Check ip of receiver with lxc list
  $ lxc list

  $ lxc launch ubuntu:focal test-rsync-sender
  $ lxc exec test-rsync-sender bash
  # apt update && apt dist-upgrade -y

  # apt install rsync -y

  - Create a random file to send over
  # dd if=/dev/urandom of=randomfile.bin bs=1M count=1000

  - Send without --trust-sender
  # rsync -av randomfile.bin ubuntu@<receiver ip>:~/file1.bin

  - Send with --trust-sender
  # rsync -av --trust-sender randomfile.bin ubuntu@<receiver ip>:~/file2.bin

  With the fix in place, --trust-sender is a valid argument and the
  transfer is notably faster as reported back by rsync.

  [Where problems could occur]

  Since this change adds a new feature in the form of an input flag,
  problems could occour when using it. This could include issues from
  skipping security checks between the sending and receiving machine.
  Another possible problem would be issues with command line input
  parsing due to the additional valid argument.

  [Other Info]
   
  The --trust-sender option is already available in Jammy and later

  [Original Description]

  OS: Ubuntu 20.04 Focal
  Package: rsync 3.1.3-8ubuntu0.5

  rsync's performance was regressed by ~7x amount after some security
  patch (debian/patches/CVE-2022-29154-*) was applied to the package,
  and introduced a list of filters that iterate on every file being
  transferred. We think that was where the performance regression came
  from.

  A Jammy version of the package (3.2.5) introduced a new flag "--trust-
  sender" that allowed user to avoid the expensive client-side filtering
  introduced by those security patches. After pulling this change
  (https://github.com/WayneD/rsync/commit/cff8f044776c5143a5b270969d4bb0f1fea8b017)
  from rsync ourselves and applied it to the Focal version, the
  performance regression went away.

  The patch we used to backport our Focal rsync is attached in this
  thread. Can you please backport it too?

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/2028810/+subscriptions