[Bug 1936407] [NEW] socat bug with SSL "filte transfers"

[Sean Reifschneider]

Public bug reported:

I'm running ganeti 3.0.1 on Ubuntu 20.04, and running into issues with
moving instances between host nodes.  Under the hood, this use socat to
copy the disk data from one node to another.  In 20.04, the copy seems
to copy nearly all the data, but then at the very end it fails with:

    Wed Jul 14 19:57:00 2021  - WARNING: import 'import-disk0-2021-07-14_19_52_25-8ockgvr6' on gnt1.example.com failed: Exited with status 1
    Wed Jul 14 19:57:00 2021 disk/0 failed to receive data: Exited with status 1 (recent output: dd: warning: partial read (65494 bytes); suggest iflag=fullblock\nsocat: E SSL_read():Connection reset by peer\n0+980142 records in\n0+980142 records out\n21976203264 bytes (22 GB, 20 GiB) copied, 273.039 s, 80.5 MB/s)

The copy seems to be almost 2MB short, expected size is 21,978,152,960
the copy above reports it copied 21,976,203,264

I've tried to isolate this down to a reproducable test case that does
not require ganeti, but I don't seem to have the socat chops to make it
happen.  I'll list what I have so far down below.

I believe this is an issue with socat version 1.7.3.3-2 because:

- If I install socat_1.7.3.2-2ubuntu2_amd64.deb from 18.04 or socat_1.7.4.1-3ubuntu1_amd64.deb from 21.04 (just download those packages and "dpkg -i", no dependency issues are reported), my VM copy will succeed.
- On the socat website ( http://www.dest-unreach.org/socat/ ) it notes that "Socat version 1.7.4.1 fixes [...] file transfer with OpenSSL.".    That seems like exactly what ganeti is doing, SSL bulk transfers.

The actual commands being run by ganeti are:

    bash -o errexit -o pipefail -c { echo -E -n
M=b0f141f7085de052088d687fc70a027e53928be8 &&{ LC_ALL=C dd bs=1048576
<&0 2>&6 & pid=${!}; echo $pid >&8; wait $pid; } } | /usr/bin/socat -ls
-d -d -b1048576 -u stdin OPENSSL:10.1.1.1:38219,connect-
timeout=20,retry=10,intervall=1,keepalive,keepidle=60,keepintvl=10,keepcnt=5,verify=1,cipher=HIGH:-DES:-3DES:-EXPORT:-DH,compress=none,key=/var/lib/ganeti/server.pem,cert=/var/lib/ganeti/server.pem,cafile=/var/run/ganeti/import-
export/export-disk0-2021-07-14_19_52_30-ehngj3q8/ca,pf=ipv4,openssl-
commonname=ganeti.example.com 2>&4

and:

    bash -o errexit -o pipefail -c /usr/bin/socat -ls -d -d -b1048576 -u
OPENSSL-
LISTEN:0,reuseaddr,forever,intervall=0.01,keepalive,keepidle=60,keepintvl=10,keepcnt=5,verify=1,cipher=HIGH:-DES:-3DES:-EXPORT:-DH,compress=none,key=/var/lib/ganeti/server.pem,cert=/var/lib/ganeti/server.pem,cafile=/var/run/ganeti/import-
export/import-disk0-2021-07-14_19_52_25-8ockgvr6/ca,pf=ipv4 stdout 2>&4
| { { read -n 42 magic && if test "$magic" !=
M=b0f141f7085de052088d687fc70a027e53928be8; then echo 'Magic value
mismatch' >&2; exit 1;fi; } && { LC_ALL=C dd bs=1048576 <&0 2>&6 &
pid=${!}; echo $pid >&8; wait $pid; } }

I tried to simplify this down to a pair of socat commands like this:

    /usr/bin/socat -ls -d -d -b1048576 -u OPENSSL-LISTEN:12345,reuseaddr,forever,intervall=0.01,keepalive,keepidle=60,keepintvl=10,keepcnt=5,verify=0,cipher=HIGH:-DES:-3DES:-EXPORT:-DH,compress=none,key=/etc/ssl/private/ssl-cert-snakeoil.key,cert=/etc/ssl/certs/ssl-cert-snakeoil.pem,pf=ipv4 stdout | sha1sum &
    dd if=/usr/bin/perl bs=1048576 | /usr/bin/socat -ls -d -d -b1048576 -u stdin OPENSSL:127.0.0.1:12345,connect-timeout=20,retry=10,intervall=1,keepalive,keepidle=60,keepintvl=10,keepcnt=5,verify=1,cipher=HIGH:-DES:-3DES:-EXPORT:-DH,compress=none,key=/etc/ssl/private/ssl-cert-snakeoil.key,cert=//etc/ssl/certs/ssl-cert-snakeoil.pem

But this fails with: "ioctl(6, IOCTL_VM_SOCKETS_GET_LOCAL_CID, ...):
Inappropriate ioctl for device" and my google searches didn't turn up
what might be the problem.

I'm going to be just installing the 21.04 socat package on my
ganeti+20.04 systems, as so far in my testing that seems to be working
fine and will prevent me from having to pin the 18.04 packages.

** Affects: socat (Ubuntu)
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to socat in Ubuntu.
https://bugs.launchpad.net/bugs/1936407

Title:
  socat bug with SSL "filte transfers"

Status in socat package in Ubuntu:
  New

Bug description:
  I'm running ganeti 3.0.1 on Ubuntu 20.04, and running into issues with
  moving instances between host nodes.  Under the hood, this use socat
  to copy the disk data from one node to another.  In 20.04, the copy
  seems to copy nearly all the data, but then at the very end it fails
  with:

      Wed Jul 14 19:57:00 2021  - WARNING: import 'import-disk0-2021-07-14_19_52_25-8ockgvr6' on gnt1.example.com failed: Exited with status 1
      Wed Jul 14 19:57:00 2021 disk/0 failed to receive data: Exited with status 1 (recent output: dd: warning: partial read (65494 bytes); suggest iflag=fullblock\nsocat: E SSL_read():Connection reset by peer\n0+980142 records in\n0+980142 records out\n21976203264 bytes (22 GB, 20 GiB) copied, 273.039 s, 80.5 MB/s)

  The copy seems to be almost 2MB short, expected size is 21,978,152,960
  the copy above reports it copied 21,976,203,264

  I've tried to isolate this down to a reproducable test case that does
  not require ganeti, but I don't seem to have the socat chops to make
  it happen.  I'll list what I have so far down below.

  I believe this is an issue with socat version 1.7.3.3-2 because:

  - If I install socat_1.7.3.2-2ubuntu2_amd64.deb from 18.04 or socat_1.7.4.1-3ubuntu1_amd64.deb from 21.04 (just download those packages and "dpkg -i", no dependency issues are reported), my VM copy will succeed.
  - On the socat website ( http://www.dest-unreach.org/socat/ ) it notes that "Socat version 1.7.4.1 fixes [...] file transfer with OpenSSL.".    That seems like exactly what ganeti is doing, SSL bulk transfers.

  The actual commands being run by ganeti are:

      bash -o errexit -o pipefail -c { echo -E -n
  M=b0f141f7085de052088d687fc70a027e53928be8 &&{ LC_ALL=C dd bs=1048576
  <&0 2>&6 & pid=${!}; echo $pid >&8; wait $pid; } } | /usr/bin/socat
  -ls -d -d -b1048576 -u stdin OPENSSL:10.1.1.1:38219,connect-
  timeout=20,retry=10,intervall=1,keepalive,keepidle=60,keepintvl=10,keepcnt=5,verify=1,cipher=HIGH:-DES:-3DES:-EXPORT:-DH,compress=none,key=/var/lib/ganeti/server.pem,cert=/var/lib/ganeti/server.pem,cafile=/var/run/ganeti/import-
  export/export-disk0-2021-07-14_19_52_30-ehngj3q8/ca,pf=ipv4,openssl-
  commonname=ganeti.example.com 2>&4

  and:

      bash -o errexit -o pipefail -c /usr/bin/socat -ls -d -d -b1048576
  -u OPENSSL-
  LISTEN:0,reuseaddr,forever,intervall=0.01,keepalive,keepidle=60,keepintvl=10,keepcnt=5,verify=1,cipher=HIGH:-DES:-3DES:-EXPORT:-DH,compress=none,key=/var/lib/ganeti/server.pem,cert=/var/lib/ganeti/server.pem,cafile=/var/run/ganeti/import-
  export/import-disk0-2021-07-14_19_52_25-8ockgvr6/ca,pf=ipv4 stdout
  2>&4 | { { read -n 42 magic && if test "$magic" !=
  M=b0f141f7085de052088d687fc70a027e53928be8; then echo 'Magic value
  mismatch' >&2; exit 1;fi; } && { LC_ALL=C dd bs=1048576 <&0 2>&6 &
  pid=${!}; echo $pid >&8; wait $pid; } }

  I tried to simplify this down to a pair of socat commands like this:

      /usr/bin/socat -ls -d -d -b1048576 -u OPENSSL-LISTEN:12345,reuseaddr,forever,intervall=0.01,keepalive,keepidle=60,keepintvl=10,keepcnt=5,verify=0,cipher=HIGH:-DES:-3DES:-EXPORT:-DH,compress=none,key=/etc/ssl/private/ssl-cert-snakeoil.key,cert=/etc/ssl/certs/ssl-cert-snakeoil.pem,pf=ipv4 stdout | sha1sum &
      dd if=/usr/bin/perl bs=1048576 | /usr/bin/socat -ls -d -d -b1048576 -u stdin OPENSSL:127.0.0.1:12345,connect-timeout=20,retry=10,intervall=1,keepalive,keepidle=60,keepintvl=10,keepcnt=5,verify=1,cipher=HIGH:-DES:-3DES:-EXPORT:-DH,compress=none,key=/etc/ssl/private/ssl-cert-snakeoil.key,cert=//etc/ssl/certs/ssl-cert-snakeoil.pem

  But this fails with: "ioctl(6, IOCTL_VM_SOCKETS_GET_LOCAL_CID, ...):
  Inappropriate ioctl for device" and my google searches didn't turn up
  what might be the problem.

  I'm going to be just installing the 21.04 socat package on my
  ganeti+20.04 systems, as so far in my testing that seems to be working
  fine and will prevent me from having to pin the 18.04 packages.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/socat/+bug/1936407/+subscriptions