[Bug 1934221] Re: systemd-resolve segfault

Thu Jul 29 16:52:17 UTC 2021

I recorded pcap and captured moment just before crash. It doesnt looks
like there was any MDNS at all at this moment, only suspicious is many
weird DNS requests. (spinesystems.solutions is my local domain set on
pc). Not sure what generated them.

Small snap of these:

02:16:36.658522 IP 127.0.0.1.39766 > 127.0.0.1.53: 17063+ A? roxquromlpczqgh.spinesystems.solutions. (56)
02:16:36.658626 IP 127.0.0.1.31846 > 127.0.0.53.53: 54480+ A? roxquromlpczqgh.spinesystems.solutions. (56)
02:16:36.659106 IP 127.0.0.53.53 > 127.0.0.1.6126: 42840 NXDomain 0/0/0 (56)
02:16:36.659603 IP 127.0.0.53.53 > 127.0.0.1.31846: 54480 NXDomain 0/0/0 (56)
02:16:36.659644 IP 127.0.0.1.53 > 127.0.0.1.54315: 17063 NXDomain 0/0/0 (56)
02:16:36.659660 IP 127.0.0.1.53 > 127.0.0.1.39766: 17063 NXDomain 0/0/0 (56)
02:16:36.660099 IP 127.0.0.53.53 > 127.0.0.1.6126: 42840 NXDomain 0/0/0 (56)
02:16:36.660603 IP 127.0.0.53.53 > 127.0.0.1.6126: 42840 NXDomain 0/0/0 (56)
02:16:36.661082 IP 127.0.0.53.53 > 127.0.0.1.6126: 42840 NXDomain 0/0/0 (56)
02:16:36.661513 IP 127.0.0.53.53 > 127.0.0.1.25474: 60255 NXDomain 0/0/1 (67)
02:16:36.661898 IP 127.0.0.53.53 > 127.0.0.1.6126: 42840 NXDomain 0/0/0 (56)
02:16:36.662282 IP 127.0.0.53.53 > 127.0.0.1.6126: 42840 NXDomain 0/0/0 (56)
02:16:36.662778 IP 127.0.0.53.53 > 127.0.0.1.6126: 42840 NXDomain 0/0/0 (56)
02:16:36.663188 IP 127.0.0.1.60869 > 127.0.0.1.53: 47647+ AAAA? xjstkkuhaqclygt.spinesystems.solutions. (56)
02:16:36.663259 IP 127.0.0.1.32600 > 127.0.0.53.53: 10939+ AAAA? xjstkkuhaqclygt.spinesystems.solutions. (56)
02:16:36.663449 IP 127.0.0.1.36798 > 127.0.0.1.53: 30847+ AAAA? roxquromlpczqgh.spinesystems.solutions. (56)
02:16:36.663481 IP 127.0.0.1.15565 > 127.0.0.53.53: 54137+ AAAA? roxquromlpczqgh.spinesystems.solutions. (56)
02:16:36.664474 IP 127.0.0.1.47179 > 127.0.0.1.53: 3316+ A? roxquromlpczqgh.spinesystems.solutions. (56)
02:16:36.664563 IP 127.0.0.1.40056 > 127.0.0.53.53: 55684+ A? roxquromlpczqgh.spinesystems.solutions. (56)
02:16:36.664969 IP 10.255.255.225.55743 > 10.255.255.1.53: 60614+ [1au] A? roxquromlpczqgh.spinesystems.solutions. (67)
02:16:36.665503 IP 10.255.255.1.53 > 10.255.255.225.55743: 60614 NXDomain 0/1/1 (140)
02:16:36.666994 IP 10.255.255.225.55743 > 10.255.255.1.53: 60614+ A? roxquromlpczqgh.spinesystems.solutions. (56)
02:16:36.667494 IP 10.255.255.1.53 > 10.255.255.225.55743: 60614 NXDomain 0/1/0 (129)

Maybe because there is massive timeouts in query to nonexisting domain it triggers this bug like in original issue in systemd ( https://github.com/systemd/systemd/issues/18427 )?
In my ISP also all port 53 requests are force-redirected to their DNS, so they might timeout sometimes, instead of NXDomain.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1934221

Title:
  systemd-resolve segfault

Status in systemd package in Ubuntu:
  Fix Released
Status in systemd source package in Focal:
  New
Status in systemd source package in Hirsute:
  New
Status in systemd source package in Impish:
  Fix Released

Bug description:
  [Impact]

   * systemd-resolved stops replying to clients on the local LAN.
   * logging segfault crashes in dmesg:
  [836786.046514] systemd-resolve[872009]: segfault at 39900000000 ip 0000039900000000 sp 00007ffd7959a6d8 error 14 in systemd-resolved[556398695000+9000]
  [836786.046524] Code: Bad RIP value.
  [840887.303994] traps: systemd-resolve[877019] general protection fault ip:55ba402e2594 sp:7ffe8cb6bbb0 error:0 in systemd-resolved[55ba402b5000+40000]
  [844395.313421] systemd-resolve[878503]: segfault at 208 ip 00005557a249f5fa sp 00007ffe686f5a90 error 6 in systemd-resolved[5557a2472000+40000]
  [844395.313431] Code: 48 85 c0 74 0e 48 8b 8d 00 01 00 00 48 89 88 00 01 00 00 48 8b 85 00 01 00 00 48 85 c0 0f 84 1d 01 00 00 48 8b 95 f8 00 00 00 <48> 89 90 f8 00 00 00 48 c7 85 00 01 00 00 00 00 00 00 48 c7 85 f8

   * The upload backports the upstream fix
  (https://github.com/systemd/systemd/pull/18832) to Focal & Hirsute.

  [Test Plan]

   * Setup /etc/systemd/resolved.conf:
  [Resolve]
  DNS=46.182.19.48#dns2.digitalcourage.de 1.1.1.1#cloudflare-dns.com 9.9.9.9#dns.quad9.net
  DNSSEC=yes
  DNSOverTLS=opportunistic
  MulticastDNS=no
  LLMNR=no
  Cache=yes
  DNSStubListener=yes
  Domains=~.

   * wait for ~24-48 hours and observe if any crash happens

  [Where problems could occur]

   * Any regression would likely cause crashes in systemd-resolved,
  making it unresponsive to DNS network name requests to local
  applications.

  [Other Info]

   * Reported upstream: https://github.com/systemd/systemd/issues/18427
   * Fixed upstream in v248: https://github.com/systemd/systemd/pull/18832

  === Original description ===

  systemd-resolve keep crashing and it is very annoying as sometimes it severely interrupt normal dns resolving.

  Last uploaded report is 2d9e7378-d89b-11eb-9e14-fa163ee63de6
  Typical error in dmesg:
  systemd-resolve[1792202]: segfault at 564ff982f3e0 ip 0000564ff982f3e0 sp 00007ffe2fd0b758 error 15

  apport hints me that problem is related to mdns

  #3  0x00007f3e903c2f11 in sd_event_dispatch () from
  /lib/systemd/libsystemd-shared-245.so

  It might be (or not) related that some hosts with mdns in my network
  have ipv6 enabled.

  ProblemType: Bug
  DistroRelease: Ubuntu 20.04
  Package: systemd 245.4-4ubuntu3.7
  ProcVersionSignature: Ubuntu 5.4.0-75.84-generic 5.4.119
  Uname: Linux 5.4.0-75-generic x86_64
  ApportVersion: 2.20.11-0ubuntu27.18
  Architecture: amd64
  CasperMD5CheckResult: skip
  CurrentDesktop: ubuntu:GNOME
  Date: Thu Jul  1 08:22:51 2021
  InstallationDate: Installed on 2018-12-05 (938 days ago)
  InstallationMedia: Ubuntu 18.10 "Cosmic Cuttlefish" - Release amd64 (20181017.3)
  MachineType: System manufacturer System Product Name
  ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.4.0-75-generic root=UUID=54b80d5c-3d61-4919-873f-0d308083e3b9 ro quiet splash vt.handoff=7
  SourcePackage: systemd
  UpgradeStatus: Upgraded to focal on 2020-05-22 (405 days ago)
  dmi.bios.date: 05/12/2020
  dmi.bios.vendor: American Megatrends Inc.
  dmi.bios.version: 1205
  dmi.board.asset.tag: Default string
  dmi.board.name: ROG STRIX X399-E GAMING
  dmi.board.vendor: ASUSTeK COMPUTER INC.
  dmi.board.version: Rev 1.xx
  dmi.chassis.asset.tag: Default string
  dmi.chassis.type: 3
  dmi.chassis.vendor: Default string
  dmi.chassis.version: Default string
  dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr1205:bd05/12/2020:svnSystemmanufacturer:pnSystemProductName:pvrSystemVersion:rvnASUSTeKCOMPUTERINC.:rnROGSTRIXX399-EGAMING:rvrRev1.xx:cvnDefaultstring:ct3:cvrDefaultstring:
  dmi.product.family: To be filled by O.E.M.
  dmi.product.name: System Product Name
  dmi.product.sku: SKU
  dmi.product.version: System Version
  dmi.sys.vendor: System manufacturer

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1934221/+subscriptions