[Bug 1934933] [NEW] cloud-init dhclient apparmor denied with noexec on /var/tmp

[Tom Kopchak]

Public bug reported:

Hello - we are seeing an issue on multiple Azure hosts where there is a
long delay during bootup.  This appears to be related to an apparmor
issue with dhclient executed via cloud-init when /var is mounted noexec.
Because /var is noexec, the original dhclient is executed rather than
the copy in /var/tmp/cloud-init, which causes the AppArmor profile to be
applied.

This prevents the instance from being able to record the DHCP lease
information to /var/tmp/cloud-init/cloud-init-dhcp-*, which prevents the
instance from being able to obtain goalstate information, and with
cloud-init 21.2-3 or later, results in an extended delay during boot due
to a recent change in azure.py (https://github.com/canonical/cloud-
init/pull/842).

This issue does not occur in default Ubuntu installations (including the
Ubuntu 20.04 default Azure image), as the dhcp.py script in cloud-init
behaves differently, copying /usr/sbin/dhclient to /var/tmp/cloud-init
/cloud-init-dhcp-xxxxx/dhclient when /var allows executables, and the
apparmor profiles then do not apply to the copied executable.

The syslog will show the following entry when the instance boots up: 
cloud-init[820]: 2021-07-07 14:50:40,661 - dhcp.py[WARNING]: dhclient did not produce expected files: dhcp.leases, dhclient.pid

The cloud-init.log file will show this entry when this issue is occurring. Since the instance has no IP address at this stage of the boot process, an unreachable network is to be expected:
azure.py[DEBUG]: Failed HTTP request with Azure endpoint http://168.63.129.16/machine/?comp=goalstate during attempt 240 with exception: HTTPConnectionPool(host='168.63.129.16', port=80): Max retries exceeded with url: /machine/?comp=goalstate (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd9e89ee190>: Failed to establish a new connection: [Errno 101] Network is unreachable'))

With the timeouts in azure.py described above, the instance will not
boot for around 20 minutes until all 240 connection attempts are
completed.

This is logged in /var/log/audit/audit.log, showing that the dhclient process executed from cloud-init is unable to write the dhclient.pid and dhcp.leases files that are needed to continue the datasource retrieval process:
type=AVC msg=audit(1625678140.496:1898): apparmor="DENIED" operation="open" profile="/{,usr/}sbin/dhclient" name="/proc/8537/task/8540/comm" pid=8537 comm="dhclient" requested_mask="wr" denied_mask="wr" fsuid=0 ouid=0

Adding the following file resolves the issue: /etc/apparmor.d/local/sbin.dhclient
/var/tmp/cloud-init/cloud-init-dhcp-*/dhclient.pid lrw,
/var/tmp/cloud-init/cloud-init-dhcp-*/dhcp.leases lrw,

This allows dhclient executed via cloud-init to write the dhclient.pid
and dhcp.leases files to /var/tmp/cloud-init and the instance to boot
normally.

** Affects: isc-dhcp (Ubuntu)
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to isc-dhcp in Ubuntu.
https://bugs.launchpad.net/bugs/1934933

Title:
  cloud-init dhclient apparmor denied with noexec on /var/tmp

Status in isc-dhcp package in Ubuntu:
  New

Bug description:
  Hello - we are seeing an issue on multiple Azure hosts where there is
  a long delay during bootup.  This appears to be related to an apparmor
  issue with dhclient executed via cloud-init when /var is mounted
  noexec.  Because /var is noexec, the original dhclient is executed
  rather than the copy in /var/tmp/cloud-init, which causes the AppArmor
  profile to be applied.

  This prevents the instance from being able to record the DHCP lease
  information to /var/tmp/cloud-init/cloud-init-dhcp-*, which prevents
  the instance from being able to obtain goalstate information, and with
  cloud-init 21.2-3 or later, results in an extended delay during boot
  due to a recent change in azure.py (https://github.com/canonical
  /cloud-init/pull/842).

  This issue does not occur in default Ubuntu installations (including
  the Ubuntu 20.04 default Azure image), as the dhcp.py script in cloud-
  init behaves differently, copying /usr/sbin/dhclient to /var/tmp
  /cloud-init/cloud-init-dhcp-xxxxx/dhclient when /var allows
  executables, and the apparmor profiles then do not apply to the copied
  executable.

  The syslog will show the following entry when the instance boots up: 
  cloud-init[820]: 2021-07-07 14:50:40,661 - dhcp.py[WARNING]: dhclient did not produce expected files: dhcp.leases, dhclient.pid

  The cloud-init.log file will show this entry when this issue is occurring. Since the instance has no IP address at this stage of the boot process, an unreachable network is to be expected:
  azure.py[DEBUG]: Failed HTTP request with Azure endpoint http://168.63.129.16/machine/?comp=goalstate during attempt 240 with exception: HTTPConnectionPool(host='168.63.129.16', port=80): Max retries exceeded with url: /machine/?comp=goalstate (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd9e89ee190>: Failed to establish a new connection: [Errno 101] Network is unreachable'))

  With the timeouts in azure.py described above, the instance will not
  boot for around 20 minutes until all 240 connection attempts are
  completed.

  This is logged in /var/log/audit/audit.log, showing that the dhclient process executed from cloud-init is unable to write the dhclient.pid and dhcp.leases files that are needed to continue the datasource retrieval process:
  type=AVC msg=audit(1625678140.496:1898): apparmor="DENIED" operation="open" profile="/{,usr/}sbin/dhclient" name="/proc/8537/task/8540/comm" pid=8537 comm="dhclient" requested_mask="wr" denied_mask="wr" fsuid=0 ouid=0

  Adding the following file resolves the issue: /etc/apparmor.d/local/sbin.dhclient
  /var/tmp/cloud-init/cloud-init-dhcp-*/dhclient.pid lrw,
  /var/tmp/cloud-init/cloud-init-dhcp-*/dhcp.leases lrw,

  This allows dhclient executed via cloud-init to write the dhclient.pid
  and dhcp.leases files to /var/tmp/cloud-init and the instance to boot
  normally.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/isc-dhcp/+bug/1934933/+subscriptions