[Bug 1650688] Re: timedatectl set-timezone fails on UC16

[Lukas Märdian]

I'm pretty much with @ddstreet here, introducing another hack to handle
Ubuntu Core quirks is not nice, as those hacks will make our systemd
more unstable over time and will break regularly after merging upstream
changes.

As stated before, we already carry such hacks since 2014 and I just want
to give a brief quote from that long standing patch: "Forwarded: OMGno,
this is a rather nasty hack until we fix system-image to get a writable
/etc" – I do not know a lot about Ubuntu Core's file system hierarchy
and why it deviates from the common setup, but maybe getting a writable
/etc is the core problem to solve here, as stated by @pitti in 2014
already.

IIUC we currently have a workaround by @ogra in place that only applies to the "timedatectl" CLI, the proposed MRs would fix this for the "timedatectl" CLI and the systemd-timedated DBus API. But what about other tools that assume /etc/localtime to be handled like on most other common systems? Do we start patching every application now and teach them about Ubuntu Core's /etc/writable quirks, e.g. glibc's "tzset(3)"?
That cannot be the correct path forward...

OTOH those MRs are rather small and clear and they solve an issue for
our users NOW (tho only one part of the issue that is related to
systemd-timedate).

As a compromise I guess I would be willing to accept the current patches
into Jammy (so they can be SRUed afterwards), IF we have a clear path
forward about solving this problem properly and replacing the hack with
an upstream solution in the not too distant future – hopefully dropping
the other long-standing /etc/writable patch at the same time.

The snapd team (Michael/Valentin) recently started investigating proper
upstream solutions to this problem (thanks for that!) and I asked
Valentin to create a tracking bug for us, that we can reference
alongside those new hacks, so we can drop them once a better solution is
in place: https://bugs.launchpad.net/snappy/+bug/1953172

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1650688

Title:
  timedatectl set-timezone fails on UC16

Status in Snappy:
  Triaged
Status in systemd package in Ubuntu:
  Confirmed
Status in systemd source package in Jammy:
  Confirmed

Bug description:
  SRU
  ===

  [Impact]

   * The bug prevents timedated from recognizing and correctly set the
  system's timezone when running Ubuntu Core 16, 18 and 20.

   * This causes by timedated fails to take Ubuntu Core's /etc/writable
  redirection into account.

   * The recognizing part is fixed by making the code take writable
  redirection into account.

   * The set part is fixed by making the code link to the absolute path
  instead of a relative one.

   * Currently core snaps worked around the set part by providing a
  wrapper script which re-create /etc/writable/localtime afterward.
  However this does not cover DBus users.

  [Test Plan]

   * On classics systems: ensure the proposed systemd package is installed.
     On Ubuntu Core systems: build a new core snap including proposed package, and install it. Replaces timedatectl with timedatectl.real to test skipping the wrapper.

  (Note that one can simulate core snap's /etc/writable redirection by
  running this image creation hook [1] on the system.)

  [1] https://git.launchpad.net/livecd-rootfs/tree/live-build/ubuntu-
  core/hooks/08-etc-writable.chroot?h=ubuntu/focal

   * On freshly boot system: query the timezone using `timedatectl`. The
  timezone should corresponds to `readlink -f /etc/localtime` and does
  not show `n/a`.

   * Set a new timezone: `sudo timedatectl set-timezone Asia/Bangkok`.
  `readlink -f /etc/localtime` should points to an existing file.

   * Run `sudo systemctl restart systemd-timedated.service`. Then, query
  the timezone again: `timedatectl`. It should show the previously set
  timezone and not `n/a`.

   * Run `sudo systemctl status systemd-timedated.service`. This should
  show no sign of timedated crashing.

  [Where problems could occur]

   * It's possible that the redirection handling code will be sub-par
  and causes crash. However, it's not likely because the similar pieces
  of code is in the previous patch since Ubuntu 16.04.

   * If it does: the patched `get_timezone()` function is used in 2
  places: the networkd's DHCP server [3] and the timedated itself.

     - Networkd is used primarily on servers where NetworkManage is
  absent. It's possible that this patch causes the user to loss access
  to the server due to networkd crash when setting up network
  interfaces, and requires physical access to fix. However, the code
  path is executed when DHCP is enabled only. I think it's not common
  for users to have networkd's DHCP server enabled: the feature seems to
  gear towards desktop users wanting to share internet connection, and
  in those cases they're more likely to use NetworkManager.

     - The timedated itself is likely used by the programs that involves
  in time-related functions. If a crash occur, in the worst case users
  won't be able to set time or timezone via timedated. However, users
  should still be able to e.g. set time using `date` or set timezone
  using /etc/localtime (assuming online guides still consider systems
  without systemd). Timedated is DBus-activated, and thus a crash should
  be self-healing.

   * The set part would also affects the clasic systems. However, I
  believe nothing else actually rely on /etc/localtime being a relative
  path, otherwise the /etc/writable redirection would causes even more
  problem, and would have been reported.

  [3] Yes, I'm surprised that there's a DHCP server inside systemd
  codebase.

  [Other Info]

   * This is also useful for UBports's Ubuntu Touch. We continue using
  system-image system where the rootfs is read-only, and thus is
  affected by this bug similarly to Ubuntu Core. I've tested the Focal
  version of the package on our (currently in development) Focal Ubuntu
  Touch image, and the fix works.

  ------------

  [Original bug description]

  On a system running UC16, the file /etc/localtime is a link that
  points to /etc/writable/localtime.

  On a freshly installed system, /etc/writable/localtime is a fully-
  qualified link that points at /usr/share/zoneinfo/UTC.

  If timedatectl is used to set the timezone to something else,
  timedated updates the localtime symbolic link with a relative path to
  the zoneinfo directory, which results in an invalid link.

  $ sudo timedatectl set-timezone America/Detroit

  $ sudo timedatectl
        Local time: Fri 2016-12-16 18:18:49 EST
    Universal time: Fri 2016-12-16 23:18:49 UTC
          RTC time: Fri 2016-12-16 23:18:49
         Time zone: America/Detroit (EST, -0500)
   Network time on: yes
  NTP synchronized: yes
   RTC in local TZ: no

  $ ls -l /etc/writable/localtime
  /etc/writable/localtime --> ../usr/share/zoneinfo/America/Detroit

  admin at localhost:/etc/writable$ date
  Fri Dec 16 23:20:07 UTC 2016

  I'm running the latest core snap from the candidate channel which
  contains snapd 2.18.1.  Hardware details and/or more debug information
  can be supplied on request.

  See 'man 3 timezone' and 'man timedatectl' for more details.

To manage notifications about this bug go to:
https://bugs.launchpad.net/snappy/+bug/1650688/+subscriptions