[Bug 1618188] Re: systemd journal should be persistent by default: /var/log/journal should be created

[Mark Stosberg]

> @xnox
> "The journald daemon has limits set for logs, meaning they will be 
> rotated and discarded and should not cause out of disk-space errors."
> 
> What are they?  AFAICT it only has limits on the number of files, but
> not how big they can overall become.

The limits are documented in `man journald.conf`.

One of them is " SystemMaxUse=, ", which is based on disk usage, not
file size.

> I'm also thinking that the duplicate writing of logs could cause other 
> regressions, one example being where high disk throughput is ongoing and 
> many things being written to the logs. Thoughts?

Additional disk writing is somewhat mitigated by the general increase in
disk performance over time in new hardware

As one user found here, SSD is about 5x faster than HDD and the newer NVMe SSDs are about
5x faster than the older SSDs. A new NVMe SSD is about 25x faster than an HDD.

https://photographylife.com/nvme-vs-ssd-vs-hdd-performance

The idea here is to be "safe by default". People are welcome to
prioritize performance and reduce logging beyond the defaults.

    Mark

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1618188

Title:
  systemd journal should be persistent by default: /var/log/journal
  should be created

Status in systemd package in Ubuntu:
  Fix Released
Status in systemd source package in Xenial:
  In Progress
Status in systemd source package in Zesty:
  Won't Fix
Status in systemd source package in Artful:
  Fix Committed
Status in systemd source package in Bionic:
  Fix Released

Bug description:
  [Impact]

   * System logs are lost across reboots because they are not stored
  persistently.

  [Test Case]

   * Fresh installations, or upgrades to this version of systemd, should create /var/log/journal and trigger automatic persistent logs.
   * Users may choose to remove said directory, or disable persistent logging in /etc/systemd/journald.conf

  [Regression Potential]

   * Persistent logging by default will cause logs to be flushed from
  /run to /var/log, meaning there will be less RAM used (/run is tmpfs
  backed), but increased disk usage (in /var/log). The journald daemon
  has limits set for logs, meaning they will be rotated and discarded
  and should not cause out of disk-space errors.

  [Other Info]
   
   * Original bug report

  After upgrading 14.04 -> 16.04, key services are now running on
  systemd and using the systemd journal for logging. In 14.04, key
  system logs like /var/log/messages and /var/log/syslog were
  persistent, but after the upgrade to 16.04 there has a been a
  regression of sorts: Logs sent to systemd's journald are now being
  thrown away during reboots.

  This behavior is controlled by the `Storage=` option in
  `/etc/systemd/journald.conf`. The default setting is `Storage=auto`
  which will persist logs in `/var/log/journal/`, *only if the directory
  already exists*. But the directory was not created as part of the
  14.04 -> 16.04 upgrade, so logging was being lost for a while before I
  realized what was happening.

  This issue could be solved by either creating /var/log/journal or
  changing the default Storage behavior to `Storage=persistent`, which
  would create the directory if need be.

  ## Related reference

   * `systemd` currently compounds the issue by having ["journal --disk-usage" report memory usage as disk usage](https://github.com/systemd/systemd/issues/4059), giving the impression that the disk is being used for logging when it isn't.
   * [User wonders where to find logs from previous boots, unaware that the logs were thrown away](http://askubuntu.com/questions/765315/how-to-find-previous-boot-log-after-ubuntu-16-04-restarts)

  ## Recommended fix

  Restoring persistent logging as the default is recommended.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1618188/+subscriptions