[Bug 1818814] Re: systemd-tmpfiles-setup.services fails to create /var/run directories
Kalle Tuulos
1818814 at bugs.launchpad.net
Thu Apr 11 06:32:34 UTC 2019
>> /var/run directories are not created properly
>what specifically do you mean by this? just that you see log errors?
Following directories were not created automatically, I had to create
them manually in order to enable services to start:
/var/run/fail2ban
/var/run/screen
/var/run/sshd
/var/run/zabbix
Following directory was created, but its ownership was not proper:
/var/run/redis
> What's the output of:
> $ sudo find /run -ls
Please see attached file: sudo_find_run_ls.txt (working situation) and find_run_ls_after_reboot.txt (after the system was rebooted i.e. not working situation).
> $ df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/simfs 52428800 7818704 44610096 15% /
devtmpfs 1572864 0 1572864 0% /dev
tmpfs 1572864 0 1572864 0% /dev/shm
tmpfs 1572864 9488 1563376 1% /run
tmpfs 5120 4 5116 1% /run/lock
tmpfs 1572864 0 1572864 0% /sys/fs/cgroup
none 1572864 0 1572864 0% /run/shm
To make this a bit more complete, I removed the 0000var.conf and restarted the system. When I checked the md5sum of files /usr/lib/tmpfiles.d, they matched to my other Ubuntu 16.04 server.
After reboot, syslog relevant parts showed following:
---8<---8<---
Apr 11 08:44:50 vspk systemd-tmpfiles[85]: [/usr/lib/tmpfiles.d/var.conf:14] Duplicate line for path "/var/log", ignoring.
Apr 11 08:44:50 vspk systemd-tmpfiles[85]: fchownat() of /run/named failed: Invalid argument
Apr 11 08:44:50 vspk systemd-tmpfiles[85]: Failed to validate path /var/run/fail2ban: Too many levels of symbolic links
Apr 11 08:44:50 vspk systemd-tmpfiles[85]: fchownat() of /run/redis failed: Invalid argument
Apr 11 08:44:50 vspk systemd-tmpfiles[85]: Failed to validate path /var/run/screen: Too many levels of symbolic links
Apr 11 08:44:50 vspk systemd-tmpfiles[85]: Failed to validate path /var/run/sshd: Too many levels of symbolic links
Apr 11 08:44:50 vspk systemd-tmpfiles[85]: Failed to validate path /var/run/sudo: Too many levels of symbolic links
Apr 11 08:44:50 vspk systemd-tmpfiles[85]: Failed to validate path /var/run/sudo/ts: Too many levels of symbolic links
Apr 11 08:44:50 vspk systemd-tmpfiles[85]: fchownat() of /run/utmp failed: Invalid argument
Apr 11 08:44:50 vspk systemd-tmpfiles[85]: fchownat() of /run/systemd/netif failed: Invalid argument
Apr 11 08:44:50 vspk systemd-tmpfiles[85]: fchownat() of /run/systemd/netif/links failed: Invalid argument
Apr 11 08:44:50 vspk systemd-tmpfiles[85]: fchownat() of /run/systemd/netif/leases failed: Invalid argument
Apr 11 08:44:50 vspk systemd-tmpfiles[85]: Failed to validate path /var/run/zabbix: Too many levels of symbolic links
Apr 11 08:44:50 vspk systemd-tmpfiles[85]: fchownat() of /run/log/journal failed: Invalid argument
Apr 11 08:44:50 vspk systemd-tmpfiles[85]: fchownat() of /run/log/journal/6d9c7cc322ee4c48af7c0ec3b492b5cc failed: Invalid argument
Apr 11 08:44:50 vspk systemd-tmpfiles[85]: fchownat() of /run/log/journal/6d9c7cc322ee4c48af7c0ec3b492b5cc/system.journal failed: Invalid argument
Apr 11 08:44:50 vspk systemd[1]: systemd-tmpfiles-setup.service: Main process exited, code=exited, status=1/FAILURE
Apr 11 08:44:50 vspk systemd[1]: Failed to start Create Volatile Files and Directories.
Apr 11 08:44:50 vspk systemd[1]: systemd-tmpfiles-setup.service: Unit entered failed state.
Apr 11 08:44:50 vspk systemd[1]: systemd-tmpfiles-setup.service: Failed with result 'exit-code'.
---8<---8<---
This caused, that sshd, redis, fail2ban and zabbix agent were not able
to start.
In order for me to make the system usable, I needed to perform following
steps:
1) Log in to system using the serial console from the VPS control panel.
root at vspk:/var/run# ls -la /var/run
lrwxrwxrwx 1 root root 4 May 14 2018 /var/run -> /run
Contents of /run directory:
root at vspk:/var/run# find /run -ls > /home/kalle/find_run_ls_after_reboot.txt (attached)
2) Give following commands:
root at vspk:/var/run# mkdir sshd
root at vspk:/var/run# /etc/init.d/ssh start
Starting ssh (via systemctl): ssh.service.
--> Now the system can be again accessed through SSH.
After that:
kalle at vspk:~$ sudo mkdir /var/run/zabbix
kalle at vspk:~$ sudo chown zabbix:zabbix /var/run/zabbix
kalle at vspk:~$ sudo chown redis:redis /var/run/redis
kalle at vspk:~$ sudo mkdir /var/run/fail2ban
kalle at vspk:~$ sudo mkdir /var/run/screen
kalle at vspk:~$ sudo chmod 777 /var/run/screen
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1818814
Title:
systemd-tmpfiles-setup.services fails to create /var/run directories
Status in systemd package in Ubuntu:
New
Bug description:
1) The release of Ubuntu you are using, via 'lsb_release -rd' or System -> About Ubuntu
Description: Ubuntu 16.04.6 LTS
Release: 16.04
2) The version of the package you are using, via 'apt-cache policy pkgname' or by checking in Software Center
systemd:
Installed: 229-4ubuntu21.16
Candidate: 229-4ubuntu21.16
Version table:
*** 229-4ubuntu21.16 500
500 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
500 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages
100 /var/lib/dpkg/status
229-4ubuntu4 500
500 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages
3) What you expected to happen
4) What happened instead
Ubuntu server (running in OpenVZ VPS farm, thus the old kernel
version) has been up and running happily, until I performed apt-get
upgrade and rebooted the server. After reboot, I could not establish
SSH connection to server, port 22 connection was refused.
I opened a HTML console to my server instance and checked logs. From
the logs, it was shown, that SSH server could not start, as it did not
have the /var/run/sshd directory. After scrolling back the
/var/log/syslog, I noticed that there were lots of other /var/run
subdirectories, which were not created. Here is cut&paste from
/var/log/syslog, related to systemd-tmpfiles:
---8<---8<---
Mar 6 12:32:54 vspk systemd-tmpfiles[81]: [/usr/lib/tmpfiles.d/00rsyslog.conf:6] Duplicate line for path "/v
ar/log", ignoring.
Mar 6 12:32:54 vspk systemd[1]: Starting Raise network interfaces...
Mar 6 12:32:54 vspk systemd-tmpfiles[81]: fchownat() of /run/named failed: Invalid argument
Mar 6 12:32:54 vspk systemd-tmpfiles[81]: Failed to validate path /var/run/fail2ban: Too many levels of symb
olic links
Mar 6 12:32:54 vspk systemd-tmpfiles[81]: Failed to validate path /var/run/screen: Too many levels of symbol
ic links
Mar 6 12:32:54 vspk systemd-tmpfiles[81]: Failed to validate path /var/run/sshd: Too many levels of symbolic
links
Mar 6 12:32:54 vspk systemd-tmpfiles[81]: Failed to validate path /var/run/sudo: Too many levels of symbolic
links
Mar 6 12:32:54 vspk systemd-tmpfiles[81]: Failed to validate path /var/run/sudo/ts: Too many levels of symbo
lic links
Mar 6 12:32:54 vspk systemd-tmpfiles[81]: fchownat() of /run/utmp failed: Invalid argument
Mar 6 12:32:54 vspk systemd-tmpfiles[81]: fchownat() of /run/systemd/netif failed: Invalid argument
Mar 6 12:32:54 vspk systemd-tmpfiles[81]: fchownat() of /run/systemd/netif/links failed: Invalid argument
Mar 6 12:32:54 vspk systemd-tmpfiles[81]: fchownat() of /run/systemd/netif/leases failed: Invalid argument
Mar 6 12:32:54 vspk systemd-tmpfiles[81]: Failed to validate path /var/run/zabbix: Too many levels of symbol
ic links
Mar 6 12:32:54 vspk systemd-tmpfiles[81]: fchownat() of /run/log/journal failed: Invalid argument
Mar 6 12:32:54 vspk systemd-tmpfiles[81]: fchownat() of /run/log/journal/6d9c7cc322ee4c48af7c0ec3b492b5cc fa
iled: Invalid argument
Mar 6 12:32:54 vspk systemd-tmpfiles[81]: fchownat() of /run/log/journal/6d9c7cc322ee4c48af7c0ec3b492b5cc/sy
stem.journal failed: Invalid argument
Mar 6 12:32:54 vspk systemd[1]: systemd-tmpfiles-setup.service: Main process exited, code=exited, status=1/F
AILURE
Mar 6 12:32:54 vspk systemd[1]: Failed to start Create Volatile Files and Directories.
Mar 6 12:32:54 vspk systemd[1]: systemd-tmpfiles-setup.service: Unit entered failed state.
Mar 6 12:32:54 vspk systemd[1]: systemd-tmpfiles-setup.service: Failed with result 'exit-code'.
---8<---8<---
My first idea was, that for some reason, systemd-tmpfiles was not able
to create the /var directory properly, so I renamed
/usr/lib/tmpfiles.d/var.conf to 0000var.conf, but it was no help. The
only difference was the first line on the above log, complaining about
duplicate line for path /var/log.
As I created this error using the "ubuntu-bug" command, I assume
necessary background information is automatically attached. I'll check
those and add more, if necessary.
ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: systemd 229-4ubuntu21.16
Uname: Linux 2.6.32-042stab128.2 x86_64
ApportVersion: 2.20.1-0ubuntu2.18
Architecture: amd64
CurrentDmesg:
Date: Wed Mar 6 12:43:51 2019
Lsusb: Error: command ['lsusb'] failed with exit code 1:
ProcEnviron:
TERM=vt220
PATH=(custom, no user)
ProcInterrupts: Error: [Errno 2] No such file or directory: '/proc/interrupts'
ProcKernelCmdLine: quiet
ProcModules:
SourcePackage: systemd
UpgradeStatus: No upgrade log present (probably fresh install)
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1818814/+subscriptions
More information about the foundations-bugs
mailing list