[Bug 1075901] Re: Incorrect ownership or permissions for spool/work directory

[Danny Guinther]

Given that this bug has already been marked in progress, this may not be
helpful, but I think this bug is more serious than suggested and should
be made a priority to fix for precise.

I've run into situations where rsyslog will run at 100% CPU with no
useful output or logging to indicate why. Attaching strace to the
process revealed the application was spinning with the following:

futex(0x151f8e0, FUTEX_WAKE_PRIVATE, 1) = 0
open("/var/spool/rsyslog/relp_queue_hostname.00000001", O_WRONLY|O_CREAT|O_NOCTTY|O_CLOEXEC, 0600) = -1 EACCES (Permission denied)

In this particular case the problem arose because the remote relp server
was down long enough to fill the local rsyslog in-memory queue and force
rsyslog to spool to disk for relp. However, I have also run into
situations where rsyslog will try to open a spool for the local logging
queue and also run into permissions issues. In either case, if one of
the queues turns to spooling, the write rate of the other queue is
drastically affected, which is possibly why I wasn't getting any useful
logging information from rsyslog once the spinning started.

Up until yesterday I took the approach of restarting the rsyslog process
to remedy the issue which I now understand worked because it cleared the
in-memory queue and thereby postponed further throttling.

This morning I forced the issue by using logger to flood rsyslog (`for i
in $(seq 1 5000); do logger "Testing $i"; done`). Within a few thousand
messages the in-memory queue must have reached its limit, as rsyslog
began to consume 100% cpu and messages only showed up in syslog at a
rate of 5-10 every minute. strace again verified rsyslog was spinning on
the permissions error above.

With rsyslog in this state I issued a `sudo chown syslog:adm
/var/spool/rsyslog` command and immediately cpu usage dropped to
expected levels and the remaining log messages rapidly appeared in
syslog (thousands per minute). With permissions correct I tried flooding
rsyslog again and this time messages went directly to syslog with no
adverse impact on cpu usage.

I am using precise with rsyslog 5.8.6-1ubuntu8.1.

Let me know if there's any other info I can provide or anything else I
can do to help.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to rsyslog in Ubuntu.
https://bugs.launchpad.net/bugs/1075901

Title:
  Incorrect ownership or permissions for spool/work directory

Status in Rsyslog:
  Fix Released
Status in “rsyslog” package in Ubuntu:
  Fix Released
Status in “rsyslog” source package in Precise:
  In Progress

Bug description:
  Impact:
  /var/spool/rsyslog has wrong ownership/permissions

  Test Case:
  ls -ld /var/spool/rsyslog -> should be owned by syslog:adm

  Regression Potential:
  Limited, the old permissions were wrong, it's still a change in behaviour though...

  ---

  Hi,

  By default, the spool/work directory is as follows:

  drwxr-xr-x 2 root root 4096 Mar  8  2012 /var/spool/rsyslog

  This is incorrect as rsyslog drops root privileges and changes to the
  syslog user. As tested, spool files are also written out as the syslog
  user:

  root at juju-canonistack-haw-instance-1:/var/spool/rsyslog# ls -la
  total 16
  drwxrwxrwx 2 syslog adm    4096 Nov  7 09:20 .
  drwxr-xr-x 6 root   root   4096 Nov  7 06:46 ..
  -rw------- 1 syslog syslog 1603 Nov  7 09:20 srvrfwd.00000001
  -rw------- 1 syslog syslog  482 Nov  7 09:20 srvrfwd.qi

  The version of rsyslog used is as follows:

  root at juju-canonistack-haw-instance-1:/var/spool# dpkg -l | grep rsyslog
  ii  rsyslog                          5.8.6-1ubuntu8                 reliable system and kernel logging daemon
  ii  rsyslog-relp                     5.8.6-1ubuntu8                 RELP protocol support for rsyslog

  Could we please have this fixed?

  Thanks,

  Haw

To manage notifications about this bug go to:
https://bugs.launchpad.net/rsyslog/+bug/1075901/+subscriptions