[Bug 1169740] Re: rsyslog hangs loading modules

[Jon Tai]

Sorry for not commenting on this sooner. For what it's worth, we were
definitely able to reproduce this bug on our production infrastructure
before, and we've been running Adam's PPA version of this package for
several weeks now without issue. Since I've already reported these
findings and I'm the original reporter of the bug, I'm not sure that
qualifies the package to be verification-done.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to rsyslog in Ubuntu.
https://bugs.launchpad.net/bugs/1169740

Title:
  rsyslog hangs loading modules

Status in “rsyslog” package in Ubuntu:
  Fix Released
Status in “rsyslog” source package in Precise:
  Fix Committed
Status in “rsyslog” source package in Quantal:
  Fix Committed
Status in “rsyslog” source package in Raring:
  Fix Released

Bug description:
  [Impact]
  We have rsyslog configured to listen for forwarded logs on TCP and UDP, and to forward logs on to a central log host. On a number of occasions we have observed rsyslog in a hung state on our production machines -- not fowarding logs and not writing them to disk locally. It doesn't happen all the time, so I believe there's some sort of race condition involved.

  I've attached a stack trace taken when the process was hung. I believe
  the issue is the same as the one discussed here: http://www.gossamer-
  threads.com/lists/rsyslog/users/5873 There is a patch attached to that
  thread.

  [Test Case]
  #!/bin/sh

  while [ 1 ]; do
    service rsyslog stop
    sleep 1
    killall -9 rsyslogd > /dev/null 2>&1
    service rsyslog start
    sleep 1
    nonce=$(date '+%s')
    logger $nonce
    if grep $nonce /var/log/hourly/* > /dev/null; then
      echo found nonce $nonce
    else
      echo FAIL
      exit 1
    fi
  done

  [Regression Potential]
  This is a fix for a deadlock issue it is pretty difficult to determine if fixing a race condition would impact anything else, however, testing from the community hasn't raised any concerns.

  [Additional Info]

  Ubuntu 12.04.1 LTS x86_64
  rsyslog 5.8.6-1ubuntu8

  Here is the (obscured) section of our rsyslog config that I believe is
  causing the issue.

  $ActionQueueType LinkedList
  $ActionQueueFileName central_syslog
  $ActionQueueMaxDiskSpace 256M
  $ActionResumeRetryCount -1
  $ActionQueueSaveOnShutdown on
  $ActionSendTCPRebindInterval 100000
  *.* @@central_syslog.domain:514

  $ModLoad imudp
  $UDPServerRun 514

  $ModLoad imtcp
  $InputTCPServerRun 514

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/rsyslog/+bug/1169740/+subscriptions