[Bug 1169740] Re: rsyslog hangs loading modules

[Louis Bouchard]

@dave

The main problem I encountered while testing with the provided test
script is that it produces false positives both with previous and
-proposed versions.  Sometimes, the test messages sent using "logger"
makes it to /var/log/syslog after the 'grep' has gone through,
identifying rsyslog as being hung while it is not.  So the script is not
reliable.

When using the following modified script :

#!/bin/sh

while [ 1 ]; do
  service rsyslog stop
  sleep 1
  killall -9 rsyslogd > /dev/null 2>&1
  service rsyslog start
  sleep 1
  nonce=$(date '+%s')
  logger $nonce
  sleep 1
  if grep $nonce /var/log/hourly/* > /dev/null; then
    echo found nonce $nonce
  else
    echo FAIL
    exit 1
  fi
done

The script has not reported any failure when running for > 12 hours on
the current released version which does not have the fix.

My conclusion is that triggering the bug using the modified script is
not possible, hence the mention in comment #24.

So both current released version and the version in -proposed have been
running correctly with my modified test script so I am not able to
reproduce the failure.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to rsyslog in Ubuntu.
https://bugs.launchpad.net/bugs/1169740

Title:
  rsyslog hangs loading modules

Status in “rsyslog” package in Ubuntu:
  Fix Released
Status in “rsyslog” source package in Precise:
  Fix Committed
Status in “rsyslog” source package in Quantal:
  Fix Committed
Status in “rsyslog” source package in Raring:
  Fix Released

Bug description:
  [Impact]
  We have rsyslog configured to listen for forwarded logs on TCP and UDP, and to forward logs on to a central log host. On a number of occasions we have observed rsyslog in a hung state on our production machines -- not fowarding logs and not writing them to disk locally. It doesn't happen all the time, so I believe there's some sort of race condition involved.

  I've attached a stack trace taken when the process was hung. I believe
  the issue is the same as the one discussed here: http://www.gossamer-
  threads.com/lists/rsyslog/users/5873 There is a patch attached to that
  thread.

  [Test Case]
  #!/bin/sh

  while [ 1 ]; do
    service rsyslog stop
    sleep 1
    killall -9 rsyslogd > /dev/null 2>&1
    service rsyslog start
    sleep 1
    nonce=$(date '+%s')
    logger $nonce
    if grep $nonce /var/log/hourly/* > /dev/null; then
      echo found nonce $nonce
    else
      echo FAIL
      exit 1
    fi
  done

  [Regression Potential]
  This is a fix for a deadlock issue it is pretty difficult to determine if fixing a race condition would impact anything else, however, testing from the community hasn't raised any concerns.

  [Additional Info]

  Ubuntu 12.04.1 LTS x86_64
  rsyslog 5.8.6-1ubuntu8

  Here is the (obscured) section of our rsyslog config that I believe is
  causing the issue.

  $ActionQueueType LinkedList
  $ActionQueueFileName central_syslog
  $ActionQueueMaxDiskSpace 256M
  $ActionResumeRetryCount -1
  $ActionQueueSaveOnShutdown on
  $ActionSendTCPRebindInterval 100000
  *.* @@central_syslog.domain:514

  $ModLoad imudp
  $UDPServerRun 514

  $ModLoad imtcp
  $InputTCPServerRun 514

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/rsyslog/+bug/1169740/+subscriptions