[Bug 2073628] Re: imjournal module works with rsyslog package of ubuntu 22.04 but not with ubuntu 24.04
Kent Spillner
2073628 at bugs.launchpad.net
Wed Oct 16 15:10:16 UTC 2024
I think we might be encountering the same issue. At least, we're also
trying to enable imjournal in rsyslog because we want all of the
structured log fields from systemd journal, and we're encountering the
same error messages when starting rsyslog.service.
We are running an x86 EC2 instance:
$ uname -a
Linux ip-10-XXX-YYY-ZZZ 6.8.0-1016-aws #17-Ubuntu SMP Mon Sep 2 13:48:07 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 24.04.1 LTS
Release: 24.04
Codename: noble
$ dpkg -l rsyslog
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-=================-============-=========================================
ii rsyslog 8.2312.0-3ubuntu9 amd64 reliable system and kernel logging daemon
I can also confirm that there are messages related to AppArmor denying rsyslog at approximately the same time in our dmesg:
[Wed Oct 16 11:15:39 2024] audit: type=1400 audit(1729077335.160:679): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="rsyslogd" pid=506096 comm="apparmor_parser"
[Wed Oct 16 11:15:39 2024] audit: type=1400 audit(1729077335.187:680): apparmor="DENIED" operation="open" class="file" profile="rsyslogd" name="/run/log/journal/" pid=506098 comm="rsyslogd" requested_mask="r" denied_mask="r" fsuid=102 ouid=0
[Wed Oct 16 11:15:39 2024] audit: type=1400 audit(1729077335.187:681): apparmor="DENIED" operation="open" class="file" profile="rsyslogd" name="/etc/machine-id" pid=506098 comm="rsyslogd" requested_mask="r" denied_mask="r" fsuid=102 ouid=0
[Wed Oct 16 11:15:39 2024] audit: type=1400 audit(1729077335.187:682): apparmor="DENIED" operation="open" class="file" profile="rsyslogd" name="/run/log/journal/" pid=506098 comm="rsyslogd" requested_mask="r" denied_mask="r" fsuid=102 ouid=0
[Wed Oct 16 11:15:39 2024] audit: type=1400 audit(1729077335.187:683): apparmor="DENIED" operation="open" class="file" profile="rsyslogd" name="/etc/machine-id" pid=506098 comm="rsyslogd" requested_mask="r" denied_mask="r" fsuid=102 ouid=0
[Wed Oct 16 11:15:39 2024] audit: type=1400 audit(1729077335.187:684): apparmor="DENIED" operation="open" class="file" profile="rsyslogd" name="/run/log/journal/" pid=506098 comm="in:imjournal" requested_mask="r" denied_mask="r" fsuid=102 ouid=0
[Wed Oct 16 11:15:39 2024] audit: type=1400 audit(1729077335.187:685): apparmor="DENIED" operation="open" class="file" profile="rsyslogd" name="/etc/machine-id" pid=506098 comm="in:imjournal" requested_mask="r" denied_mask="r" fsuid=102 ouid=0
[Wed Oct 16 11:15:39 2024] audit: type=1400 audit(1729077335.187:686): apparmor="DENIED" operation="open" class="file" profile="rsyslogd" name="/run/log/journal/" pid=506098 comm="in:imjournal" requested_mask="r" denied_mask="r" fsuid=102 ouid=0
[Wed Oct 16 11:15:39 2024] audit: type=1400 audit(1729077335.187:687): apparmor="DENIED" operation="open" class="file" profile="rsyslogd" name="/etc/machine-id" pid=506098 comm="in:imjournal" requested_mask="r" denied_mask="r" fsuid=102 ouid=0
[Wed Oct 16 11:15:39 2024] audit: type=1400 audit(1729077335.192:688): apparmor="DENIED" operation="open" class="file" profile="rsyslogd" name="/run/log/journal/" pid=506098 comm="in:imjournal" requested_mask="r" denied_mask="r" fsuid=102 ouid=0
As you may notice the rsyslog service itself is logging that it can't
create the systemd journal state file under /var/spool/rsyslog, but it
appears AppArmor is actually preventing rsyslog & imjournal from reading
/run/log/journal/ and /etc/machine-id.
I tried stopping and disabling AppArmor, and I also tried symlinking
/etc/apparmor.d/usr.sbin.rsyslog from /etc/apparmor.d/disable/ and
running apparmor_parser -R /etc/apparmor.d/usr.sbin.rsyslog, and
confirmed /usr/sbin/rsyslog was not being enforced by running aa-status.
However, that did NOT allow rsyslog & imjournal to work as now imjournal
is segfaulting:
[Wed Oct 16 11:50:35 2024] in:imjournal[516014]: segfault at 40 ip 000058bd6b96eb21 sp 000071bcd45ff9e0 error 6 in rsyslogd[58bd6b93f000+6f000] likely on CPU 1 (core 0, socket 0)
[Wed Oct 16 11:50:35 2024] Code: b7 10 66 41 89 56 08 0f b6 40 02 41 88 46 0a e9 3f fe ff ff e8 b0 1f fd ff f3 0f 1e fa 55 48 89 e5 41 54 49 89 fc 53 48 8b 1f <f0> 83 6b 40 01 0f 85 c8 01 00 00 48 8b 7b 70 48 8d 83 50 01 00 00
[Wed Oct 16 11:50:51 2024] rs:main Q:Reg[516078]: segfault at 0 ip 000055e61b25f3d0 sp 000079c6479ff5e8 error 4 in rsyslogd[55e61b225000+6f000] likely on CPU 1 (core 0, socket 0)
[Wed Oct 16 11:50:51 2024] Code: 01 4c 63 c0 41 89 c1 4d 69 c0 ab aa aa 2a 41 c1 f9 1f 49 c1 f8 21 45 29 c8 47 8d 04 40 41 c1 e0 02 44 29 c0 48 98 48 8b 04 c2 <0f> b6 00 88 01 0f be 47 01 83 e8 01 4c 63 c0 41 89 c1 4d 69 c0 ab
[Wed Oct 16 11:50:51 2024] in:imjournal[516144]: segfault at 7a160c000090 ip 00007a160c000090 sp 00007a16415ff9c8 error 15 likely on CPU 1 (core 0, socket 0)
[Wed Oct 16 11:50:51 2024] Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <a0> de 00 0c 16 7a 00 00 40 c4 00 0c 16 7a 00 00 d0 61 00 0c 16 7a
[Wed Oct 16 11:50:52 2024] in:imjournal[516155]: segfault at 73f1f40054b0 ip 000073f1f40054b0 sp 000073f23e3ff878 error 15 likely on CPU 0 (core 0, socket 0)
[Wed Oct 16 11:50:52 2024] Code: 00 00 e0 8f 00 f4 f1 73 00 00 10 01 00 00 00 00 00 00 24 00 00 00 00 00 00 00 45 13 1f cb f6 73 00 00 45 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 11 01 00 00 00 00 00 00 40 45 00 f4 f1 73
This should be pretty easy to reproduce as I can trigger it with a
minimal config in /etc/rsyslog.d/:
module(load="imjournal" StateFile="systemd_journald_state" IgnorePreviousMessages="on")
module(load="mmjsonparse")
module(load="omfwd")
template(name="systemd_journal_json" type="string" string="%$!all-
json%\n" )
action(type="mmjsonparse")
user.* action(type="omfwd" target="remote-rsyslog" port="514" protocol="tcp" template="systemd_journal_json")
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to rsyslog in Ubuntu.
https://bugs.launchpad.net/bugs/2073628
Title:
imjournal module works with rsyslog package of ubuntu 22.04 but not
with ubuntu 24.04
Status in rsyslog package in Ubuntu:
Incomplete
Bug description:
imjournal module fails to create /var/spool/rsyslog/journal-state file
in ubuntu 24.04, rsyslog version(8.2312.0) x86 and s390x both, but
works well in ubuntu 22.04 , rsyslog version(8.2112.0) x86 and s390x
*******
Ubuntu 24.04 s390x
lsb_release -rd
No LSB modules are available.
Description: Ubuntu 24.04 LTS
Release: 24.04
# apt-cache policy rsyslog
rsyslog:
Installed: 8.2312.0-3ubuntu9
Candidate: 8.2312.0-3ubuntu9
Version table:
*** 8.2312.0-3ubuntu9 500
500 http://ports.ubuntu.com/ubuntu-ports noble/main s390x Packages
100 /var/lib/dpkg/status
Have below line in /etc/rsyslog.conf
module(load="imjournal" fileCreateMode="0666"
PersistStateInterval="999"
StateFile="/var/spool/rsyslog/journal_state")
ul 19 18:39:35 latest-logs systemd[1]: Starting rsyslog.service - System Logging Service...
Jul 19 18:39:35 latest-logs rsyslogd[8647]: rsyslogd's groupid changed to 102
Jul 19 18:39:35 latest-logs rsyslogd[8647]: rsyslogd's userid changed to 102
Jul 19 18:39:35 latest-logs systemd[1]: Started rsyslog.service - System Logging Service.
Jul 19 18:39:35 latest-logs rsyslogd[8647]: [origin software="rsyslogd" swVersion="8.2312.0" x-pid="8647" x-info="https://www.rsyslog.com"] start
Jul 19 18:39:35 latest-logs rsyslogd[8647]: imjournal: No statefile exists, /var/spool/rsyslog/journal_state will be created (ignore if this is first run): No such file or directory >
Jul 19 18:39:35 latest-logs rsyslogd[8647]: imjournal: Journal indicates no msgs when positioned at head. [v8.2312.0 try https://www.rsyslog.com/e/0 ]
Jul 19 18:39:35 latest-logs rsyslogd[8647]: imjournal: journal files changed, reloading... [v8.2312.0 try https://www.rsyslog.com/e/0 ]
Jul 19 18:39:35 latest-logs rsyslogd[8647]: imjournal: No statefile exists, /var/spool/rsyslog/journal_state will be created (ignore if this is first run): No such file or directory >
Jul 19 18:39:35 latest-logs rsyslogd[8647]: imjournal: Journal indicates no msgs when positioned at head. [v8.2312.0 try https://www.rsyslog.com/e/0 ]
lines 1-25/25 (END)
FIle /var/spool/rsyslog/journal_state should have created and logs
should have redirected to rsyslog server
******
In Ubuntu 22.04 all is working as expected
# lsb_release -rd
Description: Ubuntu 22.04.4 LTS
Release: 22.04
#apt-cache policy rsyslog
rsyslog:
Installed: 8.2112.0-2ubuntu2.2
Candidate: 8.2112.0-2ubuntu2.2
Version table:
*** 8.2112.0-2ubuntu2.2 100
100 /var/lib/dpkg/status
Use the same line as above in /etc/rsyslog.conf
restart service. it did gave error about fileCreateMode which got
ignored and proceeded to create the journal-state file and continued
without any error
Jul 19 18:44:37 systemd[1]: Starting System Logging Service...
Jul 19 18:44:37 rsyslogd[13664]: error during parsing file /etc/rsyslog.conf, on or before line 16: parameter 'fileCreateMode' not known -- typo in co>
Jul 19 18:44:37 systemd[1]: Started System Logging Service.
Jul 19 18:44:37 rsyslogd[13664]: rsyslogd's groupid changed to 111
Jul 19 18:44:37 rsyslogd[13664]: rsyslogd's userid changed to 104
Jul 19 18:44:37 rsyslogd[13664]: [origin software="rsyslogd" swVersion="8.2112.0" x-pid="13664" x-info="https://www.rsyslog.com"] start
Jul 19 18:44:37 rsyslogd[13664]: imjournal: journal files changed, reloading... [v8.2112.0 try https://www.rsyslog.com/e/0 ]
/var/spool/rsyslog# ls
journal_state
*****
please help with this issue
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/rsyslog/+bug/2073628/+subscriptions
More information about the foundations-bugs
mailing list