Systemd Journal Rate Limiting

How do we know if the journal limits are actually causing us to drop log messages? Generally you will see similar messages in the log files as below.

Jan  9 09:18:07 server1 journal: Suppressed 7124 messages from /system.slice/named.service

In this particular case we have a DNS server running Bind which is logging all DNS queries. 7124 messages were suppressed and dropped (not logged) because they were coming in too fast in this example.

By default systemd allows 1,000 messages within a 30 second period.

The limits are controlled in the /etc/systemd/journald.conf file.

RateLimitInterval=30s
RateLimitBurst=1000

If more messages than the amount specified in RateLimitBurst are received within the time defined by RateLimitInterval, all further messages within the interval are dropped until the interval is over.

You can modify these values as you see fit, you can completely disable systemd journal logging rate limiting by setting both to 0.

If you make any changes to /etc/systemd/journald.conf you will need to restart the systemd-journald service to apply the changes.

systemctl restart systemd-journald

Rsyslog Rate Limiting

The systemd journal limit is hit before any default rsyslog limits as its default limits are smaller. By default rsyslog will accept 20,000 messages within a 10 minute period.

Therefore if you increase the rate limiting of the systemd journal logging as shown above you may then start to receive similar messages in your syslog logs as shown below.

....
Jan  9 22:42:35 server1 rsyslogd-2177: imjournal: begin to drop messages due to rate-limiting
Jan  9 22:51:26 server1 rsyslogd-2177: imjournal: 143847 messages lost due to rate-limiting
...

The first message states that messages will be dropped as the limit has been reached, and once the interval is over (after 10 minutes by default) the amount of messages that were lost due to rate limiting will then be logged.

The limits are controlled in the /etc/rsyslog.conf file.

$ModLoad imjournal
$imjournalRatelimitInterval 600
$imjournalRatelimitBurst 20000