Logback with rsyslog

In company I work for, developers don't have any access to Live servers. While it is good from the responsibility division reasons, it makes problem and error tracking painful. Most often we need access to log files without asking somebody from operations team every single time.

To overcome this, we went predictable way of setting up company-wide log aggregation server. While there is plethora of solutions, one is quite easy to setup if you are on Unix/Linux - rsyslog It is widely used to provide standard syslog feature so there is good chance that you have it installed already.

I will not talk much about rsyslog server. It listens usualy on UDP port 514 and has configuration file /etc/rsyslog.conf. Excelent article about installation and basic configuration is here.

Any (client host) rsyslog can be configured to forward it's syslog messages into server rsyslog. This might be right way, when you develop native or RoR application and you log into local syslog, but Java world has it's well established logging libraries, namely Logback or Log4j. Both of them provide appanders capable of sending log events into rsyslog server, while logging into local files as usualy.

Anyway, if you are already logging directly into local syslog, and you need to forward messages into log server, just add to the end of /etc/rsyslog.conf file.

*.* @logs.mycompany.com:514

You can use DNS name or IP addres of your rsyslog server. 514 is default rsyslog port.

To log from java directly to rsyslog server, skipping local syslog completely, add SyslogAppender into logback.xml.
Logback does not provide explicit configuration for TAG aka APP-NAME aka $programname, but it can be emulated by using special syntax of the first word of the log message.
It must start with ' ' (whitespace) and end with ':' (colon) See ' mycoolapp:' in suffixPattern element value. Logback configuration feature ticket exist, but it seems to be abandoned. I don't use Log4J, so here is just a link to article about Log4j and syslog. But I don't think it covers TAG usage.

Back on server, log messages from muliple hosts and applications are written, by default, into single /vsr/log/messages file, making it quite messy. Fortunately rsyslog provides easy way of routing messages coming from different hosts or applications into different files.

You can split messages using hostname or ip address they are coming from, as described here or you can split them on log message content, as described here. You can throw them away too, as described here.

What we choose, is routing messages belonging to same application into single log file. Messages are coming from different hosts because application runs in cluster. Every log line contains hostname, so origin still can be recognized.

if $programname == 'mycoolapp' then /var/log/mycoolapp/mycoolapp.log

To make this work, client must send TAG ($programname) field correctly as described above in logback.xml

Happy rsyslogging