Using Loggly with Java/Tomcat on shared host

Published on Nov 10 2014 in Java Tomcat

You may sometimes need an easy access to your Tomcat logs outside of the hosting server. Loggly is one of the solutions but setting it up on a shared server is not straightforward.

Loggly installation guide assumes you have superuser privileges (VPS/dedicated server) so that Tomcat is installed system-wide and rsyslog.conf can be easily modified. If it is not the case and you are for example using private JVM on shared host read this tutorial to have your Tomcat working with Loggly in a few rather simple (we hope ;) steps.

Basically you will need your private rsyslogd service running on one of your custom ports. Check cPanel - Java Control Panel - Ports for an unused port number or request support to assign you one. The testing environment for the below example was Centos 6 64-bit. User home directory was /home/tomcat - change it in all occurences below to your value.

Preparing for running rsyslog/Loggly in your home directory

Register for Loggly account - it may be trial.
Make sure compiler is available on your account. If not the ask support to enable it.
Make sure outgoing TCP port 514 is open in firewall for the user. Consult with support.

Building rsyslogd

Default rsyslog version in Centos 6 is 5.8.10 so let's use neweset 5.8.x for highest compatibility.

wget http://www.rsyslog.com/download/stable-download/page/files/download/rsyslog/rsyslog-5.8.13.tar.gz
tar xzf rsyslog-5.8.13.tar.gz
cd rsyslog-5.8.13

Imfile module will be needed to trace Tomcat log file. It uses inotify by default. As imfile is off by default we need to enable it in configure line.

./configure --prefix=$HOME --exec-prefix=$HOME --enable-imfile
make
make install
mkdir -p ~/var/{spool/rsyslog,run}
echo 'export PATH=~/sbin:$PATH' >> ~/.bashrc && . ~/.bashrc
which rsyslogd
/home/tomcat/sbin/rsyslogd

Create ~/etc/rsyslog.conf - replace TOKEN with your Loggly token and $WorkDirectory with your own path.

echo '#LOGGLY
# uncomment the below 3 lines if using Log4j + UDP
#$ModLoad imudp.so
#$UDPServerAddress 127.0.0.1
#$UDPServerRun your_custom_port_here
$WorkDirectory /home/tomcat/var/spool/rsyslog
$template LogglyFormat,"<%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [TOKEN@41058 tag=\"MyRsyslogd\"] %msg%\n"
*.* @@logs-01.loggly.com:514;LogglyFormat
' > ~/etc/rsyslog.conf

Do not forget to update paths in all $InputFileName lines (see also below) and TOKEN. Start your own rsyslog with:

rsyslogd -c5 -4 -i ~/var/run/rsyslogd.pid -f ~/etc/rsyslog.conf

There should be no error diplsayed and the process will go to the background. Check for first messages in loggly.

Adjusting Tomcat logging for Loggly

In conf/logging.properties add the 4 rotatable=false lines and remove periods from names like in the following patch:

echo '--- logging.properties.orig 2014-11-10 06:13:32.978341583 -0600
+++ logging.properties 2014-11-10 06:18:09.670432130 -0600
@@ -24,19 +24,23 @@

1catalina.org.apache.juli.FileHandler.level = FINE
1catalina.org.apache.juli.FileHandler.directory = ${catalina.base}/logs
-1catalina.org.apache.juli.FileHandler.prefix = catalina.
+1catalina.org.apache.juli.FileHandler.prefix = catalina
+1catalina.org.apache.juli.FileHandler.rotatable = false

2localhost.org.apache.juli.FileHandler.level = FINE
2localhost.org.apache.juli.FileHandler.directory = ${catalina.base}/logs
-2localhost.org.apache.juli.FileHandler.prefix = localhost.
+2localhost.org.apache.juli.FileHandler.prefix = localhost
+2localhost.org.apache.juli.FileHandler.rotatable = false

3manager.org.apache.juli.FileHandler.level = FINE
3manager.org.apache.juli.FileHandler.directory = ${catalina.base}/logs
-3manager.org.apache.juli.FileHandler.prefix = manager.
+3manager.org.apache.juli.FileHandler.prefix = manager
+3manager.org.apache.juli.FileHandler.rotatable = false

4host-manager.org.apache.juli.FileHandler.level = FINE
4host-manager.org.apache.juli.FileHandler.directory = ${catalina.base}/logs
-4host-manager.org.apache.juli.FileHandler.prefix = host-manager.
+4host-manager.org.apache.juli.FileHandler.prefix = host-manager
+4host-manager.org.apache.juli.FileHandler.rotatable = false

java.util.logging.ConsoleHandler.level = FINE
java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter' > logging.properties.patch
patch -p0 < $CATALINA_HOME/conf/logging.properties.patch

This way you will have catalina.log, catalina.out, host-manager.log, localhost.log and manager.log in your logs directory. Note the names no more contain variable (date) part (after Tomcat restart).

Enable montoring of the files in rsyslog.conf by appending the follwing lines to ~/etc/rsyslog.conf.

echo '
# TOMCAT (logging.properties)

$ModLoad imfile
$InputFilePollInterval 10

#Add a tag for tomcat events
$template LogglyFormatTomcat,"<%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [TOKEN@41058 tag=\"tomcat\"] %msg%\n"

# catalina.log
$InputFileName /home/tomcat/appservers/apache-tomcat-7.0.55/logs/catalina.log
$InputFileTag catalina-log
$InputFileStateFile stat-catalina-log
$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor
if $programname == 'catalina-log' then @@logs-01.loggly.com:514;LogglyFormatTomcat
if $programname == 'catalina-log' then ~

# catalina.out
$InputFileName /home/tomcat/appservers/apache-tomcat-7.0.55/logs/catalina.out
$InputFileTag catalina-out
$InputFileStateFile stat-catalina-out
$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor
if $programname == 'catalina-out' then @@logs-01.loggly.com:514;LogglyFormatTomcat
if $programname == 'catalina-out' then ~

# host-manager.log
$InputFileName /home/tomcat/appservers/apache-tomcat-7.0.55/logs/host-manager.log
$InputFileTag host-manager
$InputFileStateFile stat-host-manager
$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor
if $programname == 'host-manager' then @@logs-01.loggly.com:514;LogglyFormatTomcat
if $programname == 'host-manager' then ~

# localhost.log
$InputFileName /home/tomcat/appservers/apache-tomcat-7.0.55/logs/localhost.log
$InputFileTag localhost-log
$InputFileStateFile stat-localhost-log
$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor
if $programname == 'localhost-log' then @@logs-01.loggly.com:514;LogglyFormatTomcat
if $programname == 'localhost-log' then ~

# manager.log
$InputFileName /home/tomcat/appservers/apache-tomcat-7.0.55/logs/manager.log
$InputFileTag manager
$InputFileStateFile stat-manager
$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor
if $programname == 'manager' then @@logs-01.loggly.com:514;LogglyFormatTomcat
if $programname == 'manager' then ~
' >> ~/etc/rsyslog.conf<<EOF

Do not forget to update paths in all $InputFileName lines and TOKEN. Finally restart your rsyslogd and Tomcat:

kill -9 `pgrep -u $USER rsyslogd`
rsyslogd -c5 -4 -i ~/var/run/rsyslogd.pid -f ~/etc/rsyslog.conf
jr

You are done! See the screenshot below. Messages from Tomcat log should now start coming to Loggly.

loggly_shared_server_tomcat
Another option is logging from Log4j to rsyslogd with UDP and we describe it below.

As for keeping rsyslog up it can either be started/stopped along with Tomcat or we can setup monit task that it will take care of it to be always up.

Switching to UDP and Log4j

To switch to Log4j we will comment out (or remove) all lines below '#TOMCAT' in ~/etc/rsyslog.conf and uncomment the 3 lines at the top of the file:

$ModLoad imudp.so
$UDPServerAddress 127.0.0.1
$UDPServerRun your_custom_port_here

To restart rsyslogd:

kill -9 `pgrep -u $USER rsyslogd`
rsyslogd -c5 -4 -i ~/var/run/rsyslogd.pid -f ~/etc/rsyslog.conf

or to only reread config fie (if rsyslogd is running):

kill -HUP `pgrep -u $USER rsyslogd`

Then copy in Log4j jar (e.g. log4j-1.2.17.jar) into $CATALINA_HOME/lib unless you already have it there or inside WEB-INF/lb of a specific application you want to log for. As we are making the change for whole Tomcat we will also need tomcat-juli-adapters.jar and tomcat-juli.jar from Tomcat extras. Ensure you download them from correct extras'matching your Tomcat version.

cd $CATALINA_HOME/bin
rm -f tomcat-juli.jar
wget http://www.us.apache.org/dist/tomcat/tomcat-7/v7.0.56/bin/extras/tomcat-juli.jar
cd $CATALINA_HOME/lib
wget http://www.us.apache.org/dist/tomcat/tomcat-7/v7.0.56/bin/extras/tomcat-juli-adapters.jar
wget https://archive.apache.org/dist/logging/log4j/1.2.17/log4j-1.2.17.jar
cd $CATALINA_HOME/conf
mv logging.properties logging.properties.bak

Create or modify log4j.properties - we will use fresh Tomcat-wide version of the file placed in lib directory.

echo 'log4j.rootLogger=INFO, SYSLOG

log4j.appender.SYSLOG=org.apache.log4j.net.SyslogAppender
log4j.appender.SYSLOG.SyslogHost=localhost:10243
log4j.appender.SYSLOG.Facility=Local3
log4j.appender.SYSLOG.Header=true
log4j.appender.SYSLOG.layout=org.apache.log4j.PatternLayout
log4j.appender.SYSLOG.layout.ConversionPattern=java %d{ISO8601} %p %t %c{1}.%M - %m%n
' > $CATALINA_HOME/lib/log4j.properties

The above example only ensures logging to Loggly. Read more here about fully replacing Tomcat logging mechanism with Log4j at http://tomcat.apache.org/tomcat-7.0-doc/logging.html#Using_Log4j if you also want to have logs preserved in Tomcat's log directory and displayed on console (standard output).

Restart Tomcat and you should see new logs in Loggly.

Notes

Instead of using '-i' switch to indicated pidfile path you could update sources to have pid file in your home directory with this patch.

cat > syslogd.c.patch<<EOF 
--- tools/syslogd.c.orig 2014-11-10 05:53:45.304901299 -0600
+++ tools/syslogd.c 2014-11-10 04:31:58.790985752 -0600
@@ -195,6 +195,9 @@
# endif
#endif

+#define _PATH_VARRUN "/home/tomcat/var/run/"
+#define _PATH_LOGPID _PATH_VARRUN "rsyslogd.pid"
+
#ifndef _PATH_TTY
# define _PATH_TTY "/dev/tty"
#endif
EOF
patch -p0 < syslogd.c.patch

To test UDP logging you may use newer version of 'logger' utility that supports -P (port) switch or use a simple Perl script:

#!/usr/bin/perl -w
use strict;
use Log::Syslog::Fast ':all';
my $logger = Log::Syslog::Fast->new(LOG_UDP, "127.0.0.1", your_custom_port_here, LOG_LOCAL0, LOG_INFO, "myserver", "logger2");
$logger->send("log message", time);

Articles