Centralizing servers and applications logs is a good way to make search and monitoring easier. Coupled with a web-interface it simplifies access (think also about sharing logs…)

Moreover, as all my servers are based on arm mini-pc (cubieboard) with flash drive, Iogs are volatile (tmpfs) to maximize SdCards life. So centralizing logs on a harddrive is a must to keep history in case of failure.

Below is the recipe of how I did it on all my servers (cubieboards  + debian testing)

RSYSLOG Server installation

Centralizing implies having a specific server for that (or severals if you need failover). I choosed to store logs in mysql so its easier to query / view / use a web interface…

Packages installation

apt-get install rsyslog rsyslog-mysql mysql-server mysql-client

You will get prompted to change the root password of mysql (via dpkg-reconfigure)

The same for rsyslog-mysql plugin : give root mysql password, then enter a password for rsyslog user or let the system choose a random one…

Configure rsyslog

Edit /etc/rsyslog.conf to activate udp and tcp connections for (futur) rsyslog clients :

# provides UDP syslog reception
 $ModLoad imudp
 $UDPServerRun 514
 
# provides TCP syslog reception 
$ModLoad imtcp 
$InputTCPServerRun 514

Then (re)start rsyslog

$ /etc/init.d/rsyslog restart
[ ok ] Stopping enhanced syslogd: rsyslogd.
[ ok ] Starting enhanced syslogd: rsyslogd.

Install and configure the web interface : loganalyser

LAMP stack

loganalyser is written in PHP and officialy support apache. To keep things simple and straighforward, let use a classical LAMP architecture. For security reasons, it is advised to make the log server only available from the lan and/or for a restricted ip range (and for professional use or big network, I would also add through https + authentication).

apt-get install libapache2-mod-php5
apt-get install php5-mysql # install mysql extension for php (needed by loganalyser)
apt-get install php-apc # install APC to make php faster (opcode cache)
apt-get install php5-gd # install gd extension for graphics in loganalyser

loganalyser deployment

cd /var/www
wget http://download.adiscon.com/loganalyzer/loganalyzer-3.6.4.tar.gz
tar -zxf loganalyzer-3.6.4.tar.gz
ln -s loganalyzer-3.6.4 loganalyzer
rm loganalyzer-3.6.4.tar.gz
chown -R www-data:www-data loganalyser/src

Note that we need to temporary allow write access to apache for the installation process…

Apache configuration

Now that loganalyser is deployed, we need to configure a virtualhost in apache.
Create the conf file in /etc/apache2/sites-available/loganalyser.conf

<Directory /var/www/loganalyser/src>
   Options FollowSymLinks
   AllowOverride All
   Order allow,deny
   Allow from <your lan ip address range>
</Directory>
 
<VirtualHost *:80>
  ServerAdmin webmaster@localhost
  ServerName foo.bar.com
  DocumentRoot /var/www/loganalyzer/src
  ErrorLog ${APACHE_LOG_DIR}/error.log
  CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>

Then go to /etc/apache/site-enabled, remove the default host and activate loganalyser one

rm 000-default.conf
ln -s ../sites-available/loganalyser.conf .
/etc/init.d/apache2 restart

LogAnalyser configuration

Initial settings

Point your browser to the server ip adress and follow the instructions (which begin by a “fatal error screen”…)
Everything is explained in the installation doc : http://loganalyzer.adiscon.com/doc/install.html

Main choice I set (personal taste…) :

• Message character limit for the main view => 0 to view full message inline
• Show message details popup => no (because I want-it inline)
• Enable User Database => yes I use the same database as rsyslog for loganalyser own table

Rsysllog database informations are found in /etc/rsyslog.d/mysql.conf, it look-likes the following :

$ModLoad ommysql
 *.* :ommysql:localhost,Syslog,<user>,<password>

If all went fine, the installer will create tables prefixed with “logcon_” (if you did not modified this) and will ask to you create a user.

Configure a first source

LogAnaylser need to know from where it should read logs to display. For now, we just want to check that everything that will be logged will be displayed, so we create a simple source :

• Name = “All”
• Type = PDO
• View = Syslog
• Databse storage engine => MySQL
• table type = monitorWare
• Fill host, tablename, user and password according to /etc/rsyslog.d/mysql.conf (beware, all is case sensitive)

Ok, we have now a base installation that should already display the server’s logs. Don’t forget to remove write access to user www-data to /var/www/loganalyser/src.

MySQL tunning

As the logserver could receive an huge amount of data to write into tables, you should ajust mysql parameters after a fews days / weeks of usage.

Basicaly :

• use “mysqltunner” to check settings from time to time
• drefagment / optimize table at least once a week (mysqlcheck -o –all-databases -u root -pxxxxxxx)

Configures other servers to send their logs to the log server

Note that for each server and/or service, you can create a dedicated source in LogAnalyser to make browsing and searching easier…

Base logging : replace syslog by rsyslog

This part is the easiest one. Firstly, install rsyslog :

apt-get install rsyslog

Then, put the following at the end of /etc/rsyslog.conf:

*.* @ip.adress.of.rsyslog.server

Restart rsyslog (or kill -HUP), and then every message for syslog will be sent to the log server.

Sending Apache log to rsylog

Apache use a proprietary log system that write on its own files. To send them to rsyslog we need to use the “imfile” module of rsyslog.

Edit /etc/rsyslog.conf to add file support :

$ModLoad imfile # provide file support

Then, create a configuration file for apache (I personnaly create one file per virtualhost), in /etc/rsyslog.d/apache_<vhostname>.conf (You can find more information on http://www.rsyslog.com/using-the-text-file-input-module)

########## Error loggin ###########
$InputFileName /var/log/apache2/<filename_of_your_error_log>
$InputFileTag apache-<vhostname>
$InputFileStateFile apache_<vhostname>_error
$InputFileSeverity error
$InputFileFacility local1
$InputRunFileMonitor
 
######### Access loggin ##########
$InputFileName /var/log/apache2/<filename_of_your_access_log>
$InputFileTag apache-<vhostname>
$InputFileStateFile apache_<vhostname>_access
$InputFileSeverity info
$InputFileFacility local2
$InputRunFileMonitor

Set Exim4 to log in syslog

Edit  /etc/exim4/exim4.conf.template, and add the folowing after “main/02_exim4-config_options”

log_file_path = syslog

Then restart exim (/etc/init.d/exim4 restart) and check if its ok :

exim4 -bP log_file_path

You should read “log_file_path = syslog”

If it does not work, try to put the “log_file_path” setting in “etc/exim4/conf.d/main/02_exim4-config_options”