NXLOG Community Edition Reference Manual for v2.8.1248

Will be set to a syslog formatted string after to_syslog_bsd() or to_syslog_ietf() is called.

$Message

Type

The message part of the syslog line, filled after parse_syslog_bsd() or parse_syslog_ietf() is called.

$SyslogSeverityValue

Type

The severity part of the syslog line, filled after parse_syslog_bsd() or parse_syslog_ietf() is called. The default severity is 5 (="notice").

$SyslogSeverity

Type

The severity part of the syslog line, filled after parse_syslog_bsd() or parse_syslog_ietf() is called. The default severity is "notice".

$SeverityValue

Type

Normalized severity number of the event.

$Severity

Type

Normalized severity name of the event.

$SyslogFacilityValue

Type

The facility part of the syslog line, filled after parse_syslog_bsd() or parse_syslog_ietf() is called. The default facility is 1 (="user").

$SyslogFacility

Type

The facility part of the syslog line, filled after parse_syslog_bsd() or parse_syslog_ietf() is called. The default facility is "user".

$EventTime

Type

Will be set to the timestamp found in the syslog message after parse_syslog_bsd() or parse_syslog_ietf() is called. If the year value is missing, it is set to the current year.

$Hostname

Type

The hostname part of the syslog line, filled after parse_syslog_bsd() or parse_syslog_ietf() is called.

$SourceName

Type

The application/program part of the syslog line, filled after parse_syslog_bsd() or parse_syslog_ietf() is called.

$MessageID

Type

The MSGID part of the syslog message, filled after parse_syslog_ietf() is called.

$ProcessID

Type

The process id in the syslog line, filled after parse_syslog_bsd() or parse_syslog_ietf() is called.

Configuration examples

Example 6.20. Sending a file as BSD syslog over UDP

To send logs out in BSD syslog format over udp which are collected from files, use the to_syslog_bsd() procedure coupled with the om_udp module as in the following example.

<Extension syslog>
    Module	xm_syslog
</Extension>

<Input in>
    Module	im_file

    # We monitor all files matching the wildcard.
    # Every line is read into the $raw_event field.
    File	"/var/log/app*.log"

    # Set the $EventTime field usually found in the logs by extracting it with a regexp.
    # If this is not set, the current system time will be used which might be a little off.
    Exec	if $raw_event =~ /(\d\d\d\d\-\d\d-\d\d \d\d:\d\d:\d\d)/ $EventTime = parsedate($1);

    # Now set the severity to something custom. This defaults to 'INFO' if unset.
    Exec	if $raw_event =~ /ERROR/ $Severity = 'ERROR'; \
                else $Severity = 'INFO';

    # The facility can be also set, otherwise the default value is 'USER'.
    Exec	$SyslogFacility = 'AUDIT';

    # The SourceName field is called the TAG in RFC3164 terminology and is usually the process name.
    Exec	$SourceName = 'my_application';

    # It is also possible to rewrite the Hostname if you don't want to use the system's hostname.
    Exec	$Hostname = 'myhost';

    # The Message field is used if present, otherwise the current $raw_event is prepended with the
    # syslog headers.
    # You can do some modifications on the Message if required. Here we add the full path of the
    # source file to the end of message line.
    Exec	$Message = $raw_event + ' [' + file_name() + ']';

    # Now create our RFC3164 compliant syslog line using the fields set above and/or use sensible
    # defaults where possible. The result will be in $raw_event.
    Exec	to_syslog_bsd();
</Input>

<Output out>
    # This module just sends the contents of the $raw_event field to the destination defined here,
    # one UDP packet per message.
    Module	om_udp
    Host	192.168.1.42
    Port	1514
</Output>

<Route 66>
    Path	in => out
</Route>

Example 6.21. Collecting BSD style syslog messages over UDP

To collect BSD style syslog messages over UDP, use the parse_syslog_bsd() procedure coupled with the im_udp module as in the following example.

<Extension syslog>
    Module	xm_syslog
</Extension>

<Input in>
    Module	im_udp
    Host	0.0.0.0
    Port 	514
    Exec	parse_syslog_bsd();
</Input>

<Output out>
    Module	om_file
    File	"/var/log/logmsg.txt"
</Output>

<Route 1>
    Path	in => out
</Route>

Example 6.22. Collecting IETF style syslog messages over UDP

To collect IETF style syslog messages over UDP as defined by RFC 5424 and RFC 5426, use the parse_syslog_ietf() procedure coupled with the im_udp module as in the following example. Note that the default port is 514 (as defined by RFC 5426), this is the same as for BSD syslog.

<Extension syslog>
    Module	xm_syslog
</Extension>

<Input in>
    Module	im_udp
    Host	0.0.0.0
    Port 	514
    Exec	parse_syslog_ietf();
</Input>

<Output out>
    Module	om_file
    File	"/var/log/logmsg.txt"
</Output>

<Route 1>
    Path	in => out
</Route>

Example 6.23. Collecting both IETF and BSD style syslog messages over the same UDP port

To collect IETF and BSD style syslog messages over UDP, use the parse_syslog() procedure coupled with the im_udp module as in the following example. This procedure is capable of detecting and parsing both syslog formats. Since 514 is the default UDP port number for both BSD and IETF syslog, this can be useful to collect both formats simultaneously. If you want to accept both formats on different ports then it makes sense to use the appropriate parsers as in the previous two examples.

<Extension syslog>
    Module	xm_syslog
</Extension>

<Input in>
    Module	im_udp
    Host	0.0.0.0
    Port 	514
    Exec	parse_syslog();
</Input>

<Output out>
    Module	om_file
    File	"/var/log/logmsg.txt"
</Output>

<Route 1>
    Path	in => out
</Route>

Example 6.24. Collecting IETF style syslog messages over TLS/SSL

To collect IETF style syslog messages over TLS/SSL as defined by RFC 5424 and RFC 5425, use the parse_syslog_ietf() procedure coupled with the im_ssl module as in the following example. Note that the default port is 6514 in this case (as defined by RFC 5425). The payload format parser is handled by the Syslog_TLS input reader.

<Extension syslog>
    Module	xm_syslog
</Extension>

<Input in>
    Module	im_ssl
    Host	localhost
    Port	6514
    CAFile	%CERTDIR%/ca.pem
    CertFile	%CERTDIR%/client-cert.pem
    CertKeyFile	%CERTDIR%/client-key.pem
    KeyPass	secret
    InputType	Syslog_TLS
    Exec	parse_syslog_ietf();
</Input>

<Output out>
    Module	om_file
    File	"/var/log/logmsg.txt"
</Output>

<Route 1>
    Path	in => out
</Route>

Example 6.25. Forwarding IETF syslog over TCP

The following configuration uses the to_syslog_ietf() procedure to convert input to IETF syslog and forward it over TCP:

<Extension syslog>
    Module	xm_syslog
</Extension>

<Input in>
    Module	im_file
    File	"/var/log/input.txt"
    Exec	$TestField = "test value"; $Message = $raw_event;
</Input>

<Output out>
    Module	om_tcp
    Host	127.0.0.1
    Port	1514
    Exec	to_syslog_ietf();
    OutputType	Syslog_TLS
</Output>

<Route 1>
    Path	in => out
</Route>

Because of the Syslog_TLS framing, the raw data sent over TCP will look like the following:

130 <13>1 2012-01-01T16:15:52.873750Z  - - - [NXLOG@14506 EventReceivedTime="2012-01-01 17:15:52" TestField="test value"] test message

This example shows that all fields - except those which are filled by the syslog parser - are added to the structured data part.

Example 6.26. Conditional rewrite of the syslog facility - version 1

<Extension syslog>
    Module	xm_syslog
</Extension>

<Input in>
    Module	im_udp
    Port 	514
    Host	0.0.0.0
    Exec	parse_syslog_bsd();
</Input>

<Output fileout>
    Module	om_file
    File	"/var/log/logmsg.txt"
    Exec	if $Message =~ /error/ $SeverityValue = syslog_severity_value("error");
    Exec	to_syslog_bsd();
</Output>

<Route 1>
    Path	in => fileout
</Route>

Example 6.27. Conditional rewrite of the syslog facility - version 2

The following example does almost the same thing as the previous example, except that the syslog parsing and rewrite is moved to a processor module and rewrite only occurs if the facility was modified. This can make it work faster on multi-core systems because the processor module runs in a separate thread. This method can also minimize UDP packet loss because the input module does not need to parse syslog messages and can process UDP packets faster.

<Extension syslog>
    Module	xm_syslog
</Extension>

<Input in>
    Module	im_udp
    Host	0.0.0.0
    Port 	514
</Input>

<Processor rewrite>
    Module	pm_null
    Exec	parse_syslog_bsd();\
		if $Message =~ /error/ \
                {\
                  $SeverityValue = syslog_severity_value("error");\
                  to_syslog_bsd(); \
                }
</Processor>

<Output fileout>
    Module	om_file
    File	"/var/log/logmsg.txt"
</Output>

<Route 1>
    Path	in => rewrite => fileout
</Route>

External program execution (xm_exec)

This module provides two procedures which make it possible to execute external scripts or programs. The reason for providing these two procedures through this additional extension module is to keep the nxlog core small. A security advantage is that an administrator won't be able to execute arbitrarly scripts if this module is not loaded.

Note

The om_exec and im_exec modules also provide support for running external programs, though the purpose of these is to pipe data to and read data from programs. The procedures provided by the xm_exec module do not pipe log message data, these are intended for multiple invocations. Though data can be still passed to the executed script/program as command line arguments.

Functions and procedures exported by xm_exec

Procedures exported by xm_exec

exec(string command, varargs args);

description

Execute the command passing it the supplied arguments and wait for it to terminate. The command is executed in the caller module's context. Note that the module calling this procedure will block until the process terminates. Use the exec_async() procedure to avoid this problem. All output written to STDOUT and STDERR by the spawned process is discarded.

arguments

command: type: string
args: type: varargs

exec_async(string command, varargs args);

description

This procedure executes the command passing it the supplied arguments and does not wait for it to terminate.

arguments

command: type: string
args: type: varargs

Configuration examples

Example 6.28. nxlog acting as a cron daemon

<Extension exec>
    Module	xm_exec
    <Schedule>
	Every	1 sec
	Exec	exec_async("/bin/true");
    </Schedule>
</Extension>

Example 6.29. Sending email alerts

<Extension exec>
    Module	xm_exec
</Extension>

<Input in>
    Module	im_tcp
    Host	0.0.0.0
    Port	1514
    Exec	if $raw_event =~ /alertcondition/ {                                                    \
                   exec_async("/bin/sh", "-c", 'echo "' + $Hostname + '\n\nRawEvent:\n' + $raw_event + \
                           '"|/usr/bin/mail -a "Content-Type: text/plain; charset=UTF-8" -s "ALERT" '  \
                           + 'user@domain.com' );                                                      \
                }
</Input>

<Output out>
    Module	om_file
    File	"/var/log/messages"
</Output>

<Route r>
    Path	in => out
</Route>

For another example see this configuration for file rotation.

Perl (xm_perl)

This module makes it possible to execute perl code and process event data using the perl language via a built-in perl interpreter. The perl interpreter is only loaded if the module is declared in the configuration. While the nxlog language is already a powerful framework, it is not intended to be a full featured programming language. For example it does not provide lists, arrays, hashes and other features available in many high-level languages. Perl is widely used for log processing and it comes with a broad set of modules bundled or available from CPAN.

You can sometimes write faster code in C, but you can always write code faster in Perl. Code written in perl is also a lot safer because it is unlikely to crash. Exceptions in the perl code (croak/die) are handled properly and this will only result in an unfinished attempt at executing the log processing code but will not take down the whole nxlog process.

The module will parse the file specified in the PerlCode directive when nxlog and the module is started. This file should contain one or more methods which can be called from the Exec directive of any module which wants to do any log processing in perl. See the example below which illustrates the use of this module.

To acccess the fields and the event data from the perl code, you need to include the Log::Nxlog module. This exports the following methods:

set_field_integer(event, key, value): Sets the integer value in the field named 'key'.
set_field_string(event, key, value): Sets the string value in the field named 'key'.
set_field_boolean(event, key, value): Sets the boolean value in the field named 'key'.
get_field(event, key): Retreive the value associated with the field named 'key'. The method returns a scalar value if the key exist and the value is defined, otherwise it returns undef.
delete_field(event, key): Delete the value associated with the field named 'key'.
field_type(event, key): Return a string representing the type of the value associated with the field named 'key'.
field_names(event): Return a list of the field names contained in the event data. Can be used to iterate over all the fields.
log_debug(msg): Send the message in the argument to the internal logger on DEBUG loglevel. Does the same as the procedure named log_debug() in nxlog.
log_info(msg): Send the message in the argument to the internal logger on INFO loglevel. Does the same as the procedure named log_info() in nxlog.
log_warning(msg): Send the message in the argument to the internal logger on WARNING loglevel. Does the same as the procedure named log_warning() in nxlog.
log_error(msg): Send the message in the argument to the internal logger on ERROR loglevel. Does the same as the procedure named log_error() in nxlog.

You should be able to read the POD documentation contained in Nxlog.pm with perldoc Log::Nxlog.

Configuration

The following directives can be used to configure the xm_perl module instance:

PerlCode: This mandatory directive expects a file which contains valid perl code. This file is read and parsed by the perl interpreter. Methods defined in this file can be called with the call() procedure.

Functions and procedures exported by xm_perl

Procedures exported by xm_perl

call(string subroutine);

description

Calls the perl subroutine provided in the first argument.

arguments

subroutine: type: string

perl_call(string subroutine);

description

Calls the perl subroutine provided in the first argument.

arguments

subroutine: type: string

Configuration examples

In this example logs are parsed as syslog then the data is passed to a perl method which does a GeoIP lookup on the source address of the incoming message.

Example 6.30. Using the built-in perl interpreter

<Extension syslog>
    Module	xm_syslog
</Extension>

<Extension perl>
    Module	xm_perl
    PerlCode	modules/extension/perl/processlogs.pl
</Extension>

<Input in>
    Module	im_file
    File	'test.log'
    ReadFromLast FALSE
    SavePos	FALSE
</Input>

<Output out>
    Module	om_file
    File	'tmp/output'
    # First we parse the input natively from nxlog
    Exec	parse_syslog_bsd();
    # Now call the 'process' subroutine defined in 'processlogs.pl'
    Exec	perl_call("process");
    # You can also invoke this public procedure 'call' in case
    # of multiple xm_perl instances like this: 
    # Exec	perl->call("process");
</Output>

<Route 1>
    Path	in => out
</Route>

The contents of the processlogs.pl perl script is as follows:

use strict;
use warnings;

use Carp;
# FindBin is for adding a path to @INC, this not needed normally
use FindBin;
use lib "$FindBin::Bin/../../../../src/modules/extension/perl";

# Without Log::Nxlog you cannot access (read or modify) the event data
# so don't forget this:
use Log::Nxlog;

use Geo::IP;

my $geoip;

BEGIN
{
    # This will be called once when nxlog starts so you can use this to
    # initialize stuff here
    $geoip = Geo::IP->new(GEOIP_MEMORY_CACHE);
}


# this is the method which is invoked from 'Exec' for each event log
sub process
{
    # The event data is passed here when this method is invoked by the module
    my ( $event ) = @_;
    
    # We look up the county of the sender of the message
    my $msgsrcaddr = Log::Nxlog::get_field($event, 'MessageSourceAddress');
    if ( defined($msgsrcaddr) )
    {
	my $country = $geoip->country_code_by_addr($msgsrcaddr);
	$country = "unknown" unless ( defined($country) );
	Log::Nxlog::set_field_string($event, 'MessageSourceCountry', $country);
    }

    # Iterate over the fields
    foreach my $fname ( @{Log::Nxlog::field_names($event)} )
    {
	# Delete all fields except these
	if ( ! (($fname eq 'raw_event') ||
		($fname eq 'AccountName') ||
		($fname eq 'MessageSourceCountry')) )
	{
	    Log::Nxlog::delete_field($event, $fname);
	}
    }

    # Check a field and rename it if it matches
    my $accountname = Log::Nxlog::get_field($event, 'AccountName');
    if ( defined($accountname) && ($accountname eq 'John') )
    {
	Log::Nxlog::set_field_string($event, 'AccountName', 'johnny');
	Log::Nxlog::log_info('renamed john');
    }
}

WTMP (xm_wtmp)

This module provides a parser function to process binary wtmp files. The module registers an parser function using the name of the extension module instance which can be used as the parameter of the InputType directive in input modules such as im_file.

Configuration

The module does not have any module specific configuration directives.

Configuration examples

Example 6.31. WTMP to JSON format conversion

The following configuration accepts WTMP and converts it to JSON.

<Extension wtmp>
    Module    xm_wtmp
</Extension>

<Extension json>
    Module    xm_json
</Extension>

<Input in>
    Module    im_file
    File      '/var/log/wtmp'
    InputType wtmp
    Exec      to_json();
</Input>

<Output out>
    Module    om_file
    File      '/var/log/wtmp.txt'
</Output>

<Route processwtmp>
    Path      in => out
</Route>

The following is a sample output produced by the configuration above.

{"EventTime":"2013-10-01 09:39:59","AccountName":"root","Device":"pts/1",
 "LoginType":"login","EventReceivedTime":"2013-10-10 15:40:20",
 "SourceModuleName":"input","SourceModuleType":"im_file"}
{"EventTime":"2013-10-01 23:23:38","AccountName":"shutdown","Device":"no device",
 "LoginType":"shutdown","EventReceivedTime":"2013-10-11 10:58:00",
 "SourceModuleName":"input","SourceModuleType":"im_file"}

Input modules

Input modules are responsible for collecting event log data from various sources. The nxlog core will add a few fields in each input module, see the following section for the list of these.

Fields generated by core

The following fields are set by core:

$raw_event

Type

Filled with data received from stream modules (im_file, im_tcp, etc).

$EventReceivedTime

Type

Set to the time when the event is received. The value is not modified if the field already exists.

$SourceModuleName

Type

The name of the module instance is stored in this field for input modules. The value is not modified if the field already exists.

$SourceModuleType

Type

The type the module instance (such as 'im_file') is stored in this field for input modules. The value is not modified if the field already exists.

DBI (im_dbi)

FIXME

Configuration examples

Example 6.32. Reading from a MySQL database

<Input dbiin>
    Module	im_dbi
    SavePos	TRUE
    Driver	mysql
    Option	host 127.0.0.1
    Option	username mysql
    Option	password mysql
    Option      dbname logdb 
</Input>

<Output out>
    Module	om_file
    File	"tmp/output"
</Output>

<Route 1>
    Path	dbiin => out
</Route>

Program (im_exec)

This module will execute a program or script on startup and will read its standard output. It can be used to easily integrate with exotic log sources which can be read only with the help of scripts or programs.

Configuration

In addition to the common module directives, the following can be used to configure the im_exec module instance.

Command: This directive is mandatory. It specifies the name of the script/program to be executed.
Arg: This is an optional parameter, multiple can be specified for each argument needed to pass to the Command. Note that specifying multiple arguments with one Arg directive separated with spaces will not work because the Command will receive it as one argument, so you will need to split them up.
InputType: See the description about InputType in the global module config section.
Restart: Restart the process if it exits. There is a 1 second delay before it is restarted in order not to DOS the system when a process is not behaving nicely. Looping should be implemented in the script itself, this directive is only to provide some safety against malfunctioning scripts and programs. This boolean directive defaults to FALSE.

Configuration examples

Example 6.33. Emulating im_file

<Input input>
    Module	im_exec
    Command	/usr/bin/tail
    Arg		-f
    Arg		/var/log/messages
</Input>

<Output fileout>
    Module	om_file
    File	"tmp/output"
</Output>

<Route 1>
    Path	input => fileout
</Route>

This exact same configuration is not recommended for real use because im_file was designed to read log messages from files. This example only demonstrates the use of the im_exec module.

File (im_file)

This module can be used to read log messages from files. The file position can be persistently saved across restarts in order to avoid reading from the beginning again when nxlog is restarted. It also supports external rotation tools. When the module cannot read any more data from the file, it checks whether the opened file descriptor belongs to the same filename it opened originally. If the inodes differ, the module assumes the file was moved and reopens its input.

im_file uses a 1 second interval to monitor files for new messages. This method was implemented because polling a regular file is not supported on all platforms. If there is no more data to read, the module will sleep for 1 second.

By using wildcards, the module can read multiple files simultaneously and will open new files as they appear. It will also enter newly created directories if recursion is enabled.

Note

The module needs to scan the directory content in case of a wildcarded file monitoring. This can present a significant load if there are many files (hundreds or thousands) in the monitored directory. For this reason it is highly recommended to rotate files out of the monitored directory either using the built-in log rotation capabilities of nxlog or using external tools.

Configuration

In addition to the common module directives, the following can be used to configure the im_file module instance.

File

This mandatory directive specifies the name of the input file to open. It must be a string type expression. For relative filenames you should be aware that nxlog changes its working directory to '/' unless the global SpoolDir is set to something else. On Windows systems the directory separator is backslash. For compatibility reasons the forward slash '/' character can be also used as the directory separator, but this only works for filenames which don't contain wildcards. If the filename is specified using wildcards, you should use backslash for the directory separator.

Wildcards are supported in filenames only, directory names in the path cannot be wildcarded. Wildcards are not regular expressions, these are patterns commonly used by unix shells to expand filenames which is also known as globbing.

?: Matches a single character only.
*: Matches zero or more characters.
\*: Matches the asterisk '*' character.
\?: Matches the question mark '?' character.
[...]: Used to specify a single character. If the first character of the class description is ^ or !, the sense of the description is reversed. The rest of the class description is a list of single characters or pairs of characters separated by -. Any of those characters can have a backslash in front of them, which is ignored; this lets you use the characters ] and - in the character class, as well as ^ and ! at the beginning.

Note

The backslash character '\' is used to escape the wildcard characters, unfortunately this is the same as the directory separator on Windows. Take this into account when specifying wildcarded filenames on this platform. Lets suppose that we have log files under the directory C:\test which need to be monitored. Specifying the wildcard 'C:\test\*.log' will not match because '\*' becomes a literal asterisk, thus it is treated as a non-wildcarded filename. For this reason the directory separator needs to be escaped, so the 'C:\test\\*.log' will match our files. 'C:\\test\\*.log' will also work. When specifying the filename using double quotes, this would became "C:\\test\\\\*.log" because the backslash is also used as an escape character inside double quoted string literals. Filenames on Windows systems are treated case-insensitively. Unix/Linux is case-sensitive.

SavePos

This directive takes a boolean value of TRUE or FALSE and specifies whether the file position should be saved when nxlog exits. The file position will be read from the cache file file upon startup. The file position is saved by default if this directive is not specified in the configuration. Even if SavePos is enabled, it can be explicitly turned off with the NoCache directive.

ReadFromLast

This optional directive takes a boolean value. If it is set to TRUE, it instructs the module to only read logs which arrived after nxlog was started in case the saved position could not be read (for example on first start). When SavePos is TRUE and a previously saved position value could be read, the module will resume reading from this saved position. If this is FALSE, the module will read all logs from the file. This can result in quite a lot of messages which is usually not the expected behaviour. If this directive is not specified, it defaults to TRUE.

Recursive

This directive takes a boolean value of TRUE or FALSE and specifies whether input files should be searched recursively under subdirectories. The default value is TRUE. This option takes effect only if wildcards are used in the filename. For example if '/var/log/*.log' is specified in the File directive, then '/var/log/apache2/access.log' will also match. Because wildcards in directory names of the path are not supported, this directive makes it possible to read multiple files from different subdirectories with a single im_file module instance only.

PollInterval

This directive specifies in seconds how frequently the module will check for new files and new log entries. If this directive is not specified it defaults to 1 second. Fractional seconds may be specified, i.e. to check twice every second you should set the following: PollInterval 0.5

DirCheckInterval

This directive specifies in seconds how frequently the module will check the monitored directory for modifications to files and new files in case of a wildcarded File path. If this directive is not specified it defaults to double of the value of the PollInterval directive, i.e. it is 2 seconds if PollInterval isn't defined either. Fractional seconds may be specified. It is recommended to increase the default in case there are many files which cannot be rotated out and the nxlog process has a high CPU load.

ActiveFiles

This directive specifies how many files nxlog will actively monitor at most. If there are modifications to more files in parallel than the value of this directive, then modifications to files above this limit will only get noticed after the DirCheckInterval, i.e. all data should be collected eventually. Typically there is only one or at most a couple log sources which actively append data to log files, the rest of the files are usually dormant after being rotated, so the default value of 10 should be sufficient in most cases. This directive is also only relevant in case of a wildcarded File path.

RenameCheck

This directive takes a boolean value of TRUE or FALSE and specifies whether input files should be monitored for possible file rotation via renaming in order to avoid rereading the file contents. A file is considered to be rotated when nxlog detects a new file whose inode and size matches that of another watched file which has just been deleted. Note that this does not always work correctly and can yield false positives when a log file is deleted and another is added with the same size. The file system is likely to reuse to inode number of the deleted file and thus the module will falsely detect this as a rename/rotation. For this reason the default value of the RenameCheck directive is FALSE. When this directive is FALSE, renamed files are considered as new and the file contents will be reread.

Note

It is recommended to use a naming scheme for rotated files such that their name does not match the wildcard and are not monitored anymore after rotation instead of trying to solve the renaming issue by enabling this directive.

CloseWhenIdle

This directive takes a boolean value of TRUE or FALSE and specifies whether open input files should be closed as soon as possible after there is no more data to read. Some applications request an exclusive lock on the log file written or rotated, this directive can possibly help if the application can/does retry to acquire the lock. This directive defaults to FALSE unless specified explicitly.

InputType

See the description about InputType in the global module config section.

Functions and procedures exported by im_file

Functions exported by im_file

string file_name();

description: Return the name of the file currently open which the log was read from.
return type: string

Configuration examples

Example 6.34. Forwarding logs from a file to a remote host

<Input in>
    Module	im_file
    File	"/var/log/messages"
    SavePos	TRUE
</Input>

<Output out>
    Module	om_tcp
    Host	192.168.1.1
    Port	514
</Output>

<Route 1>
    Path	in => out
</Route>

Internal (im_internal)

This module makes it possible to insert internal log messages of nxlog into a route. nxlog generates messages about error conditions, debugging messages, etc. In addition internal messages can be also generated from the nxlog language using the log_info(), log_warning() and log_error() procedure calls.

Note

Only messages with loglevel INFO and above are supported. Debug messages are ignored due to technical reasons. For debugging purposes the direct logging facility should be used, see the global LogFile and LogLevel directives.

One must be careful about the use of the im_internal module because it is easy to cause a message loop. For example consider the situation when the internal messages are sent to a database. If the database is experiencing errors which result in internal error messages then these are again routed to the database and this will trigger further error messages and is easy to see that this will result in a loop. In order to avoid a resource exhaustion, the im_internal module will drop its messages when the queue of next module in the route is full. It is recommended to always put the im_internal module instance in a separate route.

The im_internal does not have any module specific configuration directives in addition to the common module directives.

Note

If you require internal messages in syslog format, you need to explicitely convert them with pm_transformer or using the to_syslog_bsd() procedure of the xm_syslog module, because the $raw_event field is not generated in syslog format.

Fields generated by im_internal

The following fields are set by im_internal:

$raw_event

Type

Will be set to the string passed to the log_info() and other log() procedures.

$Message

Type

Set to the same value as $raw_event.

$SeverityValue

Type

Depending on the log level of the internal message, the syslog severity is set to the value corresponding to "debug", "info", "warning", "error" or "critical".

$Severity

Type

The severity name of the event.

$EventTime

Type

Will be set to the current time.

$SourceName

Type

Will be set to 'nxlog'.

$ProcessID

Type

The field is filled with the process id of the nxlog process.

$Hostname

Type

The hostname where the log is produced

$ErrorCode

Type

If an error is logged resulting from an OS error, this field contains the error number provided by the Apache portable runtime library.

Configuration examples

Example 6.35. Forwaring internal messages over syslog udp

<Extension syslog>
    Module      xm_syslog
</Extension>

<Input internal>
    Module	im_internal
</Input>

<Output out>
    Module	om_udp
    Host	192.168.1.1
    Port	514
    Exec	to_syslog_bsd();
</Output>

<Route internal>
    Path	internal => out
</Route>

Kernel (im_kernel)

This module can collect kernel log messages from the kernel log buffer. Currently this module works on linux only. On Linux the klogctl() system call is used for this purpose. In order to be able to read kernel logs, special privileges are required. For this reason nxlog needs to be started as root. Using the User and Group global directives nxlog can then drop its root privileges but it will keep the CAP_SYS_ADMIN capability to be able to read the kernel log buffer.

Note

Unfortunately it is not possible to read from the /proc/kmsg pseudo file for an unprivileged process even if the CAP_SYS_ADMIN capability is kept. For this reason the /proc/kmsg interface is not supported by the im_kernel module. The im_file module should work fine with the /proc/kmsg pseudo file if one wishes to collect kernel logs this way, though this will require nxlog to be running as root.

The kernel messages are emitted in the following form.

<[0-7]>Some message from the kernel.

Note

Kernel messages are valid BSD syslog messages but do not contain timestamp and hostname fields. These can be parsed with pm_transformer or using the parse_syslog_bsd() procedure of the xm_syslog module, this will set the timestamp and hostname fields.

Configuration examples

Example 6.36. Storing raw kernel logs into a file

# drop privileges after being started as root
User nxlog
Group nxlog

<Input kern>
    Module      im_kernel
</Input>

<Output fileout>
    Module      om_file
    File        "tmp/output"
</Output>

<Route 1>
    Path        kern => fileout
</Route>

Mark (im_mark)

Mark messages are used to indicate periodic activity in order to be assured that the logger is running in case there are no log messages coming in from other sources.

By default, without specifying any of the module specific directives, a log message is emitted every 30 minutes containing "-- MARK --".

Note

If you require mark messages in syslog format, you need to explicitely convert them with pm_transformer or using the to_syslog_bsd() procedure of the xm_syslog module, because the $raw_event field is not generated in syslog format.

Note

The functionality of the im_mark module can be also achieved using the Schedule block with a log_info("--MARK--") Exec statement which would insert the messages via the im_internal module into a route. Using a single module for this task can simplify and possibly make the configuration easier to understand. Just wanted to point out that "there is more than one way to do it" :)

Configuration

In addition to the common module directives, the following can be used to configure the im_mark module instance.

Mark: This optional directive can set the string for the mark message. If not specified, the default is "-- MARK --".
MarkInterval: This optional directive sets the interval for mark messages in minutes. If not specified, the default value of 30 minutes is used.

Fields generated by im_mark

The following fields are set by im_mark:

$raw_event

Type

Will be set to "-- MARK --" or the value defined with the Mark configuration directive.

$Message

Type

Set to the same value as $raw_event.

$SeverityValue

Type

Its value will be set to 6 which is the "info" severity level.

$Severity

Type

The severity name of the event.

$EventTime

Type

Will be set to the current time.

$SourceName

Type

Will be set to 'nxlog'.

$ProcessID

Type

The field is filled with the process id of the nxlog process.

Configuration examples

Example 6.37. Using the im_mark module

<Input mark>
    Module	im_mark
    MarkInterval 1
    Mark	-=| MARK |=-
</Input>

<Output fileout>
    Module	om_file
    File	"tmp/output"
</Output>

<Route 1>
    Path	mark => fileout
</Route>

MS EventLog for Windows XP/2000/2003 (im_mseventlog)

This module can be used to collect EventLog messages on Microsoft Windows platforms. The module looks up the available EventLog sources stored under the registry key "SYSTEM\\CurrentControlSet\\Services\\Eventlog" and will poll logs from each of these or only the sources defined with the Sources directive.

Note

Windows Vista, Windows 2008 and later use a new EventLog API which is not backward compatible. Messages in some events produced by sources in this new format cannot be resolved with the old API which is used by this module. If such an event is encountered, a Message similar to the following will be set:

The description for EventID XXXX from source SOURCE cannot be read by im_mseventlog because this does not support the newer WIN2008/Vista EventLog API.

Though the majority of event messages can be read with this module even on Windows 2008/Vista and later, it is recommended to use the im_msvistalog module instead.

Note

Strings are stored in dll and executable files and these need to be looked up by the module when reading eventlog messages. If a program (dll/exe) is already uninstalled and cannot be opened to look up the strings in the message, the following message will appear instead:

The description for EventID XXXX from source SOURCE cannot be found.

Configuration

In addition to the common module directives, the following can be used to configure the im_mseventlog module instance.

SavePos: This directive takes a boolean value of TRUE or FALSE and specifies whether the file position should be saved when nxlog exits. The file position will be read from the cache file upon startup. The file position is saved by default if this directive is not specified in the configuration. Even if SavePos is enabled, it can be explicitly turned off with the NoCache directive.
ReadFromLast: This optional directive takes a boolean value. If it is set to TRUE, it instructs the module to only read logs which arrived after nxlog was started in case the saved position could not be read (for example on first start). When SavePos is TRUE and a previously saved position value could be read, the module will resume reading from this saved position. If this is FALSE, the module will read all logs from the EventLog. This can result in quite a lot of messages which is usually not the expected behaviour. If this directive is not specified, it defaults to TRUE.
Sources: This optional directive takes a comma separated list of eventlog file names, such as 'Security, Application', to read only specific eventlog sources. If this directive is not specified, then all available eventlog sources are read (as listed in the registry). This directive should not be confused with the SourceName containted within the eventlog and it is not a list of such names. The value of this is stored in the FileName field.
UTF8: This optional directive takes a boolean value. If it is set to TRUE, all strings will be converted to UTF-8 encoding. Internally this calls the convert_fields procedure. The xm_charconv module must be loaded for the character set conversoion to work. If this UTF8 directive is not defined, it defaults to TRUE, but conversion will only occur if the xm_charconv module is loaded, otherwise strings will be in the local codepage.

Fields generated by im_mseventlog

The following fields are set by im_mseventlog:

$raw_event

Type

Contains the timestamp, hostname, severity and message from the event

$Message

Type

Contains the message from the event

$EventTime

Type

Will be set to the TimeGenerated field of the EventRecord.

$EventTimeWritten

Type

Will be set to the TimeWritten field of the EventRecord.

$Hostname

Type

The host or computer name field of the EventRecord.

$SourceName

Type

The event source which produced the event, this is the subsystem or application name.

$EventID

Type

The event id of the EventRecord.

$CategoryNumber

Type

The category number, stored as Category in the EventRecord.

$Category

Type

The category name resolved from CategoryNumber.

$FileName

Type

The logfile source (e.g. Security, Application) of the event.

$AccountName

Type

The username associated with the event.

$AccountType

Type

The type of the account. Possible values are: User, Group, Domain, Alias, Well Known Group, Deleted Account, Invalid, Unknown, Computer.

$Domain

Type

The domain name of the user.

$SeverityValue

Type

Normalized severity number of the event.

$Severity

Type

Normalized severity name of the event.

$EventType

Type

The type of the event which is a string describing the severity. It takes the following values: "ERROR", "AUDIT_FAILURE", "AUDIT_SUCCESS", "INFO", "WARNING", "UNKNOWN"

$RecordNumber

Type

The number of the event record.

Configuration examples

Example 6.38. Forwarding EventLogs from a windows machine to a remote host

<Input in>
    Module      im_mseventlog
</Input>

<Output out>
    Module      om_tcp
    Host        192.168.1.1
    Port        514
</Output>

<Route 1>
    Path        in => out
</Route>

MS EventLog for Windows 2008/Vista and later (im_msvistalog)

This module can be used to collect EventLog messages on Microsoft Windows platforms which support the newer EventLog API (also known as the Crimson Eventlog subsystem), namely Windows 2008/Vista and later. You can refer to the official Microsoft documentation about Event Logs. The module supports reading all System, Application and Custom events. It looks up the available channels and monitors events in each unless the Query and the Channel directives are explicitely defined. Event logs can be collected from remote servers over MS RPC (Note: Enterprise Edition only).

Note

This module will not work on Windows 2003 and earlier because Windows Vista, Windows 2008 and later use a new EventLog API which is not available in earlier Windows versions. If you need to collect EventLog messages on these platforms, you should use the im_mseventlog module instead.

Note

The Windows EventLog subsystem does not support subscriptions to Debug and Analytic channels, thus it is not possible to collect these type of events with this module.

In addition to the standard set of fields which are listed under the System section, event providers can define their own additional schema which enables logging additional data under the EventData section. The Security log makes use of this new feature and such additional fields can be seen as in the following XML snippet:

<EventData>
  <Data Name="SubjectUserSid">S-1-5-18</Data> 
  <Data Name="SubjectUserName">WIN-OUNNPISDHIG$</Data> 
  <Data Name="SubjectDomainName">WORKGROUP</Data> 
  <Data Name="SubjectLogonId">0x3e7</Data> 
  <Data Name="TargetUserSid">S-1-5-18</Data> 
  <Data Name="TargetUserName">SYSTEM</Data> 
  <Data Name="TargetDomainName">NT AUTHORITY</Data> 
  <Data Name="TargetLogonId">0x3e7</Data> 
  <Data Name="LogonType">5</Data> 
  <Data Name="LogonProcessName">Advapi</Data> 
  <Data Name="AuthenticationPackageName">Negotiate</Data> 
  <Data Name="WorkstationName" /> 
  <Data Name="LogonGuid">{00000000-0000-0000-0000-000000000000}</Data> 
  <Data Name="TransmittedServices">-</Data> 
  <Data Name="LmPackageName">-</Data> 
  <Data Name="KeyLength">0</Data> 
  <Data Name="ProcessId">0x1dc</Data> 
  <Data Name="ProcessName">C:\Windows\System32\services.exe</Data> 
  <Data Name="IpAddress">-</Data> 
  <Data Name="IpPort">-</Data> 
</EventData>

nxlog can extract this data when fields are logged using this schema. The values will be available in the fields of the internal nxlog log structure. This is especially useful because there is no need to write pattern matching rules to extract this data from the message. These fields can be used in filtering rules, writing them into SQL tables or to trigger actions. Consider the following example which filters using the Exec directive:

<Input in>
    Module       im_msvistalog
    Exec         if ($TargetUserName == 'SYSTEM') OR ($EventType == 'VERBOSE') drop();
</Input>

Configuration

In addition to the common module directives, the following can be used to configure the im_msvistalog module instance.

SavePos

This directive takes a boolean value of TRUE or FALSE and specifies whether the file position should be saved when nxlog exits. The file position will be read from the cache file upon startup. The file position is saved by default if this directive is not specified in the configuration. Even if SavePos is enabled, it can be explicitly turned off with the NoCache directive.

ReadFromLast

This optional directive takes a boolean value. If it is set to TRUE, it instructs the module to only read logs which arrived after nxlog was started in case the saved position could not be read (for example on first start). When SavePos is TRUE and a previously saved position value could be read, the module will resume reading from this saved position. If this is FALSE, the module will read all logs from the EventLog. This can result in quite a lot of messages which is usually not the expected behaviour. If this directive is not specified, it defaults to TRUE.

Query

This directive specifies the query if one wishes to pull only specific eventlog sources. See the MSDN docs about Event Selection. Note that this directive needs a single-line parameter, so multi-line query XML should be specified with line continuation marks (\) as in the following example:

     Query <QueryList>                                             \
             <Query Id='1'>                                        \
              <Select Path='Security'>*[Security/Level=4]</Select> \
             </Query>                                              \
           </QueryList>

When the Query contains an XPath style expression, the Channel must also be specified. Otherwise if an XML Query is specified, the Channel should not be used.

Channel

The name of the Channel to query. If not specified, the module will read from all sources defined in the registry. See the MSDN docs about Event Selection.

PollInterval

This directive specifies in seconds how frequently the module will check for new events. If this directive is not specified it defaults to 1 second. Fractional seconds may be specified, i.e. to check twice every second you should set the following: PollInterval 0.5

Fields generated by im_msvistalog

The following fields are set by im_msvistalog:

$raw_event

Type

Contains the EventTime, Hostname, Severity EventID and Message from the event.

$Message

Type

Contains the message from the event.

$EventTime

Type

Will be set to the EvtSystemTimeCreated field.

$Hostname

Type

Contains the EvtSystemComputer field.

$SourceName

Type

The event source which produced the event, this is the EvtSystemProviderName field.

$EventID

Type

The event id as in EvtSystemEventID.

$Task

Type

The task number as in EvtSystemTask.

$Category

Type

The category name resolved from Task.

$Keywords

Type

The value of the Keywords field from EvtSystemKeywords.

$Channel

Type

The Channel (e.g. Security, Application) of the event source.

$AccountName

Type

The username associated with the event.

$AccountType

Type

The type of the account. Possible values are: User, Group, Domain, Alias, Well Known Group, Deleted Account, Invalid, Unknown, Computer.

$Domain

Type

The domain name of the user.

$UserID

Type

The SID which resolves to AccountName, stored in EvtSystemUserID.

$SeverityValue

Type

Normalized severity number of the event.

$Severity

Type

Normalized severity name of the event (CRITICAL|ERROR|WARNING|INFO|DEBUG).

$EventType

Type

The type of the event which is a string describing the severity. This is translated to its string representation from EvtSystemLevel. It takes the following values: "CRITICAL", "ERROR", "AUDIT_FAILURE", "AUDIT_SUCCESS", "INFO", "WARNING", "VERBOSE"

$ProviderGuid

Type

The GUI of the event's provider as stored in EvtSystemProviderGuid. This corresponds to the name of the provider stored in the SourceName field.

$Version

Type

The Version number of the event as in EvtSystemVersion.

$OpcodeValue

Type

The Opcode number of the event as in EvtSystemOpcode.

$Opcode

Type

The opcode string resolved from OpcodeValue.

$ActivityID

Type

The ActivityID as stored in EvtSystemActivityID.

$RelatedActivityID

Type

The RelatedActivityID as stored in EvtSystemRelatedActivityID.

$ProcessID

Type

The process identifier of the event producer as in EvtSystemProcessID.

$ThreadID

Type

The thread identifier of the event producer as in EvtSystemThreadID.

$RecordNumber

Type

The number of the event record.

Configuration examples

Example 6.39. Forwarding EventLogs from a windows machine to a remote host

<Input in>
    Module       im_msvistalog
    ReadFromLast TRUE
</Input>

<Output out>
    Module       om_tcp
    Host         192.168.1.1
    Port         514
</Output>

<Route 1>
    Path        in => out
</Route>

Null (im_null)

This module does not generate any input, so basically it does nothing. Yet it can be useful for creating a dummy route, testing purposes, and it can have Scheduled nxlog code execution as well, so it is not completely useless. This module does not have any module specific configuration directives. See this example for usage.

TLS/SSL (im_ssl)

The im_ssl module provides an SSL/TLS transport using the OpenSSL library beneath the surface. It behaves similarly to the im_tcp module, except that an SSL handshake is performed at connection time and the data is sent over a secure channel. Because log messages transferred over plain TCP can be eavasdropped or even altered with a man-in-the-middle attack, using the im_ssl module provides a secure log message transport.

Configuration

In addition to the common module directives, the following can be used to configure the im_ssl module instance.

Host: This specifies the IP address or a dns hostname which the module should listen on to accept connections.
Port: This specifies the port number which the module will listen on for incoming conenctions.
CertFile: This specifies the path of the certificate file to be used in the SSL handshake.
CertKeyFile: This specifies the path of the certificate key file to be used in the SSL handshake.
KeyPass: Optional password of the certificate key file defined in CertKeyFile. For passwordless private keys the directive is not needed.
CAFile: This specifies the path of the certificate of the CA which will be used to check the certificate of the remote socket against.
CADir: This specifies the path of CA certificates which will be used to check the certificate of the remote socket against. The cert file names in this directory must be in the OpenSSL hashed format.
CRLFile: This specifies the path of the certificate revocation list (CRL) which will be used to check the certificate of the remote socket against.
CRLDir: This specifies the path of certificate revocation lists (CRLs) which will be used to check the certificate of the remote socket against. The file names in this directory must be in the OpenSSL hashed format.
RequireCert: This takes a boolean value of TRUE or FALSE and specifies whether the remote must present a certificate. If set to TRUE and there is no certificate presented during the handshake of the accepted connection, the connection will be refused. The default value is TRUE if this directive is not specified, meaning that all connections must use a certificate by default.
AllowUntrusted: This takes a boolean value of TRUE or FALSE and specifies whether the remote connection should be allowed without certificate verification. If set to TRUE the remote will be able to connect with unknown and self-signed certificates. The default value is FALSE if this directive is not specified, meaning that all connections must present a trusted certificate by default.
InputType: See the description about InputType in the global module config section.

Fields generated by im_ssl

The following fields are set by im_ssl:

$raw_event

Type

Will be set to the string received.

$MessageSourceAddress

Type

Set to the IP address of the remote host.

Configuration examples

Example 6.40. Reading binary data forwarded from another nxlog agent

<Input ssl>
    Module	im_ssl
    Host	localhost
    Port	23456
    CAFile	%CERTDIR%/ca.pem
    CertFile	%CERTDIR%/client-cert.pem
    CertKeyFile	%CERTDIR%/client-key.pem
    KeyPass	secret
    InputType	Binary
</Input>

<Output fileout>
    Module	om_file
    File	"tmp/output"
</Output>

<Route 1>
    Path	ssl => fileout
</Route>

TCP (im_tcp)

This module accepts TCP connections on the address and port specified in the configuration. It can handle multiple simultaneous connections. The TCP transfer protocol provides more reliable log transmission than UDP. If security is a concern, consider using the im_ssl module instead.

Note

There is no access control built in the module. If you need to deny some hosts connecting to the module's TCP port, you should use appropriate firewall rules for this purpose.

Configuration

In addition to the common module directives, the following can be used to configure the im_tcp module instance.

Host: This specifies the IP address or a dns hostname which the module should listen on to accept connections.
Note
Because of security reasons the default listen address is localhost if this directive is not specified (the localhost loopback address is not accessible from the outside). You will most probably want to send logs from remote hosts, so make sure that the address specified here is accessible. The any address 0.0.0.0 is commonly used here.
Port: This specifies the port number which the module will listen on for incoming conenctions. The default port is 514 if this directive is not specified.
InputType: See the description about InputType in the global module config section.

Fields generated by im_tcp

The following fields are set by im_tcp:

$raw_event

Type

Will be set to the string received.

$MessageSourceAddress

Type

Set to the IP address of the remote host.

Configuration examples

Example 6.41. Using the im_tcp module

<Input in>
    Module	im_tcp
    Host	0.0.0.0
    Port	1514
</Input>

<Output out>
    Module	om_file
    File	"tmp/output"
</Output>

<Route r>
    Path	in => out
</Route>

UDP (im_udp)

This module accepts UDP datagrams on the address and port specified in the configuration. UDP is the transport protocol of the old BSD syslog standard as described in RFC 3164, so this module can be particularly useful to receive such messages from older devices which do not support other transports.

Note

There is no access control built in the module. If you need to deny some hosts sending logs to the module's UDP port, you should use appropriate firewall rules for this purpose.

Note

UDP packets can be dropped by the operating system because the protocol does not guarantee reliable message delivery. It is recommended to use the tcp or ssl transport modules instead if message loss is a concern.

Though nxlog was designed to minimize message loss even in the case of UDP, adjusting the kernel buffers could also help in avoiding UDP message loss on a loaded system. The Priority directive in the route block can also help in this situation.

For parsing syslog messages, take a look at the pm_transformer module or the parse_syslog_bsd() procedure of the xm_syslog.

Configuration

In addition to the common module directives, the following can be used to configure the im_udp module instance.

Host: This specifies the IP address or a dns hostname which the module should listen on to accept connections. The default address is "localhost" if this is not specified.
Port: This specifies the port number which the module will listen on for incoming conenctions. The default port is 514 if this directive is not specified.
SockBufSize: This optional directive sets the socket buffer size (SO_RCVBUF) to the value specified. Otherwise the OS defaults are used. If you are experiencing UDP packet loss at the kernel level, setting this to a high value (e.g. 150000000) may help. On Microsoft Windows systems the default socket buffer size is extremely low, using this option is highly recommended.
InputType: See the description about InputType in the global module config section.

Fields generated by im_udp

The following fields are set by im_udp:

$raw_event

Type

Will be set to the string received.

$MessageSourceAddress

Type

Set to the IP address of the remote host.

Configuration examples

Example 6.42. Using the im_udp module

<Input in>
    Module	im_udp
    Host	192.168.1.1
    Port	514
</Input>

<Output out>
    Module	om_file
    File	"tmp/output"
</Output>

<Route r>
    Path	in => out
</Route>

Unix Domain Socket (im_uds)

This module allows log messages to be received over a unix domain socket. Traditionally unix systems have a socket, typically /dev/log used by the system logger to accept messages from. Applications wishing to send messages to the system log use the syslog(3) system call.

Note

This module supports SOCK_DGRAM type sockets only. SOCK_STREAM type sockets may be supported in the future.

Note

It is recommended to disable FlowControl when this module is used to collect local syslog from the /dev/log unix domain socket. Otherwise the syslog() system call will block in all programs which are trying to write to the system log if the Output queue becomes full and this will result in an unresponsive system.

For parsing syslog messages, take a look at the pm_transformer module or the parse_syslog_bsd() procedure of the xm_syslog.

Configuration

In addition to the common module directives, the following can be used to configure the im_uds module instance.

UDS: This specifies the path of the unix domain socket. The default is /dev/log if this is not specified.
InputType: See the description about InputType in the global module config section. This defaults to dgram if not specified because unix domain sockets are SOCK_DGRAM type on Linux and the module does not yet support SOCK_STREAM sockets.

Configuration examples

Example 6.43. Using the im_uds module

<Input unix>
    Module	im_uds
    uds		/dev/log
</Input>

<Output out>
    Module	om_file
    File	"/var/log/messages"
</Output>

<Route 1>
       Path unix => out
</Route>

Processor modules

Blocker (pm_blocker)

This module can block log messages and can be used to simulate when a route is blocked. When the module blocks the data flow, log messages are first accumulated in the buffers, and then the flow-control mechanism pauses the input modules. Using the block() procedure it is possibile to programatically stop or resume the data flow. It can be useful for real-world scenarios as well as testing. See the examples below. When the module starts, the blocking mode is disabled by default, i.e. it operate just like pm_null would.

Functions and procedures exported by pm_blocker

Functions exported by pm_blocker

boolean is_blocking();

description: Return TRUE if the module is currently blocking the data flow, FALSE otherwise.
return type: boolean

Procedures exported by pm_blocker

block(boolean mode);

description

When mode is TRUE, the module will block. You should call block(FALSE) from a schedule block or another module, otherwise it might not get invoked if the queue is already full.

arguments

mode: type: boolean

Configuration examples

Example 6.44. Using the pm_blocker module

In this example we collect messages received over UDP and forward it to another host via TCP. The log data is forwarded during non-working hours between 19 pm and 8 am. During the other half of the day data is buffered on the disk to be sent out only after 19 pm.

<Input in>
    Module	im_udp
    Host	0.0.0.0
    Port	1514
</Input>

<Processor buffer>
    Module      pm_buffer
    # 100Mb disk buffer
    MaxSize	102400
    Type	disk
</Processor>

<Processor blocker>
    Module	pm_blocker
    <Schedule>
	When	0 8 * * *
	Exec	blocker->block(TRUE);
    </Schedule>
    <Schedule>
	When	0 19 * * *
	Exec	blocker->block(FALSE);
    </Schedule>
</Processor>

<Output out>
    Module	om_tcp
    Host	192.168.1.1
    Port	1514
</Output>

<Route 1>
    Path	in => buffer => blocker => out
</Route>

Buffer (pm_buffer)

Messages received over UDP may be dropped by the operating system unless packets are read from the message buffer fast enough. Some logging subsystems using a small circular buffer can also overwrite logs old logs in the buffer if it is not read, thus there is a chance of missing important log data. Such situations can lead to dropped or lost messages and other problems where buffering can help.

The pm_buffer module supports disk and memory based log message buffering. If both are required, multiple pm_buffer instances can be used with different settings. Because a memory buffer can be faster, though its size is limited, combining memory and disk based buffering can be a good idea in case buffering is frequently used.

The disk based buffering mode stores the log message data in chunks. When all the data is successfully forwarded from a chunk, it is then deleted in order to save disk space.

Note

Using pm_buffer is only recommended when there is a chance of message loss. The built-in flow-control in nxlog ensures that messages will not be read by the input module until the output side can send/store/forward. When reading from files (with im_file) or the Windows Eventlog (with im_mseventlog or im_msvistalog) it is rarely necessary to use the pm_buffer module unless there is a chance of log rotation (and thus a possibility of missing some data) while the output module (e.g. TCP or SSL) is being blocked.

Configuration

In addition to the common module directives, the following can be used to configure the pm_buffer module instance.

MaxSize: Specifies the size of the buffer in kilobytes. This paramater is mandatory.
WarnLimit: Specifies an optional limit smaller than MaxSize which will trigger a warning message when reached. The log message will not be emitted again until the buffer size drops to half of WarnLimit and reaches it again in order to protect against a warning message flood.
Type: Type can be either 'Mem' or 'Disk' to select memory or disk based buffering respectively.
Directory: Name of the directory used to store the disk buffer file chunks. This is only valid with Type set to 'Disk' mode.

Functions and procedures exported by pm_buffer

Functions exported by pm_buffer

integer buffer_size();

description: Return the size of the memory buffer in bytes.
return type: integer

integer buffer_count();

description: Return the number of log messages held in the memory buffer.
return type: integer

Configuration examples

Example 6.45. Using a memory buffer to protect against udp message loss

<Input udp>
    Module	im_udp
    Host	0.0.0.0
    Port	514
</Input>

<Processor buffer>
    Module      pm_buffer
    # 1Mb buffer
    MaxSize	1024
    Type	Mem
    # warn at 512k
    WarnLimit	512
</Processor>

<Output tcp>
    Module	om_tcp
    Host	192.168.1.1
    Port	1514
</Output>

<Route 1>
    Path	udp => buffer => tcp
</Route>

Event correlator (pm_evcorr)

The pm_evcorr module provides event correlation functionality in addition to the already available nxlog language features such as variables and statistical counters which can be also used for event correlation purposes.

This module was greatly inspired by the Perl based correlation tool SEC. Some of the rules of the pm_evcorr module were designed to mimic those available in SEC. This module aims to be a better alternative to SEC with the following advantages:

The correlation rules in SEC work with the current time. With pm_evcorr it is possible to specify a time field wich is used for elapsed time calculation making offline event correlation also possible.
SEC uses regular expressions extensively which can become quite slow in case of many correlation rules. In contrast this module can correlate preprocessed messages using fields for example from the pattern matcher and the syslog parser without requiring the use of regular expressions (though these are also available for use by correlation rules). Thus testing conditions can be significantly faster when simple comparison is used instead of regular expression based pattern matching.
This module was designed to operate on fields thus making it possible to correlate structured logs in addition to simple free-form log messages.
Most importantly, this module is written in C and SEC is pure Perl which could have major performance benefits.

The rulesets of this module can use a context. A context is an expression which is evaluated during runtime to a value and the correlation rule is checked in the context of this value. For example if we wanted to count the number of failed logins per user and alert if the failed logins exceed 3 for the user, then we'd use the $AccountName as the context. There is a separate context storage is for each correlation rule instance. If you need global contexts accessible from all rule instances, take a look at module variables and statistical counters.

Configuration

The pm_evcorr configuration contains correlation rules which are evaluated for each log message processed by the module. Currently there are five rule types supported by pm_evcorr: Simple, Suppressed, Pair, Absence and Thresholded. These rules are defined in config blocks. The order of the rules is important because the rules are evaluated in the order they are defined. For example a correlation rule can change a state, variable or field which can be then used by a later rule. File inclusion can be useful to move the correlation rules into a standalone file.

In addition to the common module directives, the following can be used to configure the pm_evcorr module instance.

TimeField

Specifies the name of the field to use for calculating elapsed time such as 'EventTime'. The name of the field must be specified without the leading dollar "$" sign. If this parameter is not specified, the current time is assumed. This directive makes it possible to accurately correlate events based on the event time recorded in the logs and to do non real-time event correlation also.

ContextCleanTime

When a Context is used in the correlation rules, these must be purged from memory after they are expired, otherwise using too many context values could result in a high memory usage. This optional directive specifies the interval between context cleanups in seconds. By default a 1 minute cleanup interval is used if any rules use a Context and this directive is not specified.

Simple

This rule is essentially the same as the Exec directive supported by all modules. Because Execs are evaluated before the correlation rules, the Simple rule was also needed to be able to evaluate a statement as the other rules do, following the rule order. The Simple block has one directive also with the same name.

Exec: One or more Exec directives must be specified which takes a statement as argument.

Suppressed

This rule matches the given condition. If the condition evaluates to TRUE, the statement specified with the Exec directive is evaluated. The rule will then ignore any log messages for the time specified with Interval directive. For example this rule is useful to suppress creating many alerts in a short period when a condition is satisfied.

Condition: This mandatory directive takes an expression as argument which must evaluate to a boolean value.
Interval: This mandatory directive takes an integer argument specifying the number of seconds to ignore the condition. The TimeField directive is used to calculate time.
Context: This optional directive specifies an expression to be used as the context. It must evaluate to a value. Most often a field is specified here.
Exec: One or more Exec directives must be specified which takes a statement as argument.

Pair

When TriggerCondition evaluates to TRUE, this rule type will wait Interval seconds for RequiredCondition to become TRUE, it then executes the statement(s) in the Exec directive(s).

TriggerCondition: This mandatory directive takes an expression as argument which must evaluate to a boolean value.
RequiredCondition: This mandatory directive takes an expression as argument which must evaluate to a boolean value. When this evaluates to TRUE after TriggerCondition evaluated to TRUE within Interval seconds, the statement(s) in the Exec directive(s) are executed.
Interval: Thisdirective takes an integer argument specifying the number of seconds to wait for RequiredCondition to become TRUE. If this directive is 0 or not specified, the rule will wait indefinitely for RequiredCondition to become TRUE. The TimeField directive is used to calculate time.
Context: This optional directive specifies an expression to be used as the context. It must evaluate to a value. Most often a field is specified here.
Exec: One or more Exec directives must be specified which takes a statement as argument.

Absence

This rule type does the opposite of Pair. When TriggerCondition evaluates to TRUE, this rule type will wait Interval seconds for RequiredCondition to become TRUE. If it does not become TRUE it then executes the statement(s) in the Exec directive(s).

TriggerCondition

This mandatory directive takes an expression as argument which must evaluate to a boolean value.

RequiredCondition

This mandatory directive takes an expression as argument which must evaluate to a boolean value. When this evaluates to TRUE after TriggerCondition evaluated to TRUE within Interval seconds, the statement(s) in the Exec directive(s) are NOT executed.

Interval

This mandatory directive takes an integer argument specifying the number of seconds to wait for RequiredCondition to become TRUE. Its value must be greater than 0. The TimeField directive is used to calculate time.

Context

This optional directive specifies an expression to be used as the context. It must evaluate to a value. Most often a field is specified here.

Exec

One or more Exec directives must be specified which takes a statement as argument.

Note

The evaluation of this Exec is not triggered by a log event, thus it does not make sense to use log data related operations such as accessing fields.

Thresholded

This rule will execute the statement(s) in the Exec directive(s) if the Condition evaluates to TRUE Threshold or more times during the Interval specified. The advantage of this rule over the use of statistical counters is that the time window is dynamic and shifts as log messages are processed. Thus the problem described in this example is not present with this rule.

Condition: This mandatory directive takes an expression as argument which must evaluate to a boolean value.
Interval: This mandatory directive takes an integer argument specifying a time window for Condition to become TRUE. Its value must be greater than 0. The TimeField directive is used to calculate time. This time window is dynamic, meaning that it will shift.
Threshold: This mandatory directive takes an integer argument specifying the number of times Condition must evaluate to TRUE within the given time Interval. When the treshold is reached, the module executes the statement(s) in the Exec directive(s).
Context: This optional directive specifies an expression to be used as the context. It must evaluate to a value. Most often a field is specified here.
Exec: One or more Exec directives must be specified which takes a statement as argument.

Stop

This rule will stop evaluating successive rules if the Condition evaluates to TRUE. The optional Exec directive will be evaluated in this case.

Condition: This mandatory directive takes an expression as argument which must evaluate to a boolean value. When it evaluates to TRUE, the correlation rule engine will stop checking any further rules.
Exec: One or more Exec directives can be specified which takes a statement as argument. This will be evaluated when the specified Condition is satisfied. This Exec directive is optional.

Configuration examples

Example 6.46. Correlation rules

This following configuration sample contains a rule for each type.

<Input in>
    Module	im_file
    File	"modules/processor/evcorr/testinput_evcorr2.txt"
    SavePos	FALSE
    ReadFromLast FALSE
    Exec	if ($raw_event =~ /^(\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d) (.+)/) {  \
                    $EventTime = parsedate($1);                                   \
                    $Message = $2;                                                \
                    $raw_event = $Message;                                        \
                }
</Input>

<Input internal>
    Module	im_internal
    Exec	$raw_event = $Message;
    Exec	$EventTime = 2010-01-01 00:01:00;
</Input>

<Output out>
    Module	om_file
    File	'tmp/output'
</Output>

<Processor evcorr>
    Module	   pm_evcorr
    TimeField	   EventTime

    <Simple>
	Exec	   if $Message =~ /^simple/ $raw_event = "got simple";
    </Simple>

    <Suppressed>
	# match input event and execute an action list, but ignore the following
    	# matching events for the next t seconds.
      	Condition  $Message =~ /^suppressed/
   	Interval   30
   	Exec	   $raw_event = "suppressing..";
    </Suppressed>

    <Pair>
	# If TriggerCondition is true, wait Interval seconds for RequiredCondition to be true and then do the Exec
    	# If Interval is 0, there is no window on matching
       	TriggerCondition  $Message =~ /^pair-first/
       	RequiredCondition $Message =~ /^pair-second/
       	Interval   30
       	Exec	   $raw_event = "got pair";
    </Pair>

    <Absence>
	# If TriggerCondition is true, wait Interval seconds for RequiredCondition to be true. 
	# If RequiredCondition does not become true within the specified interval then do the Exec
	TriggerCondition  $Message =~ /^absence-trigger/
   	RequiredCondition $Message =~ /^absence-required/
   	Interval 	  10
   	Exec	 	  log_info("'absence-required' not received within 10 secs");
    </Absence>

    <Thresholded>
	# if the number of events exceeeds the given threshold within the interval do the Exec
    	# Same as SingleWithThreshold in SEC
       	Condition  $Message =~ /^thresholded/
       	Threshold  3
       	Interval   60
       	Exec	   $raw_event = "got thresholded";
    </Thresholded>

    <Stop>
        Condition  $EventTime < 2010-01-02 00:00:00
	Exec	   log_debug("got stop");
    </Stop>

    <Simple>
        # This will be rewritten only if the previous Stop condition is FALSE
	Exec	$raw_event = "rewritten";
    </Simple>

</Processor>

<Route 1>
    Path	in, internal => evcorr => out
</Route>

The contents of the input file are the following:

2010-01-01 00:00:00 Not simple
2010-01-01 00:00:01 suppressed1 - Suppress kicks in, will log 'suppressing..'
2010-01-01 00:00:10 simple1
2010-01-01 00:00:12 pair-first - now look for pair-second
2010-01-01 00:00:13 thresholded1
2010-01-01 00:00:15 thresholded2
2010-01-01 00:00:19 simple2
2010-01-01 00:00:20 thresholded3 - will log 'got thresholded'
2010-01-01 00:00:21 suppressed2 - suppressed and logged as is
2010-01-01 00:00:22 pair-second - will log 'got pair'
2010-01-01 00:00:23 suppressed3 - suppressed and logged as is
2010-01-01 00:00:25 pair-first
2010-01-01 00:00:26 absence-trigger
2010-01-01 00:00:29 absence-required - will not log 'got absence'
2010-01-01 00:00:46 absence-trigger
2010-01-01 00:00:56 pair-second - will not log 'got pair' because it is over the interval
2010-01-01 00:00:57 absence-required - will log an additional 'absence-required not received within 10 secs'
2010-01-02 00:00:00 this will be rewritten 
2010-01-02 00:00:10 this too

After this is processed, the resulting output will contain these lines:

Not simple
suppressing..
got simple
pair-first - now look for pair-second
thresholded1
thresholded2
got simple
got thresholded
suppressed2 - suppressed and logged as is
got pair
suppressed3 - suppressed and logged as is
pair-first
absence-trigger
absence-required - will not log 'got absence'
absence-trigger
pair-second - will not log 'got pair' because it is over the interval
absence-required - will log an additional 'absence-required not received within 10 secs'
rewritten
rewritten
'absence-required' not received within 10 secs

Filter (pm_filter)

This is a simple module which forwards log messages if the specified condition is TRUE.

Note

This module has been obsoleted by the nxlog language because filtering is now possible in any module using the drop() procedure conditionally in the Exec directive.

Example 6.47. Dropping messages conditionally

if $raw_event =~ /^Debug/ drop();

Configuration

In addition to the common module directives, the following can be used to configure the pm_filter module instance.

Condition: This mandatory directive takes an expression as argument which must evaluate to a boolean value. If the expression does not evaluate to TRUE, the log message is discarded.

Configuration examples

Example 6.48. Filtering messages

<Input unix>
    Module	im_uds
    uds		/dev/log
</Input>

<Processor filter>
    Module pm_filter
    Condition $raw_event =~ /failed/ or $raw_event =~ /error/
</Processor>

<Output out>
    Module	om_file
    File	"/var/log/error"
</Output>

<Route 1>
    Path unix => filter => out
</Route>

Message deduplicator (pm_norepeat)

This module can be used to filter out repeating messages. Similarly to syslog daemons, this module checks the previous message against the current. If they match, the current message is dropped. The module waits one second for duplicated messages to arrive. If duplicates are detected, the first message is forwarded, the rest is dropped and a message containing "last message repeated X times" is sent instead.

Configuration

In addition to the common module directives, the following can be used to configure the pm_norepeat module instance.

CheckFields: This optional directive takes a comma separated list of field names which are used to compare log messages. Only the fields listed here are compared, the others are ignored. For example the 'EventTime' field will be different in repeating messages, so this field should not be used in the comparison. If this directive is not specified, the default field to be checked is 'Message'.

Fields generated by pm_norepeat

The following fields are set by pm_norepeat:

$raw_event

Type

Will be set to "last message repeated X times".

$Message

Type

Set to the same value as $raw_event.

$SeverityValue

Type

Its value will be set to 6 which is the "info" severity level.

$Severity

Type

The severity name of the event.

$EventTime

Type

Will be set to the time of the last event or the current time if EventTime was not present in the last event.

$SourceName

Type

Will be set to 'nxlog'.

$ProcessID

Type

The field is filled with the process id of the nxlog process.

Configuration examples

Example 6.49. Filtering out duplicated messages

<Input in>
    Module	im_uds
    UDS		/dev/log
</Input>

<Processor norepeat>
    Module	pm_norepeat
    CheckFields	Hostname, SourceName, Message
</Processor>

<Output out>
    Module	om_file
    File	"/var/log/messages"
</Output>


<Route 1>
    Path	in => norepeat => out
</Route>

Null (pm_null)

This module does not do any special processing, so basically it does nothing. Though the Exec and the Schedule directive is available in this module just like in any other, thus it can be quite useful. This module does not have any module specific configuration directives. See this example for usage.

Pattern matcher (pm_pattern)

This module makes it possible to execute pattern matching efficiently using a pattern database file in XML format. Using this module is more efficient than having nxlog regular expression rules listed in Exec directives, because the pm_pattern module was designed in such a way that patterns need not to be matched linearly. In addition, the module does an automatic on-the-fly pattern reordering internally for further speed improvements and it has a feature which can be used to tag messages with additional fields useful for message classification. See the Pattern matching and message classification section for additional examples.

Regular expressions are the most widely used in pattern matching. Unfortunately using a large number of regular expression based patterns does not scale well, because these need to be evaluated linearly. There are other techniques such as the radix tree which solve the linearity problem, the drawback is that usually these require a special syntax for specifying patterns which users must learn. If the log message is already parsed and is not treated as single line of message, then it is possible to process only a subset of the patterns which partially solves the linearity problem. With the other performance improvement tricks employed within the pm_pattern module, its speed can compare to the other techniques such as a radix tree based pattern matcher. Yet the pm_pattern module can keep using regular expressions which all programmers and system administrators are familiar with and this also provides an easy migration of regexp patterns from other tools and already existing patterns.

Traditionally pattern matching on log messages has employed a technique where the log message was one string and the pattern (regular expression or radix tree based pattern) was executed against it. To match patterns against logs which contain structured data (such as the Windows EventLog), this structured data (the fields of the log) must be converted to a single string. This is a simple but inefficient method used by many tools.

The nxlog patterns defined in the XML pattern database file can contain more than one field, this allows multi-dimensional pattern matching. Thus with nxlog's pm_pattern module there is no need to convert all fields into a single string as it can work with multiple fields.

Patterns can be grouped together under pattern groups. Pattern groups serve an optimization purpose. The group can have an optional matchfield block which can check a condition. If the condtion (such as $SourceName matches sshd) is satisfied, the pm_pattern module will dive into the group and check each pattern against the log. If the pattern group's condition didn't match (i.e. $SourceName wasn't sshd), the module can thus skip all patterns in the group without having to check each pattern one by one.

When the pm_pattern module finds a matching pattern, the PatternID and PatternName fields are set on the log message. These can be used later in conditional processing and correlation rules for example.

Note

The pm_pattern module does not process all patterns. It exits after the first matching pattern is found. This means that at most one pattern can match a log message. You should avoid writing a pattern to be used with pm_pattern which can match a subset of logs that match another pattern. For example if you have two regular expression patterns ^\d+ and ^\d\d, the second may be never matched because of the first. The internal order of patterns and pattern groups is changed dynamically by pm_pattern. Those patterns are placed and tried first which have the highest match count. Reasons for this operation mode are:

Performance optimization,
Setting the value of $PatternID would be problematic with multiple values because the language does not support arrays.

If you want a strictly linearly executing mattern matcher, you should use the Exec directive and write your rules there.

Configuration

In addition to the common module directives, the following can be used to configure the pm_pattern module instance.

PatternFile: This mandatory directive specifies the name of the pattern database file.

Pattern database file

Example 6.50. A simple pattern database

This pattern database contains two patterns to match ssh authentication messages. The patterns are under a group named ssh which checks whether the field SourceName is sshd and only tries to match the patterns if the logs are indeed from sshd. The patterns both extract AuthMethod, AccountName and SourceIP4Address from the log message when the pattern matches the log. Additionally TaxonomyStatus and TaxonomyAction are set. The second pattern utilizes the Exec block which is evaluated when the pattern matches.

Note

For this pattern to work, the logs must be parsed with parse_syslog() prior to feeding it to the pm_pattern module because it uses the SourceName and Message fields.

<?xml version='1.0' encoding='UTF-8'?>
<patterndb>
 <created>2010-01-01 01:02:03</created>
 <version>42</version>
 
 <group>
   <name>ssh</name>
   <id>42</id>
   <matchfield>
    <name>SourceName</name>
    <type>exact</type>
    <value>sshd</value>
   </matchfield>

   <pattern>
    <id>1</id>
    <name>ssh auth success</name>

    <matchfield>
     <name>SourceName</name>
     <type>exact</type>
     <value>sshd</value>
    </matchfield>

    <matchfield>
     <name>Message</name>
     <type>regexp</type>
        <!-- Accepted publickey for nxlogfan from 192.168.1.1 port 4242 ssh2 -->
     <value>^Accepted (\S+) for (\S+) from (\S+) port \d+ ssh2</value>
     <capturedfield>
	<name>AuthMethod</name>
	<type>string</type>
     </capturedfield>
     <capturedfield>
	<name>AccountName</name>
	<type>string</type>
     </capturedfield>
     <capturedfield>
	<name>SourceIP4Address</name>
        <type>string</type>
     </capturedfield>
    </matchfield>

    <set>
     <field>
       <name>TaxonomyStatus</name>
       <value>success</value>
       <type>string</type>
     </field>
     <field>
       <name>TaxonomyAction</name>
       <value>authenticate</value>
       <type>string</type>
     </field>
    </set>
   </pattern>

   <pattern>
    <id>2</id>
    <name>ssh auth failure</name>

    <matchfield>
     <name>SourceName</name>
     <type>exact</type>
     <value>sshd</value>
    </matchfield>

    <matchfield>
     <name>Message</name>
     <type>regexp</type>
     <value>^Failed (\S+) for invalid user (\S+) from (\S+) port \d+ ssh2</value>

     <capturedfield>
	<name>AuthMethod</name>
	<type>string</type>
     </capturedfield>
     <capturedfield>
	<name>AccountName</name>
	<type>string</type>
     </capturedfield>
     <capturedfield>
	<name>SourceIP4Address</name>
        <type>string</type>
     </capturedfield>
    </matchfield>

    <set>
     <field>
       <name>TaxonomyStatus</name>
       <value>failure</value>
       <type>string</type>
     </field>
     <field>
       <name>TaxonomyAction</name>
       <value>authenticate</value>
       <type>string</type>
     </field>
    </set>

    <exec>
      $TestField = 'test';
    </exec>
    <exec>
      $TestField = $Testfield + 'value';
    </exec>
   </pattern>

 </group>

</patterndb>

Fields generated by pm_pattern

The following fields are set by pm_pattern:

$PatternID

Type

Set to the id number of the pattern which matched the message.

$PatternName

Type