archive-org.com » ORG » N » NXLOG.ORG

Total: 215

Choose link from "Titles, links and description words view":

Or switch to "Titles and links view".

  • For each input source a separate context is maintained by the module so that multi line messages coming from several simultaneous sources can be still correctly processed An input source is a file for im file with wildcards it is one source for each file a connection for im ssl and im tcp Unfortunately im udp uses a single socket and is treated as a single source even if multiple UDP e g syslog senders are forwarding logs to it Note By using module variables it is possible to accomplish the same what this module does The advantages of using this module over module variables are the following Processes messages more efficiently It yields a more readable configuration Module event counters are correctly updated i e one increment for one multi line message and not per line It works on message source level each file for a wildcarded im file module instance and each tcp connection for an im tcp im ssl instance and not on module instance level Configuration The following directives can be used to configure the xm multiline module instance HeaderLine This directive takes a string or a regular expression literal This will be matched against each line When the match is successful the successive lines are appended until the next header line is read This directive is mandatory unless FixedLineCount is used Note Until there is a new header read the previous message is stored in the buffers because the module does not know where the message ends The im file module will forcibly flush this buffer after the configured PollInterval timeout If this behaviour is unacceptable consider using some kind of an encapsulation method JSON XML RFC5425 etc or use an end marker with EndLine if possible EndLine This is similar to the HeaderLine directive This optional directive also takes a string or a regular expression literal to be matched against each line When the match is successful the message is considered complete and is emitted FixedLineCount This directive takes a positive integer number defining the number of lines to concatenate This is mostly useful with log messages spanning a fixed number of lines When this number is defined the module knows where the event message ends thus it does not suffer from the problem described above Exec This directive is almost identical to the behavior of the Exec directive used by the other modules with the following differences Each line is passed in raw event as it is read The line includes the line terminator Other fields cannot be used If you want to store captured strings from regular expression based matching in fields you cannot do it here This is mostly useful for filtering out some lines with the drop procedure or rewriting them Configuration examples Example 6 15 Parsing multi line XML logs and converting to JSON XML is commonly formatted as indented multi line to make it more readable In the following configuration file we use the HeaderLine together with the HeaderLine directive to parse the events which are converted to JSON after some slight normalization Module xm multiline HeaderLine EndLine Module xm xml Module xm json Module im file File modules extension multiline xm multiline5 in SavePos FALSE ReadFromLast FALSE InputType multiline Discard everything that doesn t seem to be an xml event if raw event drop Parse the xml event parse xml Rewrite some fields EventTime parsedate timestamp delete timestamp delete EventReceivedTime Convert to JSON to json Module om file File tmp output Path in out An input sample 2012 11 23 23 00 00 ERROR Something bad happened Please check the system 2012 11 23 23 00 12 INFO System state is now back to normal The following output is produced SourceModuleName in SourceModuleType im file severity ERROR message n Something bad happened n Please check the system n EventTime 2012 11 23 23 00 00 SourceModuleName in SourceModuleType im file severity INFO message n System state is now back to normal n EventTime 20 12 11 23 23 00 12 Example 6 16 Parsing DICOM logs Each log message has a header TIMESTAMP INTEGER SEVERITY which is used as the message boundary A regular expression is defined for this using the HeaderLine directive Each log message is prepended with an additional line containing dashes and is output into a file Module xm multiline HeaderLine d d d d d d d d d d d d d d d s d s S s Module im file File modules extension multiline xm multiline4 in SavePos FALSE ReadFromLast FALSE InputType dicom multi Module om file File tmp output Exec raw event n raw event Path in out An input sample 2011 12 1512 22 51 000000 4296 INFO Association Request Parameteres Our Implementation Class UID 2 16 124 113543 6021 2 Our Implementation Version Name RZDCX 2 0 1 8 Their Implementation Class UID Their Implementation Version Name Application Context Name 1 2 840 10008 3 1 1 1 Requested Extended Negotiation none Accepted Extended Negotiation none 2011 12 1512 22 51 000000 4296 DEBUG Constructing Associate RQ PDU 2011 12 1512 22 51 000000 4296 DEBUG WriteToConnection length 310 bytes written 310 loop no 1 2011 12 1512 22 51 015000 4296 DEBUG PDU Type Associate Accept PDU Length 216 6 bytes PDU header 02 00 00 00 00 d8 00 01 00 00 50 41 43 53 20 20 20 20 20 20 20 20 20 20 20 20 52 5a 44 43 58 20 20 20 20 20 20 20 20 20 20 20 00 00 00 00 00 00 2011 12 1512 22 51 031000 4296 DEBUG DIMSE sendDcmDataset sending 1 46 bytes The following output is produced 2011 12 1512 22 51 000000 4296 INFO Association Request Parameteres Our Implementation Class UID 2 16 124 113543 6021 2 Our Implementation Version Name RZDCX 2 0 1 8 Their Implementation Class UID Their Implementation Version Name Application Context Name 1 2 840 10008 3 1 1 1 Requested Extended Negotiation none Accepted Extended Negotiation none 2011 12 1512 22 51 000000 4296 DEBUG Constructing Associate RQ PDU 2011 12 1512 22 51 000000 4296 DEBUG WriteToConnection length 310 bytes written 310 loop no 1 2011 12 1512 22 51 015000 4296 DEBUG PDU Type Associate Accept PDU Length 216 6 bytes PDU header 02 00 00 00 00 d8 00 01 00 00 50 41 43 53 20 20 20 20 20 20 20 20 20 20 20 20 52 5a 44 43 58 20 20 20 20 20 20 20 20 20 20 20 00 00 00 00 00 00 2011 12 1512 22 51 031000 4296 DEBUG DIMSE sendDcmDataset sending 1 46 bytes Example 6 17 Multi line messages with a fixed string header The following configuration will process messages having a fixed string header containing dashes Each event is then prepended with a sharp and is output to a file Module xm multiline HeaderLine Module im file File modules extension multiline xm multiline1 in SavePos FALSE ReadFromLast FALSE InputType multiline Exec raw event raw event Module om file File tmp output Path in out An input sample 1 1 2 aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb ccccccccccccccccccccccccccccccccccccc dddd The following output is produced 1 1 2 aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb ccccccccccccccccccccccccccccccccccccc dddd Example 6 18 Multi line messages with fixed line count The following configuration will process messages having a fixed line count of 4 Lines containing only whitespace are ignored and removed Each event is then prepended with a sharp and is output to a file Module xm multiline FixedLineCount 4 Exec if raw event s drop Module im file File modules extension multiline xm multiline2 in SavePos FALSE ReadFromLast FALSE InputType multiline Module om file File tmp output Exec raw event raw event Path in out An input sample 1 2 3 4 1asd 2asdassad 3ewrwerew 4xcbccvbc 1dsfsdfsd 2sfsdfsdrewrwe 3sdfsdfsew 4werwerwrwe The following output is produced 1 2 3 4 1asd 2asdassad 3ewrwerew 4xcbccvbc 1dsfsdfsd 2sfsdfsdrewrwe 3sdfsdfsew 4werwerwrwe Example 6 19 Multi line messages with a syslog header Multi line messages are frequently logged over syslog and they end up in log files Unfortunately from the result it looks that each line is one event with its own syslog header It can be a common requirement to merge these back into a single event message The following configuration does just that It strips the syslog header from the netstat output stored as a traditional syslog formatted file and each message is then printed again with a line of dashes used as a separator Module xm syslog Module xm multiline FixedLineCount 4 parse syslog bsd raw event Message n Module im file File modules extension multiline xm multiline3 in SavePos FALSE ReadFromLast FALSE InputType netstat Module om file File tmp output Exec raw event n raw event Path in out An input sample Nov 21 11 40 27 hostname app 26459 Iface MTU Met RX OK RX ERR RX D RP RX OVR TX OK TX ERR TX DRP TX OVR Flg Nov 21 11 40 27 hostname app 26459 eth2 1500 0 16936814 0 0 0 30486067 0 8 0 BMRU Nov 21 11 40 27 hostname app 26459 lo 16436 0 277217234 0 0 0 277217234 0 0 0 LRU Nov 21 11 40 27 hostname app 26459 tun0 1500 0 316943 0 0 0 368642 0 0 0 MOPRU Nov 21 11 40 28 hostname app 26459 Iface MTU Met RX OK RX ERR RX D RP RX OVR TX OK TX ERR TX DRP TX OVR Flg Nov 21 11 40 28 hostname app 26459 eth2 1500 0 16945117 0 0 0 30493583 0 8 0 BMRU Nov 21 11 40 28 hostname app 26459 lo 16436 0 277217234 0 0 0 277217234 0 0 0 LRU Nov 21 11 40 28 hostname app 26459 tun0 1500 0 316943 0 0 0 368642 0 0 0 MOPRU Nov 21 11 40 29 hostname app 26459 Iface MTU Met RX OK RX ERR RX D RP RX OVR TX OK TX ERR TX DRP TX OVR Flg Nov 21 11 40 29 hostname app 26459 eth2 1500 0 16945270 0 0 0 30493735 0 8 0 BMRU Nov 21 11 40 29 hostname app 26459 lo 16436 0 277217234 0 0 0 277217234 0 0 0 LRU Nov 21 11 40 29 hostname app 26459 tun0 1500 0 316943 0 0 0 368642 0 0 0 MOPRU The following output is produced Iface MTU Met RX OK RX ERR RX DRP RX OVR TX OK TX ERR TX DRP TX O VR Flg eth2 1500 0 16936814 0 0 0 30486067 0 8 0 BMRU lo 16436 0 277217234 0 0 0 277217234 0 0 0 LRU tun0 1500 0 316943 0 0 0 368642 0 0 0 MOPRU Iface MTU Met RX OK RX ERR RX DRP RX OVR TX OK TX ERR TX DRP TX O VR Flg eth2 1500 0 16945117 0 0 0 30493583 0 8 0 BMRU lo 16436 0 277217234 0 0 0 277217234 0 0 0 LRU tun0 1500 0 316943 0 0 0 368642 0 0 0 MOPRU Iface MTU Met RX OK RX ERR RX DRP RX OVR TX OK TX ERR TX DRP TX O VR Flg eth2 1500 0 16945270 0 0 0 30493735 0 8 0 BMRU lo 16436 0 277217234 0 0 0 277217234 0 0 0 LRU tun0 1500 0 316943 0 0 0 368642 0 0 0 MOPRU Syslog xm syslog This module provides support for the archaic BSD Syslog protocol as defined in RFC 3164 and the current IETF standard defined by RFC 5424 5426 This is achieved by exporting functions and procedures usable from the nxlog language The transport is handled by the respective input and output modules i e im udp this module only provides a parser and helper functions to create syslog messages and handle facility and severity values The older but still widespread BSD syslog standard defines both the format and the transport protocol in RFC 3164 The transport protocol is UDP but to provide reliability and security this line based format is also commonly transferred over TCP and SSL There is a newer standard defined in RFC 5424 also known as the IETF syslog format which obsolotes the BSD syslog format This format overcomes most of the limitations of the old BSD syslog and allows multi line messages and proper timestamps The transport method is defined in RFC 5426 for UDP and RFC 5425 for TLS SSL Because the IETF Syslog format supports multi line messages RFC 5425 defines a special format to encapsulate these by prepending the payload size in ASCII to the IETF syslog message Messages tranferred in UDP packets are self contained and do not need this additional framing The following input reader and output writer functions are provided by the xm syslog module to support this TLS transport defined in RFC 5425 While RFC 5425 explicitly defines that the TLS network transport protocol is to be used pure TCP may be used if security is not a requirement Syslog messages can be also persisted to files with this framing format using these functions InputType Syslog TLS This input reader function parses the payload size and then reads the message according to this value It is required to support Syslog TLS transport defined in RFC 5425 OutputType Syslog TLS This output writer function prepends the payload size to the message It is required to support Syslog TLS transport defined in RFC 5425 Note The Syslog TLS InputType OutputType can work with any input output such as im tcp or im file and it does not depend on SSL transport at all The name Syslog TLS is a little misleading it was chosen to refer to the octet framing method described in RFC 5425 used for TLS transport Note The pm transformer module can also parse and create BSD and IETF syslog messages but using the functions and procedures provided by this module makes it possible to solve more complex tasks which pm transformer is not capable of on its own Structured data in IETF syslog messages is parsed and put into nxlog fields The SD ID will be prepended to the field name with a dot unless it is NXLOG XXXX Consider the following syslog message 1 2011 12 04T21 16 10 000000 02 00 host app procid msgid exampleSDI D 32473 eventSource Application eventID 1011 Message part After this IETF formatted syslog message is parsed with parse syslog ietf there will be two additional fields exampleSDID eventID and exampleSDID eventSource When SD ID is NXLOG the field name will be the same as the SD PARAM name The two additional fields extracted from the structured data part of the following IETF syslog message are eventID and eventSource 1 2011 12 04T21 16 10 000000 02 00 host app procid msgid NXLOG 3247 3 eventSource Application eventID 1011 Message part All fields parsed from the structured data part are strings Configuration In addition to the common module directives the following can be used to configure the xm syslog module instance SnareDelimiter This optional directive takes a single character as argument to specify the delimiter character used to separate fields when using the to syslog snare procedure The character specification works the same way as with the xm csv module If this directive is not specified the default escape character is the tab character t In latter versions of Snare4 this has changed to so you can use this configuration directive to specify an alternative delimiter Note that there is no delimiter after the last field SnareReplacement This optional directive takes a single character as argument to specify the replacement character substituted in place of any occurences of the delimiter character inside the Message field when invoking the to syslog snare procedure The character specification works the same way as with the xm csv module If this directive is not specified the default replacement character is space IETFTimestampInGMT This optional boolean directive can be used to format the timestamps produced by to syslog ietf in GMT instead of local time This defaults to FALSE so that local time is used by default with a timezone indicator Functions and procedures exported by xm syslog Functions exported by xm syslog integer syslog facility value string arg description Convert a syslog facility string to an integer arguments arg type string return type integer string syslog facility string integer arg description Convert a syslog facility value to a string arguments arg type integer return type string integer syslog severity value string arg description Convert a syslog severity string to an integer arguments arg type string return type integer string syslog severity string integer arg description Convert a syslog severity value to a string arguments arg type integer return type string Procedures exported by xm syslog parse syslog description Parse the raw event field as either BSD Syslog RFC3164 or IETF Syslog RFC5424 format parse syslog string source description Parse the given string as either BSD Syslog RFC3164 or IETF Syslog RFC5424 format arguments source type string parse syslog bsd description Parse the raw event field as BSD Syslog RFC3164 format parse syslog bsd string source description Parse the given string as BSD Syslog RFC3164 format arguments source type string parse syslog ietf description Parse the raw event field as IETF Syslog RFC5424 format parse syslog ietf string source description Parse the given string as IETF Syslog RFC5424 format arguments source type string to syslog bsd description Create a BSD Syslog formatted log message in raw event from the fields of the event The fields that are used to construct the raw event field are EventTime Hostname SourceName ProcessID Message or raw event SyslogSeverity or SyslogSeverityValue or Severity or SeverityValue SyslogFacility or SyslogFacilityValue If the fields are not present a sensible default is used to syslog ietf description Create an IETF Syslog RFC5424 formatted log message in raw event from the fields of the event The fields that are used to construct the raw event field are EventTime Hostname SourceName ProcessID Message or raw event SyslogSeverity or SyslogSeverityValue or Severity or SeverityValue SyslogFacility or SyslogFacilityValue If the fields are not present a sensible default is used to syslog snare description Create a SNARE Syslog formatted log message in raw event Uses the following fields to construct raw event EventTime Hostname SeverityValue FileName EventID SourceName AccountName AccountType EventType Category Message Fields generated by xm syslog The following fields are set by xm syslog raw event Type string Will be set to a syslog formatted string after to syslog bsd or to syslog ietf is called Message Type string The message part of the syslog line filled after parse syslog bsd or parse syslog ietf is called SyslogSeverityValue Type integer The severity part of the syslog line filled after parse syslog bsd or parse syslog ietf is called The default severity is 5 notice SyslogSeverity Type string The severity part of the syslog line filled after parse syslog bsd or parse syslog ietf is called The default severity is notice SeverityValue Type integer Normalized severity number of the event Severity Type string Normalized severity name of the event SyslogFacilityValue Type integer The facility part of the syslog line filled after parse syslog bsd or parse syslog ietf is called The default facility is 1 user SyslogFacility Type string The facility part of the syslog line filled after parse syslog bsd or parse syslog ietf is called The default facility is user EventTime Type datetime Will be set to the timestamp found in the syslog message after parse syslog bsd or parse syslog ietf is called If the year value is missing it is set to the current year Hostname Type string The hostname part of the syslog line filled after parse syslog bsd or parse syslog ietf is called SourceName Type string The application program part of the syslog line filled after parse syslog bsd or parse syslog ietf is called MessageID Type string The MSGID part of the syslog message filled after parse syslog ietf is called ProcessID Type string The process id in the syslog line filled after parse syslog bsd or parse syslog ietf is called Configuration examples Example 6 20 Sending a file as BSD syslog over UDP To send logs out in BSD syslog format over udp which are collected from files use the to syslog bsd procedure coupled with the om udp module as in the following example Module xm syslog Module im file We monitor all files matching the wildcard Every line is read into the raw event field File var log app log Set the EventTime field usually found in the logs by extracti ng it with a regexp If this is not set the current system time will be used which might be a little off if raw event d d d d d d d d d d d d d d EventTi me parsedate 1 Now set the severity to something custom This defaults to IN FO if unset if raw event ERROR Severity ERROR else Severity INFO The facility can be also set otherwise the default value is USER SyslogFacility AUDIT The SourceName field is called the TAG in RFC3164 terminology and is usually the process name SourceName my application It is also possible to rewrite the Hostname if you don t want to use the system s hostname Hostname myhost The Message field is used if present otherwise the current r aw event is prepended with the syslog headers You can do some modifications on the Message if required Here we add the full path of the source file to the end of message line Message raw event file name Now create our RFC3164 compliant syslog line using the fields set above and or use sensible defaults where possible The result will be in raw event to syslog bsd This module just sends the contents of the raw event field to the destination defined here one UDP packet per message Module om udp Host 192 168 1 42 Port 1514 Path in out Example 6 21 Collecting BSD style syslog messages over UDP To collect BSD style syslog messages over UDP use the parse syslog bsd procedure coupled with the im udp module as in the following example Module xm syslog Module im udp Host 0 0 0 0 Port 514 Exec parse syslog bsd Module om file File var log logmsg txt Path in out Example 6 22 Collecting IETF style syslog messages over UDP To collect IETF style syslog messages over UDP as defined by RFC 5424 and RFC 5426 use the parse syslog ietf procedure coupled with the im udp module as in the following example Note that the default port is 514 as defined by RFC 5426 this is the same as for BSD syslog Module xm syslog Module im udp Host 0 0 0 0 Port 514 Exec parse syslog ietf Module om file File var log logmsg txt Path in out Example 6 23 Collecting both IETF and BSD style syslog messages over the same UDP port To collect IETF and BSD style syslog messages over UDP use the parse syslog procedure coupled with the im udp module as in the following example This procedure is capable of detecting and parsing both syslog formats Since 514 is the default UDP port number for both BSD and IETF syslog this can be useful to collect both formats simultaneously If you want to accept both formats on different ports then it makes sense to use the appropriate parsers as in the previous two examples Module xm syslog Module im udp Host 0 0 0 0 Port 514 Exec parse syslog Module om file File var log logmsg txt Path in out Example 6 24 Collecting IETF style syslog messages over TLS SSL To collect IETF style syslog messages over TLS SSL as defined by RFC 5424 and RFC 5425 use the parse syslog ietf procedure coupled with the im ssl module as in the following example Note that the default port is 6514 in this case as defined by RFC 5425 The payload format parser is handled by the Syslog TLS input reader Module xm syslog Module im ssl Host localhost Port 6514 CAFile CERTDIR ca pem CertFile CERTDIR client cert pem CertKeyFile CERTDIR client key pem KeyPass secret InputType Syslog TLS Exec parse syslog ietf Module om file File var log logmsg txt Path in out Example 6 25 Forwarding IETF syslog over TCP The following configuration uses the to syslog ietf procedure to convert input to IETF syslog and forward it over TCP Module xm syslog Module im file File var log input txt Exec TestField test value Message raw event Module om tcp Host 127 0 0 1 Port 1514 Exec to syslog ietf OutputType Syslog TLS Path in out Because of the Syslog TLS framing the raw data sent over TCP will look like the following 130 1 2012 01 01T16 15 52 873750Z NXLOG 14506 EventReceivedT ime 2012 01 01 17 15 52 TestField test value test message This example shows that all fields except those which are filled by the syslog parser are added to the structured data part Example 6 26 Conditional rewrite of the syslog facility version 1 Module xm syslog Module im udp Port 514 Host 0 0 0 0 Exec parse syslog bsd Module om file File var log logmsg txt Exec if Message error SeverityValue syslog severity value error Exec to syslog bsd Path in fileout Example 6 27 Conditional rewrite of the syslog facility version 2 The following example does almost the same thing as the previous example except that the syslog parsing and rewrite is moved to a processor module and rewrite only occurs if the facility was modified This can make it work faster on multi core systems because the processor module runs in a separate thread This method can also minimize UDP packet loss because the input module does not need to parse syslog messages and can process UDP packets faster Module xm syslog Module im udp Host 0 0 0 0 Port 514 Module pm null Exec parse syslog bsd if Message error SeverityValue syslog severity value error to syslog bsd Module om file File var log logmsg txt Path in rewrite fileout External program execution xm exec This module provides two procedures which make it possible to execute external scripts or programs The reason for providing these two procedures through this additional extension module is to keep the nxlog core small A security advantage is that an administrator won t be able to execute arbitrarly scripts if this module is not loaded Note The om exec and im exec modules also provide support for running external programs though the purpose of these is to pipe data to and read data from programs The procedures provided by the xm exec module do not pipe log message data these are intended for multiple invocations Though data can be still passed to the executed script program as command line arguments Functions and procedures exported by xm exec Procedures exported by xm exec exec string command varargs args description Execute the command passing it the supplied arguments and wait for it to terminate The command is executed in the caller module s context Note that the module calling this procedure will block until the process terminates Use the exec async procedure to avoid this problem All output written to STDOUT and STDERR by the spawned process is discarded arguments command type string args type varargs exec async string command varargs args description This procedure executes the command passing it the supplied arguments and does not wait for it to terminate arguments command type string args type varargs Configuration examples Example 6 28 nxlog acting as a cron daemon Module xm exec Every 1 sec Exec exec async bin true Example 6 29 Sending email alerts Module xm exec Module im tcp Host 0 0 0 0 Port 1514 if raw event alertcondition exec async bin sh c echo Hostname n nRawEv ent n raw event usr bin mail a Content Type text plain c harset UTF 8 s ALERT user domain com Module om file File var log messages Path in out For another example see this configuration for file rotation Perl xm perl This module makes it possible to execute perl code and process event data using the perl language via a built in perl interpreter The perl interpreter is only loaded if the module is declared in the configuration While the nxlog language is already a powerful framework it is not intended to be a full featured programming language For example it does not provide lists arrays hashes and other features available in many high level languages Perl is widely used for log processing and it comes with a broad set of modules bundled or available from CPAN You can sometimes write faster code in C but you can always write code faster in Perl Code written in perl is also a lot safer because it is unlikely to crash Exceptions in the perl code croak die are handled properly and this will only result in an unfinished attempt at executing the log processing code but will not take down the whole nxlog process The module will parse the file specified in the PerlCode directive when nxlog and the module is started This file should contain one or more methods which can be called from the Exec directive of any module which wants to do any log processing in perl See the example below which illustrates the use of this module To acccess the fields and the event data from the perl code you need to include the Log Nxlog module This exports the following methods set field integer event key value Sets the integer value in the field named key set field string event key value Sets the string value in the field named key set field boolean event key value Sets the boolean value in the field named key get field event key Retreive the value associated with the field named key The method returns a scalar value if the key exist and the value is defined otherwise it returns undef delete field event key Delete the value associated with the field named key field type event key Return a string representing the type of the value associated with the field named key field names event Return a list of the field names contained in the event data Can be used to iterate over all the fields log debug msg Send the message in the argument to the internal logger on DEBUG loglevel Does the same as the procedure named log debug in nxlog log info msg Send the message in the argument to the internal logger on INFO loglevel Does the same as the procedure named log info in nxlog log warning msg Send the message in the argument to the internal logger on WARNING loglevel Does the same as the procedure named log warning in nxlog log error msg Send the message in the argument to the internal logger on ERROR loglevel Does the same as the procedure named log error in nxlog You should be able to read the POD documentation contained in Nxlog pm with perldoc Log Nxlog Configuration The following directives can be used to configure the xm perl module instance PerlCode This mandatory directive expects a file which contains valid perl code This file is read and parsed by the perl interpreter Methods defined in this file can be called with the call procedure Functions and procedures exported by xm perl Procedures exported by xm perl call string subroutine description Calls the perl subroutine provided in the first argument arguments subroutine type string perl call string subroutine description Calls the perl subroutine provided in the first argument arguments subroutine type string Configuration examples In this example logs are parsed as syslog then the data is passed to a perl method which does a GeoIP lookup on the source address of the incoming message Example 6 30 Using the built in perl interpreter Module xm syslog Module xm perl PerlCode modules extension perl processlogs pl Module im file File test log ReadFromLast FALSE SavePos FALSE Module om file File tmp output First we parse the input natively from nxlog Exec parse syslog bsd Now call the process subroutine defined in processlogs pl Exec perl call process You can also invoke this public procedure call in case of multiple xm perl instances like this Exec perl call process Path in out The contents of the processlogs pl perl script is as follows use strict use warnings use Carp FindBin is for adding a path to INC this not needed normally use FindBin use lib FindBin Bin src modules extension perl Without Log Nxlog you cannot access read or modify the event data so don t forget this use Log Nxlog use Geo IP my geoip BEGIN This will be called once when nxlog starts so you can use this to initialize stuff here geoip Geo IP new GEOIP MEMORY CACHE this is the method which is invoked from Exec for each event log sub process The event data is passed here when this method is invoked by the m odule my event We look up the county of the sender of the message my msgsrcaddr Log Nxlog get field event MessageSourceAddress if defined msgsrcaddr my country geoip country code by addr msgsrcaddr country unknown unless defined country Log Nxlog set field string event MessageSourceCountry co untry Iterate over the fields foreach my fname Log Nxlog field names event Delete all fields except these if fname eq raw event fname eq AccountName fname eq MessageSourceCountry Log Nxlog delete field event fname Check a field and rename it if it matches my accountname Log Nxlog get field event AccountName if defined accountname accountname eq John Log Nxlog set field string event AccountName johnny Log Nxlog log info renamed john WTMP xm wtmp This module provides a parser function to process binary wtmp files The module registers an parser function using the name of the extension module instance which can be used as the parameter of the InputType directive in input modules such as im file Configuration The module does not have any module specific configuration directives Configuration examples Example 6 31 WTMP to JSON format conversion The following configuration accepts WTMP and converts it to JSON Module xm wtmp Module xm json Module im file File var log wtmp InputType wtmp Exec to json Module om file File var log wtmp txt Path in out The following is a sample output produced by the configuration above EventTime 2013 10 01 09 39 59 AccountName root Device pts 1 LoginType login EventReceivedTime 2013 10 10 15 40 20 SourceModuleName input SourceModuleType im file EventTime 2013 10 01 23 23 38 AccountName shutdown Device no device LoginType shutdown EventReceivedTime 2013 10 11 10 58 00 SourceModuleName input SourceModuleType im file Input modules Input modules are responsible for collecting event log data from various sources The nxlog core will add a few fields in each input module see the following section for the list of these Fields generated by core The following fields are set by core raw event Type string Filled with data received from stream modules im file im tcp etc EventReceivedTime Type datetime Set to the time when the event is received The value is not modified if the field already exists SourceModuleName Type string The name of the module instance is stored in this field for input modules The value is not modified if the field already exists SourceModuleType Type string The type the module instance such as im file is stored in this field for input modules The value

    Original URL path: http://nxlog.org/docs/nxlog-ce/nxlog-reference-manual.txt (2016-02-01)
    Open archived version from archive


  • Using NXLog with Elasticsearch and Kibana
    om elasticsearch The NXLog Enterprise Edition comes with a module that can load data natively into Elasticsearch The advantage of this module over om http is the support for bulk data operations and dynamic indexing Event data is sent in batches this greately reduces the latency caused by the HTTP responses and the elasticsearch server can also process bulk data faster The om elasticsearch module can insert data at a rate of 10 000EPS or more on low end hardware For Kibana s time filters to work properly we will need to apply a template This can be pushed to Elasticsearch with the following command curl XPUT localhost 9200 template nxlog d template nxlog mappings default properties EventTime type date format YYYY MM dd HH mm ss This will tell Elasticsearch to treat our EventTime field as date and parse it in the given format This will be applied to all records in indexes beginning with nxlog Further mappings can be added in the properties section The following configuration shows how to use the om elasticsearch module to load the log data into Elasticsearch Extension json Module xm json Extension Input in Module im tcp Host 0 0 0 0 Port 1514 InputType Binary Input Output es Module om elasticsearch URL http localhost 9200 bulk FlushInterval 2 FlushLimit 100 Create an index daily Index strftime EventTime nxlog Y m d IndexType My logs Use the following if you don t have EventTime set Index strftime now nxlog Y m d Output Route r Path in es Route The IndexType parameter although not mandatory can be help sorting our logs on the dashboard It expects a string type expression and is reevaluated for each event record Refer to the NXLog Enterprise Edition Reference Manual for further details The input section in the configuration above uses im tcp with NXLog s Binary data format as this can preserve structured data across the network better than JSON The client agent side configuration is not the scope of this document After saving the configuration we should restart NXLog to apply the changes etc init d nxlog restart Logs should now be sent to and indexed by Elasticsearch To verify that the data is getting ingested first we will need to configure the Kibana indexing as shown in the screenshot below Figure 1 Kibana index configuration Now we should be able to search and analyze event data on the Kibana interface Figure 2 Kibana search results Loading data with om http Data can be also submitted to Elasticsearch via its HTTP REST API using the om http module available in both NXLog editions This will send one event per HTTP request to ES in JSON format The throughput is limited by the HTTP request response latency regardless this may be still suitable for low volume environments The nxlog configuration below is for a Windows client that collects log data from a file and the Windows EventLog and sends it to the Elasticsearch server directly define

    Original URL path: http://nxlog.org/docs/elasticsearch-kibana/using-nxlog-with-elasticsearch-and-kibana.html (2016-02-01)
    Open archived version from archive


  • Download and extract the latest version of Kibana wget http download elastic co kibana kibana kibana 4 0 2 linux x64 tar gz tar xvf kibana tar gz mv kibana 4 0 2 linux x64 opt kibana Now start Kibana opt kibana bin kibana With everything in place we should be able to access Kibana from the browser at http localhost 5601 Loading data into Elasticsearch with NXLog Loading data with om elasticsearch The NXLog Enterprise Edition comes with a module that can load data natively into Elasticsearch The advantage of this module over om http is the support for bulk data operations and dynamic indexing Event data is sent in batches this greately reduces the latency caused by the HTTP responses and the elasticsearch server can also process bulk data faster The om elasticsearch module can insert data at a rate of 10 000EPS or more on low end hardware For Kibana s time filters to work properly we will need to apply a template This can be pushed to Elasticsearch with the following command curl XPUT localhost 9200 template nxlog d template nxlog mappings default properties EventTime type date format YYYY MM dd HH mm ss This will tell Elasticsearch to treat our EventTime field as date and parse it in the given format This will be applied to all records in indexes beginning with nxlog Further mappings can be added in the properties section The following configuration shows how to use the om elasticsearch module to load the log data into Elasticsearch Module xm json Module im tcp Host 0 0 0 0 Port 1514 InputType Binary Module om elasticsearch URL http localhost 9200 bulk FlushInterval 2 FlushLimit 100 Create an index daily Index strftime EventTime nxlog Y m d IndexType My logs Use the following if you don t have EventTime set Index strftime now nxlog Y m d Path in es The IndexType parameter although not mandatory can be help sorting our logs on the dashboard It expects a string type expression and is reevaluated for each event record Refer to the NXLog Enterprise Edition Reference Manual for further details The input section in the configuration above uses im tcp with NXLog s Binary data format as this can preserve structured data across the network better than JSON The client agent side configuration is not the scope of this document After saving the configuration we should restart NXLog to apply the changes etc init d nxlog restart Logs should now be sent to and indexed by Elasticsearch To verify that the data is getting ingested first we will need to configure the Kibana indexing as shown in the screenshot below Figure 1  Kibana index configuration Kibana index configuration Now we should be able to search and analyze event data on the Kibana interface Figure 2  Kibana search results Kibana search results Loading data with om http Data can be also submitted to Elasticsearch via its HTTP REST API using the om http module

    Original URL path: http://nxlog.org/docs/elasticsearch-kibana/using-nxlog-with-elasticsearch-and-kibana.txt (2016-02-01)
    Open archived version from archive

  • Collecting Windows Security Audit Log data with NXLog and Sysmon
    all drivers except if the signature contains Microsoft or Windows DriverLoad default include Signature condition contains microsoft Signature Signature condition contains windows Signature DriverLoad Do not log process termination ProcessTerminate Exclude certain processes that cause high event volumes ProcessCreate default include Image condition contains noisyprogram exe Image ProcessCreate Do not log file creation time stamps FileCreateTime Do not log network connections of a certain process or port NetworkConnect default include Image condition contains someapp exe Image DestinationPort 4041 DestinationPort NetworkConnect Rules Sysmon The Sysmon manual has more details about the configuration file and the event filtering tags Now we will want to collect and ship these logs to our SIEM or log analytics system This is where NXLog steps in The following nxlog conf configuration collects Sysmon generated event records from the Windows Eventlog For testing purposes we use a configuration which only reads Sysmon s events and writes them to a file in JSON format define ROOT C Program Files x86 nxlog Moduledir ROOT modules CacheDir ROOT data Pidfile ROOT data nxlog pid SpoolDir ROOT data LogFile ROOT data nxlog log LogLevel INFO Extension json Module xm json Extension Input in Module im msvistalog Query QueryList Query Id 0 Select Path Microsoft Windows Sysmon Operational Select Query QueryList Input Output out Module om file File C test sysmon json Exec to json Output Route 66 Path in out Route Sysmon generated events contain event details in a structured format in the EventData section as shown in the screenshot below Figure 2 EventData XML fields of a Sysmon generated Eventlog record This data is recorded in the same structure as the events under the Security log using the Data name key value Data tags NXLog will automatically parse this so that these values are accessible as NXLog fields When converted to JSON with our NXLog configuration the event record will look as follows Note that this JSON is pretty printed for readability NXLog generates single line JSON records EventTime 2015 04 27 15 23 46 Hostname WIN OUNNPISDHIG Keywords 9223372036854775808 EventType INFO SeverityValue 2 Severity INFO EventID 1 SourceName Microsoft Windows Sysmon ProviderGuid 5770385F C22A 43E0 BF4C 06F5698FFBD9 Version 3 Task 1 OpcodeValue 0 RecordNumber 2335906 ProcessID 1680 ThreadID 1728 Channel Microsoft Windows Sysmon Operational Domain NT AUTHORITY AccountName SYSTEM UserID SYSTEM AccountType Well Known Group Message Process Create r nUtcTime 2015 04 27 13 23 r nProcessGuid 00000000 3862 553E 0000 001051D40527 r nProcessId 25848 r nImage c Program Files x86 nxlog nxlog exe r nCommandLine c Program Files x86 nxlog nxlog exe f r nUser WIN OUNNPISDHIG Administrator r nLogonGuid 00000000 568E 5453 0000 0020D5ED0400 r nLogonId 0x4edd5 r nTerminalSessionId 2 r nIntegrityLevel High r nHashType SHA1 r nHash 1DCE4B0F24C40473CE7B2C57EB4F7E9E3E14BF94 r nParentProcessGuid 00000000 3862 553E 0000 001088D30527 r nParentProcessId 26544 r nParentImage C msys 1 0 bin sh exe r nParentCommandLine C msys 1 0 bin sh exe Opcode Info UtcTime 2015 04 27 13 23 ProcessGuid 00000000 3862 553E 0000 001051D40527 Image c Program Files x86 nxlog

    Original URL path: http://nxlog.org/docs/sysmon/audit-logging-on-windows-with-sysmon-and-nxlog.html (2016-02-01)
    Open archived version from archive


  • to identify malicious or anomalous activity and understand how intruders and malware operate on our network On Vista and higher events are stored in Applications and Services Logs Microsoft Windows Sysmon Operational On older systems events are written to the System event log Figure 1  An Eventlog record generated by Sysmon An Eventlog record generated by Sysmon Sysmon can be quite noisy and we may not want to record all the events it logs An XML configuration file can be used with the c or i command line switch to fine tune the behavior of Sysmon Below is an XML configuration example MD5 microsoft windows noisyprogram exe someapp exe 4041 The Sysmon manual has more details about the configuration file and the event filtering tags Now we will want to collect and ship these logs to our SIEM or log analytics system This is where NXLog steps in The following nxlog conf configuration collects Sysmon generated event records from the Windows Eventlog For testing purposes we use a configuration which only reads Sysmon s events and writes them to a file in JSON format define ROOT C Program Files x86 nxlog Moduledir ROOT modules CacheDir ROOT data Pidfile ROOT data nxlog pid SpoolDir ROOT data LogFile ROOT data nxlog log LogLevel INFO Module xm json Module im msvistalog Query Module om file File C test sysmon json Exec to json Path in out Sysmon generated events contain event details in a structured format in the EventData section as shown in the screenshot below Figure 2  EventData XML fields of a Sysmon generated Eventlog record EventData XML fields of a Sysmon generated Eventlog record This data is recorded in the same structure as the events under the Security log using the value tags NXLog will automatically parse this so that these values are accessible as NXLog fields When converted to JSON with our NXLog configuration the event record will look as follows Note that this JSON is pretty printed for readability NXLog generates single line JSON records EventTime 2015 04 27 15 23 46 Hostname WIN OUNNPISDHIG Keywords 9223372036854775808 EventType INFO SeverityValue 2 Severity INFO EventID 1 SourceName Microsoft Windows Sysmon ProviderGuid 5770385F C22A 43E0 BF4C 06F5698FFBD9 Version 3 Task 1 OpcodeValue 0 RecordNumber 2335906 ProcessID 1680 ThreadID 1728 Channel Microsoft Windows Sysmon Operational Domain NT AUTHORITY AccountName SYSTEM UserID SYSTEM AccountType Well Known Group Message Process Create r nUtcTime 2015 04 27 13 23 r nProcessGuid 00000000 3862 553E 0000 001051D40527 r nProcessId 25848 r nImage c Program Files x86 nxlog nxlog exe r nCommandLine c Program Files x86 nxlog nxlog exe f r nUser WIN OUNNPISDHIG Administrator r nLogonGuid 00000000 568E 5453 0000 0020D5ED0400 r nLogonId 0x4edd5 r nTerminalSessionId 2 r nIntegrityLevel High r nHashType SHA1 r nHash 1DCE4B0F24C40473CE7B2C57EB4F7E9E3E14BF94 r nParentProcessGuid 00000000 3862 553E 0000 001088D30527 r nParentProcessId 26544 r nParentImage C msys 1 0 bin sh exe r nParentCommandLine C msys 1 0 bin sh exe Opcode Info UtcTime 2015 04 27 13 23 ProcessGuid 00000000

    Original URL path: http://nxlog.org/docs/sysmon/audit-logging-on-windows-with-sysmon-and-nxlog.txt (2016-02-01)
    Open archived version from archive

  • NXlog GELF Udp is cutting my Doubletake logs | nxlog.co
    31 2016 21 44 45 000000 9976 139690482448128 7 2 0 Target Full Server Failover is allowed 01 31 2016 21 44 45 000000 9976 139690482448128 8 2 0 Source Replication is allowed 01 31 2016 21 44 45 000000 9976 139690482448128 9 2 0 Target Replication is allowed 01 31 2016 21 44 45 000000 9976 139690482448128 10 2 0 Heartbeat Transmission started on port 1500 interval 3seconds 01 31 2016 21 44 48 000000 9976 139690482448128 11 2 69 Kernel Started on bl db01 marsathletic com ip 10 10 100 75 1500 Version 7 1 1 1255 0 01 31 2016 21 44 48 000000 9976 139690482448128 12 2 504002 Double Take has successfully found dev dtrep0 01 31 2016 21 44 48 000000 9976 139690482448128 13 2 52501 Target module loaded successfully 01 31 2016 21 44 48 000000 9976 139690482448128 14 2 0 Disabling all replication from the driver 01 31 2016 21 44 48 000000 9976 139690482448128 15 2 0 Returning default addr 10 10 100 75 1500 01 31 2016 21 44 48 000000 9976 139690482448128 16 2 71 Originator Attempting ip 10 10 150 10 1500 01 31 2016 21 44 48 000000 9976 139689629570816 17 2 73 Connected to IP address ip 10 10 150 10 1500 01 31 2016 21 44 48 000000 9976 139690413819648 18 2 75 Connection resumed with IP address ip 10 10 150 10 1500 01 31 2016 21 44 48 000000 9976 139690482448128 19 2 80 Auto reconnecting Lvra 0d341950 6cba 412c 834d 8afa3426cc83 to ip 10 10 150 10 1500 var lib mysql opt dbtk mnt job 0d341950 6cba 412c 834d 8afa3426cc83 var lib mysql boot opt dbtk mnt job 0d341950 6cba 412c 834d 8afa3426cc83 boot opt dbtk mnt job 0d341950 6cba 412c 834d 8afa3426cc83 01 31 2016 21 44 48 000000 9976 139690482448128 20 2 800 Transmission manually resumed by client 01 31 2016 21 44 48 000000 9976 139690482448128 21 2 0 Returning default addr 10 10 100 75 1500 01 31 2016 21 44 48 000000 9976 139690482448128 22 2 87 Starting replication of set Lvra 0d341950 6cba 412c 834d 8afa3426cc83 for connection 1 01 31 2016 21 44 48 000000 9976 139690482448128 23 2 0 Activating replication on 01 31 2016 21 44 48 000000 9976 139690482448128 24 2 0 Activating replication on boot 01 31 2016 21 44 48 000000 9976 139690482448128 25 2 0 Activating replication on var lib mysql 01 31 2016 21 44 48 000000 9976 139690482448128 26 2 0 Disabling replication on var log DT 01 31 2016 21 44 48 000000 9976 139690482448128 27 2 0 Disabling replication on var cache DT 01 31 2016 21 44 48 000000 9976 139690482448128 28 2 500000 Starting a connection for a Linux Virtual Recovery job 01 31 2016 21 44 48 000000 9976 139690482448128 29 2 500000 Lvra 0d341950 6cba 412c 834d 8afa3426cc83 is connected to ip 10 10 150 10 1500 var lib mysql opt dbtk

    Original URL path: http://nxlog.org/question/1334/nxlog-gelf-udp-cutting-my-doubletake-logs (2016-02-01)
    Open archived version from archive

  • Remote collection of (restricted) file | nxlog.co
    domain I need to read DHCP logs from the DC s UNC path server name C Windows System32 dhcp DhcpSrvLog log Since it is not possible to specify alternate credentials for accessing remote files as it is for eventlog i e im msvistalog module nxlog has to be started using an account with special privileges on the DC s file system 4 options 1 for nxlog service use domain admin account local admin role does not exist on DC nxlog conf use UNC path server name C Windows System32 dhcp DhcpSrvLog log 2 for nxlog service use local admin account on the agent s host share C Windows System32 dhcp on the DC enabling read only permissions for nxlog account only nxlog conf use share name server name dhcp DhcpSrvLog log 3 install nxlog agent on the DC run nxlog as a service use local admin account 4 smaller footprint install http nxlog ce sourceforge net nxlog docs en nxlog reference manual html nxlog processor on the DC None of these options are win wins for customer production environment as they require opening the restricted environment of the DC My question is are there any nxlog configuration options which would

    Original URL path: http://nxlog.org/question/1330/remote-collection-restricted-file (2016-02-01)
    Open archived version from archive

  • 2.9.1347 | nxlog.co
    on a host in Windows domain I need to read DHCP logs from the DC s UNC path server name C Windows System32 dhcp DhcpSrvLog log 2 9 1347 windows im file remotely dhcp Asked January 28 2016 1 52pm 1 0 1 1 answer djontra KISS beginner s problems with im file and om file Hello nxlog world Shamed to say I ve spent entire yesterday trying to figure

    Original URL path: http://nxlog.org/community-forum/2.9.1347 (2016-02-01)
    Open archived version from archive