Ingesting Syslog Events

Introduction

Syslog helps system administrators monitor the health status of applications or resources using Syslog messages.

A Syslog is a daemon process that runs on either Linux or Windows servers. The process sends events and log information to a collector (Syslog server) over an IP network. The gateway gathers and processes the Syslog messages according to a set of rules, translates messages to corresponding alerts, and forwards them to the cloud.

Syslog messages

Syslog messages include specifications that help identify information such as when, where, and how the log was sent. Syslog messages usually include the following details:

• IP Address
• Priority Number
• Timestamp
• Actual log message
• Hostname
• Tag

Example of a Syslog message: <25>1 2020-06-24T12:01:39.625926+05:30 bvrmlpt098 appname - - [timeQuality tzKnown="1" isSynced="1" syncAccuracy="253500"]This is an actual syslog Message

Syslog message description

Syslog message format

A Syslog message consists of three parts:

• PRI
• HEADER
• MSG

The PRI data sent using the Syslog server help arrange and classify the message using two numeric values:

• Facility
• Severity

Facility

Classifies the message type that generated the Syslog event. The facility value is one of the fifteen predefined values or locally defined values ranging from 16-23.

Severity

Classifies the severity or importance of the messages ranging from 0-7.

The priority value(PRI) provides the facility and severity of the messages generated from a system. Multiply the facility value by eight and add the severity value to the result to calculate the PRI: {Faclity value * 8} + Severity = PRI

The lower the PRI, the higher the priority.

Syslog advantages

• Increased security
• Easier detection of any issues
• Timely action
• High performance

Syslog process flow

Syslog monitor configuration

Syslog monitor configuration is used to:

• Forward thousands of messages sent from different devices to a daemon exclusively.
• Configure Syslog listeners: This specifies that Syslog messages are received and processed using TCP/UDP from the target resources.
• Define include and exclude list: This pre-processes all incoming messages and optimizes CPU cycles to process rules for all messages.
• Define the set of rules: This provides granular-level filtering for important messages and subsequent alert generation.

Syslog is configured using Setup > Monitoring > Syslog Monitor Configuration and involves:

1. Adding a configuration profile
   Configuration profiles set the severity and facility of the Syslog messages. Profiles also instruct the Gateway to filter only those messages that match the facility and severity defined by the user.
2. Adding rules
   Rules set alert preferences, Syslog message patterns, and actions. For example, rules can be specified to receive warning alerts for user-level messages and mail system that includes the regex pattern: [(.+)] [(.+)] (.+).

Creating Syslog monitoring profiles

The Syslog Monitor Configuration functionality is used to filter messages and then generate alerts from the desired messages. For example, to stop receiving alerts for mail system messages, the system can be “trained” to not monitor those messages.

Prerequisites

• Syslog server installed in the managed environment.
• Good knowledge of regular expressions.

To create a Syslog monitoring configuration:

1. Select a client from the All Clients list.
2. Select Setup > Monitoring > Syslog Monitoring Configuration.
3. From the SYSLOG MONITORING CONFIGURATION screen, select one of the following tabs:
   • Configuration Profile
   • Rules
4. To add a profile:
   1. Click +Add.
   2. From the Configuration Profile section, provide the following information and click Next:
      • Client: Client name.
      • Management Profile: List of management profiles configured in Resources > Management Profiles.
      • Configuration Name: Name provided for the configuration profile.
      • Description: Brief description of the purpose for the profile.
   3. From the Global Filters section, provide the following information and click Save:
      • Severity: Level of intensity of the Syslog messages.
      • Facility: Type of messages to monitor (using Syslog) and use to generate alerts.
      • Resource Filter: IP range for servers used to monitor and receive messages.
        Use * to enable all devices to send Syslog messages.
      • Rules: Rules that are set to identify the Syslog messages.
5. To add a new rule:
   1. Click +Add.
   2. From the Rules section, provide the following information, and click Submit:
      • Scope: Partner or client-specific rules.
      • Name: Rule name.
      • Action: Determines how to process the Syslog messages.
        • Include: Sends only matching data.
        • Exclude: Ignores the matching data and sends the remaining data.
      • Regex Pattern: The regular expression pattern that is used to compile and compare with the incoming messages. OpsRamp sends an alert in case of a match.
      • Metric Name: Name of the metric. Note: Regular expressions can also be used to specify a metric name. For example, Cisco Emergency Responder.
      • Component: Syslog message module. For example, Authentication Failure.
      • Alert Subject: Summary content of the alert.
      • Alert Description: Brief description of the entities used to generate alerts.
      • Alert Severity: Severity of the alerts.
      • Tags: User-defined tags used for filtering.
        The SYSLOG MONITORING CONFIGURATION screen displays the configured profiles and rules.

Configuration profile and rules

The resource filters added in the configuration profile are processed sequentially. If multiple profiles are added to the same gateway, the profiles are processed alphabetically. For example, a client gateway consists of three configuration profiles namely UC-1, UC-2, and UC-3. Each contains ten rules. The gateway executes the messages using one of the following sequences:

• The gateway runs the configuration profiles in alphabetical order. For example:
  • UC-1
  • UC-2
  • UC-3
• The execution of rules for an incoming message follows the sequence as defined in the individual profiles. The gateway compares the incoming Syslog message with the rules from profile UC-1.
  • for any match, the gateway executes the rule and repeats the process for the next message.
  • If no rules in the UC-1 match, then the gateway compares the Syslog message with the profile UC-2.
  • If no rules match, the execution process continues with the rest of the profiles similarly.

The rules displayed in the Rules tab depends on one of the following scopes:

• Service provider rules
• Partner rules
• Client rules

After configuring the rules, go to Add > New Configuration Profile from the Rules tab to add configuration profiles for multiple rules. This facilitates the ease of searching and filtering multiple rules from the Configuration Profiles tab.

Extracting regular expressions dynamically

If there is a Syslog message-rule match, use regular expressions to dynamically extract groups to generate an Alert Subject, Alert Description, Component, and Metric Name. For example, from the syslog message: [08:52:18] [ERROR] Ceci doit aparaitre and the regular expression: [(.+)] [(.+)] (.+), the extracted groups are:

• Group 1 = 08:52:18
• Group 2 = ERROR
• Group 3 = Ceci doit aparaitre.

Use the extracted groups along with the regular expressions to dynamically generate the alert subject, description, and component details. For example, Severity: ${2}. Syslog raised time ${1} dynamically generates Severity: ERROR. Syslog raised time 08:52:18.

Predefined macros for syslog

• ${timestamp} - Replaces the timestamp in milliseconds.
• ${received.syslog.message} - Replaces the raw Syslog message received from the remote device.

Viewing details

After configuring the Syslog attributes, view the details (such as configuration profile and rules) on the Syslog Monitoring Configuration screen.

Viewing configuration profiles

View profile details in the Syslog Monitoring Configuration > Configuration Profiles tab.

Viewing rules

View rule details in the Syslog Monitoring Configuration > Rules tab.

Searching profiles and rules

Searching is available on the Syslog Monitoring Configuration screen. Methods used for searching include:

• Regular search
• Advanced search
• Advanced search - configuration profile
• Advanced search - rules

Search

Use the search option to find a configuration profile and rules using the configuration name and rule name. For searching with criteria, use the Advanced option.

Advanced search

Search results can be filtered by using the advanced search.

Advanced search – configuration profile

To search using more options:

1. Click Advanced.
2. From ADVANCED SEARCH window, provide the following information:
   • Client
   • Configuration Name
3. Click Search.

The Configuration Profile screen displays the search results.

Advanced search – rules

To search using more options:

1. Click Advanced.
2. From ADVANCED SEARCH, provide the following information:
   • Client
   • Action
   • Tags
3. Click Search.

The Rules screen displays the search results.

Deleting profiles and rules

Use the Remove option to delete the existing configuration profiles or rules. To delete a Syslog configuration from a single gateway management profile, remove the manage profile from the respective Syslog configuration.