investigateAccept Pipeline Guide

In this guide we will showcase how to create an investigateAccept pipeline and the use of the different plugins.

Components

In order to configure an investigateAccept pipeline, it is important to understand the different components that can be applied in order for the data to be normalised and ingested as desired.

  1. Input: Specifies how to handle the incoming data based on the input plugins available

    1. beats - handles input provided by the elastic stack beats software (winlogbeat, metricbeat, filebeat)
    2. syslog - handles input provided directly from servers sending syslog data
    3. webhook - handles input provided via Webhooks

  2. Processors: Are used to manipulate data according to the usecase and desired outcome

    1. blacklist - removes fields from an event when blacklisted
    2. extract - extracts data from a given field using regex for additional fields
    3. flatten - flattens a provided json
    4. inject - injects data to a given event
    5. logic - performs logic operations and if true, it follows the next step configured
    6. lowercase - lowercases provided field
    7. rename - renames a given field within an event to a new name
    8. rule - multiple selection of logic statements that are followed based on first hit
    9. setTime - specifies the event field by providing a field and a specified time format
    10. toJson - converts a given event / field to json. If outputField is not provided, it overwrites the current events.
    11. whitelist - adds fields to an event when whitelisted

  3. Output: Specifies where to output the processed data

    1. stdout - outputs all event information to cli output
    2. opensearch - sends the events to an opensearch instance for ingest
    3. investigate - sends the events to an investigate instance for ingest

All inputs, processors, and output plugins must have the below yaml structure, which would include the unique ID of each step as well as a specified next step as shown below:

input/processor/output:
  id: <unique ID>
  name: <name of step>
  plugin: <name of plugin used for this particular step>
  <additional specific functions>: <depending on the plugin these may defer>
  next: <specified next step>

Webhook pipeline example

input:
  id: 1
  name: webhook
  plugin: webhook
  next: [2]
  bindPort: 443
  method: POST
  prefix: <random-string for accessing the url>
  sslCertificate: <certificate file>
  sslPrivateKey:  <private key file>

processor:
  id: 2
  name: toJson
  plugin: toJson
  next: [3]

output:
  id: 3
  name: opensearch
  plugin: opensearch
  index: webhook-index
  password: password
  url: https://opensearch-url
  username: username
  verify: true

Beats pipeline example

An example beats pipeline can be downloaded from the below link:

Beats pipeline example (.yaml)