Documentation

Connect Logstash to OpenSearch

Logstash is an open source, server-side data processing pipeline that allows for the collection and transformation of data on the fly. Logstash supports a range of input sources, these range from HTTP to s3 bucket events.

Logstash is commonly used as an input pipeline for OpenSearch as it allows for on the fly data transformation. These transformations can be applied through various filter plugins.

There are two different versions of Logstash. Logstash OSS is open source under the Apache 2.0 Licence, and the other has OpenSearch proprietary extensions under a different license. Instaclustr recommends that Logstash OSS is installed and used with our Managed OpenSearch offering. This document details the minimum requirements to connect Logstash to an Instaclustr Managed OpenSearch instance, and includes an example Docker setup. However your installation may vary and this document should only be used as a guide. Details on how to set up Logstash OSS for your use case should be followed from the logstash-output-opensearch page which is maintained by the OpenSearch community.

Logstash Use Case

A diagram explaining how Logstash can be used with OpenSearch

Security Plugin Configuration

Instaclustr makes use of the OpenSearch Security Plugin, allowing for node to node encryption and role based access control. This plugin is automatically enabled on all OpenSearch clusters

You must create an account for Logstash to use for connections to the cluster. To create the account, first, get your connection credentials. Under the “Connection Info” section of the console you will be met with the default credentials for your cluster. These credentials will be required to add a Logstash user account to the security plugin.

In addition to the above credentials, if you are intending to use the OpenSearch API to create a Logstash user ID and password you will need to obtain the .pem file from the Instaclustr Console. This is needed to ensure secure communication to and from your cluster. Download the cluster certificates from the Connection Info page of the Instaclustr console.

As outlined in the Security Access Control support article, follow the steps for creating a new user, ensuring to give only necessary permissions needed for Logstash to operate. These permissions are as follows:

Cluster Permissions:
- cluster_monitor
- cluster_composite_ops
- indices:admin/template/get
- indices:admin/template/put
- cluster:admin/ingest/pipeline/put
- cluster:admin/ingest/pipeline/get
Index Permissions:
- < your index pattern name >
- write
- create
- delete
- create_index

Logstash Installation and Configuration

Logstash can be installed using a package manager on macOS and Linux, or through a Docker container. More details can be found on the Logstash installation page. Once installed, you will need to create a configuration file which can be placed in the /pipeline/ directory of your Logstash installation. This file can be called any name, but must end with a .conf file extension.

Logstash configuration file is made up of three parts, where plugins (included as part of the Logstash installation) are used in each part:

Input—Where is the data coming from. This can be a file, an API or a service such as Kafka.

Filter—What do you want to do with the incoming data.

Output—The data that gets output and where do you want to output it to?

For a basic configuration file for a connection to an Instaclustr Managed OpenSearch instance your configuration will need at minimum:

input {
    # Required - An input plugin to pass some data to OpenSearch - file or API
}

filter {
    # Can be empty
}

output {
    # All of the following connection details
    opensearch {
        # Your node IP addresses from the Instaclustr Console
        hosts => ["https://<NodeIP1>:9200","https://<NodeIP2>:9200","https://<NodeIP3>:9200"]
        # SSL enabled
        ssl => true
        ssl_certificate_verification => true
        # Path to your cluster certificate .pem downloaded earlier
        cacert => "<Path to: cluster-ca-certificate.pem>"
        # The Logstash Username and Password created earlier
        user => "<LogstashUserName>"
        password => "<LogstashPassword>"
        # The name of the index
        index => "<Intended Name Of Your Index>"
    }
}

input {

# Required - An input plugin to pass some data to OpenSearch - file or API

}

filter {

# Can be empty

}

output {

# All of the following connection details

opensearch {

# Your node IP addresses from the Instaclustr Console

hosts => ["https://<NodeIP1>:9200","https://<NodeIP2>:9200","https://<NodeIP3>:9200"]

# SSL enabled

ssl => true

ssl_certificate_verification => true

# Path to your cluster certificate .pem downloaded earlier

cacert => "<Path to: cluster-ca-certificate.pem>"

# The Logstash Username and Password created earlier

user => "<LogstashUserName>"

password => "<LogstashPassword>"

# The name of the index

index => "<Intended Name Of Your Index>"

}

Once you have created the configuration file, place the .conf file in the /pipeline/ directory and start Logstash based on your installation method. If Logstash is started directly, you will be able to see success or error messages directly in the terminal. If Logstash is started as a system process, you should check the logs of your Logstash installation for more details.

Adding Logstash Data to OpenSearch Dashboards

Navigate to your OpenSearch Dashboards instance and log in using the credentials from the Instaclustr Connection Info Page.
Head to Manage > Index Patterns > Create Index Pattern
If successful, you should see your index as defined in the OpenSearch Output plugin above.
From here, manipulating your data is identical to using any other data source in OpenSearch Dashboards.

Troubleshooting

Problem Observed	Possible Reason	Solution
No Config Files Found In Path	Logstash requires all configuration files to be placed within the /pipeline directory of the application.	Utilize docker copy or docker volumes when creating the docker image to move the desired config file to its correct destination.
Dead ES Instance	Client IP address is not added as an exception within the dashboard.	Add client IP to firewall exceptions within the Instaclustr Dashboard.
Error 401	Authentication failed due to incorrect SSL/credentials configuration.	Check to make sure SSL certificate is supplied and correct credentials are entered into the configuration file.

Note: Utilizing proprietary extensions within the open source Logstash version can lead to miscellaneous syntax errors. Ensuring ilm_enabled is set to false should help to rectify these issues. Syntax errors may also arise if anything other than a configuration file is placed within the /pipeline directory.

Example Docker Installation

Pull the desired Docker image from the Logstash Repo.

sudo docker pull opensearchproject/logstash-oss-with-opensearch-output-plugin:7.13.2

Below is a sample configuration file designed to stash a .csv to OpenSearch. For connection to OpenSearch you will require the credentials for the logstash user you created in the security plugin configuration section as well as the IP addresses of your OpenSearch data nodes.

Note: All paths are required to be absolute paths directly to a given file.

Create your configuration file and store it in a location that can be accessed by docker. The OpenSearch output for your configuration file should be as follows:

input {
    file {
        path => "/usr/share/logstash/test.csv"
        start_position => "beginning"
    }
}

filter {
    csv {
        separator => ","
        columns => [ "record_id", "duration", "src_mac", "dest_mac" ]
    }
}

output {
    opensearch {
        hosts => ["https://<NodeIP1>:9200","https://<NodeIP2>:9200","https://<NodeIP3>:9200"]
        ssl => true
        ssl_certificate_verification => true
        cacert => "<Path to: cluster-ca-certificate.pem in the container (see volume)>"
        user => "<LogstashUserName>"
        password => "<LogstashPassword>"
        index => "testdata"
    } 
    stdout {}
}

input {

file {

path => "/usr/share/logstash/test.csv"

start_position => "beginning"

}

filter {

csv {

separator => ","

columns => [ "record_id", "duration", "src_mac", "dest_mac" ]

}

output {

opensearch {

hosts => ["https://<NodeIP1>:9200","https://<NodeIP2>:9200","https://<NodeIP3>:9200"]

ssl => true

ssl_certificate_verification => true

cacert => "<Path to: cluster-ca-certificate.pem in the container (see volume)>"

user => "<LogstashUserName>"

password => "<LogstashPassword>"

index => "testdata"

}

stdout {}

}

The above logstash.conf file will need to be placed in the /pipeline directory of the docker container. This can be done using Docker volumes within the below docker compose file or using docker copy once your container has been established.

version: '3'
services:
    logstash:
    image: opensearchproject/logstash-oss-with-opensearch-output-plugin:7.13.2
    container_name: logstash
    volumes: 
    - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf 
    - ./cluster-ca-certificate.pem:/usr/share/logstash/cluster-ca-certificate.pem

version: '3'

services:

logstash:

image: opensearchproject/logstash-oss-with-opensearch-output-plugin:7.13.2

container_name: logstash

volumes:

- ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf

- ./cluster-ca-certificate.pem:/usr/share/logstash/cluster-ca-certificate.pem

Run the configured Docker container using the following command:

sudo docker-compose up

Once the container is started, as outlined within the logstash.conf file from step 2, Logstash will now attempt to pass the logs as configured in the Logstash configuration file from the path defined in the docker-compose config file to your OpenSearch cluster.

By Instaclustr Support

Previous Article Use VPC Peering (AWS) to Connect to OpenSearch Next Article How to perform a snapshot migration from Amazon OpenSearch Service

Need Support?

Learn More

Experiencing difficulties on the website or console?

Status page for known incidents

Already have an account?

Need help with your cluster?

Contact Support

Why sign up?

To experience the ease of creating and managing clusters via the Instaclustr Console

Spin up a cluster in minutes

Free Trial