Answering Key Questions About the ELK Stack

Organizations are increasingly turning to the ELK Stack to search and analyze massive data volumes. ELK is an acronym derived from the three technologies that formed the initial stack: Elasticsearch, Logstash, and Kibana. Some are incorporating the stack’s search capabilities into apps and websites, providing simple, fast ways to find information. Others are employing the ELK Stack to monitor the performance, availability, and security of apps and infrastructure by analyzing data from a growing number of sources.

But what is the ELK Stack? Why are the component technologies so popular? And what are some of the ways that you might use the ELK Stack in your organization?

What Is the ELK Stack?

The ELK Stack is a package of open source technologies for collecting, searching, analyzing, and visualizing large data volumes generated by diverse sources. The stack originally included only Elasticsearch, Logstash, and Kibana. But in 2015, Elastic added another open source technology: Beats. Rather than changing the acronym, Elastic now refers to the augmented stack as the Elastic Stack.

What Is Elasticsearch?

Elasticsearch is a modern, open source full-text search and analytics engine. The heart of the ELK Stack, Elasticsearch can be used for searching a full array of data types—from text, numbers, and geospatial data to other types of structured and unstructured data. 

Built on the Apache Lucene library, Elasticsearch has a distributed architecture, offers simple, REST APIs, and stores data as schema-free JSON documents. It is easy to use and is scalable, enabling you to rapidly search fast-growing volumes of data.

Read: OpenSearch and Elasticsearch Architecture

What Is Logstash?

Logstash is an open source, server-side data processing pipeline that dynamically ingests data transforms it, and ships it to whatever location (or “stash”) you define. It can simultaneously ingest unstructured data streaming in from numerous sources—including websites, application servers, and data stores. 

Logstash filters and parses the data it collects, transforming it into a common format. It then sends that data wherever you want it to go. Many organizations send the transformed data to Elasticsearch, where logs can be indexed and searched. Once data is available in Elasticsearch, it can also be visualized with Kibana.

What Is Kibana?

Kibana is an open source data analysis and visualization tool that turns the data stored in Elasticsearch into easily consumable charts, graphs, histograms, and other visual representations. Through a browser-based interface, you can use preconfigured dashboards to explore large data volumes.

Kibana provides a useful way to share insights across your organization. Non-technical users can easily see trends and assess KPIs, all through rich, customizable graphics.

What Are Beats?

An addition to the ELK Stack, Beats are lightweight, single-purpose data shippers. There are several different distinct Beats, each focused on specific types of data, including log files, metrics, network packets, Windows events, audit data, uptime monitoring data, and cloud data. The open source community is developing additional Beats. 

Beats can reside on servers or be deployed as functions. Each Beat has a simple task: gather data and ship it to Elasticsearch. Shipped data conforms with the Elastic Common Schema (ECS). If data needs to be transformed, you can use an ingest pipeline or ship it to Logstash.