CouchDB

ELK Stack

The ELK stack, a combination of Elasticsearch, Logstash, and Kibana, is an end-to-end solution for log analytics. Elasticsearch provides search capabilities, Logstash is a log management software, while Kibana serves as the visualization layer. The stack is commercially backed by a company called Elastic.

Elasticsearch

Elasticsearch is a Lucene-based open source distributed search engine designed for high scalability and fast search query response time. It simplifies the usage of Lucene, a highly performant search engine library, by providing a powerful REST API on top. Some of the important concepts in Elasticsearch are highlighted as follows:

  • Document: This is a JSON object stored in an index
  • Index: This is a document collectionType: This is a logical partition of an index representing a category of documents
  • Field: This is a key-value pair within a document
  • Mapping: This is used to map every field with its datatype
  • Shard: This is the physical location where an index’s data is stored (the data is stored on one primary shard and copied on a set of replica shards)

Logstash

This is a tool to collect and process the log events generated by a wide variety of systems. It includes a rich set of input and output connectors to ingest the logs and make them available for analysis. Some of its important features are:

  • The ability to convert logs to a common format for the ease of use
  • The ability to process multiple log formats, including custom ones
  • A rich set of input and output connectors

Kibana

This is an Elasticsearch-based data visualization tool with a wide variety of charting and dashboarding capabilities. It is powered by the data stored in the Elasticsearch indexes and is entirely developed using HTML and JavaScript. Some of its most important features are:

  • A graphical user interface for dashboard construction
  • A rich set of charts (map, pie charts, histograms, and so on)
  • The ability to embed charts in user applications

In a standard ELK stack pipeline, logs from various application servers are transported through Logstash to a central indexer module. This indexer then transmits the output to an Elasticsearch cluster, where it can be queried directly or visualized in a dashboard by leveraging Kibana.

Having already used Solr in Broadside we were already familiar with search engines. But while exploring potential technologies to design the solution for log repository and analytics system, Bitlogg, we chose ELK stack. Later, we could use the same stack for developing a NOC system as well