devkuma – Elasticsearch

Basic Elasticsearch Concepts

kc@example.com (kc kim) — Fri, 03 Sep 2021 09:09:00 +0900

Elasticsearch

Elasticsearch is an open source search engine developed in Java and based on Apache Lucene.
Elasticsearch is a distributed engine that searches for specific strings across multiple documents or files.
Because it stores data in an inverted index structure, it provides excellent performance for full-text search compared with an RDBMS.
It is usually used with Logstash, Kibana, and Beats as the ELK stack rather than on its own.
It is used for many purposes, including data storage, document search, location search, machine-learning-based search, log analysis, and security inspection analysis.

Full-text Search

Searching for a specific string across multiple documents or files is called full-text search.

Full-text search technologies include the grep type and the index type. Elasticsearch provides fast search by distributing index-type full-text search across multiple computers.

Grep type

This method searches multiple files sequentially. As the search target grows, search speed drops significantly. The UNIX string search command grep is an example.

Index type

Like a book index, this method creates indexes for multiple documents in advance to improve search speed. Examples using index-type full-text search are as follows.

Elasticsearch document search
Google Search: searches for specific strings across multiple documents or web pages
GitHub code search: searches for specific strings across multiple source-code documents

Here is an example of building an index from the following two documents.

Document 1: I Like search engine.
Document 2: I search keywords by google.

If a word is included in a document, mark it as 1; if it is not included, mark it as 0. The following index is created.

Index

	I	like	search	engine	keywords	by	google
Document 1	1	1	1	1	0	0	0
Document 2	1	0	1	0	1	1	1

However, this index can only be searched row by row from left to right. Therefore, to find documents that use the word google, every row must be scanned.

Elasticsearch uses an index called an inverted index. An inverted index is an inverted matrix where the rows and columns of the index are swapped, as shown below.

Inverted Index

	Document 1	Document 2
I	1	1
like	1	0
search	1	1
engine	1	0
keywords	0	1
by	0	1
google	0	1

In an inverted index, to search for documents where the word google appears, only the google row needs to be scanned.

Index and Inverted Index

In Elasticsearch, an index corresponds to a database in an RDBMS.

Index
- A sorted or listed collection that makes keywords easy to find.
Inverted Index
- A method for finding documents through keywords.

The benefit of an inverted-index structure is very fast search. Simply put, the table of contents at the front of a book corresponds to an index, while the keyword lookup at the back of a book corresponds to an inverted index. Elasticsearch parses text, creates a search-term dictionary, and stores text with the inverted-index method.

Elasticsearch Features

Elasticsearch has the following features.

Scale out
- It can scale horizontally through shards.
Availability
- Replicas ensure data stability.
Schema free
- Because searches are performed on JSON documents, there is no schema concept.
RESTful
- Data CRUD operations are performed through HTTP RESTful APIs.

Elasticsearch Use Cases

Internal and external document search

Elasticsearch is used to search company documents or service documents. Corporate use cases include the following.

Wemakeprice: an online web store that can search products for sale, including product catalogs, inventory storage, and autocomplete
- [Open Source SW Success Case 203] Wemakeprice - Hot deals delivered with open source SW
GitHub: source-code search
- GitHub uses Elasticsearch to index over 8 million code repositories | Elastic Customers

Anomaly detection

Elasticsearch can detect anomalies in access logs and similar data. Corporate use cases include the following.

Wemakeprice: unstructured threat detection through machine learning
- Wemakeprice: Introducing machine learning for unstructured threat detection in an AWS cloud environment
11st: monitoring and anomaly detection
- Order/payment monitoring and anomaly detection with Elasticsearch, alerts, and machine learning
Netflix: security log monitoring
- Elasticsearch in Netflix from Danny Yuan

Comparing Elasticsearch and RDBMS

The concepts correspond to a relational database as follows.

Relational Database	Elasticsearch
Database	Index
Table	Type
Row	Document
Column	Field
Index	Analyze
Primary key	_id
Schema	Mapping
Physical partition	Shard
Logical partition	Route
Relational	Parent/Child Nested
SQL	Query DSL

Source: https://www.slideshare.net/deview/2d1elasticsearch

Index structure differences

Elasticsearch
- Stores data in an inverted-index structure.
- Like a book index, it stores documents that contain a specific word.
RDBMS
- Uses B-Tree or similar indexes.

Elastic Stack

Elastic Stack consists of Elasticsearch and the following three open source projects. Elasticsearch is sometimes used alone for search, but it is often used as the ELK stack.

While being used together with Logstash and Kibana, Elastic was widely known for a time as the ELK Stack (Elasticsearch, Logstash, Kibana). In 2013, it formally absorbed the Logstash and Kibana projects and began developing them together under one roof. In 2015, the company changed its name from Elasticsearch to Elastic and officially named the product Elastic Stack instead of ELK Stack, while continuing to develop and expand features such as monitoring, cloud services, and machine learning. Excerpt from the official documentation

Logstash

A tool that transforms the format of data sent to Elasticsearch. It handles the Transform and Load parts of ETL.
It is used to convert logs into an appropriate data format when sending logs from application servers to Elasticsearch servers.
- It collects, aggregates, parses, and delivers logs or transaction data from various sources such as databases and CSV files to Elasticsearch.

Beats

A shipper that transfers data. It handles the Extract part of ETL.
It sends data from application servers to Logstash or Elasticsearch.

Elasticsearch

Searches and aggregates data received from Logstash and returns the information of interest.

Kibana

Visualizes and monitors data through Elasticsearch’s fast search.
It is a BI-like tool for visualizing Elasticsearch documents.

Elasticsearch Architecture

Elasticsearch architecture is divided into the following two concepts.

Logical concepts, which are software-level terms
Physical concepts, which are hardware-level terms

Logical concepts

The overall diagram of Elasticsearch logical concepts is as follows.

Logical concept diagram

Each term is described below.

Field

A field is a pair of an item name, or key, and a value. It corresponds to the contents of a file in a file system. Each key has a type.

text type for strings
date type for time values

Document

A set of fields is called a document. It corresponds to a file in a file system. A document is a JSON object.

Index

A set of documents is called an index. It corresponds to a folder in a file system. When a document is stored in an index, an inverted index is created.

Mapping

Mapping defines the value data type and similar information for each field key. One mapping exists per index.

In the mapping shown in the logical concept diagram above, the Content key is defined as the Text type, so the value of the Content key is always a string of the Text type.

Physical concepts

The overall diagram of Elasticsearch physical concepts is as follows. It also shows the relationship with the logical index concept.

Relationship between physical and logical concepts

Node

A node is one unit process that makes up Elasticsearch, and it is one Elasticsearch server.

Multiple Elasticsearch nodes can be started on one OS.

Nodes have the following four role attributes, and one node can have multiple attributes.

Master Node
- A node that manages cluster metadata and similar information.
- Only one master node exists per cluster.
- Nodes that can be promoted to master are called master-eligible nodes, but they are not master nodes.
Data Node
- A node that stores actual data.
- It handles requests such as search and aggregation.
- It routes requests to other nodes, for example when a shard exists on another node.
Ingest Node
- Performs data transformation and processing, then stores data on data nodes.
- It has the same role as Logstash.
Coordinating Node
- Handles request routing, which data nodes can also do.
- This role is used when you want a node dedicated to routing so that routing workload is not added to data nodes.

Cluster

In Elasticsearch, a cluster is the largest system unit and means a group of nodes made up of at least one node. When a search request is sent to a cluster, search processing is distributed across each node.

Different clusters remain independent systems that cannot access or exchange data with each other. Multiple servers can form one cluster, and multiple clusters can exist on one server.

A computer cluster is a group of computers connected together so that they operate like one system.

Shard

A shard is data created by splitting the index data described as a logical concept and storing it on nodes. This allows search processing to be distributed across nodes. The actual substance of a shard is a Lucene index file.

There are two types of shards.

Primary shard
- The original data.
- The shard where update processing is first performed.
Replica shard
- A copy of the primary shard.
- It is replicated after the primary shard update finishes.
- It is used for distributing search load and backing up data.

Shards and replicas

Shards and replicas are not concepts that exist only in Elasticsearch; they also exist in distributed database systems.

Shard
- Sharding means distributing and storing data. In other words, an index is split into multiple shards for scale-out.
- At least one shard exists by default, and the number of shards in a cluster is sometimes tuned to improve search performance.
- There is a limit to the amount of data that can be stored on a single node.
- A cluster can split the data of a single index and use cluster storage, memory, and processing capacity automatically.
- By default, the number of shards is set to 5. You can specify the number of shards for splitting index data when creating an index. After an index is created, the number of shards cannot be changed.
Replica
- A replica can be seen as another type of shard.
- Preparing replicas, or copies, helps resolve failures and continue operation without problems. In other words, shards are replicated for data reliability when a node is lost. Therefore, replicas are recommended to exist on different nodes.
- Each shard in an index can have zero or more replicas.
Summary
- The smallest Elasticsearch unit is a node, and a node stores data and executes queries.
- One or more nodes make up a cluster.
- Inside a node, data can be managed more efficiently through shards. Replicas can also be added in preparation for failures, increasing failover capability and data availability.
  - Availability means the degree to which information systems such as servers, networks, and programs can be used normally. It has a similar meaning to uptime.

References

Installing Elasticsearch, Kibana, and Cerebro

kc@example.com (kc kim) — Fri, 03 Sep 2021 00:00:00 +0000

Install from downloaded files

Install Java

Elasticsearch runs on the JVM, so Java 8 (JDK) must be installed.

If it is not installed, install the JDK first.

Install the Java 8 (JDK) runtime.

$ sudo yum install java-1.8.0-openjdk -y

Install the Java 8 (JDK) development environment.

$ sudo yum install java-1.8.0-openjdk-devel -y

Check the Java version.

$ java -version
openjdk version "1.8.0_252"
OpenJDK Runtime Environment (build 1.8.0_252-b09)
OpenJDK 64-Bit Server VM (build 25.252-b09, mixed mode)

Install Elasticsearch

Now install Elasticsearch.

First, download the installation file.

$ wget https://artifacts.elastic.co/downloads/kibana/kibana-7.14.1-linux-x86_64.tar.gz

Extract the downloaded file.

$ tar xvfz elasticsearch-7.14.1-linux-x86_64.tar.gz

Rename the directory to a shorter, easier-to-use name.

$ mv files/elasticsearch-7.14.1-linux-x86_64/ es-714

Move to the bin directory.

$ cd es-714/bin/

Run the startup file.

$ ./elasticsearch

[kimkc@localhost bin]$ ./elasticsearch
[2021-09-02T21:45:57,952][INFO ][o.e.n.Node               ] [localhost.localdomain] version[7.14.1], pid[116654], build[default/tar/66b55ebfa59c92c15db3f69a335d500018b3331e/2021-08-26T09:01:05.390870785Z], OS[Linux/4.18.0-338.el8.x86_64/amd64], JVM[Eclipse Foundation/OpenJDK 64-Bit Server VM/16.0.2/16.0.2+7]
[2021-09-02T21:45:57,957][INFO ][o.e.n.Node               ] [localhost.localdomain] JVM home [/home/kimkc/es-714/jdk], using bundled JDK [true]
[2021-09-02T21:45:57,958][INFO ][o.e.n.Node               ] [localhost.localdomain] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-1671236956648459411, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms166m, -Xmx166m, -XX:MaxDirectMemorySize=87031808, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/home/kimkc/es-714, -Des.path.conf=/home/kimkc/es-714/config, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true]
[2021-09-02T21:46:00,777][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [aggs-matrix-stats]
[2021-09-02T21:46:00,778][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [analysis-common]
[2021-09-02T21:46:00,778][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [constant-keyword]
[2021-09-02T21:46:00,779][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [frozen-indices]
[2021-09-02T21:46:00,779][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [ingest-common]

... omitted ...

Check whether it is accessible.

$ curl -XGET localhost:9200

$ curl -XGET localhost:9200
{
  "name" : "es-714",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "8mO5vPIJQS-Eqa3PUvBVcA",
  "version" : {
    "number" : "7.14.1",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "66b55ebfa59c92c15db3f69a335d500018b3331e",
    "build_date" : "2021-08-26T09:01:05.390870785Z",
    "build_snapshot" : false,
    "lucene_version" : "8.9.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
[kimkc@es-714 ~]$

If it runs as shown above, the installation completed successfully.

Install Kibana

Download the installation file.

$ wget https://artifacts.elastic.co/downloads/kibana/kibana-7.14.1-linux-x86_64.tar.gz

Extract the downloaded file.

$ tar xvfz kibana-7.14.1-linux-x86_64.tar.gz

Rename the directory to a shorter, easier-to-use name.

$ mv files/kibana-7.14.1-linux-x86_64/ kb-714

Move to the bin directory.

$ cd kb-714/bin/

Run the startup file.

$ ./kibana

Install with Docker

For simple learning or experimentation, installing with Docker is recommended.

Install Elasticsearch with Docker

Download the Elasticsearch Docker image

Run the following command to download the Elasticsearch Docker image. Here, version 7.14.0 is used.

docker pull docker.elastic.co/elasticsearch/elasticsearch:7.14.0

The result is as follows.

% docker pull docker.elastic.co/elasticsearch/elasticsearch:7.14.0
7.14.0: Pulling from elasticsearch/elasticsearch
ddf49b9115d7: Pull complete
693712dd4782: Pull complete
e1843b23879c: Pull complete
43b0526f3562: Pull complete
395a8174c4c5: Pull complete
671bce37ebcd: Pull complete
7b0bb020b357: Pull complete
Digest: sha256:46815a329dfdf633cac9261e287c8440512db13554cbcb4fa89842dde58e74b2
Status: Downloaded newer image for docker.elastic.co/elasticsearch/elasticsearch:7.14.0
docker.elastic.co/elasticsearch/elasticsearch:7.14.0

Create and run the Elasticsearch Docker container

Run the following command. The Docker container name is set to es714.

docker run -d -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" --name es714 docker.elastic.co/elasticsearch/elasticsearch:7.14.0

The result is as follows.

% docker run -d -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" --name es714 [docker.elastic.co/elasticsearch/elasticsearch:7.14.0](http://docker.elastic.co/elasticsearch/elasticsearch:7.14.0)
a3b75abf13c211cadf86637fe1c457b4b465556fc231a9a243362da49b117952

Check the Elasticsearch Docker container

Run the following command to check that the container is running.

docker ps

The result is as follows.

% docker ps
CONTAINER ID   IMAGE                                                  COMMAND                  CREATED              STATUS              PORTS                                                                                  NAMES
a3b75abf13c2   docker.elastic.co/elasticsearch/elasticsearch:7.14.0   "/bin/tini -- /usr/l..."   About a minute ago   Up About a minute   0.0.0.0:9200->9200/tcp, :::9200->9200/tcp, 0.0.0.0:9300->9300/tcp, :::9300->9300/tcp   es714

Check the Elasticsearch Docker settings

Run the following command to check the settings.

docker exec -i -t es714 cat /usr/share/elasticsearch/config/elasticsearch.yml

The result is as follows.

% docker exec -i -t es714 cat /usr/share/elasticsearch/config/elasticsearch.yml
cluster.name: "docker-cluster"
network.host: 0.0.0.0

Install Kibana with Docker

Pull the Kibana Docker image

Run the following command to pull the Kibana Docker image. Use the same version as Elasticsearch, 7.14.0.

docker pull docker.elastic.co/kibana/kibana:7.14.0

The result is as follows.

% docker pull docker.elastic.co/kibana/kibana:7.14.0
7.14.0: Pulling from kibana/kibana
ddf49b9115d7: Already exists
b62151c99641: Pull complete
d11b460fe76c: Pull complete
1078fcc7cd25: Pull complete
c74f7ca3ab86: Pull complete
2fb0a75edcfc: Pull complete
339c3eb65140: Pull complete
c359ec1d1586: Pull complete
7c0bddf401bf: Pull complete
59df587bba1b: Pull complete
34afeb086a05: Pull complete
4de583c3a000: Pull complete
961084329244: Pull complete
Digest: sha256:6408b11743abb2855e62406edb3cfd53848b97e48c65c0f4d843c427ccb3bb90
Status: Downloaded newer image for docker.elastic.co/kibana/kibana:7.14.0
docker.elastic.co/kibana/kibana:7.14.0

Create and run the Kibana Docker container

Run the following command.

docker run -d --link es714:elasticsearch -p 5601:5601 --name kb714 docker.elastic.co/kibana/kibana:7.14.0

The result is as follows.

% docker run -d --link es714:elasticsearch -p 5601:5601 --name kb714 docker.elastic.co/kibana/kibana:7.14.0
becb75fdb608c119407ea37e1f75f54898d64683fdb76674b08e8c875e1fca77

Check the Kibana Docker container

Run the following command.

docker ps

The result is as follows.

% docker ps
CONTAINER ID   IMAGE                                                  COMMAND                  CREATED             STATUS             PORTS                                                                                  NAMES
becb75fdb608   docker.elastic.co/kibana/kibana:7.14.0                 "/bin/tini -- /usr/l..."   13 seconds ago      Up 12 seconds      0.0.0.0:5601->5601/tcp, :::5601->5601/tcp                                              kb714
a3b75abf13c2   docker.elastic.co/elasticsearch/elasticsearch:7.14.0   "/bin/tini -- /usr/l..."   About an hour ago   Up About an hour   0.0.0.0:9200->9200/tcp, :::9200->9200/tcp, 0.0.0.0:9300->9300/tcp, :::9300->9300/tcp   es714

Check the Kibana settings

Run the following command.

docker exec -i -t kb714 cat /usr/share/kibana/config/kibana.yml

The result is as follows.

% docker exec -i -t kb714 cat /usr/share/kibana/config/kibana.yml
#
# ** THIS IS AN AUTO-GENERATED FILE **
#

# Default Kibana configuration for docker target
server.host: "0"
server.shutdownTimeout: "5s"
elasticsearch.hosts: [ "http://elasticsearch:9200" ]
monitoring.ui.container.elasticsearch.enabled: true%

Check Kibana access

Enter the following URL in a browser.

http://localhost:5601/

If the following screen is displayed, the installation completed correctly.

Install Cerebro with Docker

Cerebro is another GUI tool for viewing Elasticsearch.

Pull the Cerebro Docker image

Run the following command to pull the Cerebro Docker image.

docker pull lmenezes/cerebro

The result is as follows.

% docker pull lmenezes/cerebro
Using default tag: latest
latest: Pulling from lmenezes/cerebro
f7ec5a41d630: Pull complete
faf4c47c8c61: Pull complete
810072571faf: Pull complete
395682a2fa05: Pull complete
635a13f1552d: Pull complete
Digest: sha256:95b02e46ec91c00ab33ce21f4d23bf6083d3c900257b9371a56ea218737462e5
Status: Downloaded newer image for lmenezes/cerebro:latest
docker.io/lmenezes/cerebro:latest

Run the Cerebro Docker container

Run the following command.

docker run -d -p 9000:9000 --link es714:localhost --name cerebro -e "CEREBRO_PORT=9000" -e "ELASTICSEARCH_HOST=http://localhost:9200" lmenezes/cerebro

The result is as follows.

% docker run -d -p 9000:9000 --link es714:localhost --name cerebro -e "CEREBRO_PORT=9000" -e "ELASTICSEARCH_HOST=http://localhost:9200" lmenezes/cerebro
4f1c86c5e3156891cb129a841579f1ff6446c271c4fa53141c23c3d0004613f7

Check Cerebro access

Enter the following URL in a browser.

http://localhost:9000

When the following screen appears, enter http://localhost:9200 in the Node address field.

If the following screen is displayed, the installation completed correctly.

Control everything at once

Create a docker-compose file

Create docker-compose-elastic.yml and write the following content.

version: '3.6'
services:
  elasticsearch:
    image: 'docker.elastic.co/elasticsearch/elasticsearch:7.14.0'
    container_name: es714
    ports:
      - "9200:9200"
      - "9300:9300"
    environment:
      - node.name="es01"
      - cluster.name="es-docker-cluster"
      - bootstrap.memory_lock=true
      - discovery.type=single-node
      - "ES_JAVA_OPTS=-Xms4g -Xmx4g"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    networks:
      - elastic

  kibana:
    image: 'docker.elastic.co/kibana/kibana:7.14.0'
    container_name: kb714
    ports:
      - "5601:5601"
    environment:
      - ELASTICSEARCH_HOSTS=["http://es714:9200"]
    depends_on:
      - elasticsearch
    networks:
      - elastic

  cerebro:
    image: 'lmenezes/cerebro'
    container_name: 'cerebro'
    ports:
      - "9000:9000"
    environment:
      - "CEREBRO_PORT=9000"
      - "ELASTICSEARCH_HOST=http://es714:9200"
    networks:
      - elastic

volumes:
  data01:
    driver: local
  data02:
    driver: local
  data03:
    driver: local

networks:
  elastic:
    driver: bridge

Start, stop, and remove all services

Start command:

% docker-compose -f docker-compose-elastic.yml up -d

Stop command:

% docker-compose -f docker-compose-elastic.yml stop

Remove command:

% docker-compose -f docker-compose-elastic.yml rm

Check each service

Check whether Elasticsearch is running.

% chrome 127.0.0.1:9200

Check whether Kibana is running.

% chrome 127.0.0.1:5601

Check whether Cerebro is running.

% chrome 127.0.0.1:9000

Elasticsearch CRUD Operations + Bulk API

kc@example.com (kc kim) — Tue, 14 Dec 2021 09:00:00 +0900

Elasticsearch can be operated by calling the REST API from the following tools.

Kibana Dev Tools
cURL
Others

CRUD Operations + Bulk API

The Elasticsearch REST API supports the following CRUD operations for documents.

REST API	CRUD operation	Description
Index API	Create	Add a document to an index
Get API	Read	Retrieve a document
Update API	Update	Modify a document
Delete API	Delete	Delete a document

These are called Single document APIs, and one document is processed with a single API call.

To process multiple documents with one API call, use the Bulk API.

Index API (Create)

Use the Index API to add a document to an Elasticsearch index.

The Index API can be used as follows.

Using the Index API (curl)

curl "localhost:9200/2021-12-tweet/_doc/1?pretty" -XPUT -H "Content-Type: application/json" -d '
{
  "date":"2021/12/13 09:00 KST",
  "Tweet":"트위터를 시작했습니다.",
  "User ID":"devkuma"
}'

Using the Index API (Kibana)

PUT /2021-12-tweet/_doc/1
{
  "date":"2021/12/13 09:00 KST",
  "Tweet":"트위터를 시작했습니다.",
  "User ID":"devkuma"
}

When using Kibana Dev Tools, paste the command above and click the triangle button in the upper-right corner.

Adding a document to a data stream

A data stream is a set of multiple indexes grouped as one.

For more information about data streams, see the official documentation below.

Data streams | Elasticsearch Guide [7.16] | Elastic

To add a document to a data stream, use the REST API as follows.

Using the Index API (Kibana)

PUT /tweet/_create/1
{
  "date" : "2021/12/13 09:00 KST",
  "Tweet" : "트위터를 시작했습니다.",
  "User ID" : "devkuma"
}

Get API (Read)

Use the Get API to retrieve the document you created.

Using the Get API (curl)

curl "localhost:9200/2021-12-tweet/_doc/1?pretty"

Using the Get API (Kibana)

GET /2021-12-tweet/_doc/1

Response from the command above

{
  "_index" : "2021-12-tweet",
  "_type" : "_doc",
  "_id" : "1",
  "_version" : 1,
  "_seq_no" : 0,
  "_primary_term" : 1,
  "found" : true,
  "_source" : {
    "date" : "2021/12/13 09:00 KST",
    "Tweet" : "트위터를 시작했습니다.",
    "User ID" : "devkuma"
  }
}

Update API (Update)

Use the Update API to modify a document.

Using the Update API (curl)

curl "localhost:9200/2021-12-tweet/_update/1?pretty" -XPOST -H "Content-Type: application/json" -d '
{
  "doc" : {
    "date" : "2021/12/13 09:00 KST",
    "Tweet" : "문서를 업데이트했습니다.",
    "User ID" : "devkuma" 
  }
}'

Using the Update API (Kibana)

POST /2021-12-tweet/_update/1
{
  "doc" : {
    "date" : "2021/12/13 09:00 KST",
    "Tweet" : "문서를 업데이트했습니다.",
    "User ID" : "devkuma" 
  }
}

Check the update result with the Get API.

Checking the update result (curl)

curl "localhost:9200/2021-12-tweet/_doc/1?pretty"

Checking the update result (Kibana)

GET /2021-12-tweet/_doc/1

Response from the command above

{
  "_index" : "2021-12-tweet",
  "_type" : "_doc",
  "_id" : "1",
  "_version" : 2,
  "_seq_no" : 1,
  "_primary_term" : 1,
  "found" : true,
  "_source" : {
    "date" : "2021/12/13 09:00 KST",
    "Tweet" : "문서를 업데이트했습니다.",
    "User ID" : "devkuma"
  }
}

Delete API (Delete)

Use the Delete API to delete a document.

Using the Delete API (curl)

curl "localhost:9200/2021-12-tweet/_doc/1?pretty" -XDELETE

Using the Delete API (Kibana)

DELETE /2021-12-tweet/_doc/1

Confirm that the document was deleted with the Get API.

Checking the delete result (curl)

curl "localhost:9200/2021-12-tweet/_doc/1?pretty"

Checking the delete result (Kibana)

GET /2021-12-tweet/_doc/1

Response from the command above

{
  "_index" : "2021-12-tweet",
  "_type" : "_doc",
  "_id" : "1",
  "found" : false
}

Bulk API

So far, each REST API call has processed only one document.

With the Bulk API, you can process multiple documents in one batch through the REST API.

The Bulk API is used as follows.

Contents of bulk.json

{ "index" : { "_index" : "test", "_id" : "1" } }
{ "field1" : "value1" }
{ "delete" : { "_index" : "test", "_id" : "2" } }
{ "create" : { "_index" : "test", "_id" : "3" } }
{ "field1" : "value3" }
{ "update" : { "_id" : "1", "_index" : "test" } }
{ "doc" : { "field2" : "value2" } }

Using the Bulk API (curl)

curl "localhost:9200/_bulk?pretty" -XPOST --data-binary @bulk.json -H "Content-Type: application/json"

Using the Bulk API (Kibana)

POST /_bulk 
{ "index" : { "_index" : "test", "_id" : "1" } }
{ "field1" : "value1" }
{ "delete" : { "_index" : "test", "_id" : "2" } }
{ "create" : { "_index" : "test", "_id" : "3" } }
{ "field1" : "value3" }
{ "update" : { "_id" : "1", "_index" : "test" } }
{ "doc" : { "field2" : "value2" } }

The differences between Create, Index, and Update are as follows.

Create: Creates the document if it does not exist in the index.
Index: Replaces the document in the index.
Update: Modifies part of the document in the index.

References

4.2 CRUD - Input, Query, Update, Delete

Elasticsearch Mapping Management + Templates

kc@example.com (kc kim) — Tue, 14 Dec 2021 09:09:00 +0900

Mapping Management + Templates

As described earlier, a mapping defines the shape of a field key. Once a mapping has been created, it cannot be modified except by adding fields. To change it, you need to recreate the index.

There are four ways to create or add mappings.

Create mappings automatically
Create mappings manually
Create mappings with templates
Add fields to an existing mapping

Create mappings automatically

Elasticsearch automatically creates mappings when creating documents in an index, as shown below.

Create a document in the mapping_test index

PUT /mapping_test/_doc/1
{
  "date":"2021/12/01 09:00:00+0900",
  "Tweet":"This is a mapping test."
}

Check the mapping

GET /mapping_test

Mapping contents

GET /mapping_test
{
  "mapping_test" : {
    "aliases" : { },
    "mappings" : {
      "properties" : {
        "Tweet" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "date" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        }
      }
    },

// ... omitted ...

From the result above, you can see that the following mappings were defined. The difference between the text and keyword types is as follows.

The date and Tweet keys are text type.
The date.keyword and text.keyword keys are keyword type.

The date key in the mapping above is registered as text, but there are cases where it should be registered as the date type. In that case, you need to create the mapping manually.

Create mappings manually

Manual mapping creation is done as follows.

Create a manual mapping

PUT /mapping_test2
{
  "mappings": {
    "properties": {
      "date":    { 
        "type": "date",
        "format": "yyyy/MM/dd HH:mm:ssZ"
      },  
      "tweet":  {
        "type": "text",
        "fields": {
          "keyword":{
            "type":"keyword"
          }
        }
      }
    }
  }
}

Result of creating the mapping manually

GET /mapping_test2

Manually created mapping contents

GET /mapping_test2
{
  "mapping_test2" : {
    "aliases" : { },
    "mappings" : {
      "properties" : {
        "date" : {
          "type" : "date",
          "format" : "yyyy/MM/dd HH:mm:ssZ"
        },
        "tweet" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword"
            }
          }
        }
      }
    },

... omitted ...

If it was created correctly, you can see that the date key is mapped as the date type.

When the format differs from the one specified in the mapping

If you create a document with a format different from the one specified in the mapping, an error occurs.

When running the following example, the date field format is "yyyy/MM/dd HH:mm:ssZ", but the data is inserted with the different format "yyyy/MM/dd HH:mm:Z" because seconds are not specified.

Create a document with a format different from the mapping

PUT /mapping_test2/_doc/1
{
  "date":"2021/12/01 09:00+0900",
  "tweet":"This is a mapping test."
}

Error message

{
  "error" : {
    "root_cause" : [
      {
        "type" : "mapper_parsing_exception",
        "reason" : "failed to parse field [date] of type [date] in document with id '1'. Preview of field's value: '2021/12/01 09:00+0900'"
      }
    ],

... omitted ...

As expected, the message says that the date field cannot be parsed.

When the format matches the mapping

Now create a document with the correct format.

Create a document according to the mapping format

PUT /mapping_test2/_doc/2
{
  "date":"2020/11/01 09:00:00+0900",
  "tweet":"This is a mapping test."
}

When the document is created with the correct format, you can see that it succeeds.

Create mappings with templates

Creating the same mapping for each index is cumbersome. Templates solve this problem.

When an index with the specified name is created, a template creates the index using the template mapping. Use the template API to create a template.

Use the _template API.

PUT /_template/test_template
{
  "index_patterns": "test*",
  "mappings": {
    "properties": {
      "date":    { 
        "type": "date",
        "format": "yyyy/MM/dd HH:mm:ssZ"
      },  
      "tweet":  {
        "type": "text",
        "fields": {
          "keyword":{
            "type":"keyword"
          }
        }
      }
    }
  }
}

In the template above, because "index_patterns": "test*" is specified, the mapping is created for index names that have "test" as a prefix.

Now check how the template works.

Create the test index

PUT /test

Check the mapping of the test index

GET /test

Check the mapping result of the test index

GET /test
{
  "test" : {
    "aliases" : { },
    "mappings" : {
      "properties" : {
        "date" : {
          "type" : "date",
          "format" : "yyyy/MM/dd HH:mm:ssZ" // date field registered with the format specified in the template
        },
        "tweet" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword"
            }
          }
        }
      }
    },

... omitted ...

You can see that the date field was registered with the format specified in the template.

Add fields to an existing mapping

Use the mapping API to add a new field to an existing mapping.

Here, add the additional_field field to the mapping of the mapping_test2 index.

Contents of mapping_test2

GET /mapping_test2
{
  "mapping_test2" : {
    "aliases" : { },
    "mappings" : {
      "properties" : {
        "date" : {
          "type" : "date",
          "format" : "yyyy/MM/dd HH:mm:ssZ"
        },
        "tweet" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword"
            }
          }
        }
      }
    },
... omitted ...

Add a format to the mapping

PUT /mapping_test2/_mapping
{
  "properties":{
    "additional_field":{
      "type":"text"
    }
  }
}

Check that the additional_field format was added

GET /mapping_test2

Result confirming that the additional_field format was added

{
  "mapping_test2" : {
    "aliases" : { },
    "mappings" : {
      "properties" : {
        "additional_field" : { // additional_field was added
          "type" : "text"
        },
        "date" : {
          "type" : "date",
          "format" : "yyyy/MM/dd HH:mm:ssZ"
        },
        "tweet" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword"
            }
          }
        }
      }
    },
... omitted ...

You can confirm that the additional_field field was added.

Elasticsearch Analyzer (Sentence Analysis)

kc@example.com (kc kim) — Thu, 16 Dec 2021 08:12:00 +0900

Analyzer

An analyzer divides a sentence into word, or term, units.

This chapter explains the following topics.

Components of an analyzer
The Nori Korean morphological analyzer
Custom Analyzer

Why should an analyzer be used to split text into words?

The answer is “because documents cannot be searched accurately,” but let us look at a more concrete example.

Prepare a search sentence

POST demo_standard_analyzer/_doc
{
  "text":"올해 서울시 예산이 결정되었다."
}

Search the document above with the word “서울”.

GET demo_standard_analyzer/_search
{
  "query" : {
     "match" : {
       "text" : "서울"
    }
  }
}

The result is not found, as shown below.

{
  "took" : 3,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 0,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  }
}

The document containing “서울시” is not found by the word “서울” because the “서울” part of “서울시” is not matched.

You can use the analyze API to see how the document string is broken down by the analyzer and stored in the inverted index, so let us investigate the cause.

Break down a sentence with the default Standard Analyzer

POST demo_standard_analyzer/_analyze
{
  "text": "올해 서울시 예산이 결정되었다."
}

Result of sentence decomposition with the default Standard Analyzer

{
  "tokens" : [
    {
      "token" : "올해",
      "start_offset" : 0,
      "end_offset" : 2,
      "type" : "<HANGUL>",
      "position" : 0
    },
    {
      "token" : "서울시",
      "start_offset" : 3,
      "end_offset" : 6,
      "type" : "<HANGUL>",
      "position" : 1
    },
    {
      "token" : "예산이",
      "start_offset" : 7,
      "end_offset" : 10,
      "type" : "<HANGUL>",
      "position" : 2
    },
    {
      "token" : "결정되었다",
      "start_offset" : 11,
      "end_offset" : 16,
      "type" : "<HANGUL>",
      "position" : 3
    }
  ]
}

As shown by the values of the token key, the Standard Analyzer stores the sentence in the inverted index by separating it in this way.

Therefore, to perform accurate searches, Korean text must be broken down into words with an analyzer that supports Korean.

Components of an analyzer

An analyzer consists of the following three elements.

Char Filter: transforms the sentence
Tokenizer: splits the sentence into tokens, or words
Token filter: transforms tokens, or words

An analyzer processes these three elements from top to bottom and decomposes a sentence into tokens to create an inverted index.

Char Filter: example of sentence transformation

HTML Strip Character Filter
- Removes HTML tags.
- <p>foo</p> –(transformed)–> foo

Tokenizer: example of splitting a sentence into tokens

Standard Tokenizer
Nori Tokenizer

Token filter: example of token transformation

Lower case Token Filter
- Converts all token characters to lowercase.
- Used when you want to treat Google and google as the same token.
Stop Token Filter
- Deletes unused tokens.
- Used for tasks such as deleting particles.
Stemmer Token Filter
- Performs stemming.
- Converts forms such as “즐겁게” and “즐겁다” to “즐거움”.
Synonym Token Filter
- Normalizes synonyms.
- Converts words such as “구글” and “검색” to the same term, “검색”.

Nori Korean morphological analyzer

Elasticsearch’s default Standard Analyzer does not support Korean.

To perform Korean search correctly, use the nori Analysis Plugin, a plugin for Korean morphological analysis, to split sentences into words.

Install the nori Analysis Plugin

Command to install the nori Analysis Plugin

bin/elasticsearch-plugin install analysis-nori

The following is an execution example in Docker.

sh-4.4# bin/elasticsearch-plugin install analysis-nori
-> Installing analysis-nori
-> Downloading analysis-nori from elastic
[=================================================] 100%?? 
-> Installed analysis-nori
-> Please restart Elasticsearch to activate any plugins installed
sh-4.4#

For reference, the plugin can be removed as follows. Command to remove the nori Analysis Plugin

bin/elasticsearch-plugin remove analysis-nori

After installation, restart Elasticsearch to use the nori plugin.

The following shows how to use the systemctl command.

systemctl restart elasticsearch

The restart method differs depending on the installation environment, so restart it according to your environment.

Nori Analyzer configuration

Char Filter
- None
Tokenizer
- nori_tokenizer
Token filter
- nori_part_of_speech
- nori_readingform

How to use the Nori Tokenizer

Analyze a string with only the nori tokenizer.

Analyze a sentence with the Nori Tokenizer

GET _analyze
{
  "tokenizer": "nori_tokenizer",
  "text": [
    "올해 서울시 예산이 결정되었다."
  ]
}

Result of analyzing a sentence with the Nori Tokenizer

{
  "tokens" : [
    {
      "token" : "올",
      "start_offset" : 0,
      "end_offset" : 1,
      "type" : "word",
      "position" : 0
    },
    {
      "token" : "해",
      "start_offset" : 1,
      "end_offset" : 2,
      "type" : "word",
      "position" : 1
    },
    {
      "token" : "서울",
      "start_offset" : 3,
      "end_offset" : 5,
      "type" : "word",
      "position" : 2
    },
    {
      "token" : "시",
      "start_offset" : 5,
      "end_offset" : 6,
      "type" : "word",
      "position" : 3
    },
    {
      "token" : "예산",
      "start_offset" : 7,
      "end_offset" : 9,
      "type" : "word",
      "position" : 4
    },
    {
      "token" : "이",
      "start_offset" : 9,
      "end_offset" : 10,
      "type" : "word",
      "position" : 5
    },
    {
      "token" : "결정",
      "start_offset" : 11,
      "end_offset" : 13,
      "type" : "word",
      "position" : 6
    },
    {
      "token" : "되",
      "start_offset" : 13,
      "end_offset" : 14,
      "type" : "word",
      "position" : 7
    },
    {
      "token" : "었",
      "start_offset" : 14,
      "end_offset" : 15,
      "type" : "word",
      "position" : 8
    },
    {
      "token" : "다",
      "start_offset" : 15,
      "end_offset" : 16,
      "type" : "word",
      "position" : 9
    }
  ]
}

How to use the Nori Analyzer

Now configure the Nori Analyzer on the demo_analyzer index.

Specify nori as the analyzer

PUT demo_analyzer
{
  "mappings" : {
     "properties" : {
       "text" :{
         "type" : "text" ,
         "analyzer" : "nori"
      }
    }
  }
}

Next, create a document in the demo_analyzer index and run a Korean search.

Create a document

POST demo_analyzer/_doc
{
  "text": "올해 서울시 예산이 결정되었다."
}

Search for “울시”

GET demo_analyzer/_search
{
  "query" : {
     "match" : {
       "text" : "울시"
    }
  }
}

Search result for “울시”

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 0,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  }
}

The word “울시” does not find “서울시”. It seems to be recognized as a completely different word.

Check what kind of inverted index is created by the nori Analyzer.

Morphological analysis with the nori Analyzer

GET demo_analyzer/_analyze
{
  "analyzer" : "nori", 
  "text" : "올해 서울시 예산이 결정되었다."
}

Morphological analysis result with the nori Analyzer

{
  "tokens" : [
    {
      "token" : "올",
      "start_offset" : 0,
      "end_offset" : 1,
      "type" : "word",
      "position" : 0
    },
    {
      "token" : "해",
      "start_offset" : 1,
      "end_offset" : 2,
      "type" : "word",
      "position" : 1
    },
    {
      "token" : "서울",
      "start_offset" : 3,
      "end_offset" : 5,
      "type" : "word",
      "position" : 2
    },
    {
      "token" : "시",
      "start_offset" : 5,
      "end_offset" : 6,
      "type" : "word",
      "position" : 3
    },
    {
      "token" : "예산",
      "start_offset" : 7,
      "end_offset" : 9,
      "type" : "word",
      "position" : 4
    },
    {
      "token" : "결정",
      "start_offset" : 11,
      "end_offset" : 13,
      "type" : "word",
      "position" : 6
    }
  ]
}

It appears to analyze “서울시” as “서울” and “시”. Therefore, “울시” does not match “서울시”.

Custom Analyzer

You can freely create a custom analyzer by combining the following three elements.

Char Filter
Tokenizer
Token filter

For example, create the following Custom Analyzer.

Create a Custom Analyzer

PUT custome_analyze
{
  "settings" : {
     "analysis" : {
       "analyzer" : {
         "original_analyze" :{
           "char_filter" :[ "html_strip" ],
           "tokenizer" : "nori_tokenizer" ,
           "filter" :[ "my_stop" ]
        }
      },
      "filter" :{
         "my_stop" :{
           "type" : "stop" ,
           "stopwords" :[ "올", "해" , "이" ]  
        }
      }
    }
  }
}

Analyze a sentence with the analyzer you created.

Use the Custom Analyzer

POST custome_analyze/_analyze
{
  "analyzer" : "original_analyze",
  "text" : "<p>올해 서울시 예산이 결정되었다</p>"
}

Result of using the Custom Analyzer

{
  "tokens" : [
    {
      "token" : "서울",
      "start_offset" : 6,
      "end_offset" : 8,
      "type" : "word",
      "position" : 2
    },
    {
      "token" : "시",
      "start_offset" : 8,
      "end_offset" : 9,
      "type" : "word",
      "position" : 3
    },
    {
      "token" : "예산",
      "start_offset" : 10,
      "end_offset" : 12,
      "type" : "word",
      "position" : 4
    },
    {
      "token" : "결정",
      "start_offset" : 14,
      "end_offset" : 16,
      "type" : "word",
      "position" : 6
    },
    {
      "token" : "되",
      "start_offset" : 16,
      "end_offset" : 17,
      "type" : "word",
      "position" : 7
    },
    {
      "token" : "었",
      "start_offset" : 17,
      "end_offset" : 18,
      "type" : "word",
      "position" : 8
    },
    {
      "token" : "다",
      "start_offset" : 18,
      "end_offset" : 19,
      "type" : "word",
      "position" : 9
    }
  ]
}

The output reflects the following processing defined in the custom analyzer original_analyze.

html_strip removes the HTML <p> tag.
nori_tokenizer converts the sentence into tokens.
my_stop removes "올", "해", and "이" as defined in stopwords.

References

6.7.2 Nori Korean morphological analyzer

Elasticsearch Aggregation (Aggregation and Classification)

kc@example.com (kc kim) — Fri, 17 Dec 2021 08:55:00 +0900

Aggregation

Elasticsearch can run the following aggregation queries.

First, create the demo_agg index to try aggregation queries.

Create the demo_agg index

POST /_bulk
{ "index" : { "_index" : "demo_agg" , "_id" : "1" }}
{ "text" : "This is Elasticsearch test." , "type" : 1 }
{ "index" : { "_index" : "demo_agg" , "_id" : "2" }}
{ "text" : "Elasticsearch is God." , "type" : 2 }
{ "index" : { "_index" : "demo_agg" , "_id" : "3" }}
{ "text" : "This is a pen." , "type" : 3 }
{ "index" : { "_index" : "demo_agg" , "_id" : "4" }}
{ "text" : "I have a pen." , "type" : 2 }

Metrics

Run a metric aggregation against the demo_agg index. First, use the avg query to calculate the average value.

avg query

GET demo_agg/_search
{
  "size": 0,
  "aggs": {
    "hoge_name": {
      "avg": {
        "field":"type"
      }
    }
  }
}

avg query result

{
  "took" : 3,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 4,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  },
  "aggregations" : {
    "hoge_name" : {
      "value" : 2.0
    }
  }
}

The value is 2.0. The type values used to calculate the average are [1,2,3,2], so this correctly results in an average of 2.

Also, "size": 0 means that the _search API displays zero search results. In other words, it does not display the matching documents. If you increase this value, you can see the search results used to calculate the average.

The following aggregations are also available.

sum: returns the total value.
max: returns the maximum value.
min: returns the minimum value.
stats: returns all of the above values.
cardinary: returns the number of distinct values. For example, [1,2,3,2] has three distinct values: 1, 2, and 3.

Buckets

You can create buckets for specified conditions and place matching documents into those buckets.

Classifying by range

Create buckets

GET demo_agg/_search
{
  "size": 0,
  "aggs": {
    "bucket_name": {
      "range": {
        "field":"type",
        "ranges": [
          {
            "from": 0, 
            "to":2
          },
          {
            "from": 2
          }
        ]
      }
    }
  }
}

Bucket creation result

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 4,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  },
  "aggregations" : {
    "bucket_name" : {
      "buckets" : [
        {
          "key" : "0.0-2.0",
          "from" : 0.0,
          "to" : 2.0,
          "doc_count" : 1
        },
        {
          "key" : "2.0-*",
          "from" : 2.0,
          "doc_count" : 3
        }
      ]
    }
  }
}

From the result, you can confirm the following.

The bucket where the value of the "type" field is greater than or equal to 0 and less than 2 contains one document. (type=[1])
The bucket where the value of the "type" field is greater than or equal to 2 contains three documents. (type=[2,3,2])

Classifying by equal intervals (histogram)

You can also classify values by equal intervals with the histogram query.

histogram query

GET demo_agg/_search
{
  "size" : 0 ,
   "aggs" : {
     "bucket_name" : {
       "histogram" : {
         "field" : "type" ,
         "interval" : 1
      }
    }
  }
}

histogram query result

{
  "took" : 9,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 4,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  },
  "aggregations" : {
    "bucket_name" : {
      "buckets" : [
        {
          "key" : 1.0,
          "doc_count" : 1
        },
        {
          "key" : 2.0,
          "doc_count" : 2
        },
        {
          "key" : 3.0,
          "doc_count" : 1
        }
      ]
    }
  }
}

As shown above, documents are classified into buckets at the specified interval value of 1.