devkuma – NoSQL

MongoDB Sharding Overview

kc@example.com (kc kim) — Wed, 19 Jan 2022 09:19:00 +0900

Scaling

Before talking about sharding, let’s first look at scaling.

In general, when a system grows larger, scaling becomes necessary. Scaling methods are broadly divided into two types: vertical scaling and horizontal scaling.

Vertical scaling

Vertical scaling is a method of replacing CPU, RAM, or storage with higher-performance components. Scaling is simple, but there are limits to CPU and memory in the first place, so there is a practical performance limit.

Horizontal scaling

Horizontal scaling divides a system by data set and processes data in parallel from multiple servers. Overall performance can be better than vertical scaling, but the more it scales, the more complex it becomes, making management difficult.

What is sharding?

Sharding is a horizontal scaling structure that distributes data across multiple servers.

Sharding has three major benefits: read/write processing becomes faster and performance is easier to improve, storage is easier to expand, and availability is high. Each is described below. MongoDB supports sharding, or horizontal scaling, as a standard feature.

Reads and writes

When data is distributed and stored in a shard cluster, read/write processing, or file I/O, can be distributed. File read/write operations are slow, so distributing them enables faster reads and writes. In addition, when read/write load increases, the system can horizontally scale by adding servers.

Storage

Fragmented data is distributed and stored in the shard cluster. Even when the amount of data increases, the system can horizontally scale by adding servers.

High availability

Shard clusters allow partial reads and writes. If a shard cannot be read from or written to, data can also be retrieved from a shard server that can perform the work.

Shard cluster

MongoDB sharding consists of three components: shards, routers, and config servers. Their relationships and explanations are as follows.

Shard

Shard servers store fragmented data, or chunks, from split collections. A shard server can be configured as a replica set.

Router

mongos is a query router that provides an interface from applications to the shard cluster. When accessing a shard cluster, access always goes through mongos. Even collections that are not sharded must be accessed through mongos.

Config server

Config servers store metadata about shard cluster settings. Information such as which shard server holds the data is also stored on these servers. Since MongoDB 3.4, config servers are configured as replica sets.

Shard key

A shard key is information used as a key when distributing data and storing it on shard servers. MongoDB uses the shard key to distribute documents within a collection.

The following figure is a simple example distributed by an x shard key.

… figure …

The important points about shard keys are the following three.

They consist of immutable fields or groups of fields that exist in every document.
Once a shard key is chosen and distribution is performed, the shard key cannot be changed again.
A sharded collection requires an index that includes the shard key.

Needless to say, shard key selection greatly affects performance, efficiency, and scalability. If the shard key is chosen poorly, data can become biased toward specific shard servers, as described above, causing inefficient behavior.

Chunks

As mentioned above, MongoDB distributes data storage by shard key value, but it groups data into units called chunks and distributes those chunks across shard servers. MongoDB automatically moves chunks so that they are balanced within the shard cluster.

The image of splitting data into chunks and distributing them across shards is as follows.

… figure …

Operation constraints on sharded data

There are several restrictions on operations for sharded data.

The group command cannot be used.
Use MapReduce instead of aggregate.
updateOne or deleteOne must specify _id.
An error occurs if the shard key _id is not included.
A sharded collection requires a unique index.

Sharded and unsharded collections

MongoDB can mix sharded collections and unsharded collections. Sharded collections are distributed and stored in the shard cluster, while unsharded collections are stored on the primary shard.

Data distribution methods

There are two distribution methods for sharding: hashed sharding and ranged sharding.

Hashed sharding

Hashed sharding distributes data using the hash value of the shard key. With hashed sharding, chunks tend to become scattered even when shard key values are close. In other words, it is a distribution method that makes distribution easier when the shard key changes monotonically. On the other hand, because data is easy to distribute, broadcast operations may increase depending on the data structure and retrieval method.

Ranged sharding

Ranged sharding distributes data according to the shard key range. Unlike hashed sharding, when shard key values are close, they are more likely to exist in the same chunk. If the shard key is chosen poorly, load can become concentrated on specific servers even though sharding is being used as a distribution technique.

How to Use the MongoDB mongod Command

kc@example.com (kc kim) — Thu, 13 Jan 2022 07:57:00 +0900

This article summarizes the basic main usage and options of the mongod command.

Only the important options are covered here. For many other options, refer to the official manual.

Basic options

Option	Description
`--config <filename>`, `-f <filename>`	Specifies a configuration file that contains startup options. If the configuration file path contains spaces, enclose it in double quotes (`"`). The configuration file uses ASCII encoding. Non-ASCII encodings such as UTF-8 are not supported. If both a configuration file and startup options are specified, the startup option values take precedence.
`--port <port>`	Default: 27017 Specifies the port number on which MongoDB waits for TCP connections.
`--maxConns <number>`	Specifies the number of concurrent connections MongoDB allows.
`--logpath <path>`	Specifies the path where the log file is output.
`--logappend`	When MongoDB is restarted, logging continues at the end of the existing log file. By default, MongoDB backs up the existing log file and creates a new log file when restarted.
`--slowms <integer>`	Default: 100 Specifies the profiling threshold in milliseconds.
`--profile <level>`	Default: 0 Changes the database profiling level. - 0: Off. Does not perform profiling. - 1: On. Profiles only slow operations. - 2: On. Profiles all operations.
`--auth`	Uses database access control. User configuration changes are performed with the `mongo` command. If no user exists, only access through `localhost` is permitted.
`--bind_ip <address>`	Default: all connections Specifies the IP address where MongoDB accepts connections.
`--dbpath <path>`	Default: `C:\data\db` Specifies the storage path where database files are stored.

Windows service installation options

Option	Description
`--install`	Adds MongoDB as a Windows service. If needed, specify `--serviceName` and `--serviceDisplayName` at the same time.
`--remove`	Removes the MongoDB service from Windows services. If a non-default name was installed with `--serviceName`, specify `--serviceName`.
`--serviceName <name>`	Default: MongoDB Specifies the service name. The name specified here is used as the argument to `net start <name>` and `net stop <name>`.
`--serviceDisplayName <name>`	Default: MongoDB Specifies the name displayed in the service management tool.
`--serviceDescription <description>`	Default: MongoDB Server Specifies the string displayed in the description of the service management tool. If it contains spaces, enclose it in double quotes (`"`).
`--serviceUser <user>`	Specifies the user that runs MongoDB.
`--servicePassword <password>`	Specifies the password for the user that runs MongoDB.

References

MongoDB Features

kc@example.com (kc kim) — Sun, 09 Jan 2022 18:38:00 +0900

This article looks at the features of MongoDB.

The main features, also described on the official site, can be summarized as follows.

Document database (NoSQL)
Special queries
High performance
High availability
Horizontal scalability

Document database (NoSQL)

MongoDB is a document-format database that can store key-value combinations as they are, like JSON. Values can take many forms. For example, it can store numbers and strings, as well as arrays and objects. These value types can also be changed dynamically, and multiple types can be placed under the same field name. In other words, it is a schemaless document database.

Performance

The following features are implemented to provide performance.

Reduced I/O processing through embedded data models
Index support

One of MongoDB’s characteristics is that it supports indexes even though it is NoSQL. You can also specify a unique key in relation to indexes.

Special queries

MongoDB searches the database using special queries specified in JSON format. Queries can perform basic CRUD operations as well as the following distinctive operations.

Aggregation and Map-Reduce
String search using text indexes
Geospatial queries

Aggregation and Map-Reduce can perform various statistics on stored data. Query assembly is implemented in a distinctive way, so it takes some getting used to, but the processing order is pipelined, which makes the code easier to follow.

Geospatial queries can use the built-in GeoJSON format to search for points within a range.

Availability

MongoDB provides replication through a feature called replica sets. Replica sets provide automatic failover and data redundancy.

Horizontal scaling

Horizontal scaling means that performance can be improved by adding machines. MongoDB provides horizontal scalability through a core feature called sharding.

Support for multiple storage engines

MongoDB supports multiple storage engines. In addition to commonly used file systems, you can choose an in-memory engine. It also exposes a storage engine API, so third-party storage engines can be created.

References

What is NoSQL?

kc@example.com (kc kim) — Fri, 04 Jun 2021 08:26:25 +0900

What is NoSQL?

NoSQL, interpreted as “Not only SQL,” refers broadly to non-relational databases. In general, non-relational databases refer to all databases except relational databases.

Non-SQL and non-relational databases provide mechanisms for storing and retrieving data using consistency models that are less restrictive than traditional relational databases.

Classification by storage method

Examples of non-relational database data models are classified by storage method into documents, key-value, graphs, and others.

Document DB
- Adopts a collection data model structure such as JSON or XML.
- Originated from Lotus Notes.
- MongoDB, BaseX, and CouchDB are examples.
Key-Value DB
- The simplest type of solution where data is stored as key-value pairs.
- Originated from the Amazon Dynamo Paper.
- Redis and Memcached are examples.
Graph DB
- Originated from Euler and graph theory.
- Adopts Nodes, Relationship, and Key-Value data models.
- Neo4J and OrientDB are examples.
Wide Columnar Store
- Also called Big Table DB.
- Originated from Google’s BigTable Paper.
- Uses a Column Family data model developed from Key-Value.
- HBase, Cassandra, and ScyllaDB are examples.

Difference between RDB and NoSQL

The characteristics of RDBs, or relational databases, and NoSQL, or non-relational databases, can be summarized as follows.

RDB and NoSQL differ not only in data structure, but also in the meaning of data integrity and scaling.

Category	RDB	NoSQL
Data structure	Represents data structure in table form. It uses “tables” and “connections between tables” to represent data structure. RDB	Represents data structure in methods other than table form. Representation methods include documents (XML, JSON), key-value, and graphs. Document Key-Value Graph
ACID characteristics	Strict management is performed to prevent data inconsistency.	Strict data integrity management, called eventual integrity, is not performed. Speed is prioritized, and data is guaranteed to be consistent “eventually.”
Scaling	Vertical scaling, meaning increasing server specifications, is easy to perform.	Horizontal scaling, meaning increasing the number of servers, is a characteristic strength.

CAP theorem

Consistency
- Also called concurrency or sameness.
- Means guaranteeing that data queried by multiple clients at the same time is the same data.
Availability
- Guarantees that all clients’ read and write requests can always receive a response.
- Also called fault tolerance.
- Fault-tolerant NoSQL can provide normal service even when some nodes in the cluster fail.
Partition tolerance
- Means that in systems operating in regionally partitioned network environments, the systems in each region must continue operating normally even if the network between two regions is disconnected or network data is lost.

References

Redis Overview

kc@example.com (kc kim) — Mon, 24 May 2021 10:12:26 +0900

Overview

Redis stands for Remote Dictionary Server. It is an open source, non-relational database management system for storing and managing unstructured data in a key-value structure. It was first developed by Salvatore Sanfilippo in 2009. Since 2015, it has been supported by Redis Labs. Redis is used freely in many services.

Pareto principle

The Pareto principle says that 80% of the phenomena that occur in society are caused by 20% of the causes. This also fits website access patterns.

It is estimated that 80% of Internet traffic is access to only 20% of sites, and caching the data from these 20% of websites can dramatically improve efficiency.

Textbook for Infrastructure Engineers - Network Edition, 2017, Gilbut

Therefore, storing commonly used data in Redis as a cache can be an efficient way to use resources.

Redis features

Redis is a key-value store NoSQL DB.
- Supports a key-value structure for simple strings
- Supports writing binary data
It operates as a single thread.
- It guarantees operation atomicity by operating as a single thread.
It supports various data structures.
- Collection support: List, Set, Sorted Set, Hash, and other data structures
High speed
- It provides performance around 100,000 QPS.
Disk storage
- Data can be stored persistently on disk.
Replication
- Supports a master/slave structure that can replicate data to other nodes.
Sharding support
- Supports data sharding through Redis Cluster.
Publish/Subscribe support
- Can be used as a message broker
Expire time
- You can set an expiration time on a key so old data is automatically deleted.
Recovery is possible through data snapshots or AOF logs, so some persistence is guaranteed.
In Java Spring, it is often used for session management and caching.

Why are collections important?

Redis is an in-memory database. That means all data is stored and queried in memory. It is much faster than traditional relational databases such as Oracle and MySQL because memory access is faster than disk access. However, speed is only one of Redis’s characteristics. The biggest difference from other in-memory databases such as Memcached is that it supports various data structures. Redis stores various data structures in key-value form as shown below.

Redis basically provides String, Bitmap, Hash, List, Set, and Sorted Set, and as versions have advanced, it now also supports types such as Geospatial Index, HyperLogLog, and Stream.

Why is providing many data structures important? The reason is development convenience and difficulty.

For example, if you implement a real-time ranking server with an RDBMS, you need to store data in the DB, sort it by the stored SCORE value, and read it again. As the number of records increases, this process becomes slower because it uses disk. You could write code directly to process data on the server with an in-memory approach, but using Redis Sorted Set is faster and simpler.

Redis persistence

Redis uses RDB (Snapshot) and AOF (Append Only File) methods to store data on disk.
Both methods internally use the fork() function, so sufficient memory is required.

RDB (Snapshot)
- A method of moving data in memory to disk.
- Because snapshots capture the memory state as it is, they are useful for backup and recovery at a specific point in time, and they can load data into memory faster than AOF.
- If the server goes down, data changed between backed-up snapshots is lost.
- There are two methods.
  - SAVE: a blocking method that stops all Redis operations sequentially and stores the snapshot at that time on disk.
  - BGSAVE: a non-blocking method that stores the memory snapshot at the time of execution on disk through a separate process.
AOF (Append Only File)
- A method of recording all Redis write/update operations as a log file.
- When the server starts, operations are replayed sequentially to recover data.
- Because it records every operation as it runs, logs can be kept up to the current point.
- Because it only appends to the log file, write speed is fast and data loss does not occur even if the server goes down.
- Because logs are recorded for all operations, the amount of data is very large.
- When the server restarts, all operations must be executed again, so restart speed is slow.

Using Redis in the cloud

If you use Amazon Web Services or Microsoft Azure for application development, each cloud service provides a Redis service, so you can use it.

Cloud service	Service name
Amazon Web Services	Amazon ElastiCache for Redis
Microsoft Azure	Azure Redis Cache

Of course, you can also install Redis directly on each cloud service’s compute instances, but considering management cost, choosing these services is recommended.

References

Redis | Wikipedia
[redisgate][http://redisgate.kr/]

Designing Data-Intensive Applications

kc@example.com (kc kim) — Wed, 03 Aug 2022 15:06:00 +0900

Core ideas behind reliable, scalable, and maintainable systems

https://wikibook.co.kr/data-intensive-applications/

Group study members: Kim Kyung-cheol, Kim Min-gyu, Kim Jung-su, Kim Eun-taek, Park Su-min, Park Hyun-do, Lee Seung-ik, Lee Ho-jun, Jo Sung-jik, Hwang Yoon-ho

These materials were presented during a group study.

Installing Redis with Docker

kc@example.com (kc kim) — Fri, 31 Mar 2023 17:46:26 +0900

Installing Redis with Docker

If you want a simple setup for learning or experimentation, using Docker to install Redis is recommended.

Downloading the Redis Docker image

Run the following command to download Redis Docker. This example uses the alpine version.

If you want another version tag, refer to https://hub.docker.com/_/redis/tags.

% docker pull redis:alpine

After downloading, check the image and you can see the list as follows.

% docker images
REPOSITORY         TAG       IMAGE ID       CREATED        SIZE
redis              alpine    a4cf5af74f5e   2 days ago     30.4MB

Creating and running a Redis Docker container

Run the following command.

% docker run -d -p 6379:6379 --name=redis redis:alpine

-p: specify the port to expose on the host
--name: specify the container name

After running it, check the running container.

% docker container ls
CONTAINER ID   IMAGE          COMMAND                  CREATED          STATUS          PORTS                    NAMES
1df277cb625c   redis:alpine   "docker-entrypoint.s…"   30 seconds ago   Up 29 seconds   0.0.0.0:6379->6379/tcp   redis

Connecting to Redis Docker

Connecting inside the Redis Docker container

Before connecting to Redis, connect inside the Docker container where Redis is installed.

% docker exec -it redis /bin/bash

If you see an error message like the following, try connecting with /bin/sh.

OCI runtime exec failed: exec failed: unable to start container process: exec: "/bin/bash": stat /bin/bash: no such file or directory: unknown

% docker exec -it redis /bin/sh

Connecting to Redis

Once connected inside the container, connect to Redis. Use the redis-cli command to connect to Redis.

/data # redis-cli
127.0.0.1:6379>

When connected, the prompt changes to IP:port> as shown above.

To check Redis information, enter info as follows.

127.0.0.1:6379> info
# Server
redis_version:7.0.10

... omitted ...

Press Ctrl+C to exit.

Installing MongoDB with Docker

kc@example.com (kc kim) — Mon, 10 Jan 2022 08:53:00 +0900

Installing MongoDB with Docker

Downloading the MongoDB Docker image

Run the following command to download the MongoDB Docker image. If you do not specify a version in the tag, the latest version is downloaded.

docker pull mongo

The result is as follows.

% docker pull mongo
Using default tag: latest
latest: Pulling from library/mongo
ea362f368469: Pull complete
ecab26900ceb: Pull complete
1847fcb70562: Pull complete
a7de23811c0d: Pull complete
29dd51833fb9: Pull complete
5eccd2be8afb: Pull complete
cd8a8cd6879f: Pull complete
e6ca3abc397d: Pull complete
7481c3682d3c: Pull complete
af377cb9eb7d: Pull complete
Digest: sha256:6743836d42756b2ae50549b2eb4585c688fce81a243cedd152b56266c2fb3d17
Status: Downloaded newer image for mongo:latest
docker.io/library/mongo:latest

To specify the MongoDB version, specify the version in the tag. You can check available MongoDB versions on Docker Hub.

Check the downloaded Docker image with the following command.

docker images

The result is as follows.

% docker images
REPOSITORY                                      TAG            IMAGE ID       CREATED         SIZE
mongo                                           latest         ee13a1eacac9   2 days ago      696MB

Creating and running a MongoDB Docker container

docker run --name my-mongodb -v ~/mongodb/db:/data/db -d -p 27017:27017 mongo

-v ~/mongodb/db:/data/db mounts the ~/mongodb/db directory on the host, meaning the local computer running the container, to the /data/db directory in the container. If you do not set up a volume this way, the stored data is deleted when the container is deleted. Once deleted, container data cannot be recovered.

% docker run --name my-mongodb -v ~/mongodb/db:/data/db -d -p 27017:27017 mongo
ade97bdb08fde4eaa72ceb0135b0a570d97b21616affb7949a0fc479fec25e0e

Starting, stopping, and restarting a MongoDB Docker container

Stop the MongoDB Docker container.

docker stop my-mongodb

Start the MongoDB Docker container.

docker start my-mongodb

Restart the MongoDB Docker container.

docker restart my-mongodb

Checking whether the MongoDB Docker container is running

Run the following command.

% docker ps -a
CONTAINER ID   IMAGE                                                  COMMAND                  CREATED          STATUS                      PORTS                                       NAMES
ade97bdb08fd   mongo                                                  "docker-entrypoint.s..."   37 seconds ago   Up 36 seconds               0.0.0.0:27017->27017/tcp                    my-mongodb

Connecting to the MongoDB Docker container

Run the following command.

docker exec -it my-mongodb bash

The result is as follows.

% docker exec -it my-mongodb bash
root@ade97bdb08fd:/#

Redis redis-cli

kc@example.com (kc kim) — Fri, 10 Mar 2023 14:39:26 +0900

Installing redis-cli

The following shows how to install it with Homebrew in a macOS environment.

brew tap aoki/redis-cli
brew update && brew doctor
brew install redis-cli

github.com/aoki/homebrew-redis-cli

Server control

Connecting with redis-cli

If the host name and port number are omitted, it connects to localhost on port 6379.

$ redis-cli

Remote access can be done as follows.

$ redis-cli -h #{host name} -p #{port number}

Other main options are as follows.

-n: database number
-a: password
-s: socket
-u: server URL and other connection options can be set

Querying server information

redis-cli info

> info
# Server
redis_version:6.0.9
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:ffd199d8341c2d8f
redis_mode:standalone
... omitted ....

Querying connected clients

> client list

Disconnecting a connected client

> client kill 10.0.0.8:33333

Dumping all requests received after running the command

redis-cli monitor

> monitor
OK

Key operations

Setting a key and value

set {key name} {value}

> set key01 value01
OK

Querying a value by key

get {key name}

> get key01
value01

Querying key list

> keys *
key01

Deleting a key

> del key01
1

Checking whether a key exists, where 1 means true

exists {key name} {value}

> exists key01
0

Setting data only if the key does not already exist

setnx {key name} {value}

> setnx key01 value02
1
> setnx key01 value02
0

If the result is 0, it means the key exists and the data was not updated.

Renaming an existing key

rename {old key name} {new key name}

> rename key01 key02 
OK
> rename key03 key04
ERR no such key

Returning the number of keys in the current database

> dbsize
2

Moving a specific key to another database

> move key01 1
1

Selecting a DB number

> select 1
OK

Deleting all keys in the current database

> flushdb
OK

Deleting all keys in all databases

> flushall
OK

Regular expressions

The regular expressions available in Redis are only the following.

[]: one of the characters inside the brackets
*: any string
?: a single character

Examples

# Query all Key lists.
$ keys *

# Query keys that start with a number.
$ keys [0-9]*

Multiple Key

Setting multiple key values together

> mset key01 value01 key02 value02 key03 value03 
OK

Querying multiple key values at once

> mget key01 key02 key03
value01
value02
value03

expire

Setting a key with an expiration period

The following example sets it to 60 seconds.

> setex key01 60 value01 
OK

Setting expiration on an existing key

> expire key02 30

Math

Adding 1 to a key value

The following example sets key02 to 1 and increments it by 1.

> set key02 1 
OK
> incr key02 
2

Adding a specified number to a key value

> incrby key02 100 
102

Subtracting 1 from a key value

> decr key02 
101

Subtracting a specified number from a key value

> decrby key02 51 
50

List

Adding values to the head of a List

Multiple values can be specified separated by spaces.

> lpush list01 value01
1

Adding values to the end of a List

Multiple values can be specified separated by spaces.

> rpush list01 value02
2

Returning the number of List elements

> llen list01
2

Returning the element at a specified index in a List

> lindex list01 0
value01

Returning elements from a specified start index to end index in a List

> lrange list01 0 1
value01
value02

Changing the element at a specified index in a List to a specified value

> lset list01 1 value03
OK
> lindex list01 1
value03

Returning and deleting the first element of a List

> lpop list01
value01

Returning and deleting the last element of a List

> rpop list01
value03

Trimming a List to elements from a specified start index to end index

> ltrim list01 1 3
OK

Hash

This is a data format managed by a parent Key, child Key (field), and value. For example, if you use the parent Key as a product ID and child keys to store product attribute information, you can use it with a way of thinking close to RDB data management.

Adding a value to a specified Hash field

> hset hash01 field01 value01
(integer) 1

Returning the value of a specified Hash field

> hget hash01 field01
"value01"

Adding multiple fields to a Hash

> hmset hash01 field02 value02 field03 value03
OK

Returning the values of multiple Hash fields

> hmget hash01 field01 field02 field03
value01
value02
value03

Returning all Hash fields

> hkeys hash01
field01
field02
field03

Returning all values of Hash fields

> hvals hash01
value01
value02
value03

Adding a specified number to the value of a specified Hash field

> hincrby hash01 field04 100
101

Checking whether a specified Hash field exists

> hexists hash01 field01
1

Returning the number of Hash fields

> hlen hash01
4

Removing a specified Hash field

> hdel hash01 field04
1

Other operations

How to search for keys when keys cannot be used

scan 0 match devkuma-* count 1000

Key names can be searched with wildcards, and devkuma-* searches for keys that start with devkuma-.

Querying and Storing Redis Data Structures

kc@example.com (kc kim) — Fri, 31 Mar 2023 17:46:26 +0900

Data structures (Collection)

Redis provides various data structures.

In Redis, the basic type for data representation consists of one Key and one or more Field/Element values.

Key
- Stores an ASCII value.
Value (Field/Element)
- Basically stores String data.
- Stores Container-type data.
Container type
- Includes Hash, List, Set, Sorted Set, and others.

Strings

String is the most basic data type for Redis values. It stores characters, numbers, and other values. When storing, there is no separate distinction between numbers and characters.
Redis Strings are binary safe. This means a Redis string can contain any kind of data, such as a JPEG image or a serialized Ruby object.

A String value can have a maximum length of 512 MB.

Redis commands for Strings

Let’s store and query data in a key-value structure.

Command	Syntax	Description
`set`	`set key`	Stores a key-value pair.
`get`	`get key`	Queries the value for the key.
`del`	`del key`	Deletes the key.

127.0.0.1:6379> set devkuma 1000
OK
127.0.0.1:6379> get devkuma
"1000"
127.0.0.1:6379> del devkuma
(integer) 1
127.0.0.1:6379> get devkuma
(nil)
127.0.0.1:6379>

Lists

A List is simply a String list ordered by insertion order. You can think of the String type as becoming an array structure.
New elements can be added to the front or back of a list. It is said to support up to about 4 billion elements.

Command	Syntax	Description
`lpush`	`LPUSH key element [element ...]`	Left push. Stores data from the left, with the index starting at 0.
`lpop`	`LPOP key`	Left pop. Extracts data from index 0 of the list.
`rpush`	`RPUSH key element [element ...]`	Right push. Stores data from the right, meaning from the last index.
`rpop`	`RPOP key`	Right pop. Extracts data from the last index of the list.
`lrange`	`LRANGE key start stop`	Extracts list data from start to end. If end is declared as -1 in lrange, all data in the list is extracted.

The LPUSH command adds a new element before the head, and RPUSH adds it at the end. If executed on an empty key, a new list is created and added.

127.0.0.1:6379> lpush fruit apple
(integer) 1
127.0.0.1:6379> lpush fruit banana
(integer) 2
127.0.0.1:6379> lpush fruit orange
(integer) 3
127.0.0.1:6379> lrange fruit 0 2
1) "orange"
2) "banana"
3) "apple"
127.0.0.1:6379>

Sets

A Set is an unordered String collection. It is like a List type without indexes. Values are not duplicated. It can be processed faster than Lists.

Because data can be extracted randomly, it may be usable for areas such as machine learning.

Command	Syntax	Description
`sadd`	`SADD key member [member...]`	Adds members to the set key.
`srem`	`SREM key member [member...]`	Deletes members from the set key.
`smembers`	`SMEMBERS key`	Queries all members of the set key.

Add elements to key animal with the sadd command.

127.0.0.1:6379> sadd animal dog
(integer) 1
127.0.0.1:6379> smembers animal
1) "dog"
127.0.0.1:6379> sadd animal cat cow
(integer) 2
127.0.0.1:6379> smembers animal
1) "dog"
2) "cat"
3) "cow"

Delete an element from key animal with the srem command.

127.0.0.1:6379> srem animal cat
(integer) 1
127.0.0.1:6379> smembers animal
1) "dog"
2) "cow"
127.0.0.1:6379>

Use the del command to delete everything.

127.0.0.1:6379> del animal
(integer) 1
127.0.0.1:6379> smembers animal
(empty array)
127.0.0.1:6379>

Sorted Sets

A Sorted Set is a set of non-repeating Strings similar to a Redis Set. It is a Set type with the concept of score added. Values are not duplicated here either. It can be sorted by score or processed with threshold values on scores. It seems usable for real-time ranking displays in games.

Hashes

A Hash consists of multiple field-value pairs under one key. It is a Set with the concept of fields that can be specified as strings.

Every hash can store up to 2^32 - 1 field-value pairs, or more than about 4 billion.

Command	Syntax	Description
`hset`	`HSET key field [field value...]`	Sets one or more values.
`hget`	`HGET key field`	Gets one or more values.

Bitmaps and HyperLogLogs

Redis also supports Bitmaps and HyperLogLogs, data types that are actually based on the String base type but have their own meanings.

References

Redis data types | Redis

How to Use the MongoDB mongo Command

kc@example.com (kc kim) — Tue, 11 Jan 2022 18:21:00 +0900

This article explains how to connect using mongo, the client module for connecting to MongoDB.

Basic options

Option	Description
`--host <hostname>`	Specifies the host name of the MongoDB server to connect to. If no host name is specified, it connects to localhost.
`--port <port>`	Specifies the port number used to connect to the MongoDB server. If no port number is specified, it connects to `27017`.
`--username <username>, -u <username>`	Specifies the user name when accessing MongoDB with authentication enabled.
`--password <password>, -p <password>`	Specifies the password when connecting to MongoDB with authentication enabled.
`--authenticationDatabase <dbname>`	Specifies the database name used for authentication.

Files

.dbshell

This stores the history of commands executed in the mongo shell. The file is automatically created directly under the home directory (%UserProfile%). On Windows, for example, it is located in the following place.

%UserProfile%

C:\Users\<user name>\.dbshell

.mongorc.js

If you place the .mongorc.js file directly under a defined directory (%UserProfile% or %ProgramData%), .mongorc.js is automatically loaded and executed when mongo starts. This lets you predefine variables or functions that you want to use commonly while the shell is running. .mongorc.js reads global definitions (%ProgramData%) before user definitions (%UserProfile%).

If you do not want to load .mongorc.js, start mongo with the --norc option.

On Windows, for example, the directories are as follows. %UserProfile%

C:\Users\<user name>\.mongorc.js

%ProgramData%

C:\ProgramData\MongoDB\.mongorc.js

References

Redis Command SETEX key Seconds value

kc@example.com (kc kim) — Fri, 31 Mar 2023 17:46:26 +0900

SETEX

The SETEX command sets a string value with a key and seconds. The data is deleted after the specified number of seconds.
It is useful for storing data with an expiration time.

Available version: version 2.0.0 and later
Logical processing time complexity: O(1)

It behaves the same as the following two commands.

SET mykey value
EXPIRE mykey seconds

SETEX usage

The usage is as follows.

SETEX [key] [seconds] [value]

When creating a key to store a string value, you can also set a timeout. The key is deleted after the timeout period has passed. It has the same effect as the following two commands.

Data is deleted after the specified time: specify in seconds. Data is deleted after the specified number of seconds. It is useful for storing data with an expiration time. The usage is SETEX key seconds value.

Example Command> setex key 5 value Result> OK Command> ttl key Result> 2 shows the remaining time in seconds Command> get key Result> (nil) data was deleted after 5 seconds View animation

The TTL command shows the remaining time in seconds.

How to Apply Indexes in MongoDB

kc@example.com (kc kim) — Wed, 12 Jan 2022 08:39:00 +0900

Because MongoDB is a document database, it has indexes that work a little differently. This article explains index concepts that do not exist in relational databases.

MongoDB indexes

In MongoDB, the _id property is basically created automatically as a unique index whenever a document is created. If this is not enough, you can create your own indexes with db.collection.createIndex(). Indexes created with db.collection.createIndex() are B-tree data structure indexes.

MongoDB supports several index types in addition to simple key specifications. The supported index types are as follows.

Single key
Compound key
Multikey
Geospatial
Text
Hashed

“Compound key” and “multikey” have similar names, but they are different. A compound key is a key made from a combination of multiple fields, while a multikey is a key that targets all values included in an array for a field that handles arrays.

Below, we will look at concrete examples of the basic single-field index, compound index, and multikey index.

Basic index creation

You can create an index on a specified collection by running the following command in the mongo shell.

The function is the same in JavaScript code, but JavaScript can specify a callback as the third argument.

Syntax

db.collection.createIndex(keys, options)

Arguments

Argument	Type	Description
`keys`	`string`/`object`	Specifies the field name that has a value. Specify `1` for an ascending key and `-1` for a descending key.
`options`	`object`	Specifies options for index creation. The available options include `unique` and `collation`. There are more options, but remember these for now.

Examples

Below, we will look at examples of single-field indexes, compound indexes, and multikey indexes.

Creating a single-field index

This section covers three sample codes for creating indexes on a sample document like the following. Because MongoDB is a document database, it can create indexes on embedded fields and embedded documents that do not exist in relational databases.

{
  "score": 1034,
  "location": { city: "Seoul", county: "Korea Republic of" }
}

Single field

This is the simplest sample code for creating an index on a single field.

> db.collection.createIndex({ score: 1 })

> db.collection.find({ score: 1034 })
> db.collection.find({ score: { $gt: 1000 } })

Embedded field

MongoDB can also specify a document as an attribute value. In this case, fields inside the document, or nested elements, can also be indexed.

> db.collection.createIndex({ "location.city": 1 })

> db.collection.find({ "location.city": "Kyunggi" })

Embedded document

This is similar to an embedded field, but you can also specify the original object. However, note that in this case an exact match is required, including the number and order of attributes.

> db.collection.createIndex({ location: 1 })

> db.collection.find({ city: "Seoul", county: "Korea Republic of" })

Creating a compound-key index

This is a compound-key index, which also exists in relational databases. In MongoDB, it can be created by specifying multiple fields as keys. In a MongoDB compound-key index, the index is effective not only for patterns that match all specified compound keys, but also for patterns that match part of them.

Let’s look at an example of creating a compound-key index for the following sample data.

{
  "item": "Apple",
  "location": "xxx store",
  "stock": 4,
  "type": "cases"
}

> db.collection.createIndex({ item: 1, location: 1, stock: 1 })

> db.collection.find({ item: "Apple" })
> db.collection.find({ item: "Apple", location: "xxx store" })
> db.collection.find({ item: "Apple", location: "xxx store", stock: { $gt: 0 } })

In the example above, the index is created as item -> location -> stock, so the leading parts such as item and item -> location can also use the index.

Creating a multikey index

MongoDB also has fields that use arrays as values. Whether an array contains primitive values or objects, a multikey index can be created.

Below, we will look at examples of creating an index on a collection that includes an array field named ratings and searching it.

{ "item": "Apple",  ratings: [ 2, 9 ] }
{ "item": "Banana", ratings: [ 4, 3 ] }

An index can be created simply by calling createIndex() for ratings.

> db.collection.createIndex({ ratings: 1 })

Searches on array fields usually use $elemMatch. The following example extracts documents where at least one ratings record is greater than or equal to 3 and less than or equal to 6.

> db.collection.find({ ratings: { $elemMatch: { $gte: 3, $lte: 6 } } })
{ "_id" : ObjectId("5a2d24459c684f917e3ec0c2"), "item" : "XYZ", "ratings" : [ 4, 3 ] }

If you enter the following, it extracts documents where at least one record satisfies greater than or equal to 3 or less than or equal to 6, so all documents are extracted as a result.

> db.collection.find({ ratings: { $gte: 3, $lte: 6 } })
{ "_id" : ObjectId("5a2d24399c684f917e3ec0c1"), "item" : "ABC", "ratings" : [ 2, 9 ] }
{ "_id" : ObjectId("5a2d24459c684f917e3ec0c2"), "item" : "XYZ", "ratings" : [ 4, 3 ] }

References

How to Apply Unique Constraints in MongoDB

kc@example.com (kc kim) — Wed, 12 Jan 2022 08:59:00 +0900

This article explains how to apply unique constraints in MongoDB.

Applying unique constraints

To apply a unique constraint in MongoDB, specify true for the unique option of createIndex() when creating an index.

The examples below show the cases of a single-key index and a compound-key index.

Single-key index

For a single-key index, you can create it by simply running the following code in the mongo shell.

> db.members.createIndex({ userId: 1 }, { unique: true })

Compound-key index

For a compound-key index, the code is the same as for a single-key index.

> db.members.createIndex({ firstname: 1, lastname: 1 }, { unique: true })

With a compound-key index, the combination of compound keys only needs to be unique, so the following can be set without issue.

> db.members.insert({ firstname: "kc", lastname: "Kim" })
> db.members.insert({ firstname: "et", lastname: "Kim" })
> db.members.insert({ firstname: "kc" })

Limits on setting unique constraints

As with relational databases, you cannot apply a unique constraint to a collection that already contains duplicate data. This may be obvious once duplicate data already exists.

Also, setting a unique constraint on a hashed index is not recommended. See Considerations for Hashed Indexes.

Behavior when no value is set for the unique constraint key

As in the example above where a unique constraint was set with a compound-key index, if there is a field with no value, MongoDB internally treats it as null.

For example, suppose there is a collection with a unique constraint on field x as follows.

> db.collection.createIndex({ x: 1 }, { unique: true })

Assume that data is inserted without specifying field x.

> db.collection.insert({ y: 1 })
WriteResult({ "nInserted" : 1 })

The first insert succeeds because x: null is assigned. What happens if another document without field x is inserted?

> db.collection.insert({ z: 1 })
WriteResult({
    "nInserted" : 0,
    "writeError" : {
        "code" : 11000,
        "errmsg" : "E11000 duplicate key error collection: test.collection index: x_1 dup key: { : null }"
    }
})

As shown above, an error occurs and the insert cannot be performed. The error message also shows that null is assigned when no value is specified.

References

MongoDB - Unique Indexes

Apache HBase Overview

kc@example.com (kc kim) — Fri, 08 Jul 2022 09:37:00 +0900

What is HBase?

Apache HBase is an open source distributed database management system developed and published by the Apache Software Foundation. It is one of the NoSQL systems, with a structure different from mainstream relational databases.

It models Google’s BigTable, which was developed and used internally at Google, and reimplements a similar database management system as open source software.

HBase builds a database on HDFS (Hadoop Distributed File System), a distributed file system also developed and published by the ASF. Like a relational database, it structures data in tables, but unlike a typical RDBMS, it stores the values of each column together in storage. This approach is called column-oriented storage.

Each row in a table stores data as pairs of qualifiers, which correspond to column names, and cells, which correspond to field values. Multiple qualifiers are grouped into a column family. Cell data is automatically versioned, so data from an arbitrary point in the past can be retrieved.

HBase is designed for a distributed environment where multiple server computers manage one database. It automatically shards data across multiple nodes, so applications do not need to worry about the target location. It also provides strong data consistency, preventing stale data from being externally referenced during processing.

Database operations are performed not with SQL, as in RDBs, but by calling APIs such as the Java API or RESTful API from an application. HBase is well suited to large-scale distributed big data processing such as Apache Hadoop MapReduce.

HBase is not a replacement for SQL databases, but it can add an SQL layer by using software such as Apache Phoenix. With an SQL layer, JDBC drivers and similar tools can be used to handle various analytical workloads more easily and in a familiar way.

Apache HBase|GitHub

HBase History

Year	Event
November 2006	Google published the BigTable paper.
February 2007	The initial HBase prototype was created as a Hadoop contribution.
October 2007	The first usable HBase was released with Hadoop 0.15.0.
January 2008	HBase became a subproject of Hadoop.
October 2008	HBase 0.18.1 was released.
January 2009	HBase 0.19.0 was released.
September 2009	HBase 0.20.0 was released.
May 2010	HBase became an Apache top-level project.
June 2010	The first developer version, HBase 0.89.20100621, was released.
January 2011	HBase 0.90.0 was released.
Mid-2011	HBase 0.92.0 was released.

HBase Features

The following are some important characteristics of Apache HBase.

Distributed

Apache HBase can run in two modes: pseudo-distributed mode and fully distributed mode. Pseudo-distributed mode is used for testing and runs on a single node, while fully distributed mode is used in production and runs on a cluster of nodes.

Big Data Store

Apache HBase is designed to store very large data in tables with billions of rows and millions or billions of columns. Because it runs on Hadoop HDFS, it provides low-latency real-time reads and writes for data. It also provides better performance for read operations on massive data stored in tables.

Non-Relational

As already noted, Apache HBase is a non-relational database, so it does not follow the relational database model. In relational database management, data is stored in tables as rows and columns. SQL can be used to access that data, but Apache HBase uses storage and query mechanisms where data storage is not fixed to that format. In Apache HBase, schemas are flexible and can be extended according to requirements.

Flexible Data Model

Apache HBase provides a flexible data model for storing data in tables. A table has one or more column families. User data is stored in rows, which are collections of key/value pairs. In a table, each row is uniquely identified by a row key.

Scalable

Apache HBase regions are horizontally scalable, with rows distributed by region. A table can be stored across multiple regions, and when a region becomes very large, the data is split into two regions around the middle row key.

HBase Functions

Apache HBase provides the following functions.

Supports linear and modular scalability.
Provides strict consistency for reads and writes.
Provides automatic and configurable table sharding.
Supports automatic failover between RegionServers.
Provides convenient base classes for supporting Hadoop MapReduce jobs with Apache HBase tables.
Makes the Java API easy to use for client access.
Provides block cache and Bloom filters for real-time queries.
Provides query predicate pushdown through server-side filters.
Provides an extensible JRuby-based (JIRB) shell.

HBase Pros and Cons

HBase Advantages

Effective for reliably handling large volumes of data.
- A master that controls the overall distributed system manages overall data consistency and guarantees consistency among replicated data.
Suitable for supporting large-scale data analysis processing.
- Easy to use as MapReduce input.
- Optimized for use with HDFS, MapReduce, and related tools.
If performance issues occur, performance can be maintained by adding only Region servers.
Failover is easy, and management is convenient.

HBase Disadvantages

It is convenient as MapReduce input, but when used together with file input, CPU usage can rise and Region servers can go down easily.
There is material about conditions for appropriate HBase settings, but because cluster size and basic specifications differ, it is difficult to apply directly.
Regions for a specific table can easily become concentrated on a specific Region server, leading to performance degradation.
- Proper HBase design is required during configuration.

Differences Between RDBMS and HBase

The following table summarizes the major differences between relational databases and HBase.

RDBMS	HBase
Uses tables as the database.	Uses regions as the database.
Supported file systems are FAT, NTFS, and EXT.	The supported file system is HDFS.
Uses commit logs to store logs.	Uses WAL (Write-Ahead Logs) to store logs.
The reference system used is a coordinate system.	The reference system is the one used by ZooKeeper.
Uses a primary key.	Uses a row key.
Partitioning is supported.	Sharding is supported.
Uses Row, Column, and Cell.	Uses Row, Column family, Column, and Cell.

Differences Between HDFS and HBase

The following table summarizes the major differences between HDFS and HBase.

HDFS	HBase
HDFS provides a file system for distributed storage.	HBase provides table-form, column-based data storage.
HDFS provides storage optimized for large files.	HBase provides optimization for table-form data.
Uses flat files.	Uses key-value pair data.
The data model is not flexible.	Provides a flexible data model.
Uses a file system and processing framework.	Uses table-form storage with built-in Hadoop MapReduce support.
Mostly optimized for write-once-read-many workloads.	Optimized for many reads and writes.

Differences Between Row-Oriented and Column-Oriented Data Stores

The following table summarizes the major differences between row-oriented and column-oriented data stores.

Row-oriented data store	Column-oriented data store
Row-oriented data stores are efficient for adding and modifying records.	Column-oriented data stores are efficient for reading data.
They read pages containing entire rows.	They read only the required columns.
Best suited for OLTP systems.	Not yet suitable for OLTP systems.
Serializes the full value of a row.	Serializes the full value of a column.
Stores rows in contiguous pages of memory.	Stores columns from pages in memory.

What is OLTP (Online Transactional Processing)? It refers to online transaction processing and means batch transaction processing against databases by online users over a network.

Pros and Cons of Column-Oriented Databases

The following are the pros and cons of column-oriented databases.

Advantages

Built-in support for efficiency and data compression.
Supports fast data retrieval.
Management and configuration are simplified in column-oriented databases.
Suitable for high performance in aggregate queries, such as COUNT, SUM, AVG, MIN, and MAX.
Provides an automatic sharding mechanism that distributes larger regions into smaller regions, making it efficient for partitioning.

Disadvantages

JOIN queries and data across multiple tables are not optimized.
Storage efficiency decreases because partitions must be created for frequent deletes and updates.
Because of the non-relational nature, designing partitions and indexes is very difficult.

HBase Use Cases

Used to guarantee data consistency or for analysis.
- Example: Facebook, eBay, Adobe, LINE, etc.
Monitoring systems.
Tracking user actions.
Audit logging systems.
Real-time analytics.
- Cases where Hadoop must be used to analyze large amounts of data.
Used as the primary storage for large-scale social networking services.
- Message-oriented systems, such as Twitter-like messages and statuses.
Content management systems that serve content from HBase.
Standard use cases such as storing web pages during web crawling.

References

Apache HBase Architecture

kc@example.com (kc kim) — Fri, 08 Jul 2022 09:40:00 +0900

Apache HBase Architecture

Apache HBase is a NoSQL database. NoSQL is a general expression indicating that a database is not an RDBMS whose primary access language is SQL. Apache HBase is a data store rather than a traditional database. HBase can scale linearly and modularly by adding commodity nodes to the cluster. If the number of nodes increases from 20 to 40, storage and capacity also increase at the same time in the HBase cluster.

HBase Cluster Architecture

Apache HBase Components

HBase consists of two types of servers: Master servers and Region servers.
Its structure allows simple scale-out by adding Region servers. HBase data is divided into units called Regions, and server types are separated according to how Regions are handled.

Master Server - HBase HMaster

Apache HBase HMaster is an important component of an HBase cluster that is responsible for monitoring RegionServers, handling failover, and managing region splits.

Data for each table is managed by RegionServers, and the entire cluster is managed by HMaster.
HMaster manages metadata that records which Region server stores each piece of HBase data (Region).
It assigns Regions to Region servers and detects failures in Region servers.

HMaster functions include the following.

Monitors RegionServers.
- Coordinates Region servers.
- Manages Region startup.
Handles RegionServer failover.
Assigns or unassigns Regions.
Monitors all Region servers in the cluster.
- Management functions.
- Table creation, deletion, and update.
Handles metadata changes.
Provides an interface for all metadata changes.
Performs load balancing during idle time.
HMaster provides a web user interface that shows information about the HBase cluster.

These tasks are handled by ZooKeeper, software that runs inside the Master server.
ZooKeeper is software used in many distributed applications besides HBase.

Region Server - HBase Region Server

RegionServer is responsible for storing the actual data assigned by the Master server. As in a Hadoop cluster, NameNode stores metadata and DataNode stores actual data. HBase is similar: the Master holds metadata, and RegionServers store actual data. RegionServer runs on DataNodes in a distributed cluster environment.

RegionServer performs the following tasks.

Handles assigned regions (tables).
Handles read and write requests made by clients.
Flushes cache to HDFS.
Is responsible for region split processing.
Maintains HLog.

The client first queries the Master server, or more precisely ZooKeeper, to learn the location of the Region server that holds the target data (Region).

Then that Region server handles the client request.
A Region is always managed by a single Region server and maintains data consistency for reads and writes.

If a Region server goes down, the assigned Regions fail over to another Region server. This work is performed by the Master server.
However, until failover completes, clients cannot read from or write to the target Region.

RegionServer Components

The following are the components of a RegionServer.

WAL (Write-Ahead Logging)

What is WAL? It is a standard method used by databases to guarantee data integrity. In systems that use WAL, changes are first recorded in the WAL before data is modified. These records make it possible to determine when a failure occurred, how far processing succeeded, and where subsequent work should resume.

Various database systems, including PostgreSQL, HBase, and MongoDB, use the WAL method.

Apache HBase WAL is an intermediate file also called an edit log file. When reading or modifying data in HBase, data is not written directly to disk but kept in memory for a period of time. If the system goes down, all data in memory may be lost. To overcome this problem, Apache HBase first records data in a Write-Ahead Logging file before recording it in memory.

HFile

HFile is the format for storing data in HBase. It belongs to a Column Family, and a Column Family can have multiple HFiles. It is the actual file where row data is physically stored, and data is stored in key/value format. When MemStore fills sufficiently, it creates a new HFile in HDFS and stores the data there.
This process is very fast because the structure minimizes random access.

Store

Store corresponds to a Column family for an HBase table. HFiles are stored here.

MemStore

MemStore resides in main memory and records current data operations. After data is stored in the WAL, key/value data is sorted and stored in the RegionServer memory store. This data is then stored in HFile as-is. There is one MemStore per Column Family.

Region

A Region is a table split that is divided based on keys and hosted by RegionServers.

Client

Clients can be written in Java or other languages and can connect through external APIs to the RegionServer that manages the actual row data. A client queries catalog tables to find a Region. Once the Region is found, the client connects directly to RegionServers, performs data operations, and caches data for fast lookups.

Catalog Tables

Catalog tables are used to maintain metadata for all RegionServers and Regions.

HBase has two types of catalog tables.

-ROOT- This table contains information about the location of the META table.
.META This table contains information about all Regions and their locations.

ZooKeeper

Apache ZooKeeper acts like the coordinator for HBase. HBase uses ZooKeeper to manage the state of servers that make up the cluster. It provides services such as maintaining configuration information, naming, distributed synchronization, and server error notification. Clients communicate with region servers through ZooKeeper.

Apache ZooKeeper is a high-performance centralized coordination service system for distributed applications that provides distributed synchronization and group services to HBase. Despite cluster coordination, it lets users focus on application logic. It also provides APIs that allow users to interact with the Master server.

The Apache ZooKeeper API provides consistency, ordering, and durability, and also provides synchronization and concurrency for distributed cluster systems.

HBase Data Write/Read Process

This section explains how HBase reads and writes data.

HBase Data Writes

When storing data in HBase, the data is stored in two places: the WAL (Write Ahead Log) and the Region server’s MemStore.

Using these two locations, the write process is considered complete when changes have occurred in both the WAL and MemStore.

When a data load request arrives in HBase, the appropriate Region server is found first. After the Region server is found, a commit log is added and the data is added to the in-memory MemStore. Data stored in MemStore is flushed to disk as an HFile when it becomes full according to configured values. Memory is cleared, the server waits for the next request, and a corresponding record is also written to the WAL. MemStore flush occurs when the value exceeds hbase.hregion.memstore.flush.size; the unit is bytes and the setting is 134217728 (128 MB).

MemStore is a memory area of the Region server and acts like a cache. When a certain amount of data accumulates in MemStore, the data is exported to disk as an HFile and persisted.
However, data before it is exported to HFile is stored in server memory (MemStore), so it disappears if the server goes down. Therefore, HBase writes a log called HLog when writing data, and if the server goes down, it can recover data based on HLog.

HBase Data Reads

The process of reading data from HBase is similar to the write process described above.

When a request enters HBase, MemStore is checked first. If the desired data is found in MemStore, that data is returned. Otherwise, HBase searches from the most recently flushed file to older files until it finds suitable data that satisfies the query or there are no more flushed files.

References

HBase Introduction|joinc.co.kr

Apache HBase Regions

kc@example.com (kc kim) — Wed, 21 Jun 2023 11:05:33 +0900

Apache HBase stores large amounts of data in tables sorted by row key in lexicographic order. Tables are distributed across multiple Regions, and Regions are further distributed across multiple RegionServers. When you create a table in Apache HBase, a default Region is assigned.

Apache HBase Regions are horizontally scalable. They include a start key and an end key, and rows are sorted and stored in a continuous format based on those keys. Because HBase provides strong consistency, it does not store the same row key in multiple Regions. Regions distribute load across multiple RegionServers and also perform load balancing and failover according to requirements. As data grows, Regions are split manually or automatically.

Number of Regions

It is recommended to use a small number of medium to large Regions, 5 to 20 GB, per RegionServer, usually 20 to 200 Regions. The standard number of Regions is 100.

Some factors to consider for Regions are as follows.

The limiting factor when choosing the number of Regions is available heap space. MemStore requires almost 2 MB per column family per Region. If there are 100 Regions and 3 column families per Region, MemStore heap space requirements are 600 MB. Fewer Regions require less MemStore heap.
A large number of Regions creates many small flushes. If each flush creates a StoreFile, many StoreFiles are created, requiring more compaction. MemStore and StoreFile indexes also require more heap space.
A large number of Regions creates load on the master because the master must assign and reassign Regions to RegionServers. The master must also move Regions for load balancing.

Region Assignment

In Apache HBase, Regions are assigned by the master as follows.

The master starts the assignment manager.
The assignment manager checks existing assignments in hbase:meta metadata.
Assignments are stored when RegionServers are available.
If a RegionServer is not online, the load balancer is called to assign Regions to another RegionServer.
The hbase:meta metadata is updated with the new assignment.

Region failover

Because a RegionServer can fail and multiple RegionServers provide data, a RegionServer’s Regions can become unavailable. ZooKeeper detects RegionServer failures, and the master starts failover on another RegionServer with a similar Row Key for the Region.

Region Locality

Region Locality indicates the proximity of a Region to a RegionServer. It is achieved through HDFS block replication across the cluster.

The replica placement policy is the HDFS replica placement policy, as follows.

The first replica is placed on the local node.
The second replica is placed on a random node in a different rack.
The third replica is placed in the same rack as the second replica, but on a different node.

Benefits of Regions

The benefits of Regions include distributed data storage, partitioning, automatic sharding and scalability, and Region splitting.

Let’s look at each in detail.

Distributed data store

The design of a distributed data store matches Apache HBase’s design of using multiple Regions for a table. Distributing Regions of larger tables across a cluster of nodes provides high availability.

Partitioning

Data tables are stored in multiple Regions, and Regions partition the data. To access data in that table, access must now happen from different Regions. Having multiple Regions provides the benefit of delivering data quickly.

Automatic sharding and scalability

The automatic sharding process is used to split a Region into roughly two halves when the number of row keys in the Region becomes too large. In HBase, the basic unit of horizontal scalability is the Region, and rows are shared by Region.

Region splitting

When a threshold is exceeded, a Region is split. This is handled by the RegionServer, which splits the Region and takes the split Region offline. After that, two split Regions are added to hbase:meta, opened by the RegionServer, and reported to the master. Region splitting is automatic by default, but it can also be run manually. The HBase Region split policy is configured with hbase.regionserver.region.split.policy.

HBase Data Model

kc@example.com (kc kim) — Tue, 20 Jun 2023 19:07:12 +0900

Data structure

HBase tables have no fixed type and are stored as byte arrays (byte[]).
HBase rows are sorted in ascending order by a unique row key, and this row key is used when reading and writing table values.
As mentioned earlier, data managed by HBase is divided into Regions by fixed ranges, and a table has multiple Regions. HBase columns are also grouped in units called column families.
Table data is divided by Region and by column family, then exported as files. Files are separate, but if they belong to the same Region, they are stored on the same Region server.

Each HBase data cell value has a timestamp, so versions are managed. Files are stored in the following format.

Row(Row Key): Column family: Column: timestamp: value

Data model

The Apache HBase data model is a distributed, multidimensional, persistent, sorted map indexed by column key, row key, and timestamp. This is why Apache HBase is also called a key-value storage system.

The basic unit of HBase is a column. Columns form a Column Family, and Column Families form a table. Each Row in a table has a Row Key and can be identified by it. See the figure below.

This table has two column families: Customer and Sales.
The Customer column family has two columns: Name and City.
The Sales column family has two columns: Product and Amount.
A Row consists of Row Key, Customer CF, and Sales CF.

The following are data model terms used in Apache HBase.

Table

An Apache HBase table consists of multiple Rows. It organizes data into tables made of characters that are easy to use with the file system.

A set of Rows, with a Row Key and multiple column families.
In schema definition, only Column Families are defined.

Row

Apache HBase stores data based on Rows, and each Row has a unique Row Key. The Row Key is represented as a byte array.

Row Key

Because data is collected based on the Row Key, the Row Key must be designed so that data can be distributed appropriately. The goal of Row Key design is to store data so similar Rows are near each other.

Sorted lexicographically as arbitrary bytes.
An empty byte string means the start and end of the table.
Anything can be a row key, including strings, integers, binary data, and serialized data structures.

In general, a Row Key pattern can also be organized in website domain format. If domains are stored in reverse order, such as org.apache.mair and org.apache.jira, Apache domain data can be stored near each other.

Column

A Column consists of a Column Family and a Column Qualifier.

Column Family

A Column Family is used to store Rows and also provides the structure for storing data in Apache HBase.

It consists of characters and strings and can be used with file system paths. Each row in a table has the same column families, but a row does not need to store data in every column family.

Column Qualifier

A Column Qualifier provides an index for data stored in a Column Family. The Column qualifier is not a fixed value, so various data can be entered. It can be thought of as a map object.

A group of Columns where all ColumnFamily members use the same prefix.
NOSQL:Cassandra and NOSQL:HBASE are member columns of the NOSQL column family.
The column family prefix must consist of displayable characters.
Part of the table schema definition must be specified first.
All column family members are physically stored together in the file system.
New column family members can be added dynamically.

Cell

A Cell is a data unit consisting of Column family, Row key, and Column qualifier. The value of each column is called a cell.

A data cell includes a value and a timestamp, which is the version of the value. Because there is a timestamp, previous values are stored together and maintained for a certain period.

A tuple with ROW KEY, Column, and Version specified.
The value is an arbitrary byte array and Timestamp.
Table cells are versioned, only cells.

Timestamp

Versions are assigned to values where the same data is stored in a Cell. Each version is identified by a version number assigned during creation time. If Timestamp is not mentioned while writing data, the current time is assigned.

HBase data types

Apache HBase has no concept of data types. Everything is a byte array. When values are inserted, it is a kind of byte-in and byte-out database converted to byte arrays using the Put and Result interfaces. Apache HBase uses a serialization framework to convert user data into byte arrays.

Apache HBase cells can store values up to 10 to 15 MB. If a value is larger, it can be stored in Hadoop HDFS and file path metadata information can be stored in Apache HBase.

HBase data storage

The following introduces how Apache HBase is physically stored.

Conceptual view

You can see that a table is represented as a series of Rows at the conceptual level.

The following is a conceptual diagram of how data is stored in HBase.

Physical view

In the physical view, a table is physically stored by column family.

The following example shows a table that will be stored as a column-family-based table.

Namespace

A namespace is a logical group of tables. It is similar to grouping related tables in a relational database.

The representation of a namespace is as follows.

HBase Namespaces
- Table
- Region Server Group
- Permission
- Quota

Let’s look at each component of namespace space.

Table

All tables are part of a namespace. If no namespace is defined, the table is assigned to the default namespace.

RegionServer group

A default RegionServer group can be assigned to a namespace. In this case, created tables become members of the RegionServer group.

Permission

Using namespaces, users can define access control lists such as read, delete, and update permissions, and with write permission, users can create tables.

Quota

This component is used to define the quota that a namespace can include for tables and Regions.

Predefined namespaces

There are two predefined special namespaces.

hbase: A system namespace used to include HBase internal tables.
default: This namespace is for all tables where no namespace is defined.

Data model operations

The main data model operations are Get, Put, Scan, and Delete. These operations can be used to read, write, and delete records from tables.

Let’s look at each operation in detail.

Get

The Get operation is similar to the Select statement in a relational database. It is used to fetch the contents of an HBase table.

You can run the Get command in the HBase shell as follows.

hbase(main) :001:0> get 'table name', 'row key' <filters>

Put

The Put operation is used to write data to a table.

Scan

The Scan operation is used to read multiple rows from a table. It differs from Get, where the set of rows to read must be specified. Scan can iterate over a row range or all rows in a table.

Delete

The Delete operation is used to delete a row or set of rows from an HBase table. It can be executed through HTable.delete().

When a delete command is executed, it is marked as deleted, and when compaction occurs, the row is finally deleted from the table.

Internal delete types are as follows.

Delete: Used for a specific version of a column.
Delete column: Can be used for all column versions.
Delete family: Used for all columns in a specific ColumnFamily.

Installing HBase with Docker for Local Practice

kc@example.com (kc kim) — Tue, 13 Jun 2023 18:40:33 +0900

Installing HBase with Docker

Let’s install HBase in a Docker environment using the Docker image below.

HBase Docker image to install
- https://hub.docker.com/r/dajobe/hbase

Downloading HBase Docker run commands

Download, or clone, the HBase Docker run commands from the GitHub repository below.

GitHub
- https://github.com/dajobe/hbase-docker

git clone git@github.com:dajobe/hbase-docker.git

Running HBase Docker

With Docker running, execute the following command as-is.

./start-hbase.sh

It is recommended to use the start-hbase.sh script, which starts the container, inspects it to determine all local API ports and web UI ports, and suggests editing /etc/hosts to add an alias for the container IP if it does not already exist.

It is recommended to use the start-hbase.sh script, which starts the container, determines all local API ports and web UI ports, and suggests editing /etc/hosts to add an alias for the container IP if it does not already exist.

% ./start-hbase.sh
start-hbase.sh: Starting HBase container
Error: No such container: hbase-docker
Unable to find image 'dajobe/hbase:latest' locally
latest: Pulling from dajobe/hbase
6cf436f81810: Pulling fs layer

... omitted ...

159beed970fa: Pull complete
Digest: sha256:daa36a6d90b118ced866b6c76fcd918e7da73302b0e4971f506f0f61f645a9fe
Status: Downloaded newer image for dajobe/hbase:latest
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
start-hbase.sh: Container has ID 3804d552250334cc850189e076b77cb0d3066850cef332bec7a125dc29228361
./start-hbase.sh: line 32: python: command not found

Here, the container ID is 3804d552250334cc850189e076b77cb0d3066850cef332bec7a125dc29228361. Record it because it is needed when running the shell.

When you run the command above, data is created in the current path where it was run, ports are opened, and HBase should run.

Manual run method

docker run --name=hbase-docker \
  -h hbase-docker \
  -p 9095:9095 \
  -p 8085:8085 \
  -p 16010:16010  \
  -v $PWD/data:/data \
  -d dajobe/hbase

Running the HBase Shell

Next, let’s run the shell command that can connect to HBase using the Docker image.

Put the container ID recorded above into $id and run it.

docker run --rm -it --link $id:hbase-docker dajobe/hbase hbase shell

On macOS Apple Silicon, such as M1 or M2, the following error may occur.

% docker run --rm -it --link 3804d552250334cc850189e076b77cb0d3066850cef332bec7a125dc29228361:hbase-docker dajobe/hbase hbase shell
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested

In that case, delete the Docker container that is already running, add the --platform linux/amd64 option, and run it again.

% docker run --rm -it --platform linux/amd64 --link 3804d552250334cc850189e076b77cb0d3066850cef332bec7a125dc29228361:hbase-docker dajobe/hbase hbase shell

2023-06-16 09:38:17,680 WARN  [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
HBase Shell
Use "help" to get list of supported commands.
Use "exit" to quit this interactive shell.
For Reference, please visit: http://hbase.apache.org/2.0/book.html#shell
Version 2.1.2, r1dfc418f77801fbfb59a125756891b9100c1fc6d, Sun Dec 30 21:45:09 PST 2018
Took 0.0480 seconds
hbase(main):001:0>

The command to exit the shell is quit.

hbase(main):005:0> quit

How to use commands will be explained on the next page.

HBase General Shell Commands - status, version, whoami

kc@example.com (kc kim) — Fri, 16 Jun 2023 14:55:22 +0900

General HBase shell commands

This section introduces general HBase commands.

`status`

Displays cluster status. Additional options, "summary", "simple", and "detailed", can be used to check detailed information. The default is "summary".

You can check cluster status with the status command.

Option	Description
simple	Server information
summary	Number of servers, load
detailed	Server information, meta information
replication	Source and sink information

status

hbase(main):001:0> status
1 servers, 0 dead, 2.0000 average load

status 'simple'

hbase(main):002:0> status 'simple'
1 live servers
    532f4d895a98:40371 1686636094968
        requestsPerSecond=0.0, numberOfOnlineRegions=2, usedHeapMB=19, maxHeapMB=966, numberOfStores=2, numberOfStorefiles=2, storefileUncompressedSizeMB=0, storefileSizeMB=0, memstoreSizeMB=0, storefileIndexSizeMB=0, readRequestsCount=42, writeRequestsCount=5, rootIndexSizeKB=0, totalStaticIndexSizeKB=0, totalStaticBloomSizeKB=0, totalCompactingKVs=0, currentCompactedKVs=0, compactionProgressPct=NaN, coprocessors=[]
0 dead servers
Aggregate load: 0, regions: 2

status 'summary'

hbase(main):003:0> status 'summary'
1 servers, 0 dead, 2.0000 average load

status 'detailed'

hbase(main):004:0> status 'detailed'
version 2.0.0-SNAPSHOT
0 regionsInTransition
master coprocessors: []
1 live servers
    532f4d895a98:40371 1686636094968
        requestsPerSecond=0.0, numberOfOnlineRegions=2, usedHeapMB=21, maxHeapMB=966, numberOfStores=2, numberOfStorefiles=2, storefileUncompressedSizeMB=0, storefileSizeMB=0, memstoreSizeMB=0, storefileIndexSizeMB=0, readRequestsCount=42, writeRequestsCount=5, rootIndexSizeKB=0, totalStaticIndexSizeKB=0, totalStaticBloomSizeKB=0, totalCompactingKVs=0, currentCompactedKVs=0, compactionProgressPct=NaN, coprocessors=[]
        "hbase:meta,,1"
            numberOfStores=1, numberOfStorefiles=1, storefileUncompressedSizeMB=0, storefileSizeMB=0, memstoreSizeMB=0, storefileIndexSizeMB=0, readRequestsCount=36, writeRequestsCount=3, rootIndexSizeKB=0, totalStaticIndexSizeKB=0, totalStaticBloomSizeKB=0, totalCompactingKVs=0, currentCompactedKVs=0, compactionProgressPct=NaN, completeSequenceId=10
        "hbase:namespace,,1686636105191.93358ca1f9bf1816320d88c48e39a3b4."
            numberOfStores=1, numberOfStorefiles=1, storefileUncompressedSizeMB=0, storefileSizeMB=0, memstoreSizeMB=0, storefileIndexSizeMB=0, readRequestsCount=6, writeRequestsCount=2, rootIndexSizeKB=0, totalStaticIndexSizeKB=0, totalStaticBloomSizeKB=0, totalCompactingKVs=0, currentCompactedKVs=0, compactionProgressPct=NaN, completeSequenceId=8
0 dead servers

version

Checks the installed HBase version information.

version

hbase(main):001:0> version
2.0.0-SNAPSHOT, r31ed817447b4c3edef2019d1580aa3ede83a82da, Fri Sep 26 02:07:26 UTC 2014

whoami

Checks the current HBase user.

whoami

hbase(main):008:0> whoami
root (auth:SIMPLE)
    groups: root

HBase Table Management Commands - create, list, exists, disable/enable, describe, alter, drop

kc@example.com (kc kim) — Fri, 16 Jun 2023 14:55:22 +0900

Table Management commands

This section introduces HBase Shell commands for table management.

create - Create a Table

You can create a table by using the create command. Here, you must specify the table name and Column Family name. The syntax for creating a table in the HBase shell is as follows.

create '<table name>', '<column family>'

The following is a sample schema for a table named order. It has two column families: customer and sales.

You can create this table in the HBase shell as follows.

create 'order', 'customer', 'sales'

When executed, it runs as follows.

hbase(main):002:0> create 'order', 'customer', 'sales'
Created table order
Took 2.4344 seconds
=> Hbase::Table - order

create Options

You can also add options when creating a table.

Create t1 and create column family cf1, storing up to 5 latest version values.

create 't1', {NAME => 'f1', VERSIONS => 5}

Create t1 and create column families cf1, cf2, and cf3.

create 't1', {NAME => 'f1', NAME => 'f2', NAME => 'f3'}

The following commands specify other options.

create 't1', {NAME => 'cf2', VERSION => 1, TTL => 2592000, BLOCKCACHE =>true}

create 't1', {NAME => 'cf2', CONFIGURATION => {'hbase.hstore.blockingStoreFiles' ='10'}}

list - View Table List

list is a command used to list all tables in HBase. The syntax of the list command is as follows.

list

If you enter this command and run it at the HBase prompt, all HBase tables are displayed as follows.

hbase(main):004:0> list
TABLE
order
1 row(s) in 0.1070 seconds
=> ["order"]

You can also add options as follows.

hbase(main):005:0> list 'customer.*'
TABLE
order
1 row(s) in 0.0140 seconds
=> ["order"]

exists - Check Table Existence

You can check whether a table exists by using the exists command. The following is the syntax for checking whether a table exists.

exists '<table name>'

The following example checks whether the order table exists.

hbase(main):022:0> exists 'order'
Table order does exist
0 row(s) in 0.0680 seconds

The following example checks whether the user table exists, where it does not actually exist.

hbase(main):023:0> exists 'user'
Table user does not exist
0 row(s) in 0.0110 seconds

disable - Disable a Table

To delete a table or change its settings, you must first disable the table by using the disable command. You can enable it again by using the enable command.

The syntax for disabling a table is as follows.

disable '<table name>'

The following example shows how to disable the order table.

hbase(main):004:0> disable 'order'
0 row(s) in 1.4570 seconds

Even after disabling a table, you can check that the table exists with the list and exists commands.

hbase(main):009:0> list
TABLE
order
1 row(s) in 0.0140 seconds
=> ["order"]

A disabled table cannot be scanned. The following error occurs.

hbase(main):010:0> scan 'order'
ROW                                         COLUMN+CELL

ERROR: order is disabled.

// omitted

is_disabled

This command is used to check whether a table is disabled. The syntax is as follows.

is_disabled '<table name>'

The following example checks whether a table named order is disabled. It returns true if disabled and false otherwise.

hbase(main):011:0> is_disabled 'order'
true
0 row(s) in 0.0360 seconds

disable_all

The disable_all command is used to disable all tables matching a given regular expression. The syntax of the disable_all command is as follows.

disable_all ‘<regex table name>’

A simple example is shown below. The following command disables all tables that start with r.

disable_all 'r.*'

Assume HBase has four tables that start with dev: dev_follow, dev_friend, dev_test, and dev_user. The following code disables all tables that start with dev.

hbase(main):017:0> disable_all 'dev.*'
dev_follow
dev_friend
dev_test
dev_user

Disable the above 4 tables (y/n)?
y
4 tables successfully disabled

enable - Enable a Table

To enable a disabled table, use the enable command. The syntax for enabling a table is as follows.

enable '<table name>'

The following is an example of enabling a table.

hbase(main):018:0> enable 'order'
0 row(s) in 0.6580 seconds

After enabling the table, scan it. If no error occurs, the table has been enabled normally.

hbase(main):020:0> scan 'order'
ROW                                         COLUMN+CELL
0 row(s) in 0.0730 seconds

is_enabled

The is_enabled command is used to check whether a table is enabled. The syntax is as follows.

is_enabled '<table name>'

The following command checks whether a table named order is enabled. It returns true if enabled and false otherwise.

hbase(main):021:0> is_enabled 'order'
true
0 row(s) in 0.0750 seconds

describe - Describe a Table

The describe command returns a description of a table. The syntax is as follows.

describe '<table name>'

The following is the output of the describe command for the order table.

hbase(main):006:0> describe 'order'
DESCRIPTION                                                                                                  ENABLED
 'order', {NAME => 'customer', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCO true
 PE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELL
 S => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'sales', DATA_BLO
 CK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NON
 E', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY =
 > 'false', BLOCKCACHE => 'true'}
1 row(s) in 0.1450 seconds

alter - Change a Table

alter is a command used to change an existing table. You can use this command to change the maximum number of cells in a column family, set and remove table-scope operators, and delete column families from a table.

Change the Maximum Number of Cells in a Column Family

The following is the syntax for changing the maximum number of cells in a column family.

alter '<table name>', NAME => '<column family>', VERSIONS => 5

The following example sets the maximum number of cells to 5.

hbase(main):007:0>alter 'order', NAME => 'customer', VERSIONS => 5
Updating all regions with the new schema...
0/1 regions updated.
1/1 regions updated.
Done.
0 row(s) in 2.4240 seconds

hbase(main):009:0>

Table-Scope Operators

With alter, you can set and remove table-scope operators such as MAX_FILESIZE, READONLY, MEMSTORE_FLUSHSIZE, and DEFERRED_LOG_FLUSH.

READONLY - Set Read-Only

The following is the syntax for making a table read-only.

alter '<table name>', READONLY(option)

The following example makes the order table read-only.

hbase(main):009:0> alter 'order', READONLY
Updating all regions with the new schema...
0/1 regions updated.
1/1 regions updated.
Done.
0 row(s) in 2.3650 seconds

MAX_FILESIZE - Change Region Setting

Change the region setting with MAX_FILESIZE.

alter 't1', MAX_FILESIZE => '123217728'

This changes the maximum region size to 128 MB.

Delete a Column Family

With alter, you can also delete a column family. The syntax for deleting a column family with alter is as follows.

alter '<table name>', 'delete' => '<column family>'

The following is an example of deleting a column family from the order table.

hbase(main):010:0> alter 'order', 'delete' => 'sales'
Updating all regions with the new schema...
0/1 regions updated.
1/1 regions updated.
Done.
0 row(s) in 2.3790 seconds

If you check with the describe command, you can see that 'sales' has disappeared.

hbase(main):011:0> describe 'order'
DESCRIPTION                                                                                                  ENABLED
 'order', {NAME => 'READONLY', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCO true
 PE => '0', COMPRESSION => 'NONE', VERSIONS => '1', TTL => 'FOREVER', MIN_VERSIONS => '0', KEEP_DELETED_CELL
 S => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'customer', DATA_
 BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '5', COMPRESSION => '
 NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMOR
 Y => 'false', BLOCKCACHE => 'true'}
1 row(s) in 0.0850 seconds

drop - Delete a Table

You can delete a table by using the drop command. The syntax is as follows.

drop '<table name>'

The table must be disabled before it is deleted.

hbase(main):024:0> disable 'order'
0 row(s) in 1.3960 seconds
hbase(main):025:0> drop 'order'
0 row(s) in 0.2860 seconds

Use the exists command to check whether the table has been deleted.

hbase(main):026:0> exists 'order'
Table order does not exist
0 row(s) in 0.0710 seconds

drop_all

The drop_all command is used to delete all tables matching a given regular expression. The syntax is as follows.

drop_all '<regex table name>'

A simple example is shown below. The following command deletes all tables that start with t.

drop_all 't.*'

Note: All target tables must be disabled before deletion.

HBase Data Manipulation Commands - put, get, delete, deleteall, scan, count, truncate

kc@example.com (kc kim) — Fri, 16 Jun 2023 14:55:22 +0900

Data Manipulation commands

This section introduces HBase Shell commands for manipulating data.

put - Insert/Update Data

This section introduces how to create and update data in an HBase table.

The following HBase table will be created.

You can insert and update a row in a table by using the put command. The syntax is as follows.

put '<table name>', '<row id>', '<column family>:<column name>', '<value>'

Insert the First Row

create 'order', 'customer', 'sales'

put 'order', '101', 'customer:name', 'John White'
put 'order', '101', 'customer:city', 'Los Angeles'
put 'order', '101', 'sales:product', 'Chairs'
put 'order', '101', 'sales:amount', '400.00'

First, create the table.

hbase(main):001:0> create 'order', 'customer', 'sales'
Created table order
Took 5.0510 seconds
=> Hbase::Table - order

Then insert the first row values into the table.

hbase(main):002:0> put 'order', '101', 'customer:name', 'John White'
Took 0.6725 seconds
hbase(main):003:0> put 'order', '101', 'customer:city', 'Los Angeles'
Took 0.0318 seconds
hbase(main):004:0> put 'order', '101', 'sales:product', 'Chairs'
Took 0.0988 seconds
hbase(main):005:0> put 'order', '101', 'sales:amount', '400.00'
Took 0.0423 seconds

If you scan the whole table, the following result is displayed.

hbase(main):006:0> scan 'order'
ROW                               COLUMN+CELL
 101                              column=customer:city, timestamp=1686908392705, value=Los Angeles
 101                              column=customer:name, timestamp=1686908388954, value=John White
 101                              column=sales:amount, timestamp=1686908401071, value=400.00
 101                              column=sales:product, timestamp=1686908398256, value=Chairs
1 row(s)
Took 0.4121 seconds

Update Data

You can change an existing cell value by using the put command. The syntax is the same as when inserting data.

Specify a new value as follows.

put 'order','101','customer:city','LA'

The command execution is as follows.

hbase(main):007:0> put 'order','101','customer:city','LA'
Took 0.0677 seconds

hbase(main):008:0> scan 'order'
ROW                               COLUMN+CELL
 101                              column=customer:city, timestamp=1686909532774, value=LA
 101                              column=customer:name, timestamp=1686908388954, value=John White
 101                              column=sales:amount, timestamp=1686908401071, value=400.00
 101                              column=sales:product, timestamp=1686908398256, value=Chairs
1 row(s)
Took 0.1186 seconds

get - Retrieve Data

The get command is used to read data from an HBase table. The syntax is as follows.

get '<table name>', '<row id>'

The following example uses the get command to retrieve row 101 from the order table.

hbase(main):009:0> get 'order', '101'
COLUMN                            CELL
 customer:city                    timestamp=1686909532774, value=LA
 customer:name                    timestamp=1686908388954, value=John White
 sales:amount                     timestamp=1686908401071, value=400.00
 sales:product                    timestamp=1686908398256, value=Chairs
1 row(s)
Took 0.3162 seconds

Retrieve a Specific Row

The following is the syntax for using the get command to retrieve a specific column.

get '<table name>', '<row id>', {COLUMN => '<column family>:<column name>'}

The following is an example of retrieving a specific column from an HBase table.

hbase(main):010:0> get 'order', '101', {COLUMN => 'customer:name'}
COLUMN                            CELL
 customer:name                    timestamp=1686908388954, value=John White
1 row(s)
Took 0.1450 seconds

delete - Delete a Specific Cell from a Table

You can delete a specific cell from a table by using the delete command. The syntax of the delete command is as follows.

delete '<table name>', '<row id>', '<column name>', '<timestamp>'

The following is an example of deleting a specific cell. Here, city is deleted.

hbase(main):011:0> delete 'order', '101', 'customer:city', 1686909532774
Took 0.0917 seconds

If you query again, you can see that the 'LA' value changed with the previous put command has been deleted and the existing 'Los Angeles' value is visible again.

hbase(main):012:0> get 'order', '101'
COLUMN                            CELL
 customer:city                    timestamp=1686908392705, value=Los Angeles
 customer:name                    timestamp=1686908388954, value=John White
 sales:amount                     timestamp=1686908401071, value=400.00
 sales:product                    timestamp=1686908398256, value=Chairs
1 row(s)
Took 0.2196 seconds

deleteall - Delete All Cells in a Table Row

You can delete all cells in a row by using the deleteall command. The following is the syntax of the deleteall command.

deleteall '<table name>', '<row id>'

The following is an example of the deleteall command that deletes all cells in row 1 of the order table.

hbase(main):013:0> deleteall 'order','101'
Took 0.1271 seconds
hbase(main):015:0>

Check the table by using the scan command. After deleting the table data, the table snapshot is as follows.

hbase(main):015:0> scan 'order'
ROW                               COLUMN+CELL
0 row(s)
Took 0.0518 seconds

scan - Retrieve Data

The scan command is used to view data in HTable. You can use the scan command to fetch table data. The syntax is as follows.

scan '<table name>'

First, insert data as follows.

put 'order', '101', 'customer:name', 'John White'
put 'order', '101', 'customer:city', 'Los Angeles'
put 'order', '101', 'sales:product', 'Chairs'
put 'order', '101', 'sales:amount', '400.00'

The following example uses the scan command to retrieve data from the table.

hbase(main):019:0> scan 'order'
ROW                               COLUMN+CELL
 101                              column=customer:city, timestamp=1686912065208, value=Los Angeles
 101                              column=customer:name, timestamp=1686912059341, value=John White
 101                              column=sales:amount, timestamp=1686912074891, value=400.00
 101                              column=sales:product, timestamp=1686912070718, value=Chairs
1 row(s)
Took 0.2649 seconds

count - Count Data

You can count the number of rows in a table by using the count command. The syntax is as follows.

count ‘<table name>'

hbase(main):020:0> count 'order'
1 row(s)
Took 0.2992 seconds
=> 1

truncate - Delete All

The truncate command disables, drops, and recreates a table. The syntax of truncate is as follows.

truncate ‘<table name>'

The following is an example of the truncate command. Here, the order table was deleted and recreated.

hbase(main):021:0> truncate 'order'
Truncating 'order' table (it may take a while):
Disabling table...
Truncating table...
Took 5.6828 seconds

After deleting the table data with truncate, check it by using the scan command. You can confirm that the table has no rows.

hbase(main):023:0> scan 'order'
ROW                               COLUMN+CELL
0 row(s)
Took 1.6790 seconds

HBase Security Commands - grant, revoke, user_permission

kc@example.com (kc kim) — Sun, 18 Jun 2023 16:13:38 +0900

Security tools

In HBase, permissions can be granted to and revoked from users. There are three commands for security purposes.

grant
revoke
user_permission

grant - granting permissions

The grant command grants users specific permissions such as read, write, execute, and admin permissions for tables. The syntax of the grant command is as follows.

grant '<user>' '<permissions>' ['<table>' ['<column family>' ['<column; qualifier>']]

RWXCA can be used to grant permissions to users.

R: READ permission
W: WRITE permission
X: EXEC permission
C: CREATE permission
A: ADMIN permission

grant 'devkuma', 'RWXCA'

grant 'devkuma', 'RW' 't1', 'cf1', 'qualifier1'

revoke - deleting permissions

Deletes a user’s access permissions.

revoke <user> <permissions> [<table> [<column family> [<column; qualifier>]]

revoke 'devkuma', 't1', 'cf1', 'qualifier1'

user_permission - querying permissions for a specific table

The user_permission command is used to query all permissions for a specific table. The syntax is as follows.

user_permission '<table name>'

Shows all permissions that exist for a specific table.

user_permission 't1'

HBase Compression, Compaction, and Data Block Encoding

kc@example.com (kc kim) — Thu, 22 Jun 2023 19:13:23 +0900

Compression and Compaction

Regions stored on an HBase RegionServer have an HStore for each Column Family, and this consists of MemStore and HFile for that CF.

MemStore is Column Family data in memory, and HFile is Column Family data stored on local disk, meaning HDFS. The reason for this layered structure is that data must be sorted by Row key according to the HBase table structure, so it cannot be written directly to disk and must be sorted in memory first.

When write operations such as put or update are performed in HBase, data is first recorded in the WAL (Write Ahead Log), then written to MemStore. For read operations such as scan, HBase checks MemStore first and then searches HFiles for the requested content.

When repeated writes cause MemStore data size to exceed the configured threshold, MemStore data is stored as an HFile. In a Region composed of many HFiles, as HFiles increase, compaction occurs and merges multiple HFiles into larger ones. As a result, many small HFiles are reduced to fewer large HFiles. This operation is called compaction.

HBase compaction is performed as minor compaction and major compaction.

Minor Compaction

Minor compaction is the process of merging several recently created small files into a larger configured HFile size.
As data is inserted, many small HFiles are created. Too many files can reduce performance, so HBase automatically manages the number of HFiles by remaking multiple HFiles into a few larger HFiles.
Because data stored in HFiles is sorted, it can be merged quickly using merge sort.

Major Compaction

Major compaction gathers all HFiles for a Column Family in a specific Region and creates one HFile per column. It runs once every 7 days by default.
During this process, unnecessary cells and expired cells are removed to improve overall read performance. Data with Tombstone Markers is finally deleted at this time.
When this work is performed, large amounts of file read/write activity occur, which can increase disk I/O and network traffic.
It can be scheduled to run automatically, and this can be used to schedule it on weekends or at night to minimize the impact of sudden I/O increases on the service.

Compaction settings

Minor compaction uses the following settings to configure the number of HFiles that will be compacted.

hbase.hstore.compaction.min
- Sets the minimum number of HFiles for minor compaction.
- The default is 3, and it must be set to 2 or higher.
- If the value is too large, minor compaction is delayed and later processing all at once can create load.
hbase.hstore.compaction.max
- Sets the maximum number of HFiles for minor compaction.

The following settings specify the size of HFiles to compact.

hbase.hstore.compaction.min.size: HFiles smaller than this value are always included when searching for minor compaction through ExploringCompactionPolicy. This value is used together with hbase.hstore.compaction.ratio to choose HFiles for compaction.
hbase.hstore.compaction.max.size: HFiles larger than this size are excluded from minor compaction. If minor compaction occurs often but does not have much effect, this value can be reduced so large files are excluded from minor compaction. Default: 9223372036854775807, LONG.MAX_VALUE, bytes.

Other settings:

hbase.hregion.majorcompaction
- HBase can have multiple StoreFiles for one Region. Periodically, to improve performance, it gathers these files and combines them into one larger file. This process involves significant CPU usage and disk I/O, and response time may decrease. Therefore, when response time matters, it is better to run Major compaction manually during off-peak time.
- The default is 86,400,000 ms. If this value is changed to 0, periodic Major Compaction will not run.
hbase.hregion.majorcompaction.jitter
- The property defaults to 0.2, or 20%. It spreads out the timing when Major compaction runs for each stored file.
- Without it, major compaction runs simultaneously every 24 hours.

HBase compression and data block encoding

The codecs mentioned in this section are for encoding and decoding data blocks or row keys. For more information about replication codecs, see cluster.replication.preserving.tags.

HBase supports several compression algorithms that can be enabled for a ColumnFamily. Data block encoding uses HBase’s underlying design and patterns, such as sorted row keys and the schema of a given table, to limit information duplication in keys. Compressors can reduce the size of large opaque byte arrays in Cells and significantly reduce the storage space needed for uncompressed data.

Compressors and data block encoding can be used together in the same ColumnFamily.

Changes are applied during compaction

If compression or encoding is changed for a ColumnFamily, the change is applied during compaction.

Some codecs use features built into Java, such as GZip compression. Other codecs depend on native libraries. Native libraries can be provided through codec dependencies installed in the HBase library directory, or as part of Hadoop when using Hadoop codecs. Hadoop codecs usually have native code components, so follow the Hadoop native binary support installation instructions to use Hadoop native libraries in HBase.

This section describes common codecs used and tested in HBase.

Whichever codec you use, you must test that the codec is installed correctly and available on all nodes in the cluster. Additional operational steps may be needed to verify codec availability on newly deployed nodes. The compression.test utility can be used to verify that a specified codec is installed correctly.

To configure HBase to use compressors, see compressor.install. To enable a compressor for a ColumnFamily, see changing.compression. To enable data block encoding for a ColumnFamily, see data.block.encoding.enable.

Block Compressors

NONE

This compression type constant selects no compression and is the default.

BROTLI

Brotli is a general-purpose lossless compression algorithm. It combines a modern variant of LZ77, Huffman coding, and second-order context modeling, and provides compression ratios comparable to the best general-purpose compression methods currently available. It has similar speed to GZ but provides denser compression.

BZIP2

Bzip2 compresses files using the Burrows-Wheeler block-sorting text compression algorithm and Huffman coding. It usually has a much better compression ratio than dictionary-based compression such as LZ, but both compression and decompression may be slower than other options.

GZ(GZIP)

Focuses on compression ratio.
gzip is based on the DEFLATE algorithm, a combination of LZ77 and Huffman coding. This algorithm is universally available in the Java runtime environment, so it is a good lowest-common-denominator option. However, it is quite slow compared with newer algorithms such as Zstandard.

LZ4

Focuses on compression/decompression speed.
LZ4 is a lossless data compression algorithm focused on compression and decompression speed. It belongs to the LZ77 family of compression algorithms, like Brotli, DEFLATE, and Zstandard. In microbenchmarks, LZ4 is the fastest option in that family for both compression and decompression and is generally recommended.

LZMA

LZMA is a dictionary compression method somewhat similar to LZ77. It achieves a very high compression ratio with a computationally expensive prediction model and variable-size compression dictionary, while maintaining decompression speed similar to other commonly used compression algorithms. LZMA beats all other options at typical compression ratios, but as a compressor it can be very slow, especially when configured to operate at high compression levels.

LZO

Focuses on compression/decompression speed and requires additional library installation.
LZO is another LZ-variant data compression algorithm implemented with a focus on decompression speed. It is not as fast as LZ4 but is close.

SNAPPY

Focuses on compression/decompression speed.
SNAPPY is based on LZ77 ideas but optimized for very fast compression speed, achieving only a reasonable level of compression in the tradeoff. It is as fast as LZ4, but its compression ratio is not very high. HBase provides a pure Java Snappy codec that can be used instead of GZ as a universally available option for all Java runtimes on all hardware architectures.

ZSTD

Zstandard combines a dictionary matching stage (LZ77) with a large search window and fast entropy coding stage using both finite-state entropy and Huffman coding. Compression speed can differ by more than 20x between the fastest and slowest levels, but decompression is uniformly fast, with less than a 20% difference between the fastest and slowest levels.
ZStandard is the most flexible available compression codec option. At level 1 it provides compression ratio similar to LZ4, with slightly lower performance. At mid levels it provides compression ratio similar to DEFLATE with better performance, and at high levels it provides dense compression similar to LZMA with LZMA-like compression speed, while providing universally fast decompression.

Data Block Encoding Types

HBase provides five data block encoding types:

NONE
Prefix
Diff
Fast Diff
Prefix Tree

None

None means no encoding is applied and is the default.

ColumnFamily without encoding

The following table shows a hypothetical ColumnFamily without data block encoding.

Key Length	Value Length	Key	Value
24	…	RowKey:Family:Qualifier0	…
24	…	RowKey:Family:Qualifier1	…
25	…	RowKey:Family:QualifierN	…
25	…	RowKey2:Family:Qualifier1	…
25	…	RowKey2:Family:Qualifier2	…
…	…	…	…

Prefix

Usually keys often have the same prefix and only the last part differs.

For example, the first key may be RowKey:Family:Qualifier0, and the next key may be RowKey:Family:Qualifier1. Prefix encoding adds an extended column that stores the length of the prefix shared between the current key and the previous key. In this example, if the first key is completely different from the previous key, the prefix length is 0. Since the first 23 characters of the second key are the same, the prefix length becomes 23.

Of course, if two keys have nothing in common, Prefix provides no benefit.

ColumnFamily with Prefix encoding
The following is the same data with Prefix data encoding applied.

Key Length	Value Length	Prefix Length	Key	Value
24	…	0	RowKey:Family:Qualifier0	…
1	…	23	1	…
1	…	23	N	…
19	…	6	2:Family:Qualifier1	…
1	…	24	…	…
…	…	…	…	…

Diff

Diff encoding extends Prefix encoding. Rather than treating the Key as one sequence of bytes, it splits each Key Field so each part of the Key can be compressed more efficiently.

Two new fields, timestamp and type, are added.

If the ColumnFamily is the same as the previous row, it is omitted from the current row. If key length, value length, and type are the same as the previous row, the fields are omitted.

Also, to improve compression, timestamp is not stored in full but as a difference from the previous row’s timestamp. In the prefix example, if there are two row keys and timestamp matches exactly and type is the same, the second row does not need to store value length or type, and the timestamp value in the second row becomes 0 instead of a full timestamp.

Because write and scan speed become slower, Diff encoding is disabled by default even though more data can be cached.

ColumnFamily with Diff encoding
This table shows the same ColumnFamily with Diff encoding applied.

Flags	Key Length	Value Length	Prefix Length	Key	Timestamp	Type Value
0	24	512	0	RowKey:Family:Qualifier0	1340466835163	4	…
5	-	320	23	1	0	-	…
3	-	-	23	N	120	8	…
0	25	576	6	2:Family:Qualifier1	25	4	…
5	-	3384	24	2	1124	-	…
…	…	…	…	…	…	…	…

Fast Diff

Fast Diff works similarly to Diff but is implemented faster. It also adds one more field that stores a single bit to track whether the data itself is the same as the previous row. If it is the same, the data is not stored again.

Use the Fast Diff codec when keys are long or there are many columns.

The data format is almost the same as Diff encoding, so there is no separate image to explain.

Prefix Tree

Prefix Tree encoding was introduced as an experimental feature in HBase 0.96. Prefix Tree encoding provides memory savings similar to Prefix, Diff, and Fast Diff encoders, but provides faster random access at the cost of slower encoding speed. It was a good idea, but it was rarely used and was removed in HBase 2.0.0.

References

Appendix D: Compression and Data Block Encoding In HBase

HBase Data Versioning

kc@example.com (kc kim) — Thu, 22 Jun 2023 19:13:23 +0900

Data Versioning

As one of HBase’s special features, multiple versions can be stored for a specific column value in each cell.

This is implemented using a timestamp for each version and is sorted in descending order. It uses Unix time converted to milliseconds as a long integer type.

HBase manages data versions by default. When data is stored with the same Row key, the most recent data is versioned by timestamp and stored in descending order, so the most recent value is found first when reading from storage files.

Timestamp can also be entered explicitly. By default, HBase keeps up to three changes for each cell, and because scan queries in descending order, the latest data is queried.

You can also query all stored versions of data as follows.

get 't1', 'rowkey1', {COLUMN => 'cf1', TIMESTAMP => ts1}

It is also possible to set a timestamp range and view the value at that point.

get 't1', 'rowkey1', {TIMERANGE => [start_timestamp, end_timestamp]}

Versioning is usually assigned automatically, but it can also be specified manually.

Automatic versioning
- Version differences can occur if server times in the cluster are not the same.
- Timestamp can be set when executing the Put method, but generally automatic versioning within the server is recommended.
- The most recent three versions are managed by default, but because Major Compaction has a long cycle, old versions that have not yet been deleted may still exist.
Manual versioning
- Can be implemented by overriding the timestamp.

Does deleting reveal an older version of a column even when HBase VERSIONS=1?

Earlier, it was mentioned that the most recent three versions are managed by default, but because Major Compaction has a long cycle, old versions that have not yet been deleted may still exist.

This section introduces a related issue that can occur.

Applying VERSIONS

Apply VERSIONS to the ColumnFamily as follows.

create 't1', {NAME => 'f1', VERSIONS => 2}

hbase(main):001:0> create 't1', {NAME => 'f1', VERSIONS => 2}
Created table t1
Took 5.9091 seconds
=> Hbase::Table - t1

Next, register data in order.

put 't1', '101', 'f1:name1', 'test1'
put 't1', '101', 'f1:name1', 'test2'
put 't1', '101', 'f1:name1', 'test2'
put 't1', '101', 'f1:name1', 'test4'

hbase(main):002:0> put 't1', '101', 'f1:name1', 'test1'
Took 0.7153 seconds
hbase(main):003:0> put 't1', '101', 'f1:name1', 'test2'
Took 0.0318 seconds
hbase(main):004:0> put 't1', '101', 'f1:name1', 'test3'
Took 0.0418 seconds
hbase(main):005:0> put 't1', '101', 'f1:name1', 'test4'
Took 0.0255 seconds

Then delete one by one with the delete command and check the result.

delete 't1', '101', 'f1:name1'

hbase(main):013:0> scan 't1'
ROW                                                                  COLUMN+CELL
 101                                                                 column=f1:name1, timestamp=1687426826665, value=test4
1 row(s)
Took 0.3049 seconds
hbase(main):014:0> delete 't1', '101', 'f1:name1'
Took 0.0160 seconds
hbase(main):015:0> scan 't1'
ROW                                                                  COLUMN+CELL
 101                                                                 column=f1:name1, timestamp=1687426822568, value=test3
1 row(s)
Took 0.1760 seconds
hbase(main):016:0> delete 't1', '101', 'f1:name1'
Took 0.0605 seconds
hbase(main):017:0> scan 't1'
ROW                                                                  COLUMN+CELL
 101                                                                 column=f1:name1, timestamp=1687426817569, value=test2
1 row(s)
Took 0.0710 seconds
hbase(main):018:0> delete 't1', '101', 'f1:name1'
Took 0.0337 seconds
hbase(main):019:0> scan 't1'
ROW                                                                  COLUMN+CELL
 101                                                                 column=f1:name1, timestamp=1687426813634, value=test1
1 row(s)
Took 0.0265 seconds
hbase(main):020:0>

You can see that the previously registered data is queried in order.

Question about VERSIONS

The question here is that VERSIONS was said to store only one version, but even when VERSIONS => 2 is specified, more versions of the column appear to be maintained. At first glance, this looks suspiciously like a bug.

Searching online, a person with the same question was found.
Delete reveals older version of a column even when VERSIONS=1

The answer to that question says the following.

Older versions do not actually disappear until compaction occurs. Compaction should be performed once a day unless the major compaction setting is changed, or whenever a Region is split.

This issue occurs because Major Compaction has not yet been performed.

How to check VERSIONS application

Then how can VERSIONS be checked? Use the get command and query details with VERSIONS => 4 as follows.

get 't1', '101', {COLUMN=>'f1:name1',VERSIONS => 4 }

hbase(main):005:0> get 't1', '101', {COLUMN=>'f1:name1', VERSIONS => 4 }
COLUMN                                                               CELL
 f1:name1                                                            timestamp=1687427184339, value=test4
 f1:name1                                                            timestamp=1687427181122, value=test2
1 row(s)
Took 0.5028 seconds
hbase(main):006:0>

As shown above, you can see two records according to VERSIONS => 2 specified when creating the table.

HBase BloomFilter

kc@example.com (kc kim) — Fri, 23 Jun 2023 19:01:58 +0900

What is BloomFilter?

BloomFilter is a fast search algorithm for multi-hash-function mapping proposed by Bloom in 1970. This algorithm is generally used in situations where it is necessary to quickly determine whether an element belongs to a set, starting from the premise that strict 100% accuracy is not required.

For detailed theory, refer to the Bloom Filter data structure explained simply page.

HBase BloomFilter

HBase BloomFilter data is stored in StoreFile metadata, and once written, a StoreFile is immutable, so it cannot be updated.

BloomFilter is a configuration property at the Column family level. If BloomFilter is configured on a table, HBase includes a data fragment of the BloomFilter structure when creating a StoreFile called a MetaBlock. MetaBlock and DataBlock, which contain actual KeyValue data, are used together in LRU BlockCacheMaintenance.

Therefore, configuring BloomFilter introduces specific storage and memory cache overhead.

How to configure HBase BloomFilter

When searching for whether a specific Row key is included, it reduces unnecessary block loading compared to the existing block index, improving overall cluster throughput.

Bloom filters should be selected according to cell size, number of cells, data storage method, read method, and similar factors.

According to the HBase documentation, BloomFilter is recommended in most cases.

BloomFilter has three parameters: NONE (default), ROW, and ROWCOL.

ROW: Indicates a row-level Bloom filter that filters StoreFiles based on the KeyValue row.
ROWCOL: Indicates a column-level Bloom filter that filters StoreFiles based on the KeyValue row plus column.

Therefore, the space overhead of ROWCOL is higher than that of ROW.

The more StoreFiles there are in the Region, the better the BloomFilter effect. The fewer StoreFiles there are in the Region, the better HBase read performance becomes.

Configure the Column family in HBase to enable the BloomFilter command as follows.

create 't1',{name => 'c1', BLOOMFILTER => '<ROW or ROWCOL>'}

Cassandra

kc@example.com (kc kim) — Fri, 28 Oct 2022 01:02:00 +0900

Cassandra overview

Cassandra is an open source distributed database management system with high scalability and availability.
Cassandra was originally developed at Facebook to store large amounts of data, and its source code was released in 2008. After Facebook adopted HBase, a separate product modeled on Google’s Bigtable, the number of companies adopting Cassandra was limited for a while. Recently, however, companies such as Apple and Netflix have adopted Cassandra for their large-scale systems from the perspective of availability, so it is attracting attention again.

Because Cassandra is an open source project, the source code can be obtained and used in production for free. There is also a commercial product edition from DataStax that provides tested support.

The major release 4.0.0 was released in July 2021.

This major release advanced features as follows.

Added experimental support for Java 11 and transient replication
Added Virtual Tables, which publish metrics and YAML configuration information
Added audit logging useful for compliance and debugging
Added Full Query Logging (FQL) for live traffic capture and replay
Optimized the internode messaging protocol
Improved streaming used to exchange data between cluster nodes

Main features of Cassandra

The main features of Cassandra are as follows.

Masterless architecture

Most database cluster architectures use a master-slave method, but Cassandra uses a masterless method.
In this method, each node is equivalent, and there is no master that supervises the nodes. Therefore, there is no single point of failure. Also, depending on replication settings, data registered on each node is automatically propagated to other nodes, enabling high availability.

Schema definitions exist

Unlike document DBs such as MongoDB and key-value stores such as Redis, Cassandra has schema definitions. This makes it easier for developers and operators to understand data contents during application development and operation. Also, unlike RDBs, Cassandra does not completely depend on schemas and can have somewhat flexible table structures, such as inserting multiple values into one column.

Performance scales linearly

Cassandra can scale processing by adding nodes.
Cassandra officially claims that performance scales linearly, and a benchmark conducted by Netflix supports this.
This scaling performance is thought to be one reason Cassandra is adopted in large-scale systems such as Apple’s iCloud.

SQL-like queries available (CQL)

Cassandra can be operated with SQL-like queries called CQL (Cassandra Query Language). Basic queries such as SELECT, UPDATE, and DELETE can be executed almost exactly like SQL. GROUP BY has been implemented and supported since version 3.4.3. However, aggregations such as JOIN are not implemented.
There are also restrictions on functions and sorting such as ORDER BY, so care is required.

Cassandra license

Cassandra is released under the Apache License version 2, and anyone can freely use, modify, and redistribute it for free, regardless of commercial or non-commercial use.

Cassandra operating environment

Cassandra is a Java application.

The latest version of Oracle Java Standard Edition 8 or OpenJDK 8 is required. Java 11 is still experimental support and is not recommended for production use.
To use cqlsh, the latest version of Python 3.6 or later is required.

Cassandra download

http://cassandra.apache.org/download/