devkuma – HBase

Apache HBase Overview

kc@example.com (kc kim) — Fri, 08 Jul 2022 09:37:00 +0900

What is HBase?

Apache HBase is an open source distributed database management system developed and published by the Apache Software Foundation. It is one of the NoSQL systems, with a structure different from mainstream relational databases.

It models Google’s BigTable, which was developed and used internally at Google, and reimplements a similar database management system as open source software.

HBase builds a database on HDFS (Hadoop Distributed File System), a distributed file system also developed and published by the ASF. Like a relational database, it structures data in tables, but unlike a typical RDBMS, it stores the values of each column together in storage. This approach is called column-oriented storage.

Each row in a table stores data as pairs of qualifiers, which correspond to column names, and cells, which correspond to field values. Multiple qualifiers are grouped into a column family. Cell data is automatically versioned, so data from an arbitrary point in the past can be retrieved.

HBase is designed for a distributed environment where multiple server computers manage one database. It automatically shards data across multiple nodes, so applications do not need to worry about the target location. It also provides strong data consistency, preventing stale data from being externally referenced during processing.

Database operations are performed not with SQL, as in RDBs, but by calling APIs such as the Java API or RESTful API from an application. HBase is well suited to large-scale distributed big data processing such as Apache Hadoop MapReduce.

HBase is not a replacement for SQL databases, but it can add an SQL layer by using software such as Apache Phoenix. With an SQL layer, JDBC drivers and similar tools can be used to handle various analytical workloads more easily and in a familiar way.

Apache HBase|GitHub

HBase History

Year	Event
November 2006	Google published the BigTable paper.
February 2007	The initial HBase prototype was created as a Hadoop contribution.
October 2007	The first usable HBase was released with Hadoop 0.15.0.
January 2008	HBase became a subproject of Hadoop.
October 2008	HBase 0.18.1 was released.
January 2009	HBase 0.19.0 was released.
September 2009	HBase 0.20.0 was released.
May 2010	HBase became an Apache top-level project.
June 2010	The first developer version, HBase 0.89.20100621, was released.
January 2011	HBase 0.90.0 was released.
Mid-2011	HBase 0.92.0 was released.

HBase Features

The following are some important characteristics of Apache HBase.

Distributed

Apache HBase can run in two modes: pseudo-distributed mode and fully distributed mode. Pseudo-distributed mode is used for testing and runs on a single node, while fully distributed mode is used in production and runs on a cluster of nodes.

Big Data Store

Apache HBase is designed to store very large data in tables with billions of rows and millions or billions of columns. Because it runs on Hadoop HDFS, it provides low-latency real-time reads and writes for data. It also provides better performance for read operations on massive data stored in tables.

Non-Relational

As already noted, Apache HBase is a non-relational database, so it does not follow the relational database model. In relational database management, data is stored in tables as rows and columns. SQL can be used to access that data, but Apache HBase uses storage and query mechanisms where data storage is not fixed to that format. In Apache HBase, schemas are flexible and can be extended according to requirements.

Flexible Data Model

Apache HBase provides a flexible data model for storing data in tables. A table has one or more column families. User data is stored in rows, which are collections of key/value pairs. In a table, each row is uniquely identified by a row key.

Scalable

Apache HBase regions are horizontally scalable, with rows distributed by region. A table can be stored across multiple regions, and when a region becomes very large, the data is split into two regions around the middle row key.

HBase Functions

Apache HBase provides the following functions.

Supports linear and modular scalability.
Provides strict consistency for reads and writes.
Provides automatic and configurable table sharding.
Supports automatic failover between RegionServers.
Provides convenient base classes for supporting Hadoop MapReduce jobs with Apache HBase tables.
Makes the Java API easy to use for client access.
Provides block cache and Bloom filters for real-time queries.
Provides query predicate pushdown through server-side filters.
Provides an extensible JRuby-based (JIRB) shell.

HBase Pros and Cons

HBase Advantages

Effective for reliably handling large volumes of data.
- A master that controls the overall distributed system manages overall data consistency and guarantees consistency among replicated data.
Suitable for supporting large-scale data analysis processing.
- Easy to use as MapReduce input.
- Optimized for use with HDFS, MapReduce, and related tools.
If performance issues occur, performance can be maintained by adding only Region servers.
Failover is easy, and management is convenient.

HBase Disadvantages

It is convenient as MapReduce input, but when used together with file input, CPU usage can rise and Region servers can go down easily.
There is material about conditions for appropriate HBase settings, but because cluster size and basic specifications differ, it is difficult to apply directly.
Regions for a specific table can easily become concentrated on a specific Region server, leading to performance degradation.
- Proper HBase design is required during configuration.

Differences Between RDBMS and HBase

The following table summarizes the major differences between relational databases and HBase.

RDBMS	HBase
Uses tables as the database.	Uses regions as the database.
Supported file systems are FAT, NTFS, and EXT.	The supported file system is HDFS.
Uses commit logs to store logs.	Uses WAL (Write-Ahead Logs) to store logs.
The reference system used is a coordinate system.	The reference system is the one used by ZooKeeper.
Uses a primary key.	Uses a row key.
Partitioning is supported.	Sharding is supported.
Uses Row, Column, and Cell.	Uses Row, Column family, Column, and Cell.

Differences Between HDFS and HBase

The following table summarizes the major differences between HDFS and HBase.

HDFS	HBase
HDFS provides a file system for distributed storage.	HBase provides table-form, column-based data storage.
HDFS provides storage optimized for large files.	HBase provides optimization for table-form data.
Uses flat files.	Uses key-value pair data.
The data model is not flexible.	Provides a flexible data model.
Uses a file system and processing framework.	Uses table-form storage with built-in Hadoop MapReduce support.
Mostly optimized for write-once-read-many workloads.	Optimized for many reads and writes.

Differences Between Row-Oriented and Column-Oriented Data Stores

The following table summarizes the major differences between row-oriented and column-oriented data stores.

Row-oriented data store	Column-oriented data store
Row-oriented data stores are efficient for adding and modifying records.	Column-oriented data stores are efficient for reading data.
They read pages containing entire rows.	They read only the required columns.
Best suited for OLTP systems.	Not yet suitable for OLTP systems.
Serializes the full value of a row.	Serializes the full value of a column.
Stores rows in contiguous pages of memory.	Stores columns from pages in memory.

What is OLTP (Online Transactional Processing)? It refers to online transaction processing and means batch transaction processing against databases by online users over a network.

Pros and Cons of Column-Oriented Databases

The following are the pros and cons of column-oriented databases.

Advantages

Built-in support for efficiency and data compression.
Supports fast data retrieval.
Management and configuration are simplified in column-oriented databases.
Suitable for high performance in aggregate queries, such as COUNT, SUM, AVG, MIN, and MAX.
Provides an automatic sharding mechanism that distributes larger regions into smaller regions, making it efficient for partitioning.

Disadvantages

JOIN queries and data across multiple tables are not optimized.
Storage efficiency decreases because partitions must be created for frequent deletes and updates.
Because of the non-relational nature, designing partitions and indexes is very difficult.

HBase Use Cases

Used to guarantee data consistency or for analysis.
- Example: Facebook, eBay, Adobe, LINE, etc.
Monitoring systems.
Tracking user actions.
Audit logging systems.
Real-time analytics.
- Cases where Hadoop must be used to analyze large amounts of data.
Used as the primary storage for large-scale social networking services.
- Message-oriented systems, such as Twitter-like messages and statuses.
Content management systems that serve content from HBase.
Standard use cases such as storing web pages during web crawling.

References

Apache HBase Architecture

kc@example.com (kc kim) — Fri, 08 Jul 2022 09:40:00 +0900

Apache HBase Architecture

Apache HBase is a NoSQL database. NoSQL is a general expression indicating that a database is not an RDBMS whose primary access language is SQL. Apache HBase is a data store rather than a traditional database. HBase can scale linearly and modularly by adding commodity nodes to the cluster. If the number of nodes increases from 20 to 40, storage and capacity also increase at the same time in the HBase cluster.

HBase Cluster Architecture

Apache HBase Components

HBase consists of two types of servers: Master servers and Region servers.
Its structure allows simple scale-out by adding Region servers. HBase data is divided into units called Regions, and server types are separated according to how Regions are handled.

Master Server - HBase HMaster

Apache HBase HMaster is an important component of an HBase cluster that is responsible for monitoring RegionServers, handling failover, and managing region splits.

Data for each table is managed by RegionServers, and the entire cluster is managed by HMaster.
HMaster manages metadata that records which Region server stores each piece of HBase data (Region).
It assigns Regions to Region servers and detects failures in Region servers.

HMaster functions include the following.

Monitors RegionServers.
- Coordinates Region servers.
- Manages Region startup.
Handles RegionServer failover.
Assigns or unassigns Regions.
Monitors all Region servers in the cluster.
- Management functions.
- Table creation, deletion, and update.
Handles metadata changes.
Provides an interface for all metadata changes.
Performs load balancing during idle time.
HMaster provides a web user interface that shows information about the HBase cluster.

These tasks are handled by ZooKeeper, software that runs inside the Master server.
ZooKeeper is software used in many distributed applications besides HBase.

Region Server - HBase Region Server

RegionServer is responsible for storing the actual data assigned by the Master server. As in a Hadoop cluster, NameNode stores metadata and DataNode stores actual data. HBase is similar: the Master holds metadata, and RegionServers store actual data. RegionServer runs on DataNodes in a distributed cluster environment.

RegionServer performs the following tasks.

Handles assigned regions (tables).
Handles read and write requests made by clients.
Flushes cache to HDFS.
Is responsible for region split processing.
Maintains HLog.

The client first queries the Master server, or more precisely ZooKeeper, to learn the location of the Region server that holds the target data (Region).

Then that Region server handles the client request.
A Region is always managed by a single Region server and maintains data consistency for reads and writes.

If a Region server goes down, the assigned Regions fail over to another Region server. This work is performed by the Master server.
However, until failover completes, clients cannot read from or write to the target Region.

RegionServer Components

The following are the components of a RegionServer.

WAL (Write-Ahead Logging)

What is WAL? It is a standard method used by databases to guarantee data integrity. In systems that use WAL, changes are first recorded in the WAL before data is modified. These records make it possible to determine when a failure occurred, how far processing succeeded, and where subsequent work should resume.

Various database systems, including PostgreSQL, HBase, and MongoDB, use the WAL method.

Apache HBase WAL is an intermediate file also called an edit log file. When reading or modifying data in HBase, data is not written directly to disk but kept in memory for a period of time. If the system goes down, all data in memory may be lost. To overcome this problem, Apache HBase first records data in a Write-Ahead Logging file before recording it in memory.

HFile

HFile is the format for storing data in HBase. It belongs to a Column Family, and a Column Family can have multiple HFiles. It is the actual file where row data is physically stored, and data is stored in key/value format. When MemStore fills sufficiently, it creates a new HFile in HDFS and stores the data there.
This process is very fast because the structure minimizes random access.

Store

Store corresponds to a Column family for an HBase table. HFiles are stored here.

MemStore

MemStore resides in main memory and records current data operations. After data is stored in the WAL, key/value data is sorted and stored in the RegionServer memory store. This data is then stored in HFile as-is. There is one MemStore per Column Family.

Region

A Region is a table split that is divided based on keys and hosted by RegionServers.

Client

Clients can be written in Java or other languages and can connect through external APIs to the RegionServer that manages the actual row data. A client queries catalog tables to find a Region. Once the Region is found, the client connects directly to RegionServers, performs data operations, and caches data for fast lookups.

Catalog Tables

Catalog tables are used to maintain metadata for all RegionServers and Regions.

HBase has two types of catalog tables.

-ROOT- This table contains information about the location of the META table.
.META This table contains information about all Regions and their locations.

ZooKeeper

Apache ZooKeeper acts like the coordinator for HBase. HBase uses ZooKeeper to manage the state of servers that make up the cluster. It provides services such as maintaining configuration information, naming, distributed synchronization, and server error notification. Clients communicate with region servers through ZooKeeper.

Apache ZooKeeper is a high-performance centralized coordination service system for distributed applications that provides distributed synchronization and group services to HBase. Despite cluster coordination, it lets users focus on application logic. It also provides APIs that allow users to interact with the Master server.

The Apache ZooKeeper API provides consistency, ordering, and durability, and also provides synchronization and concurrency for distributed cluster systems.

HBase Data Write/Read Process

This section explains how HBase reads and writes data.

HBase Data Writes

When storing data in HBase, the data is stored in two places: the WAL (Write Ahead Log) and the Region server’s MemStore.

Using these two locations, the write process is considered complete when changes have occurred in both the WAL and MemStore.

When a data load request arrives in HBase, the appropriate Region server is found first. After the Region server is found, a commit log is added and the data is added to the in-memory MemStore. Data stored in MemStore is flushed to disk as an HFile when it becomes full according to configured values. Memory is cleared, the server waits for the next request, and a corresponding record is also written to the WAL. MemStore flush occurs when the value exceeds hbase.hregion.memstore.flush.size; the unit is bytes and the setting is 134217728 (128 MB).

MemStore is a memory area of the Region server and acts like a cache. When a certain amount of data accumulates in MemStore, the data is exported to disk as an HFile and persisted.
However, data before it is exported to HFile is stored in server memory (MemStore), so it disappears if the server goes down. Therefore, HBase writes a log called HLog when writing data, and if the server goes down, it can recover data based on HLog.

HBase Data Reads

The process of reading data from HBase is similar to the write process described above.

When a request enters HBase, MemStore is checked first. If the desired data is found in MemStore, that data is returned. Otherwise, HBase searches from the most recently flushed file to older files until it finds suitable data that satisfies the query or there are no more flushed files.

References

HBase Introduction|joinc.co.kr

Apache HBase Regions

kc@example.com (kc kim) — Wed, 21 Jun 2023 11:05:33 +0900

Apache HBase stores large amounts of data in tables sorted by row key in lexicographic order. Tables are distributed across multiple Regions, and Regions are further distributed across multiple RegionServers. When you create a table in Apache HBase, a default Region is assigned.

Apache HBase Regions are horizontally scalable. They include a start key and an end key, and rows are sorted and stored in a continuous format based on those keys. Because HBase provides strong consistency, it does not store the same row key in multiple Regions. Regions distribute load across multiple RegionServers and also perform load balancing and failover according to requirements. As data grows, Regions are split manually or automatically.

Number of Regions

It is recommended to use a small number of medium to large Regions, 5 to 20 GB, per RegionServer, usually 20 to 200 Regions. The standard number of Regions is 100.

Some factors to consider for Regions are as follows.

The limiting factor when choosing the number of Regions is available heap space. MemStore requires almost 2 MB per column family per Region. If there are 100 Regions and 3 column families per Region, MemStore heap space requirements are 600 MB. Fewer Regions require less MemStore heap.
A large number of Regions creates many small flushes. If each flush creates a StoreFile, many StoreFiles are created, requiring more compaction. MemStore and StoreFile indexes also require more heap space.
A large number of Regions creates load on the master because the master must assign and reassign Regions to RegionServers. The master must also move Regions for load balancing.

Region Assignment

In Apache HBase, Regions are assigned by the master as follows.

The master starts the assignment manager.
The assignment manager checks existing assignments in hbase:meta metadata.
Assignments are stored when RegionServers are available.
If a RegionServer is not online, the load balancer is called to assign Regions to another RegionServer.
The hbase:meta metadata is updated with the new assignment.

Region failover

Because a RegionServer can fail and multiple RegionServers provide data, a RegionServer’s Regions can become unavailable. ZooKeeper detects RegionServer failures, and the master starts failover on another RegionServer with a similar Row Key for the Region.

Region Locality

Region Locality indicates the proximity of a Region to a RegionServer. It is achieved through HDFS block replication across the cluster.

The replica placement policy is the HDFS replica placement policy, as follows.

The first replica is placed on the local node.
The second replica is placed on a random node in a different rack.
The third replica is placed in the same rack as the second replica, but on a different node.

Benefits of Regions

The benefits of Regions include distributed data storage, partitioning, automatic sharding and scalability, and Region splitting.

Let’s look at each in detail.

Distributed data store

The design of a distributed data store matches Apache HBase’s design of using multiple Regions for a table. Distributing Regions of larger tables across a cluster of nodes provides high availability.

Partitioning

Data tables are stored in multiple Regions, and Regions partition the data. To access data in that table, access must now happen from different Regions. Having multiple Regions provides the benefit of delivering data quickly.

Automatic sharding and scalability

The automatic sharding process is used to split a Region into roughly two halves when the number of row keys in the Region becomes too large. In HBase, the basic unit of horizontal scalability is the Region, and rows are shared by Region.

Region splitting

When a threshold is exceeded, a Region is split. This is handled by the RegionServer, which splits the Region and takes the split Region offline. After that, two split Regions are added to hbase:meta, opened by the RegionServer, and reported to the master. Region splitting is automatic by default, but it can also be run manually. The HBase Region split policy is configured with hbase.regionserver.region.split.policy.

HBase Data Model

kc@example.com (kc kim) — Tue, 20 Jun 2023 19:07:12 +0900

Data structure

HBase tables have no fixed type and are stored as byte arrays (byte[]).
HBase rows are sorted in ascending order by a unique row key, and this row key is used when reading and writing table values.
As mentioned earlier, data managed by HBase is divided into Regions by fixed ranges, and a table has multiple Regions. HBase columns are also grouped in units called column families.
Table data is divided by Region and by column family, then exported as files. Files are separate, but if they belong to the same Region, they are stored on the same Region server.

Each HBase data cell value has a timestamp, so versions are managed. Files are stored in the following format.

Row(Row Key): Column family: Column: timestamp: value

Data model

The Apache HBase data model is a distributed, multidimensional, persistent, sorted map indexed by column key, row key, and timestamp. This is why Apache HBase is also called a key-value storage system.

The basic unit of HBase is a column. Columns form a Column Family, and Column Families form a table. Each Row in a table has a Row Key and can be identified by it. See the figure below.

This table has two column families: Customer and Sales.
The Customer column family has two columns: Name and City.
The Sales column family has two columns: Product and Amount.
A Row consists of Row Key, Customer CF, and Sales CF.

The following are data model terms used in Apache HBase.

Table

An Apache HBase table consists of multiple Rows. It organizes data into tables made of characters that are easy to use with the file system.

A set of Rows, with a Row Key and multiple column families.
In schema definition, only Column Families are defined.

Row

Apache HBase stores data based on Rows, and each Row has a unique Row Key. The Row Key is represented as a byte array.

Row Key

Because data is collected based on the Row Key, the Row Key must be designed so that data can be distributed appropriately. The goal of Row Key design is to store data so similar Rows are near each other.

Sorted lexicographically as arbitrary bytes.
An empty byte string means the start and end of the table.
Anything can be a row key, including strings, integers, binary data, and serialized data structures.

In general, a Row Key pattern can also be organized in website domain format. If domains are stored in reverse order, such as org.apache.mair and org.apache.jira, Apache domain data can be stored near each other.

Column

A Column consists of a Column Family and a Column Qualifier.

Column Family

A Column Family is used to store Rows and also provides the structure for storing data in Apache HBase.

It consists of characters and strings and can be used with file system paths. Each row in a table has the same column families, but a row does not need to store data in every column family.

Column Qualifier

A Column Qualifier provides an index for data stored in a Column Family. The Column qualifier is not a fixed value, so various data can be entered. It can be thought of as a map object.

A group of Columns where all ColumnFamily members use the same prefix.
NOSQL:Cassandra and NOSQL:HBASE are member columns of the NOSQL column family.
The column family prefix must consist of displayable characters.
Part of the table schema definition must be specified first.
All column family members are physically stored together in the file system.
New column family members can be added dynamically.

Cell

A Cell is a data unit consisting of Column family, Row key, and Column qualifier. The value of each column is called a cell.

A data cell includes a value and a timestamp, which is the version of the value. Because there is a timestamp, previous values are stored together and maintained for a certain period.

A tuple with ROW KEY, Column, and Version specified.
The value is an arbitrary byte array and Timestamp.
Table cells are versioned, only cells.

Timestamp

Versions are assigned to values where the same data is stored in a Cell. Each version is identified by a version number assigned during creation time. If Timestamp is not mentioned while writing data, the current time is assigned.

HBase data types

Apache HBase has no concept of data types. Everything is a byte array. When values are inserted, it is a kind of byte-in and byte-out database converted to byte arrays using the Put and Result interfaces. Apache HBase uses a serialization framework to convert user data into byte arrays.

Apache HBase cells can store values up to 10 to 15 MB. If a value is larger, it can be stored in Hadoop HDFS and file path metadata information can be stored in Apache HBase.

HBase data storage

The following introduces how Apache HBase is physically stored.

Conceptual view

You can see that a table is represented as a series of Rows at the conceptual level.

The following is a conceptual diagram of how data is stored in HBase.

Physical view

In the physical view, a table is physically stored by column family.

The following example shows a table that will be stored as a column-family-based table.

Namespace

A namespace is a logical group of tables. It is similar to grouping related tables in a relational database.

The representation of a namespace is as follows.

HBase Namespaces
- Table
- Region Server Group
- Permission
- Quota

Let’s look at each component of namespace space.

Table

All tables are part of a namespace. If no namespace is defined, the table is assigned to the default namespace.

RegionServer group

A default RegionServer group can be assigned to a namespace. In this case, created tables become members of the RegionServer group.

Permission

Using namespaces, users can define access control lists such as read, delete, and update permissions, and with write permission, users can create tables.

Quota

This component is used to define the quota that a namespace can include for tables and Regions.

Predefined namespaces

There are two predefined special namespaces.

hbase: A system namespace used to include HBase internal tables.
default: This namespace is for all tables where no namespace is defined.

Data model operations

The main data model operations are Get, Put, Scan, and Delete. These operations can be used to read, write, and delete records from tables.

Let’s look at each operation in detail.

Get

The Get operation is similar to the Select statement in a relational database. It is used to fetch the contents of an HBase table.

You can run the Get command in the HBase shell as follows.

hbase(main) :001:0> get 'table name', 'row key' <filters>

Put

The Put operation is used to write data to a table.

Scan

The Scan operation is used to read multiple rows from a table. It differs from Get, where the set of rows to read must be specified. Scan can iterate over a row range or all rows in a table.

Delete

The Delete operation is used to delete a row or set of rows from an HBase table. It can be executed through HTable.delete().

When a delete command is executed, it is marked as deleted, and when compaction occurs, the row is finally deleted from the table.

Internal delete types are as follows.

Delete: Used for a specific version of a column.
Delete column: Can be used for all column versions.
Delete family: Used for all columns in a specific ColumnFamily.

Installing HBase with Docker for Local Practice

kc@example.com (kc kim) — Tue, 13 Jun 2023 18:40:33 +0900

Installing HBase with Docker

Let’s install HBase in a Docker environment using the Docker image below.

HBase Docker image to install
- https://hub.docker.com/r/dajobe/hbase

Downloading HBase Docker run commands

Download, or clone, the HBase Docker run commands from the GitHub repository below.

GitHub
- https://github.com/dajobe/hbase-docker

git clone git@github.com:dajobe/hbase-docker.git

Running HBase Docker

With Docker running, execute the following command as-is.

./start-hbase.sh

It is recommended to use the start-hbase.sh script, which starts the container, inspects it to determine all local API ports and web UI ports, and suggests editing /etc/hosts to add an alias for the container IP if it does not already exist.

It is recommended to use the start-hbase.sh script, which starts the container, determines all local API ports and web UI ports, and suggests editing /etc/hosts to add an alias for the container IP if it does not already exist.

% ./start-hbase.sh
start-hbase.sh: Starting HBase container
Error: No such container: hbase-docker
Unable to find image 'dajobe/hbase:latest' locally
latest: Pulling from dajobe/hbase
6cf436f81810: Pulling fs layer

... omitted ...

159beed970fa: Pull complete
Digest: sha256:daa36a6d90b118ced866b6c76fcd918e7da73302b0e4971f506f0f61f645a9fe
Status: Downloaded newer image for dajobe/hbase:latest
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
start-hbase.sh: Container has ID 3804d552250334cc850189e076b77cb0d3066850cef332bec7a125dc29228361
./start-hbase.sh: line 32: python: command not found

Here, the container ID is 3804d552250334cc850189e076b77cb0d3066850cef332bec7a125dc29228361. Record it because it is needed when running the shell.

When you run the command above, data is created in the current path where it was run, ports are opened, and HBase should run.

Manual run method

docker run --name=hbase-docker \
  -h hbase-docker \
  -p 9095:9095 \
  -p 8085:8085 \
  -p 16010:16010  \
  -v $PWD/data:/data \
  -d dajobe/hbase

Running the HBase Shell

Next, let’s run the shell command that can connect to HBase using the Docker image.

Put the container ID recorded above into $id and run it.

docker run --rm -it --link $id:hbase-docker dajobe/hbase hbase shell

On macOS Apple Silicon, such as M1 or M2, the following error may occur.

% docker run --rm -it --link 3804d552250334cc850189e076b77cb0d3066850cef332bec7a125dc29228361:hbase-docker dajobe/hbase hbase shell
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested

In that case, delete the Docker container that is already running, add the --platform linux/amd64 option, and run it again.

% docker run --rm -it --platform linux/amd64 --link 3804d552250334cc850189e076b77cb0d3066850cef332bec7a125dc29228361:hbase-docker dajobe/hbase hbase shell

2023-06-16 09:38:17,680 WARN  [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
HBase Shell
Use "help" to get list of supported commands.
Use "exit" to quit this interactive shell.
For Reference, please visit: http://hbase.apache.org/2.0/book.html#shell
Version 2.1.2, r1dfc418f77801fbfb59a125756891b9100c1fc6d, Sun Dec 30 21:45:09 PST 2018
Took 0.0480 seconds
hbase(main):001:0>

The command to exit the shell is quit.

hbase(main):005:0> quit

How to use commands will be explained on the next page.

HBase General Shell Commands - status, version, whoami

kc@example.com (kc kim) — Fri, 16 Jun 2023 14:55:22 +0900

General HBase shell commands

This section introduces general HBase commands.

`status`

Displays cluster status. Additional options, "summary", "simple", and "detailed", can be used to check detailed information. The default is "summary".

You can check cluster status with the status command.

Option	Description
simple	Server information
summary	Number of servers, load
detailed	Server information, meta information
replication	Source and sink information

status

hbase(main):001:0> status
1 servers, 0 dead, 2.0000 average load

status 'simple'

hbase(main):002:0> status 'simple'
1 live servers
    532f4d895a98:40371 1686636094968
        requestsPerSecond=0.0, numberOfOnlineRegions=2, usedHeapMB=19, maxHeapMB=966, numberOfStores=2, numberOfStorefiles=2, storefileUncompressedSizeMB=0, storefileSizeMB=0, memstoreSizeMB=0, storefileIndexSizeMB=0, readRequestsCount=42, writeRequestsCount=5, rootIndexSizeKB=0, totalStaticIndexSizeKB=0, totalStaticBloomSizeKB=0, totalCompactingKVs=0, currentCompactedKVs=0, compactionProgressPct=NaN, coprocessors=[]
0 dead servers
Aggregate load: 0, regions: 2

status 'summary'

hbase(main):003:0> status 'summary'
1 servers, 0 dead, 2.0000 average load

status 'detailed'

hbase(main):004:0> status 'detailed'
version 2.0.0-SNAPSHOT
0 regionsInTransition
master coprocessors: []
1 live servers
    532f4d895a98:40371 1686636094968
        requestsPerSecond=0.0, numberOfOnlineRegions=2, usedHeapMB=21, maxHeapMB=966, numberOfStores=2, numberOfStorefiles=2, storefileUncompressedSizeMB=0, storefileSizeMB=0, memstoreSizeMB=0, storefileIndexSizeMB=0, readRequestsCount=42, writeRequestsCount=5, rootIndexSizeKB=0, totalStaticIndexSizeKB=0, totalStaticBloomSizeKB=0, totalCompactingKVs=0, currentCompactedKVs=0, compactionProgressPct=NaN, coprocessors=[]
        "hbase:meta,,1"
            numberOfStores=1, numberOfStorefiles=1, storefileUncompressedSizeMB=0, storefileSizeMB=0, memstoreSizeMB=0, storefileIndexSizeMB=0, readRequestsCount=36, writeRequestsCount=3, rootIndexSizeKB=0, totalStaticIndexSizeKB=0, totalStaticBloomSizeKB=0, totalCompactingKVs=0, currentCompactedKVs=0, compactionProgressPct=NaN, completeSequenceId=10
        "hbase:namespace,,1686636105191.93358ca1f9bf1816320d88c48e39a3b4."
            numberOfStores=1, numberOfStorefiles=1, storefileUncompressedSizeMB=0, storefileSizeMB=0, memstoreSizeMB=0, storefileIndexSizeMB=0, readRequestsCount=6, writeRequestsCount=2, rootIndexSizeKB=0, totalStaticIndexSizeKB=0, totalStaticBloomSizeKB=0, totalCompactingKVs=0, currentCompactedKVs=0, compactionProgressPct=NaN, completeSequenceId=8
0 dead servers

version

Checks the installed HBase version information.

version

hbase(main):001:0> version
2.0.0-SNAPSHOT, r31ed817447b4c3edef2019d1580aa3ede83a82da, Fri Sep 26 02:07:26 UTC 2014

whoami

Checks the current HBase user.

whoami

hbase(main):008:0> whoami
root (auth:SIMPLE)
    groups: root

HBase Table Management Commands - create, list, exists, disable/enable, describe, alter, drop

kc@example.com (kc kim) — Fri, 16 Jun 2023 14:55:22 +0900

Table Management commands

This section introduces HBase Shell commands for table management.

create - Create a Table

You can create a table by using the create command. Here, you must specify the table name and Column Family name. The syntax for creating a table in the HBase shell is as follows.

create '<table name>', '<column family>'

The following is a sample schema for a table named order. It has two column families: customer and sales.

You can create this table in the HBase shell as follows.

create 'order', 'customer', 'sales'

When executed, it runs as follows.

hbase(main):002:0> create 'order', 'customer', 'sales'
Created table order
Took 2.4344 seconds
=> Hbase::Table - order

create Options

You can also add options when creating a table.

Create t1 and create column family cf1, storing up to 5 latest version values.

create 't1', {NAME => 'f1', VERSIONS => 5}

Create t1 and create column families cf1, cf2, and cf3.

create 't1', {NAME => 'f1', NAME => 'f2', NAME => 'f3'}

The following commands specify other options.

create 't1', {NAME => 'cf2', VERSION => 1, TTL => 2592000, BLOCKCACHE =>true}

create 't1', {NAME => 'cf2', CONFIGURATION => {'hbase.hstore.blockingStoreFiles' ='10'}}

list - View Table List

list is a command used to list all tables in HBase. The syntax of the list command is as follows.

list

If you enter this command and run it at the HBase prompt, all HBase tables are displayed as follows.

hbase(main):004:0> list
TABLE
order
1 row(s) in 0.1070 seconds
=> ["order"]

You can also add options as follows.

hbase(main):005:0> list 'customer.*'
TABLE
order
1 row(s) in 0.0140 seconds
=> ["order"]

exists - Check Table Existence

You can check whether a table exists by using the exists command. The following is the syntax for checking whether a table exists.

exists '<table name>'

The following example checks whether the order table exists.

hbase(main):022:0> exists 'order'
Table order does exist
0 row(s) in 0.0680 seconds

The following example checks whether the user table exists, where it does not actually exist.

hbase(main):023:0> exists 'user'
Table user does not exist
0 row(s) in 0.0110 seconds

disable - Disable a Table

To delete a table or change its settings, you must first disable the table by using the disable command. You can enable it again by using the enable command.

The syntax for disabling a table is as follows.

disable '<table name>'

The following example shows how to disable the order table.

hbase(main):004:0> disable 'order'
0 row(s) in 1.4570 seconds

Even after disabling a table, you can check that the table exists with the list and exists commands.

hbase(main):009:0> list
TABLE
order
1 row(s) in 0.0140 seconds
=> ["order"]

A disabled table cannot be scanned. The following error occurs.

hbase(main):010:0> scan 'order'
ROW                                         COLUMN+CELL

ERROR: order is disabled.

// omitted

is_disabled

This command is used to check whether a table is disabled. The syntax is as follows.

is_disabled '<table name>'

The following example checks whether a table named order is disabled. It returns true if disabled and false otherwise.

hbase(main):011:0> is_disabled 'order'
true
0 row(s) in 0.0360 seconds

disable_all

The disable_all command is used to disable all tables matching a given regular expression. The syntax of the disable_all command is as follows.

disable_all ‘<regex table name>’

A simple example is shown below. The following command disables all tables that start with r.

disable_all 'r.*'

Assume HBase has four tables that start with dev: dev_follow, dev_friend, dev_test, and dev_user. The following code disables all tables that start with dev.

hbase(main):017:0> disable_all 'dev.*'
dev_follow
dev_friend
dev_test
dev_user

Disable the above 4 tables (y/n)?
y
4 tables successfully disabled

enable - Enable a Table

To enable a disabled table, use the enable command. The syntax for enabling a table is as follows.

enable '<table name>'

The following is an example of enabling a table.

hbase(main):018:0> enable 'order'
0 row(s) in 0.6580 seconds

After enabling the table, scan it. If no error occurs, the table has been enabled normally.

hbase(main):020:0> scan 'order'
ROW                                         COLUMN+CELL
0 row(s) in 0.0730 seconds

is_enabled

The is_enabled command is used to check whether a table is enabled. The syntax is as follows.

is_enabled '<table name>'

The following command checks whether a table named order is enabled. It returns true if enabled and false otherwise.

hbase(main):021:0> is_enabled 'order'
true
0 row(s) in 0.0750 seconds

describe - Describe a Table

The describe command returns a description of a table. The syntax is as follows.

describe '<table name>'

The following is the output of the describe command for the order table.

hbase(main):006:0> describe 'order'
DESCRIPTION                                                                                                  ENABLED
 'order', {NAME => 'customer', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCO true
 PE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELL
 S => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'sales', DATA_BLO
 CK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NON
 E', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY =
 > 'false', BLOCKCACHE => 'true'}
1 row(s) in 0.1450 seconds

alter - Change a Table

alter is a command used to change an existing table. You can use this command to change the maximum number of cells in a column family, set and remove table-scope operators, and delete column families from a table.

Change the Maximum Number of Cells in a Column Family

The following is the syntax for changing the maximum number of cells in a column family.

alter '<table name>', NAME => '<column family>', VERSIONS => 5

The following example sets the maximum number of cells to 5.

hbase(main):007:0>alter 'order', NAME => 'customer', VERSIONS => 5
Updating all regions with the new schema...
0/1 regions updated.
1/1 regions updated.
Done.
0 row(s) in 2.4240 seconds

hbase(main):009:0>

Table-Scope Operators

With alter, you can set and remove table-scope operators such as MAX_FILESIZE, READONLY, MEMSTORE_FLUSHSIZE, and DEFERRED_LOG_FLUSH.

READONLY - Set Read-Only

The following is the syntax for making a table read-only.

alter '<table name>', READONLY(option)

The following example makes the order table read-only.

hbase(main):009:0> alter 'order', READONLY
Updating all regions with the new schema...
0/1 regions updated.
1/1 regions updated.
Done.
0 row(s) in 2.3650 seconds

MAX_FILESIZE - Change Region Setting

Change the region setting with MAX_FILESIZE.

alter 't1', MAX_FILESIZE => '123217728'

This changes the maximum region size to 128 MB.

Delete a Column Family

With alter, you can also delete a column family. The syntax for deleting a column family with alter is as follows.

alter '<table name>', 'delete' => '<column family>'

The following is an example of deleting a column family from the order table.

hbase(main):010:0> alter 'order', 'delete' => 'sales'
Updating all regions with the new schema...
0/1 regions updated.
1/1 regions updated.
Done.
0 row(s) in 2.3790 seconds

If you check with the describe command, you can see that 'sales' has disappeared.

hbase(main):011:0> describe 'order'
DESCRIPTION                                                                                                  ENABLED
 'order', {NAME => 'READONLY', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCO true
 PE => '0', COMPRESSION => 'NONE', VERSIONS => '1', TTL => 'FOREVER', MIN_VERSIONS => '0', KEEP_DELETED_CELL
 S => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'customer', DATA_
 BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '5', COMPRESSION => '
 NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMOR
 Y => 'false', BLOCKCACHE => 'true'}
1 row(s) in 0.0850 seconds

drop - Delete a Table

You can delete a table by using the drop command. The syntax is as follows.

drop '<table name>'

The table must be disabled before it is deleted.

hbase(main):024:0> disable 'order'
0 row(s) in 1.3960 seconds
hbase(main):025:0> drop 'order'
0 row(s) in 0.2860 seconds

Use the exists command to check whether the table has been deleted.

hbase(main):026:0> exists 'order'
Table order does not exist
0 row(s) in 0.0710 seconds

drop_all

The drop_all command is used to delete all tables matching a given regular expression. The syntax is as follows.

drop_all '<regex table name>'

A simple example is shown below. The following command deletes all tables that start with t.

drop_all 't.*'

Note: All target tables must be disabled before deletion.

HBase Data Manipulation Commands - put, get, delete, deleteall, scan, count, truncate

kc@example.com (kc kim) — Fri, 16 Jun 2023 14:55:22 +0900

Data Manipulation commands

This section introduces HBase Shell commands for manipulating data.

put - Insert/Update Data

This section introduces how to create and update data in an HBase table.

The following HBase table will be created.

You can insert and update a row in a table by using the put command. The syntax is as follows.

put '<table name>', '<row id>', '<column family>:<column name>', '<value>'

Insert the First Row

create 'order', 'customer', 'sales'

put 'order', '101', 'customer:name', 'John White'
put 'order', '101', 'customer:city', 'Los Angeles'
put 'order', '101', 'sales:product', 'Chairs'
put 'order', '101', 'sales:amount', '400.00'

First, create the table.

hbase(main):001:0> create 'order', 'customer', 'sales'
Created table order
Took 5.0510 seconds
=> Hbase::Table - order

Then insert the first row values into the table.

hbase(main):002:0> put 'order', '101', 'customer:name', 'John White'
Took 0.6725 seconds
hbase(main):003:0> put 'order', '101', 'customer:city', 'Los Angeles'
Took 0.0318 seconds
hbase(main):004:0> put 'order', '101', 'sales:product', 'Chairs'
Took 0.0988 seconds
hbase(main):005:0> put 'order', '101', 'sales:amount', '400.00'
Took 0.0423 seconds

If you scan the whole table, the following result is displayed.

hbase(main):006:0> scan 'order'
ROW                               COLUMN+CELL
 101                              column=customer:city, timestamp=1686908392705, value=Los Angeles
 101                              column=customer:name, timestamp=1686908388954, value=John White
 101                              column=sales:amount, timestamp=1686908401071, value=400.00
 101                              column=sales:product, timestamp=1686908398256, value=Chairs
1 row(s)
Took 0.4121 seconds

Update Data

You can change an existing cell value by using the put command. The syntax is the same as when inserting data.

Specify a new value as follows.

put 'order','101','customer:city','LA'

The command execution is as follows.

hbase(main):007:0> put 'order','101','customer:city','LA'
Took 0.0677 seconds

hbase(main):008:0> scan 'order'
ROW                               COLUMN+CELL
 101                              column=customer:city, timestamp=1686909532774, value=LA
 101                              column=customer:name, timestamp=1686908388954, value=John White
 101                              column=sales:amount, timestamp=1686908401071, value=400.00
 101                              column=sales:product, timestamp=1686908398256, value=Chairs
1 row(s)
Took 0.1186 seconds

get - Retrieve Data

The get command is used to read data from an HBase table. The syntax is as follows.

get '<table name>', '<row id>'

The following example uses the get command to retrieve row 101 from the order table.

hbase(main):009:0> get 'order', '101'
COLUMN                            CELL
 customer:city                    timestamp=1686909532774, value=LA
 customer:name                    timestamp=1686908388954, value=John White
 sales:amount                     timestamp=1686908401071, value=400.00
 sales:product                    timestamp=1686908398256, value=Chairs
1 row(s)
Took 0.3162 seconds

Retrieve a Specific Row

The following is the syntax for using the get command to retrieve a specific column.

get '<table name>', '<row id>', {COLUMN => '<column family>:<column name>'}

The following is an example of retrieving a specific column from an HBase table.

hbase(main):010:0> get 'order', '101', {COLUMN => 'customer:name'}
COLUMN                            CELL
 customer:name                    timestamp=1686908388954, value=John White
1 row(s)
Took 0.1450 seconds

delete - Delete a Specific Cell from a Table

You can delete a specific cell from a table by using the delete command. The syntax of the delete command is as follows.

delete '<table name>', '<row id>', '<column name>', '<timestamp>'

The following is an example of deleting a specific cell. Here, city is deleted.

hbase(main):011:0> delete 'order', '101', 'customer:city', 1686909532774
Took 0.0917 seconds

If you query again, you can see that the 'LA' value changed with the previous put command has been deleted and the existing 'Los Angeles' value is visible again.

hbase(main):012:0> get 'order', '101'
COLUMN                            CELL
 customer:city                    timestamp=1686908392705, value=Los Angeles
 customer:name                    timestamp=1686908388954, value=John White
 sales:amount                     timestamp=1686908401071, value=400.00
 sales:product                    timestamp=1686908398256, value=Chairs
1 row(s)
Took 0.2196 seconds

deleteall - Delete All Cells in a Table Row

You can delete all cells in a row by using the deleteall command. The following is the syntax of the deleteall command.

deleteall '<table name>', '<row id>'

The following is an example of the deleteall command that deletes all cells in row 1 of the order table.

hbase(main):013:0> deleteall 'order','101'
Took 0.1271 seconds
hbase(main):015:0>

Check the table by using the scan command. After deleting the table data, the table snapshot is as follows.

hbase(main):015:0> scan 'order'
ROW                               COLUMN+CELL
0 row(s)
Took 0.0518 seconds

scan - Retrieve Data

The scan command is used to view data in HTable. You can use the scan command to fetch table data. The syntax is as follows.

scan '<table name>'

First, insert data as follows.

put 'order', '101', 'customer:name', 'John White'
put 'order', '101', 'customer:city', 'Los Angeles'
put 'order', '101', 'sales:product', 'Chairs'
put 'order', '101', 'sales:amount', '400.00'

The following example uses the scan command to retrieve data from the table.

hbase(main):019:0> scan 'order'
ROW                               COLUMN+CELL
 101                              column=customer:city, timestamp=1686912065208, value=Los Angeles
 101                              column=customer:name, timestamp=1686912059341, value=John White
 101                              column=sales:amount, timestamp=1686912074891, value=400.00
 101                              column=sales:product, timestamp=1686912070718, value=Chairs
1 row(s)
Took 0.2649 seconds

count - Count Data

You can count the number of rows in a table by using the count command. The syntax is as follows.

count ‘<table name>'

hbase(main):020:0> count 'order'
1 row(s)
Took 0.2992 seconds
=> 1

truncate - Delete All

The truncate command disables, drops, and recreates a table. The syntax of truncate is as follows.

truncate ‘<table name>'

The following is an example of the truncate command. Here, the order table was deleted and recreated.

hbase(main):021:0> truncate 'order'
Truncating 'order' table (it may take a while):
Disabling table...
Truncating table...
Took 5.6828 seconds

After deleting the table data with truncate, check it by using the scan command. You can confirm that the table has no rows.

hbase(main):023:0> scan 'order'
ROW                               COLUMN+CELL
0 row(s)
Took 1.6790 seconds

HBase Security Commands - grant, revoke, user_permission

kc@example.com (kc kim) — Sun, 18 Jun 2023 16:13:38 +0900

Security tools

In HBase, permissions can be granted to and revoked from users. There are three commands for security purposes.

grant
revoke
user_permission

grant - granting permissions

The grant command grants users specific permissions such as read, write, execute, and admin permissions for tables. The syntax of the grant command is as follows.

grant '<user>' '<permissions>' ['<table>' ['<column family>' ['<column; qualifier>']]

RWXCA can be used to grant permissions to users.

R: READ permission
W: WRITE permission
X: EXEC permission
C: CREATE permission
A: ADMIN permission

grant 'devkuma', 'RWXCA'

grant 'devkuma', 'RW' 't1', 'cf1', 'qualifier1'

revoke - deleting permissions

Deletes a user’s access permissions.

revoke <user> <permissions> [<table> [<column family> [<column; qualifier>]]

revoke 'devkuma', 't1', 'cf1', 'qualifier1'

user_permission - querying permissions for a specific table

The user_permission command is used to query all permissions for a specific table. The syntax is as follows.

user_permission '<table name>'

Shows all permissions that exist for a specific table.

user_permission 't1'

HBase Compression, Compaction, and Data Block Encoding

kc@example.com (kc kim) — Thu, 22 Jun 2023 19:13:23 +0900

Compression and Compaction

Regions stored on an HBase RegionServer have an HStore for each Column Family, and this consists of MemStore and HFile for that CF.

MemStore is Column Family data in memory, and HFile is Column Family data stored on local disk, meaning HDFS. The reason for this layered structure is that data must be sorted by Row key according to the HBase table structure, so it cannot be written directly to disk and must be sorted in memory first.

When write operations such as put or update are performed in HBase, data is first recorded in the WAL (Write Ahead Log), then written to MemStore. For read operations such as scan, HBase checks MemStore first and then searches HFiles for the requested content.

When repeated writes cause MemStore data size to exceed the configured threshold, MemStore data is stored as an HFile. In a Region composed of many HFiles, as HFiles increase, compaction occurs and merges multiple HFiles into larger ones. As a result, many small HFiles are reduced to fewer large HFiles. This operation is called compaction.

HBase compaction is performed as minor compaction and major compaction.

Minor Compaction

Minor compaction is the process of merging several recently created small files into a larger configured HFile size.
As data is inserted, many small HFiles are created. Too many files can reduce performance, so HBase automatically manages the number of HFiles by remaking multiple HFiles into a few larger HFiles.
Because data stored in HFiles is sorted, it can be merged quickly using merge sort.

Major Compaction

Major compaction gathers all HFiles for a Column Family in a specific Region and creates one HFile per column. It runs once every 7 days by default.
During this process, unnecessary cells and expired cells are removed to improve overall read performance. Data with Tombstone Markers is finally deleted at this time.
When this work is performed, large amounts of file read/write activity occur, which can increase disk I/O and network traffic.
It can be scheduled to run automatically, and this can be used to schedule it on weekends or at night to minimize the impact of sudden I/O increases on the service.

Compaction settings

Minor compaction uses the following settings to configure the number of HFiles that will be compacted.

hbase.hstore.compaction.min
- Sets the minimum number of HFiles for minor compaction.
- The default is 3, and it must be set to 2 or higher.
- If the value is too large, minor compaction is delayed and later processing all at once can create load.
hbase.hstore.compaction.max
- Sets the maximum number of HFiles for minor compaction.

The following settings specify the size of HFiles to compact.

hbase.hstore.compaction.min.size: HFiles smaller than this value are always included when searching for minor compaction through ExploringCompactionPolicy. This value is used together with hbase.hstore.compaction.ratio to choose HFiles for compaction.
hbase.hstore.compaction.max.size: HFiles larger than this size are excluded from minor compaction. If minor compaction occurs often but does not have much effect, this value can be reduced so large files are excluded from minor compaction. Default: 9223372036854775807, LONG.MAX_VALUE, bytes.

Other settings:

hbase.hregion.majorcompaction
- HBase can have multiple StoreFiles for one Region. Periodically, to improve performance, it gathers these files and combines them into one larger file. This process involves significant CPU usage and disk I/O, and response time may decrease. Therefore, when response time matters, it is better to run Major compaction manually during off-peak time.
- The default is 86,400,000 ms. If this value is changed to 0, periodic Major Compaction will not run.
hbase.hregion.majorcompaction.jitter
- The property defaults to 0.2, or 20%. It spreads out the timing when Major compaction runs for each stored file.
- Without it, major compaction runs simultaneously every 24 hours.

HBase compression and data block encoding

The codecs mentioned in this section are for encoding and decoding data blocks or row keys. For more information about replication codecs, see cluster.replication.preserving.tags.

HBase supports several compression algorithms that can be enabled for a ColumnFamily. Data block encoding uses HBase’s underlying design and patterns, such as sorted row keys and the schema of a given table, to limit information duplication in keys. Compressors can reduce the size of large opaque byte arrays in Cells and significantly reduce the storage space needed for uncompressed data.

Compressors and data block encoding can be used together in the same ColumnFamily.

Changes are applied during compaction

If compression or encoding is changed for a ColumnFamily, the change is applied during compaction.

Some codecs use features built into Java, such as GZip compression. Other codecs depend on native libraries. Native libraries can be provided through codec dependencies installed in the HBase library directory, or as part of Hadoop when using Hadoop codecs. Hadoop codecs usually have native code components, so follow the Hadoop native binary support installation instructions to use Hadoop native libraries in HBase.

This section describes common codecs used and tested in HBase.

Whichever codec you use, you must test that the codec is installed correctly and available on all nodes in the cluster. Additional operational steps may be needed to verify codec availability on newly deployed nodes. The compression.test utility can be used to verify that a specified codec is installed correctly.

To configure HBase to use compressors, see compressor.install. To enable a compressor for a ColumnFamily, see changing.compression. To enable data block encoding for a ColumnFamily, see data.block.encoding.enable.

Block Compressors

NONE

This compression type constant selects no compression and is the default.

BROTLI

Brotli is a general-purpose lossless compression algorithm. It combines a modern variant of LZ77, Huffman coding, and second-order context modeling, and provides compression ratios comparable to the best general-purpose compression methods currently available. It has similar speed to GZ but provides denser compression.

BZIP2

Bzip2 compresses files using the Burrows-Wheeler block-sorting text compression algorithm and Huffman coding. It usually has a much better compression ratio than dictionary-based compression such as LZ, but both compression and decompression may be slower than other options.

GZ(GZIP)

Focuses on compression ratio.
gzip is based on the DEFLATE algorithm, a combination of LZ77 and Huffman coding. This algorithm is universally available in the Java runtime environment, so it is a good lowest-common-denominator option. However, it is quite slow compared with newer algorithms such as Zstandard.

LZ4

Focuses on compression/decompression speed.
LZ4 is a lossless data compression algorithm focused on compression and decompression speed. It belongs to the LZ77 family of compression algorithms, like Brotli, DEFLATE, and Zstandard. In microbenchmarks, LZ4 is the fastest option in that family for both compression and decompression and is generally recommended.

LZMA

LZMA is a dictionary compression method somewhat similar to LZ77. It achieves a very high compression ratio with a computationally expensive prediction model and variable-size compression dictionary, while maintaining decompression speed similar to other commonly used compression algorithms. LZMA beats all other options at typical compression ratios, but as a compressor it can be very slow, especially when configured to operate at high compression levels.

LZO

Focuses on compression/decompression speed and requires additional library installation.
LZO is another LZ-variant data compression algorithm implemented with a focus on decompression speed. It is not as fast as LZ4 but is close.

SNAPPY

Focuses on compression/decompression speed.
SNAPPY is based on LZ77 ideas but optimized for very fast compression speed, achieving only a reasonable level of compression in the tradeoff. It is as fast as LZ4, but its compression ratio is not very high. HBase provides a pure Java Snappy codec that can be used instead of GZ as a universally available option for all Java runtimes on all hardware architectures.

ZSTD

Zstandard combines a dictionary matching stage (LZ77) with a large search window and fast entropy coding stage using both finite-state entropy and Huffman coding. Compression speed can differ by more than 20x between the fastest and slowest levels, but decompression is uniformly fast, with less than a 20% difference between the fastest and slowest levels.
ZStandard is the most flexible available compression codec option. At level 1 it provides compression ratio similar to LZ4, with slightly lower performance. At mid levels it provides compression ratio similar to DEFLATE with better performance, and at high levels it provides dense compression similar to LZMA with LZMA-like compression speed, while providing universally fast decompression.

Data Block Encoding Types

HBase provides five data block encoding types:

NONE
Prefix
Diff
Fast Diff
Prefix Tree

None

None means no encoding is applied and is the default.

ColumnFamily without encoding

The following table shows a hypothetical ColumnFamily without data block encoding.

Key Length	Value Length	Key	Value
24	…	RowKey:Family:Qualifier0	…
24	…	RowKey:Family:Qualifier1	…
25	…	RowKey:Family:QualifierN	…
25	…	RowKey2:Family:Qualifier1	…
25	…	RowKey2:Family:Qualifier2	…
…	…	…	…

Prefix

Usually keys often have the same prefix and only the last part differs.

For example, the first key may be RowKey:Family:Qualifier0, and the next key may be RowKey:Family:Qualifier1. Prefix encoding adds an extended column that stores the length of the prefix shared between the current key and the previous key. In this example, if the first key is completely different from the previous key, the prefix length is 0. Since the first 23 characters of the second key are the same, the prefix length becomes 23.

Of course, if two keys have nothing in common, Prefix provides no benefit.

ColumnFamily with Prefix encoding
The following is the same data with Prefix data encoding applied.

Key Length	Value Length	Prefix Length	Key	Value
24	…	0	RowKey:Family:Qualifier0	…
1	…	23	1	…
1	…	23	N	…
19	…	6	2:Family:Qualifier1	…
1	…	24	…	…
…	…	…	…	…

Diff

Diff encoding extends Prefix encoding. Rather than treating the Key as one sequence of bytes, it splits each Key Field so each part of the Key can be compressed more efficiently.

Two new fields, timestamp and type, are added.

If the ColumnFamily is the same as the previous row, it is omitted from the current row. If key length, value length, and type are the same as the previous row, the fields are omitted.

Also, to improve compression, timestamp is not stored in full but as a difference from the previous row’s timestamp. In the prefix example, if there are two row keys and timestamp matches exactly and type is the same, the second row does not need to store value length or type, and the timestamp value in the second row becomes 0 instead of a full timestamp.

Because write and scan speed become slower, Diff encoding is disabled by default even though more data can be cached.

ColumnFamily with Diff encoding
This table shows the same ColumnFamily with Diff encoding applied.

Flags	Key Length	Value Length	Prefix Length	Key	Timestamp	Type Value
0	24	512	0	RowKey:Family:Qualifier0	1340466835163	4	…
5	-	320	23	1	0	-	…
3	-	-	23	N	120	8	…
0	25	576	6	2:Family:Qualifier1	25	4	…
5	-	3384	24	2	1124	-	…
…	…	…	…	…	…	…	…

Fast Diff

Fast Diff works similarly to Diff but is implemented faster. It also adds one more field that stores a single bit to track whether the data itself is the same as the previous row. If it is the same, the data is not stored again.

Use the Fast Diff codec when keys are long or there are many columns.

The data format is almost the same as Diff encoding, so there is no separate image to explain.

Prefix Tree

Prefix Tree encoding was introduced as an experimental feature in HBase 0.96. Prefix Tree encoding provides memory savings similar to Prefix, Diff, and Fast Diff encoders, but provides faster random access at the cost of slower encoding speed. It was a good idea, but it was rarely used and was removed in HBase 2.0.0.

References

Appendix D: Compression and Data Block Encoding In HBase

HBase Data Versioning

kc@example.com (kc kim) — Thu, 22 Jun 2023 19:13:23 +0900

Data Versioning

As one of HBase’s special features, multiple versions can be stored for a specific column value in each cell.

This is implemented using a timestamp for each version and is sorted in descending order. It uses Unix time converted to milliseconds as a long integer type.

HBase manages data versions by default. When data is stored with the same Row key, the most recent data is versioned by timestamp and stored in descending order, so the most recent value is found first when reading from storage files.

Timestamp can also be entered explicitly. By default, HBase keeps up to three changes for each cell, and because scan queries in descending order, the latest data is queried.

You can also query all stored versions of data as follows.

get 't1', 'rowkey1', {COLUMN => 'cf1', TIMESTAMP => ts1}

It is also possible to set a timestamp range and view the value at that point.

get 't1', 'rowkey1', {TIMERANGE => [start_timestamp, end_timestamp]}

Versioning is usually assigned automatically, but it can also be specified manually.

Automatic versioning
- Version differences can occur if server times in the cluster are not the same.
- Timestamp can be set when executing the Put method, but generally automatic versioning within the server is recommended.
- The most recent three versions are managed by default, but because Major Compaction has a long cycle, old versions that have not yet been deleted may still exist.
Manual versioning
- Can be implemented by overriding the timestamp.

Does deleting reveal an older version of a column even when HBase VERSIONS=1?

Earlier, it was mentioned that the most recent three versions are managed by default, but because Major Compaction has a long cycle, old versions that have not yet been deleted may still exist.

This section introduces a related issue that can occur.

Applying VERSIONS

Apply VERSIONS to the ColumnFamily as follows.

create 't1', {NAME => 'f1', VERSIONS => 2}

hbase(main):001:0> create 't1', {NAME => 'f1', VERSIONS => 2}
Created table t1
Took 5.9091 seconds
=> Hbase::Table - t1

Next, register data in order.

put 't1', '101', 'f1:name1', 'test1'
put 't1', '101', 'f1:name1', 'test2'
put 't1', '101', 'f1:name1', 'test2'
put 't1', '101', 'f1:name1', 'test4'

hbase(main):002:0> put 't1', '101', 'f1:name1', 'test1'
Took 0.7153 seconds
hbase(main):003:0> put 't1', '101', 'f1:name1', 'test2'
Took 0.0318 seconds
hbase(main):004:0> put 't1', '101', 'f1:name1', 'test3'
Took 0.0418 seconds
hbase(main):005:0> put 't1', '101', 'f1:name1', 'test4'
Took 0.0255 seconds

Then delete one by one with the delete command and check the result.

delete 't1', '101', 'f1:name1'

hbase(main):013:0> scan 't1'
ROW                                                                  COLUMN+CELL
 101                                                                 column=f1:name1, timestamp=1687426826665, value=test4
1 row(s)
Took 0.3049 seconds
hbase(main):014:0> delete 't1', '101', 'f1:name1'
Took 0.0160 seconds
hbase(main):015:0> scan 't1'
ROW                                                                  COLUMN+CELL
 101                                                                 column=f1:name1, timestamp=1687426822568, value=test3
1 row(s)
Took 0.1760 seconds
hbase(main):016:0> delete 't1', '101', 'f1:name1'
Took 0.0605 seconds
hbase(main):017:0> scan 't1'
ROW                                                                  COLUMN+CELL
 101                                                                 column=f1:name1, timestamp=1687426817569, value=test2
1 row(s)
Took 0.0710 seconds
hbase(main):018:0> delete 't1', '101', 'f1:name1'
Took 0.0337 seconds
hbase(main):019:0> scan 't1'
ROW                                                                  COLUMN+CELL
 101                                                                 column=f1:name1, timestamp=1687426813634, value=test1
1 row(s)
Took 0.0265 seconds
hbase(main):020:0>

You can see that the previously registered data is queried in order.

Question about VERSIONS

The question here is that VERSIONS was said to store only one version, but even when VERSIONS => 2 is specified, more versions of the column appear to be maintained. At first glance, this looks suspiciously like a bug.

Searching online, a person with the same question was found.
Delete reveals older version of a column even when VERSIONS=1

The answer to that question says the following.

Older versions do not actually disappear until compaction occurs. Compaction should be performed once a day unless the major compaction setting is changed, or whenever a Region is split.

This issue occurs because Major Compaction has not yet been performed.

How to check VERSIONS application

Then how can VERSIONS be checked? Use the get command and query details with VERSIONS => 4 as follows.

get 't1', '101', {COLUMN=>'f1:name1',VERSIONS => 4 }

hbase(main):005:0> get 't1', '101', {COLUMN=>'f1:name1', VERSIONS => 4 }
COLUMN                                                               CELL
 f1:name1                                                            timestamp=1687427184339, value=test4
 f1:name1                                                            timestamp=1687427181122, value=test2
1 row(s)
Took 0.5028 seconds
hbase(main):006:0>

As shown above, you can see two records according to VERSIONS => 2 specified when creating the table.

HBase BloomFilter

kc@example.com (kc kim) — Fri, 23 Jun 2023 19:01:58 +0900

What is BloomFilter?

BloomFilter is a fast search algorithm for multi-hash-function mapping proposed by Bloom in 1970. This algorithm is generally used in situations where it is necessary to quickly determine whether an element belongs to a set, starting from the premise that strict 100% accuracy is not required.

For detailed theory, refer to the Bloom Filter data structure explained simply page.

HBase BloomFilter

HBase BloomFilter data is stored in StoreFile metadata, and once written, a StoreFile is immutable, so it cannot be updated.

BloomFilter is a configuration property at the Column family level. If BloomFilter is configured on a table, HBase includes a data fragment of the BloomFilter structure when creating a StoreFile called a MetaBlock. MetaBlock and DataBlock, which contain actual KeyValue data, are used together in LRU BlockCacheMaintenance.

Therefore, configuring BloomFilter introduces specific storage and memory cache overhead.

How to configure HBase BloomFilter

When searching for whether a specific Row key is included, it reduces unnecessary block loading compared to the existing block index, improving overall cluster throughput.

Bloom filters should be selected according to cell size, number of cells, data storage method, read method, and similar factors.

According to the HBase documentation, BloomFilter is recommended in most cases.

BloomFilter has three parameters: NONE (default), ROW, and ROWCOL.

ROW: Indicates a row-level Bloom filter that filters StoreFiles based on the KeyValue row.
ROWCOL: Indicates a column-level Bloom filter that filters StoreFiles based on the KeyValue row plus column.

Therefore, the space overhead of ROWCOL is higher than that of ROW.

The more StoreFiles there are in the Region, the better the BloomFilter effect. The fewer StoreFiles there are in the Region, the better HBase read performance becomes.

Configure the Column family in HBase to enable the BloomFilter command as follows.

create 't1',{name => 'c1', BLOOMFILTER => '<ROW or ROWCOL>'}