devkuma – MongoDB

MongoDB Sharding Overview

kc@example.com (kc kim) — Wed, 19 Jan 2022 09:19:00 +0900

Scaling

Before talking about sharding, let’s first look at scaling.

In general, when a system grows larger, scaling becomes necessary. Scaling methods are broadly divided into two types: vertical scaling and horizontal scaling.

Vertical scaling

Vertical scaling is a method of replacing CPU, RAM, or storage with higher-performance components. Scaling is simple, but there are limits to CPU and memory in the first place, so there is a practical performance limit.

Horizontal scaling

Horizontal scaling divides a system by data set and processes data in parallel from multiple servers. Overall performance can be better than vertical scaling, but the more it scales, the more complex it becomes, making management difficult.

What is sharding?

Sharding is a horizontal scaling structure that distributes data across multiple servers.

Sharding has three major benefits: read/write processing becomes faster and performance is easier to improve, storage is easier to expand, and availability is high. Each is described below. MongoDB supports sharding, or horizontal scaling, as a standard feature.

Reads and writes

When data is distributed and stored in a shard cluster, read/write processing, or file I/O, can be distributed. File read/write operations are slow, so distributing them enables faster reads and writes. In addition, when read/write load increases, the system can horizontally scale by adding servers.

Storage

Fragmented data is distributed and stored in the shard cluster. Even when the amount of data increases, the system can horizontally scale by adding servers.

High availability

Shard clusters allow partial reads and writes. If a shard cannot be read from or written to, data can also be retrieved from a shard server that can perform the work.

Shard cluster

MongoDB sharding consists of three components: shards, routers, and config servers. Their relationships and explanations are as follows.

Shard

Shard servers store fragmented data, or chunks, from split collections. A shard server can be configured as a replica set.

Router

mongos is a query router that provides an interface from applications to the shard cluster. When accessing a shard cluster, access always goes through mongos. Even collections that are not sharded must be accessed through mongos.

Config server

Config servers store metadata about shard cluster settings. Information such as which shard server holds the data is also stored on these servers. Since MongoDB 3.4, config servers are configured as replica sets.

Shard key

A shard key is information used as a key when distributing data and storing it on shard servers. MongoDB uses the shard key to distribute documents within a collection.

The following figure is a simple example distributed by an x shard key.

… figure …

The important points about shard keys are the following three.

They consist of immutable fields or groups of fields that exist in every document.
Once a shard key is chosen and distribution is performed, the shard key cannot be changed again.
A sharded collection requires an index that includes the shard key.

Needless to say, shard key selection greatly affects performance, efficiency, and scalability. If the shard key is chosen poorly, data can become biased toward specific shard servers, as described above, causing inefficient behavior.

Chunks

As mentioned above, MongoDB distributes data storage by shard key value, but it groups data into units called chunks and distributes those chunks across shard servers. MongoDB automatically moves chunks so that they are balanced within the shard cluster.

The image of splitting data into chunks and distributing them across shards is as follows.

… figure …

Operation constraints on sharded data

There are several restrictions on operations for sharded data.

The group command cannot be used.
Use MapReduce instead of aggregate.
updateOne or deleteOne must specify _id.
An error occurs if the shard key _id is not included.
A sharded collection requires a unique index.

Sharded and unsharded collections

MongoDB can mix sharded collections and unsharded collections. Sharded collections are distributed and stored in the shard cluster, while unsharded collections are stored on the primary shard.

Data distribution methods

There are two distribution methods for sharding: hashed sharding and ranged sharding.

Hashed sharding

Hashed sharding distributes data using the hash value of the shard key. With hashed sharding, chunks tend to become scattered even when shard key values are close. In other words, it is a distribution method that makes distribution easier when the shard key changes monotonically. On the other hand, because data is easy to distribute, broadcast operations may increase depending on the data structure and retrieval method.

Ranged sharding

Ranged sharding distributes data according to the shard key range. Unlike hashed sharding, when shard key values are close, they are more likely to exist in the same chunk. If the shard key is chosen poorly, load can become concentrated on specific servers even though sharding is being used as a distribution technique.

How to Use the MongoDB mongod Command

kc@example.com (kc kim) — Thu, 13 Jan 2022 07:57:00 +0900

This article summarizes the basic main usage and options of the mongod command.

Only the important options are covered here. For many other options, refer to the official manual.

Basic options

Option	Description
`--config <filename>`, `-f <filename>`	Specifies a configuration file that contains startup options. If the configuration file path contains spaces, enclose it in double quotes (`"`). The configuration file uses ASCII encoding. Non-ASCII encodings such as UTF-8 are not supported. If both a configuration file and startup options are specified, the startup option values take precedence.
`--port <port>`	Default: 27017 Specifies the port number on which MongoDB waits for TCP connections.
`--maxConns <number>`	Specifies the number of concurrent connections MongoDB allows.
`--logpath <path>`	Specifies the path where the log file is output.
`--logappend`	When MongoDB is restarted, logging continues at the end of the existing log file. By default, MongoDB backs up the existing log file and creates a new log file when restarted.
`--slowms <integer>`	Default: 100 Specifies the profiling threshold in milliseconds.
`--profile <level>`	Default: 0 Changes the database profiling level. - 0: Off. Does not perform profiling. - 1: On. Profiles only slow operations. - 2: On. Profiles all operations.
`--auth`	Uses database access control. User configuration changes are performed with the `mongo` command. If no user exists, only access through `localhost` is permitted.
`--bind_ip <address>`	Default: all connections Specifies the IP address where MongoDB accepts connections.
`--dbpath <path>`	Default: `C:\data\db` Specifies the storage path where database files are stored.

Windows service installation options

Option	Description
`--install`	Adds MongoDB as a Windows service. If needed, specify `--serviceName` and `--serviceDisplayName` at the same time.
`--remove`	Removes the MongoDB service from Windows services. If a non-default name was installed with `--serviceName`, specify `--serviceName`.
`--serviceName <name>`	Default: MongoDB Specifies the service name. The name specified here is used as the argument to `net start <name>` and `net stop <name>`.
`--serviceDisplayName <name>`	Default: MongoDB Specifies the name displayed in the service management tool.
`--serviceDescription <description>`	Default: MongoDB Server Specifies the string displayed in the description of the service management tool. If it contains spaces, enclose it in double quotes (`"`).
`--serviceUser <user>`	Specifies the user that runs MongoDB.
`--servicePassword <password>`	Specifies the password for the user that runs MongoDB.

References

MongoDB Features

kc@example.com (kc kim) — Sun, 09 Jan 2022 18:38:00 +0900

This article looks at the features of MongoDB.

The main features, also described on the official site, can be summarized as follows.

Document database (NoSQL)
Special queries
High performance
High availability
Horizontal scalability

Document database (NoSQL)

MongoDB is a document-format database that can store key-value combinations as they are, like JSON. Values can take many forms. For example, it can store numbers and strings, as well as arrays and objects. These value types can also be changed dynamically, and multiple types can be placed under the same field name. In other words, it is a schemaless document database.

Performance

The following features are implemented to provide performance.

Reduced I/O processing through embedded data models
Index support

One of MongoDB’s characteristics is that it supports indexes even though it is NoSQL. You can also specify a unique key in relation to indexes.

Special queries

MongoDB searches the database using special queries specified in JSON format. Queries can perform basic CRUD operations as well as the following distinctive operations.

Aggregation and Map-Reduce
String search using text indexes
Geospatial queries

Aggregation and Map-Reduce can perform various statistics on stored data. Query assembly is implemented in a distinctive way, so it takes some getting used to, but the processing order is pipelined, which makes the code easier to follow.

Geospatial queries can use the built-in GeoJSON format to search for points within a range.

Availability

MongoDB provides replication through a feature called replica sets. Replica sets provide automatic failover and data redundancy.

Horizontal scaling

Horizontal scaling means that performance can be improved by adding machines. MongoDB provides horizontal scalability through a core feature called sharding.

Support for multiple storage engines

MongoDB supports multiple storage engines. In addition to commonly used file systems, you can choose an in-memory engine. It also exposes a storage engine API, so third-party storage engines can be created.

References

Installing MongoDB with Docker

kc@example.com (kc kim) — Mon, 10 Jan 2022 08:53:00 +0900

Installing MongoDB with Docker

Downloading the MongoDB Docker image

Run the following command to download the MongoDB Docker image. If you do not specify a version in the tag, the latest version is downloaded.

docker pull mongo

The result is as follows.

% docker pull mongo
Using default tag: latest
latest: Pulling from library/mongo
ea362f368469: Pull complete
ecab26900ceb: Pull complete
1847fcb70562: Pull complete
a7de23811c0d: Pull complete
29dd51833fb9: Pull complete
5eccd2be8afb: Pull complete
cd8a8cd6879f: Pull complete
e6ca3abc397d: Pull complete
7481c3682d3c: Pull complete
af377cb9eb7d: Pull complete
Digest: sha256:6743836d42756b2ae50549b2eb4585c688fce81a243cedd152b56266c2fb3d17
Status: Downloaded newer image for mongo:latest
docker.io/library/mongo:latest

To specify the MongoDB version, specify the version in the tag. You can check available MongoDB versions on Docker Hub.

Check the downloaded Docker image with the following command.

docker images

The result is as follows.

% docker images
REPOSITORY                                      TAG            IMAGE ID       CREATED         SIZE
mongo                                           latest         ee13a1eacac9   2 days ago      696MB

Creating and running a MongoDB Docker container

docker run --name my-mongodb -v ~/mongodb/db:/data/db -d -p 27017:27017 mongo

-v ~/mongodb/db:/data/db mounts the ~/mongodb/db directory on the host, meaning the local computer running the container, to the /data/db directory in the container. If you do not set up a volume this way, the stored data is deleted when the container is deleted. Once deleted, container data cannot be recovered.

% docker run --name my-mongodb -v ~/mongodb/db:/data/db -d -p 27017:27017 mongo
ade97bdb08fde4eaa72ceb0135b0a570d97b21616affb7949a0fc479fec25e0e

Starting, stopping, and restarting a MongoDB Docker container

Stop the MongoDB Docker container.

docker stop my-mongodb

Start the MongoDB Docker container.

docker start my-mongodb

Restart the MongoDB Docker container.

docker restart my-mongodb

Checking whether the MongoDB Docker container is running

Run the following command.

% docker ps -a
CONTAINER ID   IMAGE                                                  COMMAND                  CREATED          STATUS                      PORTS                                       NAMES
ade97bdb08fd   mongo                                                  "docker-entrypoint.s..."   37 seconds ago   Up 36 seconds               0.0.0.0:27017->27017/tcp                    my-mongodb

Connecting to the MongoDB Docker container

Run the following command.

docker exec -it my-mongodb bash

The result is as follows.

% docker exec -it my-mongodb bash
root@ade97bdb08fd:/#

How to Use the MongoDB mongo Command

kc@example.com (kc kim) — Tue, 11 Jan 2022 18:21:00 +0900

This article explains how to connect using mongo, the client module for connecting to MongoDB.

Basic options

Option	Description
`--host <hostname>`	Specifies the host name of the MongoDB server to connect to. If no host name is specified, it connects to localhost.
`--port <port>`	Specifies the port number used to connect to the MongoDB server. If no port number is specified, it connects to `27017`.
`--username <username>, -u <username>`	Specifies the user name when accessing MongoDB with authentication enabled.
`--password <password>, -p <password>`	Specifies the password when connecting to MongoDB with authentication enabled.
`--authenticationDatabase <dbname>`	Specifies the database name used for authentication.

Files

.dbshell

This stores the history of commands executed in the mongo shell. The file is automatically created directly under the home directory (%UserProfile%). On Windows, for example, it is located in the following place.

%UserProfile%

C:\Users\<user name>\.dbshell

.mongorc.js

If you place the .mongorc.js file directly under a defined directory (%UserProfile% or %ProgramData%), .mongorc.js is automatically loaded and executed when mongo starts. This lets you predefine variables or functions that you want to use commonly while the shell is running. .mongorc.js reads global definitions (%ProgramData%) before user definitions (%UserProfile%).

If you do not want to load .mongorc.js, start mongo with the --norc option.

On Windows, for example, the directories are as follows. %UserProfile%

C:\Users\<user name>\.mongorc.js

%ProgramData%

C:\ProgramData\MongoDB\.mongorc.js

References

How to Apply Indexes in MongoDB

kc@example.com (kc kim) — Wed, 12 Jan 2022 08:39:00 +0900

Because MongoDB is a document database, it has indexes that work a little differently. This article explains index concepts that do not exist in relational databases.

MongoDB indexes

In MongoDB, the _id property is basically created automatically as a unique index whenever a document is created. If this is not enough, you can create your own indexes with db.collection.createIndex(). Indexes created with db.collection.createIndex() are B-tree data structure indexes.

MongoDB supports several index types in addition to simple key specifications. The supported index types are as follows.

Single key
Compound key
Multikey
Geospatial
Text
Hashed

“Compound key” and “multikey” have similar names, but they are different. A compound key is a key made from a combination of multiple fields, while a multikey is a key that targets all values included in an array for a field that handles arrays.

Below, we will look at concrete examples of the basic single-field index, compound index, and multikey index.

Basic index creation

You can create an index on a specified collection by running the following command in the mongo shell.

The function is the same in JavaScript code, but JavaScript can specify a callback as the third argument.

Syntax

db.collection.createIndex(keys, options)

Arguments

Argument	Type	Description
`keys`	`string`/`object`	Specifies the field name that has a value. Specify `1` for an ascending key and `-1` for a descending key.
`options`	`object`	Specifies options for index creation. The available options include `unique` and `collation`. There are more options, but remember these for now.

Examples

Below, we will look at examples of single-field indexes, compound indexes, and multikey indexes.

Creating a single-field index

This section covers three sample codes for creating indexes on a sample document like the following. Because MongoDB is a document database, it can create indexes on embedded fields and embedded documents that do not exist in relational databases.

{
  "score": 1034,
  "location": { city: "Seoul", county: "Korea Republic of" }
}

Single field

This is the simplest sample code for creating an index on a single field.

> db.collection.createIndex({ score: 1 })

> db.collection.find({ score: 1034 })
> db.collection.find({ score: { $gt: 1000 } })

Embedded field

MongoDB can also specify a document as an attribute value. In this case, fields inside the document, or nested elements, can also be indexed.

> db.collection.createIndex({ "location.city": 1 })

> db.collection.find({ "location.city": "Kyunggi" })

Embedded document

This is similar to an embedded field, but you can also specify the original object. However, note that in this case an exact match is required, including the number and order of attributes.

> db.collection.createIndex({ location: 1 })

> db.collection.find({ city: "Seoul", county: "Korea Republic of" })

Creating a compound-key index

This is a compound-key index, which also exists in relational databases. In MongoDB, it can be created by specifying multiple fields as keys. In a MongoDB compound-key index, the index is effective not only for patterns that match all specified compound keys, but also for patterns that match part of them.

Let’s look at an example of creating a compound-key index for the following sample data.

{
  "item": "Apple",
  "location": "xxx store",
  "stock": 4,
  "type": "cases"
}

> db.collection.createIndex({ item: 1, location: 1, stock: 1 })

> db.collection.find({ item: "Apple" })
> db.collection.find({ item: "Apple", location: "xxx store" })
> db.collection.find({ item: "Apple", location: "xxx store", stock: { $gt: 0 } })

In the example above, the index is created as item -> location -> stock, so the leading parts such as item and item -> location can also use the index.

Creating a multikey index

MongoDB also has fields that use arrays as values. Whether an array contains primitive values or objects, a multikey index can be created.

Below, we will look at examples of creating an index on a collection that includes an array field named ratings and searching it.

{ "item": "Apple",  ratings: [ 2, 9 ] }
{ "item": "Banana", ratings: [ 4, 3 ] }

An index can be created simply by calling createIndex() for ratings.

> db.collection.createIndex({ ratings: 1 })

Searches on array fields usually use $elemMatch. The following example extracts documents where at least one ratings record is greater than or equal to 3 and less than or equal to 6.

> db.collection.find({ ratings: { $elemMatch: { $gte: 3, $lte: 6 } } })
{ "_id" : ObjectId("5a2d24459c684f917e3ec0c2"), "item" : "XYZ", "ratings" : [ 4, 3 ] }

If you enter the following, it extracts documents where at least one record satisfies greater than or equal to 3 or less than or equal to 6, so all documents are extracted as a result.

> db.collection.find({ ratings: { $gte: 3, $lte: 6 } })
{ "_id" : ObjectId("5a2d24399c684f917e3ec0c1"), "item" : "ABC", "ratings" : [ 2, 9 ] }
{ "_id" : ObjectId("5a2d24459c684f917e3ec0c2"), "item" : "XYZ", "ratings" : [ 4, 3 ] }

References

How to Apply Unique Constraints in MongoDB

kc@example.com (kc kim) — Wed, 12 Jan 2022 08:59:00 +0900

This article explains how to apply unique constraints in MongoDB.

Applying unique constraints

To apply a unique constraint in MongoDB, specify true for the unique option of createIndex() when creating an index.

The examples below show the cases of a single-key index and a compound-key index.

Single-key index

For a single-key index, you can create it by simply running the following code in the mongo shell.

> db.members.createIndex({ userId: 1 }, { unique: true })

Compound-key index

For a compound-key index, the code is the same as for a single-key index.

> db.members.createIndex({ firstname: 1, lastname: 1 }, { unique: true })

With a compound-key index, the combination of compound keys only needs to be unique, so the following can be set without issue.

> db.members.insert({ firstname: "kc", lastname: "Kim" })
> db.members.insert({ firstname: "et", lastname: "Kim" })
> db.members.insert({ firstname: "kc" })

Limits on setting unique constraints

As with relational databases, you cannot apply a unique constraint to a collection that already contains duplicate data. This may be obvious once duplicate data already exists.

Also, setting a unique constraint on a hashed index is not recommended. See Considerations for Hashed Indexes.

Behavior when no value is set for the unique constraint key

As in the example above where a unique constraint was set with a compound-key index, if there is a field with no value, MongoDB internally treats it as null.

For example, suppose there is a collection with a unique constraint on field x as follows.

> db.collection.createIndex({ x: 1 }, { unique: true })

Assume that data is inserted without specifying field x.

> db.collection.insert({ y: 1 })
WriteResult({ "nInserted" : 1 })

The first insert succeeds because x: null is assigned. What happens if another document without field x is inserted?

> db.collection.insert({ z: 1 })
WriteResult({
    "nInserted" : 0,
    "writeError" : {
        "code" : 11000,
        "errmsg" : "E11000 duplicate key error collection: test.collection index: x_1 dup key: { : null }"
    }
})

As shown above, an error occurs and the insert cannot be performed. The error message also shows that null is assigned when no value is specified.

References

MongoDB - Unique Indexes