devkuma – BigData

Apache Hadoop

kc@example.com (kc kim) — Sat, 24 Dec 2022 10:22:54 +0900

Apache Hadoop Overview

Hadoop is a framework implemented in Java for distributed storage and analysis of large-scale data. Hadoop originated from Google’s MapReduce and Google File System, which were distributed processing foundations for efficiently processing large amounts of data.

Google published papers about these systems in 2004, and Doug Cutting and Mike Cafarella developed Hadoop based on them. The name Hadoop came from the name Doug’s son gave to a yellow elephant stuffed toy. It was adopted because it had no meaning, was simple, and was not used elsewhere. The yellow elephant is also Hadoop’s mascot.

Because Hadoop is a distributed processing platform, each process is divided across machines in a cluster (Map), and the results processed by each machine are aggregated (Reduce) to obtain the final result.
Recently, demand for data mining has increased, such as extracting target data from large amounts of data (BigData) or reading trends from stored data. There is also growing demand not only to process BigData, but also to produce such information in a shorter time.
Previously, dedicated products such as data warehouses were needed to process BigData. Hadoop makes this kind of data processing possible by connecting multiple ordinary server machines together, or scaling out.

Although a Hadoop system consists of multiple servers, distribution across multiple machines increases system flexibility. To improve processing performance, you only need to add systems to the Hadoop cluster. Because a Hadoop cluster can be composed of ordinary server machines, hardware procurement is easy. On the software side, a Hadoop cluster can be scaled up simply by installing and configuring Hadoop on servers added to the cluster. These characteristics make it highly scalable in both hardware and software.

Because cloud services now make it easy to start multiple servers, a Hadoop cluster can be built in the cloud only when data processing is needed. If performance is insufficient, servers can be added; if resources remain, servers can be reduced; and when a job is complete, all machines in the Hadoop cluster can be released. For this reason, Hadoop is expected to be used in more and more scenarios.

Until Hadoop version 1, MapReduce was the only parallel processing framework, but from Hadoop version 2, other parallel processing frameworks such as Storm, Spark, Tez, and Impala became available. Interfaces for processing data in Hadoop other than MapReduce (Java) also increased. For example, through Hive and Pig running on Impala and Tez, users can access data using queries almost equivalent to familiar SQL. In addition, Storm and Spark enable real-time data processing through streaming, making it possible to use Hadoop systems even with data outside HDFS.

Hadoop Features

Hadoop consists of the following four core modules.

Hadoop Distributed File System (HDFS)
Hadoop MapReduce
Hadoop Common
Hadoop YARN

There are also the following modules that are separate Hadoop projects.

Apache Ozone
Apache Submarine

HDFS (Hadoop Distributed File System)

HDFS is Hadoop’s own distributed file system. To users it appears as one large file system, but it stores files across nodes. To prevent data loss if one node fails, the same data is stored on three nodes by default.

MapReduce

MapReduce is a framework for processing distributed data in parallel. In the Map step, each slave node processes its data, and in the Reduce step, the processing results distributed and executed across multiple nodes in the Map step are aggregated.

Hadoop Common

Hadoop Common is a set of utilities that support Hadoop functionality.

YARN (Yet Another Resource Negotiator)

Until Hadoop version 1, YARN was not an independent component, but in Hadoop version 2 it became an independent module dedicated to resource management. It can manage MapReduce resources and job scheduling, as well as resources for other distributed processing frameworks such as Giraph, Storm, Spark, Tez, and Impala.

Apache Ozone

Apache Ozone is a project for implementing distributed object storage in Hadoop. It is designed to scale to hundreds of billions of files and blocks, and also supports operation in container environments such as YARN and Kubernetes. It can be accessed using multiple protocols, including S3 and the Hadoop File System API. It was originally a Hadoop subproject, but became an independent Apache top-level project.

Apache Submarine

Apache Submarine is a project that enables deep learning applications such as TensorFlow, PyTorch, and MxNet to run on resource management platforms such as YARN. It was originally a Hadoop subproject, but became an independent Apache top-level project. It can be used with Hadoop 2.7.3 or later.

Hadoop Use Cases

Hadoop can use Apache Spark, which can process faster than MapReduce. For details, see https://openstandia.jp/solution/hadoop-spark/.

Hadoop Operating Environment

Hadoop is written in Java, so it requires a JVM. As of April 2022, Hadoop 3.3.2, the stable version at that time, supports Java 8 and Java 11. Any OS is acceptable as long as the JVM runs on it.

Operating Systems Where Hadoop Runs

Major Linux distributions
Windows
MacOSX

Hadoop has been confirmed to work normally on OpenJDK. Verification results for each JDK can be checked on the Hadoop Wiki page below.
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Java+Versions

Hadoop provides both compiled binary packages and source versions that users compile themselves.
Compiled binary packages can be used immediately, but some settings cannot be extended, so users may need to build from source to enable the required features.

Hadoop License

Hadoop is one of Apache’s top-level projects.
The license is Apache License 2.0, and users are not restricted in using, distributing, modifying, or distributing derivative versions of the software.

Hadoop Official Site

Hadoop’s official site is the URL below. http://hadoop.apache.org/

The official Hadoop Wiki page also contains various information about Hadoop. https://cwiki.apache.org/confluence/display/HADOOP/Home

Hadoop Download

https://hadoop.apache.org/releases.html

Three Layers That Make Up Hadoop

The Hadoop architecture mainly consists of the following three layers.

Distributed processing engine (Hadoop uses Hadoop MapReduce)
Resource manager (Hadoop uses Hadoop YARN)
Distributed file system (Hadoop uses HDFS)

Hadoop also often uses query engines to access data.

Hadoop installs the above components on every computer and distributes data reading, writing, and processing.

Distributed Processing Engine

The distributed processing engine is the software group responsible for parallel distributed processing in Hadoop.

By default, a distributed processing engine called MapReduce runs.

MapReduce Processing Overview

MapReduce performs distributed processing as follows. - Map: Outputs input in key-value format and can distribute each Map by node - Shuffle: Sorts Map output - Reduce: Aggregates identical keys

Representative distributed processing engines have the following characteristics. Lower items are faster.

MapReduce: Writes intermediate results to HDFS storage
Tez: Writes intermediate results to YARN container storage
Spark: Writes intermediate results to memory

MapReduce appears to be a technology that will not disappear, but using Tez or Spark is recommended.

Resource Manager

The resource manager is responsible for managing resources such as CPU and memory in Hadoop.

The resource manager used by MapReduce is Hadoop YARN, which manages application-level containers.

Apache Mesos also exists and manages OS-level containers. It uses technologies such as Docker and Linux containers.

Distributed File System

The distributed file system is responsible for distributed data reading and writing in Hadoop. Distributed file systems used in Hadoop include the following.

HDFS: Hadoop’s standard file system
EMRFS: A file system that uses Amazon S3 as storage
MapR-FS: A file system that rewrites HDFS in C. It is fast.

Cloud Storage or Blob Storage also appear to be usable as storage, but it is unclear which distributed file system is used internally.

Hadoop Ecosystem List

Software that composes Hadoop beyond the defaults, or related surrounding software, is called the Hadoop ecosystem.

The Hadoop ecosystem can be combined as follows to perform various kinds of distributed processing.

Data warehouse configuration example: Hadoop + Tez + Hive
- Hive enables Hadoop to be operated with SQL.
Machine learning configuration example: Hadoop + Spark
- Spark’s in-memory processing is efficient for iterative processing that often occurs in machine learning.
Full-text search configuration example: Hadoop + Elasticsearch
- Elasticsearch for Apache Hadoop can be used to implement a full-text search service.
- An Elasticsearch cluster is used as Hadoop’s distributed file system.
Stream processing configuration example: each server and IoT device –> Kafka –> Hadoop
- Kafka is used to perform stream processing from multiple servers and IoT devices and aggregate data into Hadoop.

Representative Hadoop ecosystem and related systems and their functions are introduced below.

Hadoop ecosystem	Function implemented
Apache Accumulo	KVS-type NoSQL. Emphasizes security
Apache Atlas	Governance control and compliance support
Cascading	API that makes MapReduce easier to handle
Apache Drill	Distributed SQL engine for manipulating edge device data
Apache Falcon	Data lifecycle management
Apache Flume	Aggregates unstructured data from multiple data sources into Hadoop (stream data processing)
Apache HBase	KVS-type NoSQL
Apache Hive	Manipulates data with SQL-like HiveQL queries. Emphasizes fault tolerance. Implements DWH
Apache Hue	Works with Hadoop and the Hadoop ecosystem through a GUI
Apache Impala	Manipulates data with SQL-like Impala SQL queries. Emphasizes speed. Implements real-time processing
Apache Kafka	Aggregates unstructured data from multiple data sources into Hadoop (stream data processing). Difference from Flume is noted separately
Apache Knox	Centralized authentication and access management
Apache Mahout	Linear algebra, statistical analysis, and machine learning library
Apache Mesos	Resource manager that manages OS-level containers
Apache Oozie	Job scheduler
Apache Phoenix	Real-time RDB that uses HBase as a datastore
Apache Pig	Data processing (ETL) tool
Apache Ranger	Grants attribute-based access rights to authenticated users
Apache Sentry	Grants role-based access rights to authenticated users
Apache Slider	Controls YARN applications, such as killing long-running applications
Apache Solr	Full-text search, also used with Elasticsearch
Apache Spark	Processes machine learning, SQL operations, R language, and graphs in memory
Apache Sqoop	Imports and exports structured data between RDBMS and Hadoop
Apache Tez	Distributed processing framework faster than MapReduce
Presto	SQL query engine that outputs intermediate results to memory

Apache Spark

kc@example.com (kc kim) — Sat, 24 Dec 2022 10:22:54 +0900

Apache Spark Overview

Apache Spark is a distributed processing framework for cluster computing that processes large-scale data such as big data and machine learning workloads. Spark development began in 2009 at AMPLab at the University of California, Berkeley, by Mate Zaharia, who was also a Hadoop committer. It is now managed and developed as one of the top-level projects of the Apache Software Foundation.

Spark was developed to improve the slow processing speed of traditional MapReduce and to support a more flexible processing style that is not bound to repeatedly using Map and Reduce.

Spark can run independently as a distributed processing framework, so it is attracting attention as a post-Hadoop technology. At the same time, it can also be used as a replacement for MapReduce within the Hadoop core system, which consists of MapReduce, HDFS, YARN, and related components.

Key Features of Apache Spark

Spark’s major features include the ability to easily program flexible processing models using the concise APIs provided by Spark, and the ability to process large-scale data in far less time than traditional MapReduce.

In traditional MapReduce, Map and Reduce had to be performed as one processing model, so applications running on Hadoop had to be developed according to that style. This made it difficult to develop flexible processing models.

MapReduce also writes processing results to disk after each Map and Reduce operation, making it difficult to improve processing speed. In contrast, Spark can run multiple Map operations consecutively on datasets loaded into memory (RDDs), reduce the results, and then run the next Map operation on the same dataset while keeping it in memory rather than writing it to disk. For this reason, Spark can sometimes achieve more than 100 times faster processing than MapReduce, although it can also write results to disk like traditional MapReduce.

Spark’s features are as follows.

Speed
- Fast in-memory processing
Ease of Use
- Ease of use through support for various languages such as Java, Scala, Python, R, and SQL
Generality
- Provides various components such as SQL, Streaming, machine learning, and graph computation
Run Everywhere
- Can run on various clusters such as YARN, Mesos, and Kubernetes
- Supports various file formats and storage systems such as HDFS, Cassandra, and HBase

Apache Spark Component Structure

Spark is a distributed processing framework consisting of the following components.

Spark Core, including Scala, Java, Python, and R APIs
Spark SQL + DataFrames
Spark Streaming
MLlib
GraphX

Spark Core

Spark keeps data to be processed in the form of RDDs (Resilient Distributed Datasets).
An RDD is an immutable collection that can be executed in parallel and is distributed across computers.
In the Spark programming model, processing is performed by applying various methods provided by Spark Core to these RDDs. When developers manipulate RDDs through the APIs provided by Spark Core, they can perform distributed processing without being conscious of distributed data.
This is one of Spark’s strengths: it makes flexible processing easy to program.
Spark Core APIs are provided not only for Scala, Spark’s development language, but also for Java, Python, and R as standard APIs. Some third-party libraries also provide Spark API access from Clojure, a functional language that runs on the JVM like Scala, and APIs for other languages are expected to continue increasing.

Spark SQL + DataFrames

In addition to manipulating RDDs through Spark APIs, Spark can also use a SQL-like language called Spark SQL to manipulate abstract datasets called DataFrames, which have named columns like database tables.
This is an interface that allows users who have not learned languages such as Scala, Java, Python, or R to process data with Spark if they have SQL knowledge.

Spark Streaming

Spark Streaming is an engine that provides real-time distributed processing for streaming data continuously sent to Spark.
Apache Storm is a similar framework for processing streaming data. While Apache Storm is specialized for streaming data processing, Spark Streaming is Spark’s engine for real-time data processing.
Apache Flink is another streaming processing framework. Because it can also perform batch processing and includes machine learning and graph processing libraries, it has a structure quite similar to Spark and is considered a Spark competitor.

Apache Storm
http://storm.apache.org/

Apache Flink
http://flink.apache.org/introduction.html

MLlib

MLlib is Spark’s machine learning library. It allows users to write programs that perform machine learning using Spark’s flexible processing style.
Before this, Mahout existed as software for machine learning in cooperation with Hadoop, but Hadoop + Mahout required machine learning programs to be written using the MapReduce programming model, which caused slower processing.
Spark can process faster than Hadoop, and machine learning using Spark and MLlib is attracting attention because it is efficient.

Apache Mahout
http://mahout.apache.org/

GraphX

GraphX provides APIs for parallel processing of graph data through Spark.
It enables parallel processing of graph data with Spark’s fast processing speed.

Some components described above do not include storage components. Spark can use various existing storage systems for reading and writing. The following are some storage systems that can be integrated with Spark, including through third-party libraries.

HDFS, Cassandra, HBase, S3, MongoDB, Couchbase, Riak, Neo4j, OrientDB

Readable data sources also vary widely, from files such as CSV and XML to search engines such as Solr and Elasticsearch.

List of packages for integration between Spark and various data sources
https://spark-packages.org/?q=tags%3A%22Data%20Sources%22

In addition to packages that enable integration with various data sources, various packages for extending the existing Spark ecosystem are also provided. These packages are published as Spark Packages at the following site.
https://spark-packages.org/

Apache Spark Operating Environment

Spark officially supports the following OS environments. Java must also be installed to run Spark.

Major Linux distributions
Windows
Linux
MacOSX

Versions supported by the APIs provided by Spark are as follows.

Java 8, 11, 17 (versions below Java 8u201 are deprecated in Spark 3.2.0)
Scala 2.12, 2.13 (Spark 3.3.0 must use a compatible Scala 2.12.x version)
Python 3.7 or later (with Python 3.9, Apache Arrow and pandas UDFs might not work)
R 3.5 or later

Apache Spark License Type

Spark is one of Apache’s top-level projects.
The license is Apache License 2.0, and users are not restricted in using, distributing, modifying, or distributing derivative versions of the software.

Apache Spark Reference Information

It is provided by Databricks, a company started by Spark developers.

PySpark Concepts and Key Features

kc@example.com (kc kim) — Fri, 06 Jan 2023 12:36:13 +0900

What Is PySpark?

PySpark is the Python API for Apache Spark, an open source distributed computing framework and set of libraries for real-time large-scale data processing. If you are already familiar with Python and libraries such as Pandas, PySpark is a good language for learning how to build more scalable analytics and pipelines.

Apache Spark is basically a compute engine that works with huge datasets by processing them in parallel and batch systems. Spark is written in Scala, and PySpark was released to support collaboration between Spark and Python. In addition to providing APIs for Spark, PySpark uses the Py4J library to support interfaces with RDDs (Resilient Distributed Datasets).

The main data type used in PySpark is the Spark DataFrame. This object can be thought of as a table distributed across a cluster, and it has functionality similar to data frames in R and Pandas. To perform distributed computation with PySpark, you must run operations on Spark DataFrames rather than other Python data types.

One of the main differences between Pandas and Spark DataFrames is eager execution versus lazy execution. In PySpark, operations are delayed until results are actually requested in the pipeline. For example, you can specify a job that loads a dataset from Amazon S3 and applies multiple transformations to a DataFrame, but those operations are not applied immediately. Instead, the transformation graph is recorded, and when the data is actually needed, such as when writing results back to S3, the transformations are applied as a single pipeline job. This approach is used to avoid bringing the entire DataFrame into memory and to enable more effective processing across a cluster of systems. With Pandas DataFrames, everything is brought into memory and all Pandas operations are applied immediately.

PySpark Features and Libraries

Py4J is a widely used library integrated into PySpark that enables Python to dynamically interface with JVM (Java Virtual Machine) objects. PySpark provides many libraries for writing efficient programs. It is also compatible with various external libraries, including the following.

PySparkSQL

PySparkSQL is a PySpark library that applies SQL-like analysis to large volumes of structured or semi-structured data. SQL queries can also be used with PySparkSQL.

MLlib

MLlib is the wrapper machine learning (ML) library for PySpark and Spark. MLlib supports many machine learning algorithms for classification, regression, clustering, collaborative filtering, dimensionality reduction, and basic optimization primitives.

GraphFrames

GraphFrames is a graph processing library that provides a set of APIs for efficiently performing graph analysis using PySpark Core and PySparkSQL. It is optimized for fast distributed computing.

Conclusion

For data engineers who know Python but not Scala, PySpark is much easier to use than pure Spark, but it also has drawbacks. PySpark errors show both Java stack trace errors and references to Python code, so debugging PySpark applications can be very difficult.

Spark includes more processing overhead and more complex setup than other data processing options. Ray and Dask have emerged recently. Because Dask is a pure Python framework, most data engineers can use Dask immediately.

devkuma – BigData

Apache Hadoop

Apache Hadoop Overview

Hadoop Features

HDFS (Hadoop Distributed File System)

MapReduce

Hadoop Common

YARN (Yet Another Resource Negotiator)

Apache Ozone

Apache Submarine

Hadoop Use Cases

Hadoop Operating Environment

Operating Systems Where Hadoop Runs

Hadoop License

Hadoop Official Site

Hadoop Download

Three Layers That Make Up Hadoop

Distributed Processing Engine

Resource Manager

Distributed File System

Hadoop Ecosystem List

Apache Spark

Apache Spark Overview

Key Features of Apache Spark

Apache Spark Component Structure

Spark Core

Spark SQL + DataFrames

Spark Streaming

MLlib

GraphX

Apache Spark Operating Environment

Apache Spark License Type

Apache Spark Reference Information

PySpark Concepts and Key Features

What Is PySpark?

PySpark Features and Libraries

PySparkSQL

MLlib

GraphFrames

Conclusion

References