Apache Spark

Tags:

Apache Spark Overview

Apache Spark is a distributed processing framework for cluster computing that processes large-scale data such as big data and machine learning workloads. Spark development began in 2009 at AMPLab at the University of California, Berkeley, by Mate Zaharia, who was also a Hadoop committer. It is now managed and developed as one of the top-level projects of the Apache Software Foundation.

Spark was developed to improve the slow processing speed of traditional MapReduce and to support a more flexible processing style that is not bound to repeatedly using Map and Reduce.

Spark can run independently as a distributed processing framework, so it is attracting attention as a post-Hadoop technology. At the same time, it can also be used as a replacement for MapReduce within the Hadoop core system, which consists of MapReduce, HDFS, YARN, and related components.

Key Features of Apache Spark

Spark’s major features include the ability to easily program flexible processing models using the concise APIs provided by Spark, and the ability to process large-scale data in far less time than traditional MapReduce.

In traditional MapReduce, Map and Reduce had to be performed as one processing model, so applications running on Hadoop had to be developed according to that style. This made it difficult to develop flexible processing models.

MapReduce also writes processing results to disk after each Map and Reduce operation, making it difficult to improve processing speed. In contrast, Spark can run multiple Map operations consecutively on datasets loaded into memory (RDDs), reduce the results, and then run the next Map operation on the same dataset while keeping it in memory rather than writing it to disk. For this reason, Spark can sometimes achieve more than 100 times faster processing than MapReduce, although it can also write results to disk like traditional MapReduce.

Spark’s features are as follows.

  • Speed
    • Fast in-memory processing
  • Ease of Use
    • Ease of use through support for various languages such as Java, Scala, Python, R, and SQL
  • Generality
    • Provides various components such as SQL, Streaming, machine learning, and graph computation
  • Run Everywhere
    • Can run on various clusters such as YARN, Mesos, and Kubernetes
    • Supports various file formats and storage systems such as HDFS, Cassandra, and HBase

Apache Spark Component Structure

Spark is a distributed processing framework consisting of the following components.

  • Spark Core, including Scala, Java, Python, and R APIs
  • Spark SQL + DataFrames
  • Spark Streaming
  • MLlib
  • GraphX

Spark Core

Spark keeps data to be processed in the form of RDDs (Resilient Distributed Datasets).
An RDD is an immutable collection that can be executed in parallel and is distributed across computers.
In the Spark programming model, processing is performed by applying various methods provided by Spark Core to these RDDs. When developers manipulate RDDs through the APIs provided by Spark Core, they can perform distributed processing without being conscious of distributed data.
This is one of Spark’s strengths: it makes flexible processing easy to program.
Spark Core APIs are provided not only for Scala, Spark’s development language, but also for Java, Python, and R as standard APIs. Some third-party libraries also provide Spark API access from Clojure, a functional language that runs on the JVM like Scala, and APIs for other languages are expected to continue increasing.

Spark SQL + DataFrames

In addition to manipulating RDDs through Spark APIs, Spark can also use a SQL-like language called Spark SQL to manipulate abstract datasets called DataFrames, which have named columns like database tables.
This is an interface that allows users who have not learned languages such as Scala, Java, Python, or R to process data with Spark if they have SQL knowledge.

Spark Streaming

Spark Streaming is an engine that provides real-time distributed processing for streaming data continuously sent to Spark.
Apache Storm is a similar framework for processing streaming data. While Apache Storm is specialized for streaming data processing, Spark Streaming is Spark’s engine for real-time data processing.
Apache Flink is another streaming processing framework. Because it can also perform batch processing and includes machine learning and graph processing libraries, it has a structure quite similar to Spark and is considered a Spark competitor.

Apache Storm
http://storm.apache.org/

Apache Flink
http://flink.apache.org/introduction.html

MLlib

MLlib is Spark’s machine learning library. It allows users to write programs that perform machine learning using Spark’s flexible processing style.
Before this, Mahout existed as software for machine learning in cooperation with Hadoop, but Hadoop + Mahout required machine learning programs to be written using the MapReduce programming model, which caused slower processing.
Spark can process faster than Hadoop, and machine learning using Spark and MLlib is attracting attention because it is efficient.

Apache Mahout
http://mahout.apache.org/

GraphX

GraphX provides APIs for parallel processing of graph data through Spark.
It enables parallel processing of graph data with Spark’s fast processing speed.

Some components described above do not include storage components. Spark can use various existing storage systems for reading and writing. The following are some storage systems that can be integrated with Spark, including through third-party libraries.

HDFS, Cassandra, HBase, S3, MongoDB, Couchbase, Riak, Neo4j, OrientDB

Readable data sources also vary widely, from files such as CSV and XML to search engines such as Solr and Elasticsearch.

List of packages for integration between Spark and various data sources
https://spark-packages.org/?q=tags%3A%22Data%20Sources%22

In addition to packages that enable integration with various data sources, various packages for extending the existing Spark ecosystem are also provided. These packages are published as Spark Packages at the following site.
https://spark-packages.org/

Apache Spark Operating Environment

Spark officially supports the following OS environments. Java must also be installed to run Spark.

  • Major Linux distributions
  • Windows
  • Linux
  • MacOSX

Versions supported by the APIs provided by Spark are as follows.

  • Java 8, 11, 17 (versions below Java 8u201 are deprecated in Spark 3.2.0)
  • Scala 2.12, 2.13 (Spark 3.3.0 must use a compatible Scala 2.12.x version)
  • Python 3.7 or later (with Python 3.9, Apache Arrow and pandas UDFs might not work)
  • R 3.5 or later

Apache Spark License Type

Spark is one of Apache’s top-level projects.
The license is Apache License 2.0, and users are not restricted in using, distributing, modifying, or distributing derivative versions of the software.

Apache Spark Reference Information

It is provided by Databricks, a company started by Spark developers.