Jump to content

Apache Spark

From Wikipedia, the free encyclopedia
This is an old revision of this page, as edited by Cclauss (talk | contribs) at 12:35, 24 June 2015 (ten times as fast as --> ten times faster than). The present address (URL) is a permanent link to this revision, which may differ significantly from the current revision.
Apache Spark
Developer(s)Apache Software Foundation, UC Berkeley AMPLab, Databricks
Stable release
v1.4.0 / June 11, 2015 (2015-06-11)
Repository
Written inScala, Java, Python
Operating systemLinux, Mac OS, Windows
Typedata analytics, machine learning algorithms
LicenseApache License 2.0
Websitespark.apache.org

Apache Spark is an open-source cluster computing framework originally developed in the AMPLab at UC Berkeley. In contrast to Hadoop's two-stage disk-based MapReduce paradigm, Spark's in-memory primitives provide performance up to 100 times faster for certain applications.[1] By allowing user programs to load data into a cluster's memory and query it repeatedly, Spark is well suited to machine learning algorithms.[2]

Spark requires a cluster manager and a distributed storage system. For cluster management, Spark supports standalone (native Spark cluster), Hadoop YARN, or Apache Mesos.[3] For distributed storage, Spark can interface with a wide variety, including Hadoop Distributed File System (HDFS),[4] Cassandra,[5] OpenStack Swift, and Amazon S3. Spark also supports a pseudo-distributed local mode, usually used only for development or testing purposes, where distributed storage is not required and the local file system can be used instead; in this scenario, Spark is running on a single machine with one executor per CPU core.

Spark had over 465 contributors in 2014,[6] making it the most active project in the Apache Software Foundation and among Big Data open source projects.

History

Spark was initially started by Matei Zaharia at UC Berkeley AMPLab in 2009, and open sourced in 2010 under a BSD license.

In 2013, the project was donated to the Apache Software Foundation and switched its license to Apache 2.0. In February 2014, Spark became an Apache Top-Level Project.[7]

In November 2014, the engineering team at Databricks used Spark and set a new world record in large scale sorting.[8]

Project Components

The Spark project consists of multiple components.

Spark Core and Resilient Distributed Datasets (RDDs)

Spark Core is the foundation of the overall project. It provides distributed task dispatching, scheduling, and basic I/O functionalities. The fundamental programming abstraction is called Resilient Distributed Datasets, a logical collection of data partitioned across machines. RDDs can be created by referencing datasets in external storage systems, or by applying coarse-grained transformations (e.g. map, filter, reduce, join) on existing RDDs.

The RDD abstraction is exposed through a language-integrated API in Java, Python, Scala similar to local, in-process collections. This simplifies programming complexity because the way applications manipulate RDDs is similar to manipulating local collections of data.

Spark SQL

Spark SQL is a component on top of Spark Core that introduces a new data abstraction called SchemaRDD, which provides support for structured and semi-structured data. Spark SQL provides a domain-specific language to manipulate SchemaRDDs in Scala, Java, or Python. It also provides SQL language support, with command-line interfaces and ODBC/JDBC server. In Spark version 1.3, SchemaRDD is renamed to DataFrame.

Spark Streaming

Spark Streaming leverages Spark Core's fast scheduling capability to perform streaming analytics. It ingests data in mini-batches and performs RDD transformations on those mini-batches of data. This design enables the same set of application code written for batch analytics to be used in streaming analytics, on a single engine.

MLlib Machine Learning Library

MLlib is a distributed machine learning framework on top of Spark that because of the distributed memory-based Spark architecture is, according to benchmarks done by the MLlib developers, ten times faster than Hadoop disk-based Apache Mahout and even scales better than Vowpal Wabbit.[9] It implements many common machine learning and statistical algorithms to simplify large scale machine learning pipelines, including:

  • summary statistics, correlations, stratified sampling, hypothesis testing, random data generation
  • classification and regression: SVMs, logistic regression, linear regression, decision trees, naive Bayes
  • collaborative filtering: alternating least squares (ALS)
  • clustering: k-means, Latent Dirichlet Allocation (LDA)
  • dimensionality reduction: singular value decomposition (SVD), principal component analysis (PCA)
  • feature extraction and transformation
  • optimization primitives: stochastic gradient descent, limited-memory BFGS (L-BFGS)

GraphX

GraphX is a distributed graph processing framework on top of Spark. It provides an API for expressing graph computation that can model the Pregel abstraction. It also provides an optimized runtime for this abstraction.

GraphX started initially as a research project at UC Berkeley AMPLab and Databricks, and was later donated to the Spark project.[10]

Features

  • Java, Scala, and Python APIs.
  • Scalability to over 8000 nodes in production.[11]
  • Ability to cache datasets in memory for interactive data analysis: extract a working set, cache it, query it repeatedly.
  • Interactive command line interface (in Scala or Python) for low-latency horizontally scalable data exploration.
  • Higher level library for stream processing, through Spark Streaming.
  • Support for structured and relational query processing (SQL), through Spark SQL.
  • Higher level libraries for machine learning and graph processing.

References

  1. ^ Xin, Reynold; Rosen, Josh; Zaharia, Matei; Franklin, Michael; Shenker, Scott; Stoica, Ion (June 2013). "Shark: SQL and Rich Analytics at Scale" (PDF). {{cite journal}}: Cite journal requires |journal= (help); Unknown parameter |conference= ignored (help)
  2. ^ Matei Zaharia. Spark: In-Memory Cluster Computing for Iterative and Interactive Applications. Invited Talk at NIPS 2011 Big Learning Workshop: Algorithms, Systems, and Tools for Learning at Scale.{{cite AV media}}: CS1 maint: location (link)
  3. ^ "Cluster Mode Overview - Spark 1.2.0 Documentation - Cluster Manager Types". apache.org. Apache Foundation. 2014-12-18. Retrieved 2015-01-18.
  4. ^ Figure showing Spark in relation to other open-source Software projects including Hadoop
  5. ^ Doan, DuyHai (2014-09-10). "Re: cassandra + spark / pyspark". Cassandra User (Mailing list). Retrieved 2014-11-21. {{cite mailing list}}: Unknown parameter |mailinglist= ignored (|mailing-list= suggested) (help)
  6. ^ Open HUB Spark development activity
  7. ^ "The Apache Software Foundation Announces Apache&#8482 Spark&#8482 as a Top-Level Project". apache.org. Apache Software Foundation. 27 February 2014. Retrieved 4 March 2014.
  8. ^ Spark officially sets a new record in large-scale sorting
  9. ^ Sparks, Evan; Talwalkar, Ameet (2013-08-06). "Spark Meetup: MLbase, Distributed Machine Learning with Spark". slideshare.net. Spark User Meetup, San Francisco, California. Retrieved 10 February 2014.
  10. ^ Gonzalez, Joseph; Xin, Reynold; Dave, Ankur; Crankshaw, Daniel; Franklin, Michael; Stoica, Ion (Oct 2014). "GraphX: Graph Processing in a Distributed Dataflow Framework" (PDF). {{cite journal}}: Cite journal requires |journal= (help); Unknown parameter |conference= ignored (help)
  11. ^ "Apache Spark FAQ". apache.org. Apache Software Foundation. Retrieved 5 December 2014.
Retrieved from "https://en.wikipedia.org/w/index.php?title=Apache_Spark&oldid=668460196"