Jump to content

Apache Arrow

From Wikipedia, the free encyclopedia
This is an old revision of this page, as edited by Andrej Shadura (talk | contribs) at 19:53, 18 December 2019. The present address (URL) is a permanent link to this revision, which may differ significantly from the current revision.
  • Comment: The sources added don't really cover AA as the primary topic of their attention but more the topic in general. SITH (talk) 10:25, 29 May 2019 (UTC)
  • Comment: There's some relevant discussion of the sources on the talk page. Abadi's blog may meet the "published expert in the field" clause for reliable self-published sources. Huon (talk) 22:46, 13 April 2019 (UTC)
  • Comment: Nothing of substance has changed since the last time this was declined. There's one paragraph added, which is referenced to three sources which do not mee WP:RS, i.e. a blog post, etc. -- RoySmith (talk) 04:30, 30 January 2019 (UTC)
  • Comment: REVIEWERS: Please note that the submitting editor is the chief marketing officer and vice president of strategy at this company. [1] Missvain (talk) 04:25, 18 March 2018 (UTC)
Apache Arrow
Developer(s)Apache Software Foundation
Initial releaseOctober 10, 2016; 8 years ago (2016-10-10)
Stable release
v0.15.1...[1] / November 1, 2019; 5 years ago (2019-11-01)
Repositoryhttps://github.com/apache/arrow
Written inC++ (reference implementation)
TypeData format, algorithms
LicenseApache License 2.0
Websitearrow.apache.org

Apache Arrow is a cross-language development platform for in-memory columnar data. It attempts to specify a standard memory format to represent flat and hierarchical data optimised for efficient analytic operations on modern CPU and GPU hardware. The project aims to provide a common framework for data representation to be used by other libraries, systems, languages, and frameworks.[2] Arrow can be used with Apache Parquet, Apache Spark, NumPy, PySpark, pandas and other data processing libraries. The project provides an open source software library written in C++ with bindings for many other programming languages, e.g. Python and Java. Arrow allows for zero-copy reads and fast data access and interchange without serialisation overhead between these languages and systems. Arrow has been used in diverse domains, e.g. analytics,[3] genomics,[4] and cloud computing.[5] Arrow was announced by Cloudera[6] and donated to the Apache Software Foundation[7] in 2016, where it has been maintained and extended since.[7][8][9][10][11] In October 2019 the Apache Arrow team announced that it plans to split the Arrow format and library versioning starting with the planned v1.0 release [12].

Comparison to Apache Parquet and ORC

Apache Parquet and Apache ORC are popular examples of on-disk columnar data formats. Arrow is designed as a complement to these formats for processing data in-memory.[13] The hardware resource engineering trade-offs for in-memory processing vary from those associated with on-disk storage.[14] The Arrow and Parquet projects includes libraries that allow for reading and writing data between the two formats.[15]

Daniel Abadi, Darnell-Kanal Professor of Computer Science at the University of Maryland[16] and a pioneer of column-oriented databases,[17] reviewed Apache Arrow in March 2018.[18] "The time is right for database systems architects to agree on and adhere to a main memory data representation standard," he concluded. "[If your] workloads are typically scanning through a few attributes of many entities, I do not see any reason not to embrace the Arrow standard."

References

  1. ^ "Github releases".
  2. ^ "Apache Arrow and Distributed Compute with Kubernetes". 13 Dec 2018.
  3. ^ Dinsmore T.W. (2016). "In-Memory Analytics". In-Memory Analytics. In: Disruptive Analytics. Apress, Berkeley, CA. pp. 97–116. doi:10.1007/978-1-4842-1311-7_5. ISBN 978-1-4842-1312-4.
  4. ^ Versaci F, Pireddu L, Zanetti G (2016). "Scalable genomics: from raw data to aligned reads on Apache YARN" (PDF). IEEE International Conference on Big Data: 1232–1241.
  5. ^ Maas M, Asanović K, Kubiatowicz J (2017). "Return of the runtimes: rethinking the language runtime system for the cloud 3.0 era" (PDF). Proceedings of the 16th Workshop on Hot Topics in Operating Systems (ACM): 138–143. doi:10.1145/3110000/3103003/p138-Maas (inactive 2019-08-19).{{cite journal}}: CS1 maint: DOI inactive as of August 2019 (link)
  6. ^ "Introducing Apache Arrow".
  7. ^ a b Martin, Alexander J. (17 February 2016). "Apache Foundation rushes out Apache Arrow as top-level project". The Register.
  8. ^ "Big data gets a new open-source project, Apache Arrow: It offers performance improvements of more than 100x on analytical workloads, the foundation says".
  9. ^ Yegulalp, Serdar (27 February 2016). "Apache Arrow aims to speed access to big data". InfoWorld.
  10. ^ LeDem, Julien (28 November 2016). "The first release of Apache Arrow". SD Times.
  11. ^ "Julien Le Dem on the Future of Column-Oriented Data Processing with Apache Arrow".
  12. ^ pmc (2019-10-06). "Apache Arrow 0.15.0 Release". Apache Arrow. Retrieved 2019-12-18.
  13. ^ LeDem, Julien. "Apache Arrow and Apache Parquet: Why We Needed Different Projects for Columnar Data, On Disk and In-Memory". KDnuggets.
  14. ^ "Apache Arrow vs. Parquet and ORC: Do we really need a third Apache project for columnar data representation?".
  15. ^ "PyArrow:Reading and Writing the Apache Parquet Format".
  16. ^ "Daniel Abadi". Department of Computer Science, University of Maryland.
  17. ^ "Prof. Abadi Wins VLDB 10-Year Best Paper Award".
  18. ^ "An analysis of the strengths and weaknesses of Apache Arrow".
Retrieved from "https://en.wikipedia.org/w/index.php?title=Apache_Arrow&oldid=931415889"