Zum Inhalt springen

Apache Arrow

aus Wikipedia, der freien Enzyklopädie
Dies ist eine alte Version dieser Seite, zuletzt bearbeitet am 13. April 2020 um 08:24 Uhr durch en>Rodw (Disambiguated: Go → Go (programming language), Python → Python (programming language)). Sie kann sich erheblich von der aktuellen Version unterscheiden.

Vorlage:Undisclosed paid

Vorlage:COI

Apache Arrow

Basisdaten

Entwickler Wes McKinney, Antoine Pitrou, Sutou Kouhei, Matt Topol[1]
Erscheinungsjahr 17. Februar 2016[2]
Aktuelle Version 22.0.0[3]
(24. Oktober 2025)
Lizenz Apache-Lizenz, Version 2.0
arrow.apache.org

Apache Arrow is a language-agnostic software framework for developing data analytics applications that process columnar data. It contains a standardized column-oriented memory format that is able to represent flat and hierarchical data for efficient analytic operations on modern CPU and GPU hardware.[4][5][6][7][8] This reduces or eliminates factors that limit the feasibility of working with large sets of data, such as the cost, volatility, or physical constraints of dynamic random-access memory.[9]

Interoperability

Arrow can be used with Apache Parquet, Apache Spark, NumPy, PySpark, pandas and other data processing libraries. The project includes native software libraries written in C++, C# .NET, Go, Java, JavaScript, and Rust with bindings for other programming languages, such as Python, R, and Ruby. Arrow allows for zero-copy reads and fast data access and interchange without serialization overhead between these languages and systems.[4]

Applications

Arrow has been used in diverse domains, including analytics,[10] genomics,[11][9] and cloud computing.[12]

Comparison to Apache Parquet and ORC

Apache Parquet and Apache ORC are popular examples of on-disk columnar data formats. Arrow is designed as a complement to these formats for processing data in-memory.[13] The hardware resource engineering trade-offs for in-memory processing vary from those associated with on-disk storage.[14] The Arrow and Parquet projects includes libraries that allow for reading and writing data between the two formats.[15]

Governance

Apache Arrow was announced by The Apache Software Foundation on February 17, 2016[16], with development led by a coalition of developers from other open source data analytics projects [17][18][8][19][20]. The initial codebase and Java library was seeded by code from Apache Drill [16].

References

Vorlage:Reflist

  1. ↑ github.com.
  2. ↑ Origin and History of Apache Arrow. (abgerufen am 16. November 2025).
  3. ↑ Release 22.0.0. 24. Oktober 2025 (abgerufen am 11. November 2025).
  4. ↑ a b Apache Arrow and Distributed Compute with Kubernetes. 13. Dezember 2018;.
  5. ↑ Tony Baer: Apache Arrow: Lining Up The Ducks In A Row... Or Column. In: Seeking Alpha. 17. Februar 2016;.
  6. ↑ Tony Baer: Apache Arrow: The little data accelerator that could. In: ZDNet. 25. Februar 2019;.
  7. ↑ Susan Hall: Apache Arrow's Columnar Layouts of Data Could Accelerate Hadoop, Spark. In: The New Stack. 23. Februar 2016;.
  8. ↑ a b Serdar Yegulalp: Apache Arrow aims to speed access to big data. In: InfoWorld. 27. Februar 2016;.
  9. ↑ a b Tanveer Ahmad: ArrowSAM: In-Memory Genomics Data Processing through Apache Arrow Framework. In: bioRxiv. 2019, S. 741843, doi:10.1101/741843 (biorxiv.org).
  10. ↑ Dinsmore T.W.: In-Memory Analytics. In: Disruptive Analytics. Apress, Berkeley, CA, 2016, ISBN 978-1-4842-1312-4, In-Memory Analytics, S. 97–116, doi:10.1007/978-1-4842-1311-7_5.
  11. ↑ Versaci F, Pireddu L, Zanetti G: Scalable genomics: from raw data to aligned reads on Apache YARN. In: IEEE International Conference on Big Data. 2016, S. 1232–1241 (biorxiv.org [PDF]).
  12. ↑ Maas M, Asanović K, Kubiatowicz J: Return of the runtimes: rethinking the language runtime system for the cloud 3.0 era. In: Proceedings of the 16th Workshop on Hot Topics in Operating Systems (ACM). 2017, S. 138–143, doi:10.1145/3102980.3103003 (acm.org [PDF]).
  13. ↑ Julien Le Dem: Apache Arrow and Apache Parquet: Why We Needed Different Projects for Columnar Data, On Disk and In-Memory. In: KDnuggets.
  14. ↑ Apache Arrow vs. Parquet and ORC: Do we really need a third Apache project for columnar data representation? 31. Oktober 2017;.
  15. ↑ PyArrow:Reading and Writing the Apache Parquet Format.
  16. ↑ a b The Apache® Software Foundation Announces Apache Arrow™ as a Top-Level Project. In: The Apache Software Foundation Blog.
  17. ↑ Alexander J. Martin: Apache Foundation rushes out Apache Arrow as top-level project. In: The Register. 17. Februar 2016;.
  18. ↑ Big data gets a new open-source project, Apache Arrow: It offers performance improvements of more than 100x on analytical workloads, the foundation says. 17. Februar 2016;.
  19. ↑ Julien Le Dem: The first release of Apache Arrow. In: SD Times. 28. November 2016;.
  20. ↑ Julien Le Dem on the Future of Column-Oriented Data Processing with Apache Arrow.
Abgerufen von „https://de.wikipedia.org/w/index.php?title=Apache_Arrow&oldid=261611099“