Jump to content

Data mesh

From Wikipedia, the free encyclopedia
This is an old revision of this page, as edited by Mrpamm (talk | contribs) at 12:44, 12 December 2022. The present address (URL) is a permanent link to this revision, which may differ significantly from the current revision.

Data mesh is a sociotechnical approach to build a decentralized data architecture by leveraging a domain-oriented, self-serve design (in a software development perspective), and borrows Eric Evans’ theory of domain-driven design[1] and Manuel Pais’ and Matthew Skelton’s theory of team topologies.[2] Data mesh mainly concerns about the data itself, taking the Data Lake and the pipelines as a secondary concern. [3] The main proposition is scaling analytical data by domain-oriented decentralization.[4] With data mesh, the responsibility for analytical data is shifted from the central data team to the domain teams, supported by a data platform team that provides a domain-agnostic data platform.[5]

History

The term data mesh was first defined by Zhamak Dehghani in 2019[6] while she was working as a principal consultant at the technology company Thoughtworks.[7][8] Dehghani introduced the term in 2019 and then provided greater detail on its principles and logical architecture throughout 2020. The process was predicted to be a “big contender” for companies in 2022.[9][10] Data meshes have been implemented by companies such as Zalando,[11] Netflix,[12] Intuit,[13] VistaPrint and others.

Principles

Data mesh is based on four core principles:[14]

In addition to these principles, Dehghani writes that the data products created by each domain team should be discoverable, addressable, trustworthy, possess self-describing semantics and syntax, be interoperable, secure, and governed by global standards and access controls.[16] In other words, the data should be treated as a product that is ready to use and reliable.[17]

Data Maps

The term Data Maps[18] has emerged from algorithmic trading but it still has roots similar to those of Data mesh in the sense that Data Maps also considers that its nature emerges from business units. However, Data Maps combines business divisions with data science and IT to exploit synergies across projects. Unlocking synergies across projects allows it to generate algorithmic services.

See also

References

  1. ^ Evans, Eric (2004). Domain-driven design : tackling complexity in the heart of software. Boston: Addison-Wesley. ISBN 0-321-12521-5. OCLC 52134890.
  2. ^ Skelton, Matthew (2019). Team topologies : organizing business and technology teams for fast flow. Manuel Pais. Portland, OR. ISBN 978-1-942788-84-3. OCLC 1108538721.{{cite book}}: CS1 maint: location missing publisher (link)
  3. ^ Machado, Inês Araújo; Costa, Carlos; Santos, Maribel Yasmina (2022-01-01). "Data Mesh: Concepts and Principles of a Paradigm Shift in Data Architectures". Procedia Computer Science. International Conference on ENTERprise Information Systems / ProjMAN - International Conference on Project MANagement / HCist - International Conference on Health and Social Care Information Systems and Technologies 2021. 196: 263–271. doi:10.1016/j.procs.2021.12.013. ISSN 1877-0509.
  4. ^ "Data Mesh Architecture". datamesh-architecture.com. Retrieved 2022-06-13.
  5. ^ Dehghani, Zhamak (2022). Data Mesh. Sebastopol, CA. ISBN 978-1-4920-9236-0. OCLC 1260236796.{{cite book}}: CS1 maint: location missing publisher (link)
  6. ^ "How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh". martinfowler.com. Retrieved 28 January 2022.
  7. ^ Baer (dbInsight), Tony. "Data Mesh: Should you try this at home?". ZDNet. Retrieved 2022-02-10.
  8. ^ Andy Mott (2022-01-12). "Driving Faster Insights with a Data Mesh". RTInsights. Retrieved 2022-03-01.
  9. ^ "Developments that will define data governance and operational security in 2022". Help Net Security. 2021-12-28. Retrieved 2022-03-01.
  10. ^ Bane, Andy. "Council Post: Where Is Industrial Transformation Headed In 2022?". Forbes. Retrieved 2022-03-01.
  11. ^ Schultze, Max; Wider, Arif (2021). Data Mesh in Practice. ISBN 978-1-09-810849-6.
  12. ^ Netflix Data Mesh: Composable Data Processing - Justin Cunningham, retrieved 2022-04-29
  13. ^ Baker, Tristan (2021-02-22). "Intuit's Data Mesh Strategy". Intuit Engineering. Retrieved 2022-04-29.
  14. ^ Dehghani, Zhamak (2022). Data Mesh. Sebastopol, CA. ISBN 978-1-4920-9236-0. OCLC 1260236796.{{cite book}}: CS1 maint: location missing publisher (link)
  15. ^ "Data Mesh defined | James Serra's Blog". 16 February 2021. Retrieved 28 January 2022.
  16. ^ "Analytics in 2022 Means Mastery of Distributed Data Politics". The New Stack. 2021-12-29. Retrieved 2022-03-03.
  17. ^ "Developments that will define data governance and operational security in 2022". Help Net Security. 2021-12-28. Retrieved 2022-03-01.
  18. ^ "Data MAPs". www.scitheworld.com. Retrieved 2022-12-12.
Retrieved from "https://en.wikipedia.org/w/index.php?title=Data_mesh&oldid=1127017179"