Jump to content

Comparison of distributed file systems

Add links
From Wikipedia, the free encyclopedia
This is an old revision of this page, as edited by 2001:14bb:190:2476:ddb2:9e81:3b60:cb66 (talk) at 12:47, 12 September 2020 (→Comparison: Simplified, corrected text). The present address (URL) is a permanent link to this revision, which may differ significantly from the current revision.
See also: Comparison of distributed parallel fault-tolerant file systems

In computing, a distributed file system (DFS) or network file system is any file system that allows access to files from multiple hosts sharing via a computer network. This makes it possible for multiple users on multiple machines to share files and storage resources.

Distributed file systems differ in their performance, mutability of content, handling of concurrent writes, handling of permanent or temporary loss of nodes or storage, and their policy of storing content.

Locally managed

FOSS

Client Written in License Access API High availability Shards Initial release year Memory requirements (GB)
Alluxio (Virtual Distributed File System) Java Apache License 2.0 HDFS, FUSE, HTTP/REST, S3 hot standby No 2013
Ceph C++ LGPL librados (C, C++, Python, Ruby), S3, Swift, FUSE Yes Yes 2010 1 per TB of storage
Coda C GPL C Yes Yes 1987
GlusterFS C GPLv3 libglusterfs, FUSE, NFS, SMB, Swift, libgfapi Yes Yes 2005
MooseFS C GPLv2 POSIX, FUSE master No 2008
Quantcast File System C Apache License 2.0 C++ client, FUSE (C++ server: MetaServer and ChunkServer are both in C++) master No 2012
Kertish-DFS Go GPLv3 HTTP(REST), CLI, C# Client, Go Client Yes 2020
LizardFS C++ GPLv3 POSIX, FUSE, NFS-Ganesha, Ceph FSAL (via libcephfs) master No 2013
Lustre C GPLv2 POSIX, NFS-Ganesha, NFS, SMB Yes Yes 2003
MinIO Go Apache Licence 2.0 AWS S3 API Yes Yes 2014
OpenAFS C IBM Public License Virtual file system, Installable File System 2000 [1]
OpenIO[2] C AGPLv3 / LGPLv3 Native (Python, C, Java), HTTP/REST, S3, Swift, FUSE (POSIX, NFS, SMB, FTP) Yes 2015 0.5
SeaweedFS Go, Java Apache License 2.0 HTTP (REST), POSIX, FUSE, S3, HDFS requires CockroachDB, undocumented config 2015
Tahoe-LAFS Python GNU GPL [3] HTTP (browser or CLI), SFTP, FTP, FUSE via SSHFS, pyfilesystem 2007
HDFS Java Apache License 2.0 Java and C client, HTTP transparent master failover No 2005
XtreemFS Java, C++ BSD License libxtreemfs (Java, C++), FUSE 2009
Ori[4] C, C++ MIT libori, FUSE 2012

Proprietary

Client Written in License Access API
BeeGFS C / C++ FRAUNHOFER FS (FhGFS) EULA,[5]

GPLv2 client

POSIX
ObjectiveFS[6] C Proprietary POSIX, FUSE
Spectrum Scale (GPFS) C, C++ Proprietary POSIX, NFS, SMB, Swift
MapR-FS C, C++ Proprietary POSIX, NFS, FUSE, S3
PanFS C, C++ Proprietary DirectFlow, POSIX, NFS, SMB/CIFS, HTTP, CLI
Infinit[7] C++ Proprietary (to be open sourced)[8] FUSE, Installable File System, NFS/SMB, POSIX, CLI, SDK (libinfinit)
Isilon OneFS C/C++ Proprietary POSIX, NFS, SMB/CIFS, HDFS, HTTP, FTP, SWIFT Object, CLI, Rest API
Scality C Proprietary FUSE, NFS, REST, AWS S3
Quobyte Java, C++ Proprietary POSIX, FUSE, NFS, SMB/CIFS, HDFS, AWS S3, TensorFlow Plugin, CLI, Rest API

Remote access

Name Run by Access API
Amazon S3 Amazon.com HTTP (REST/SOAP)
Google Cloud Storage Google HTTP (REST)
SWIFT (part of OpenStack) Rackspace, Hewlett-Packard, others HTTP (REST)
Microsoft Azure Microsoft HTTP (REST)
IBM Cloud Object Storage IBM (formerly Cleversafe)[9] HTTP (REST)

Comparison

Some researchers have made a functional and experimental analysis of several distributed file systems including HDFS, Ceph, Gluster, Lustre and old (1.6.x) version of MooseFS, although this document is from 2013 and a lot of information are outdated (e.g. MooseFS had no HA for Metadata Server at that time).[10]

The cloud based remote distributed storage from major vendors have different APIs and different consistency models.[11]

See also

References

  1. ^ https://www.openafs.org/release/openafs-1.0.html
  2. ^ "OpenIO SDS Documentation". docs.openio.io.
  3. ^ "About Tahoe-LAFS".
  4. ^ "Ori: A Secure Distributed File System".
  5. ^ "FRAUNHOFER FS (FhGFS) END USER LICENSE AGREEMENT". Fraunhofer Society. 2012-02-22.
  6. ^ "ObjectiveFS official website".
  7. ^ "The Infinit Storage Platform".
  8. ^ "Infinit's Open Source Projects".
  9. ^ "IBM Plans to Acquire Cleversafe for Object Storage in Cloud". www-03.ibm.com. 2015-10-05. Retrieved 2019-05-06.
  10. ^ Séguin, Cyril; Depardon, Benjamin; Le Mahec, Gaël. "Analysis of Six Distributed File Systems" (PDF). HAL.
  11. ^ "Data Consistency Models of Public Cloud Storage Services: Amazon S3, Google Cloud Storage and Windows Azure Storage". SysTutorials. Retrieved 19 June 2017.
Stub icon

This computer-storage-related article is a stub. You can help Wikipedia by expanding it.

Retrieved from "https://en.wikipedia.org/w/index.php?title=Comparison_of_distributed_file_systems&oldid=978025981"