Distributed Hierarchical File Systems strike back in the CloudShow others and affiliations
2020 (English)In: 2020 IEEE 40th international conference on distributed computing systems (ICDCS), Institute of Electrical and Electronics Engineers (IEEE) , 2020, p. 820-830Conference paper, Published paper (Refereed)
Abstract [en]
Cloud service providers have aligned on availability zones as an important unit of failure and replication for storage systems. An availability zone (AZ) has independent power, networking, and cooling systems and consists of one or more data centers. Multiple AZs in close geographic proximity form a region that can support replicated low latency storage services that can survive the failure of one or more AZs. Recent reductions in inter-AZ latency have made synchronous replication protocols increasingly viable, instead of traditional quorum-based replication protocols. We introduce HopsFS-CL, a distributed hierarchical file system with support for high-availability (HA) across AZs, backed by AZ-aware synchronously replicated metadata and AZ-aware block replication. HopsFS-CL is a redesign of HopsFS, a version of HDFS with distributed metadata, and its design involved making replication protocols and block placement protocols AZ-aware at all layers of its stack: the metadata serving, the metadata storage, and block storage layers. In experiments on a real-world workload from Spotify, we show that HopsFS-CL, deployed in HA mode over 3 AZs, reaches 1.66 million ops/s, and has similar performance to HopsFS when deployed in a single AZ, while preserving the same semantics.
Place, publisher, year, edition, pages
Institute of Electrical and Electronics Engineers (IEEE) , 2020. p. 820-830
Series
IEEE International Conference on Distributed Computing Systems, ISSN 1063-6927
National Category
Computer Systems
Identifiers
URN: urn:nbn:se:kth:diva-299114DOI: 10.1109/ICDCS47774.2020.00108ISI: 000667971400075Scopus ID: 2-s2.0-85101968318OAI: oai:DiVA.org:kth-299114DiVA, id: diva2:1582683
Conference
40th IEEE International Conference on Distributed Computing Systems (ICDCS), NOV 29-DEC 01, 2020, ELECTR NETWORK
Note
QC 20210803
Not duplicate with DiVA 1467134
2021-08-032021-08-032022-06-25Bibliographically approved