Change search
ReferencesLink to record
Permanent link

Direct link
Scaling HDFS with a Strongly Consistent Relational Model for Metadata
KTH, School of Information and Communication Technology (ICT), Software and Computer systems, SCS.
KTH, School of Information and Communication Technology (ICT), Software and Computer systems, SCS.ORCID iD: 0000-0002-9484-6714
2014 (English)In: Distributed Applications and Interoperable Systems, 2014, 38-51 p.Conference paper (Refereed)
Abstract [en]

The Hadoop Distributed File System (HDFS) scales to store tens of petabytes of data despite the fact that the entire file system's metadata must fit on the heap of a single Java virtual machine. The size of HDFS' metadata is limited to under 100 GB in production, as garbage collection events in bigger clusters result in heartbeats timing out to the metadata server (NameNode). In this paper, we address the problem of how to migrate the HDFS' metadata to a relational model, so that we can support larger amounts of storage on a shared-nothing, in-memory, distributed database. Our main contribution is that we show how to provide at least as strong consistency semantics as HDFS while adding support for a multiple-writer, multiple-reader concurrency model. We guarantee freedom from deadlocks by logically organizing inodes (and their constituent blocks and replicas) into a hierarchy and having all metadata operations agree on a global order for acquiring both explicit locks and implicit locks on subtrees in the hierarchy. We use transactions with pessimistic concurrency control to ensure the safety and progress of metadata operations. Finally, we show how to improve performance of our solution by introducing a snapshotting mechanism at NameNodes that minimizes the number of roundtrips to the database.

Place, publisher, year, edition, pages
2014. 38-51 p.
, Lecture Notes in Computer Science, ISSN 0302-9743 ; 8460
National Category
Computer Science
URN: urn:nbn:se:kth:diva-155504DOI: 10.1007/978-3-662-43352-2_4ISI: 000342900800004ScopusID: 2-s2.0-84902601785ISBN: 978-3-662-43352-2; 978-3-662-43351-5OAI: diva2:761919
14th IFIP WG 6.1 International Conference on Distributed Applications and Interoperable Systems (DAIS), JUN 03-05, 2014, Berlin, GERMANY

QC 20141110

Available from: 2014-11-10 Created: 2014-11-06 Last updated: 2014-11-10Bibliographically approved

Open Access in DiVA

No full text

Other links

Publisher's full textScopus

Search in DiVA

By author/editor
Peiro Sajjad, HoomanDowling, Jim
By organisation
Software and Computer systems, SCS
Computer Science

Search outside of DiVA

GoogleGoogle Scholar
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

Altmetric score

Total: 175 hits
ReferencesLink to record
Permanent link

Direct link