Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Scalable Block Reporting for HopsFS
KTH, School of Electrical Engineering and Computer Science (EECS), Software and Computer systems, SCS.ORCID iD: 0000-0002-6578-3902
KTH.
Logical Clocks AB.ORCID iD: 0000-0002-1672-6899
KTH, School of Electrical Engineering and Computer Science (EECS), Software and Computer systems, SCS.ORCID iD: 0000-0002-6718-0144
Show others and affiliations
2019 (English)In: 2019 IEEE International Congress on Big Data (BigData Congress), 2019, p. 157-164Conference paper, Published paper (Refereed)
Abstract [en]

Distributed hierarchical file systems typically de- couple the storage of the file system’s metadata from the data (file system blocks) to enable the scalability of the file system. This decoupling, however, requires the introduction of a periodic synchronization protocol to ensure the consistency of the file system’s metadata and its blocks. Apache HDFS and HopsFS implement a protocol, called block reporting, where each data server periodically sends ground truth information about all its file system blocks to the metadata servers, allowing the metadata to be synchronized with the actual state of the data blocks in the file system. The network and processing overhead of the existing block reporting protocol, however, increases with cluster size, ultimately limiting cluster scalability. In this paper, we introduce a new block reporting protocol for HopsFS that reduces the protocol bandwidth and processing overhead by up to three orders of magnitude, compared to HDFS/HopsFS’ existing protocol. Our new protocol removes a major bottleneck that prevented HopsFS clusters scaling to tens of thousands of servers.

Place, publisher, year, edition, pages
2019. p. 157-164
National Category
Computer Systems
Identifiers
URN: urn:nbn:se:kth:diva-254924DOI: 10.1109/BigDataCongress.2019.00035ISBN: 978-1-7281-2771-2 (electronic)OAI: oai:DiVA.org:kth-254924DiVA, id: diva2:1336096
Conference
IEEE International Congress on Big Data, IEEE BigData Congress 2019, Milan, Italy, July 8- July 13, 2019
Note

QC 20190902

Available from: 2019-07-09 Created: 2019-07-09 Last updated: 2019-09-02Bibliographically approved

Open Access in DiVA

No full text in DiVA

Other links

Publisher's full text

Authority records BETA

Ismail, MahmoudNiazi, SalmanHaridi, SeifDowling, Jim

Search in DiVA

By author/editor
Ismail, MahmoudBonds, AugustNiazi, SalmanHaridi, SeifDowling, Jim
By organisation
Software and Computer systems, SCSKTH
Computer Systems

Search outside of DiVA

GoogleGoogle Scholar

doi
isbn
urn-nbn

Altmetric score

doi
isbn
urn-nbn
Total: 77 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf