Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
A Distributed Namespace for a Distributed File System
KTH, School of Information and Communication Technology (ICT).
2012 (English)Independent thesis Advanced level (degree of Master (Two Years)), 20 credits / 30 HE creditsStudent thesis
Abstract [en]

Due to the rapid growth of data in recent years, distributed file systems have gained widespread adoption. The new breed of distributed file systems reliably store petabytes of data on commodity hardware, and also provide rich abstractions for massively parallel data analytics. The Hadoop Distributed File System (HDFS) is one such system which provides the storage layer for MapReduce, Hive, HBase and Mahout. The metadata server in HDFS, called the NameNode, is a centralized server which stores information about the whole namespace. The centralized architecture not only makes the NameNode a bottleneck and a single-point-of-failure, but also restricts the overall capacity of the filesystem. To solve the availability and scalability issues of HDFS, a new architecture is required. In this report, we propose a distributed implementation of the HDFS NameNode, where the filesystem metadata is stored in a distributed, in-memory, replicated database called MySQL Cluster. The NameNodes are state-less, and the throughput of the system can be increased by either adding NameNodes, or by adding more data nodes in NDB. HDFS clients can access the metadata by connecting to any one of the NameNodes. The evaluation section shows that the new architecture, as compared to HDFS, can handle more requests per second, store ten times more files and recover from failures within a few seconds.

Place, publisher, year, edition, pages
2012. , 46 p.
Series
Trita-ICT-EX, 2012:173
National Category
Engineering and Technology
Identifiers
URN: urn:nbn:se:kth:diva-101482OAI: oai:DiVA.org:kth-101482DiVA: diva2:548037
Educational program
Master of Science - Distributed Computing
Uppsok
Technology
Examiners
Available from: 2012-08-29 Created: 2012-08-29 Last updated: 2012-08-29Bibliographically approved

Open Access in DiVA

fulltext(597 kB)756 downloads
File information
File name FULLTEXT01.pdfFile size 597 kBChecksum SHA-512
d2bd2e2bc8d08cc98bba3ad24bbe9945fd070bd9899b0394c7c9d03ed1175d3922b795ba09025ad08f534b4485f06558fa13043a2c26048ad5e8bf6af82e7141
Type fulltextMimetype application/pdf

By organisation
School of Information and Communication Technology (ICT)
Engineering and Technology

Search outside of DiVA

GoogleGoogle Scholar
Total: 756 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

urn-nbn

Altmetric score

urn-nbn
Total: 289 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf