Win a copy of The Little Book of Impediments (e-book only) this week in the Agile and Other Processes forum!
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

NameNode,DataNode and hdfs concepts

 
nawab wajid
Greenhorn
Posts: 13
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
1)I have fair understanding of the hadooop and MapReduce concepts .but still I need to understand these concepts on a higher technical level. I know that NameNode stores the metadata information about the data and DataNode stores the actual data.BUT I am not understanding what makes the node as NameNode and Data Node? i mean what special things do we need to make it a NameNode and DataNode?

2)What is the difference between HDFS and local file system in hadoop ? I have understood that the mapper output is stored in local file system and the reducer output is stored on the HDFS. Also if there is no reducer then the mapper output is stored directly on the HDFS .What I am not understanding is where is HDFS and where is local file system located? How do I differentiate between the two as far as hadoop is concerned?

thanks
 
Abhinav Srivastava
Ranch Hand
Posts: 354
Eclipse IDE Java Oracle
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
nawab wajid wrote:I have fair understanding of the hadooop and MapReduce concepts .but still I need to understand these concepts on a higher technical level. I know that NameNode stores the metadata information about the data and DataNode stores the actual data.BUT I am not understanding what makes the node as NameNode and Data Node? i mean what special things do we need to make it a NameNode and DataNode?


There are configuration files that specify the slave/data nodes.

nawab wajid wrote:
2)What is the difference between HDFS and local file system in hadoop ? I have understood that the mapper output is stored in local file system and the reducer output is stored on the HDFS. Also if there is no reducer then the mapper output is stored directly on the HDFS .What I am not understanding is where is HDFS and where is local file system located? How do I differentiate between the two as far as hadoop is concerned?


HDFS and Local Filesystems are non-overlapping. However, the File I/O is generic so you could use that to write to local FS as well as HDFS (see scheme in fs.default.name). The HDFS would not even be visible to the native 'ls' command.
 
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic