Win a copy of Zero to AI - A non-technical, hype-free guide to prospering in the AI era this week in the Artificial Intelligence and Machine Learning forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Liutauras Vilda
  • Paul Clapham
  • Bear Bibeault
  • Jeanne Boyarsky
Sheriffs:
  • Ron McLeod
  • Tim Cooke
  • Devaka Cooray
Saloon Keepers:
  • Tim Moores
  • Tim Holloway
  • Jj Roberts
  • Stephan van Hulst
  • Carey Brown
Bartenders:
  • salvin francis
  • Scott Selikoff
  • fred rosenberger

Is it same in Hadoop to have two data nodes of 50 GB each and have 1 data node of 100 MB?

 
Ranch Foreman
Posts: 1766
9
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Is it same in Hadoop to have two data nodes of 50 GB each and have 1 data node of 100 MB? If not so which one is better(faster processing).

thanks
 
Bartender
Posts: 2407
36
Scala Python Oracle Postgres Database Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hadoop is all about distributing your data and your processing across multiple cheap machines. The data is replicated so there are e.g. 3 copies of each block of data, with diifferent copies on different machines. If you have more nodes than replicas, e.g. 3 replicas across 6 nodes, then on average each node only contains half the total original data volume. Hadoop knows where your data is replicated, so it can decide to process different subsets of your data on different nodes at the same time. This is how Hadoop allows you to exploit the power of distributed processing.

If you only have two nodes, and your replication factor is 2 or more, then each node contains all your data anyway, so Hadoop cannot decide how to break up the processing in this way. And if you only have one node, then nothing is distributed at all.
 
Ranch Hand
Posts: 765
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

chris webster wrote:Hadoop is all about distributing your data and your processing across multiple cheap machines. The data is replicated so there are e.g. 3 copies of each block of data, with diifferent copies on different machines. If you have more nodes than replicas, e.g. 3 replicas across 6 nodes, then on average each node only contains half the total original data volume. Hadoop knows where your data is replicated, so it can decide to process different subsets of your data on different nodes at the same time. This is how Hadoop allows you to exploit the power of distributed processing.

If you only have two nodes, and your replication factor is 2 or more, then each node contains all your data anyway, so Hadoop cannot decide how to break up the processing in this way. And if you only have one node, then nothing is distributed at all.



The first case you mentioned i.e of 3 replicas across 6 nodes, you mentioned Hadoop can decide what to process where.

Whereas, in your last example, i.e two nodes with replication factor is 2 or more, in this case you said, Hadoop cannot decide how to breakup processing.

My question, why in 2nd case, Hadoop cannot decide ? If both nodes are deployed on two separate machines, and one machine is loaded and not have good resources as compare to the other, then don't you think YARN will select the second machine to process the task ?

Thanks.

Viki.
 
Java Cowboy
Posts: 16084
88
Android Scala IntelliJ IDE Spring Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
One node with 100 MB will be faster than two nodes with 50 GB, because in the first case, there is 1000x as little data.

You probably meant 100 GB instead of 100 MB.
 
Monica Shiralkar
Ranch Foreman
Posts: 1766
9
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

One node with 100 MB will be faster than two nodes with 50 GB, because in the first case, there is 1000x as little data.

You probably meant 100 GB instead of 100 MB.



Yes I meant 100 GB. So will One node  of 100 GB be faster or two nodes of 50 GB each?

thanks
 
Marshal
Posts: 70598
287
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
MS: please don't edit old threads like that: add a new post saying that “MB” was a misspelling.
 
Monica Shiralkar
Ranch Foreman
Posts: 1766
9
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Campbell Ritchie wrote:MS: please don't edit old threads like that: add a new post saying that “MB” was a misspelling.



Am I supposed to reply saying “MB” was a misspelling and then edit the subject or only reply ?
 
Campbell Ritchie
Marshal
Posts: 70598
287
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Please simply say MB was a mistake. You have replied, and I think you have done everything needed.
 
Popeye has his spinach. I have this tiny ad:
the value of filler advertising in 2020
https://coderanch.com/t/730886/filler-advertising
reply
    Bookmark Topic Watch Topic
  • New Topic