Win a copy of The Little Book of Impediments (e-book only) this week in the Agile and Other Processes forum!
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

How to execute a simple program in hadoop cluster 2.6

 
Anup Wadhekar
Greenhorn
Posts: 3
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I have a Linux,Unix and a windows machine having installed hadoop 2.6 in each of them.Soon I will configure them to be in cluster so that Linux will be master node and other one or both OS will be slave.

I have been unable to execute even simple word count program in standalone mode on Linux machine.

I wish to execute any simple program in cluster.

I would be grateful if someone provide me sample program and exact steps (or pointer to appropriate resources) to execute it successfully in my cluster.

My configuration :

Hadoop 2.6
Linux 14.04 (Master)
Mac OS X 10.x (One slave)
Windows 10 (another slave)
 
Karthik Shiraly
Bartender
Posts: 1210
25
Android C++ Java Linux PHP Python
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Anup Wadhekar wrote:I have been unable to execute even simple word count program in standalone mode on Linux machine.

Have you worked through the single node setup tutorial? What problems did you face?

I feel it's better to get the single node working and familiarizing with it before jumping to cluster, because there are many more things that can go wrong in cluster.
Install single node in a VirtualBox VM if you want to keep your physical machines sanitized for the eventual cluster setup.
 
Anup Wadhekar
Greenhorn
Posts: 3
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi Karthik.. thanks for the reply.

I have successfully configured hadoop 2.6 in standalone mode and pseudo distributed mode on Linux,Mac OS X and windows 10 and I can see all required daemons using jps command.

configuration doesn't seem to be problem for me so far.

I tried once executing a sample word count program in standalone linux but couldn't execute anyhow.. don't remember the exact error/problem though.

I will again execute it and will post the exact problem.

thanks again.
 
abhi k tripathi
Greenhorn
Posts: 8
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi Anup,

Try the following simple wordcount program to test your environment -
1. Please start the hadoop services - start-dfs.sh
2. Please start the yarn services - start-yarn.sh
3. Create a input directory
$ hadoop fs -mkdir /user/hadoop/input
-- If it will give you error that directory not found-Create the parent directory first
4. Upload the input file
$ hadoop fs -put $HADOOP_INSTALL/LICENSE.txt /user/hadoop/input/License
5. Use the hadoop example jar available with hadoop installation package to run the code
$ hadoop jar $HADOOP_INSTALL/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar wordcount /user/hadoop/input/License /home/hadoop/Output
Note:- Define the input file and output directory based on your system/OS
 
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic