• Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

hadoop 2.6 installation on small cluster demo

 
Jp bordi
Greenhorn
Posts: 13
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hello,

i dont know if i can find something here, but i try

i get problem with apache hadoop 2.6 (hortonwork)

i install small cluster prototype for learn my self,

1 master namenode 8GB ram, 2 slave datanode 16GB ram, i would expand my master server to 16GB, wait for memory

i configure min max mapreduce 128-2048 Mb, resource manager to 4096MB, it is enought for 2 container map reduce

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.6.0/bk_installing_manually_book/content/rpm-chap1-11.html

yarn-site.xml
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>128</value>

<name>yarn.scheduler.maximum-allocation-mb</name>
<value>2048</value>

<name>yarn.nodemanager.resource.memory-mb</name>
<value>4096</value>


mapred-site.xml

<name>mapreduce.reduce.memory.mb</name>
<value>1024</value>

<name>yarn.app.mapreduce.am.command-opts</name>
<value>-Xmx768m</value>


<name>mapreduce.map.memory.mb</name>
<value>1024</value>

<name>mapreduce.map.java.opts</name>
<value>-Xmx768m</value>


<name>yarn.app.mapreduce.am.resource.mb</name>
<value>1024</value>

<name>mapreduce.reduce.java.opts</name>
<value>-Xmx768m</value>


i have 4 core for all pc, i use only 1-2 core for map reduce, jvm are set to a max 768MB, map/reduce resources 1024MB


i installed apache hadoop 2.6, oozie 4.0.1, pig 0.13.1, hue gui,

i dont use apache ambarri

the hdfs hadoop is working fine, all node are active

but when i launch task pig (word count) with Hue gui, sometime the job is succesful working fine, but lot of time, the job hang for ever, i see no error, some time working with hear beat.

before each test i stop/start yarn cluster. the result is aleatory and i dont understand,

I think problem with configuration resource. i am surely not far to get the cluster working, next step, i would use
the ecosystem (hive,hbase,mahoot) with small demo program

thanks for feedback, remarks, idea and your support.

i continu the figth with my hadoop cluster

KR
JP
 
Jp bordi
Greenhorn
Posts: 13
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
but most of the time i fail because it can launch subtask pig or it launch to many task and wait container available

Seem to be a problem of configuration Resource Manager scheduler and cpu doesnt work normaly for distribute task on cluster
may be cpu and and scheduller



Sucessful job pig work on my cluster,

Apache Pig version 0.10.1 (r1426282)
compiled Dec 27 2012, 11:23:06

Run pig script using PigRunner.run() for Pig version 0.8+
2015-04-08 10:06:09,906 [main] INFO org.apache.pig.Main - Apache Pig version 0.10.1 (r1426282) compiled Dec 27 2012, 11:23:06
2015-04-08 10:06:09,906 [main] INFO org.apache.pig.Main - Apache Pig version 0.10.1 (r1426282) compiled Dec 27 2012, 11:23:06
2015-04-08 10:06:09,909 [main] INFO org.apache.pig.Main - Logging error messages to: /tmp/hadoop-hduser-hue/nm-local-dir/usercache/hduser/appcache/application_1428479880621_0002/container_1428479880621_0002_01_000002/pig-job_1428479880621_0002.log
2015-04-08 10:06:09,909 [main] INFO org.apache.pig.Main - Logging error messages to: /tmp/hadoop-hduser-hue/nm-local-dir/usercache/hduser/appcache/application_1428479880621_0002/container_1428479880621_0002_01_000002/pig-job_1428479880621_0002.log
2015-04-08 10:06:09,943 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - user.name is deprecated. Instead, use mapreduce.job.user.name
2015-04-08 10:06:10,372 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://stargate:9000
2015-04-08 10:06:10,372 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://stargate:9000
2015-04-08 10:06:10,380 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2015-04-08 10:06:12,418 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: GROUP_BY,ORDER_BY,FILTER
2015-04-08 10:06:12,418 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: GROUP_BY,ORDER_BY,FILTER
2015-04-08 10:06:12,641 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.textoutputformat.separator is deprecated. Instead, use mapreduce.output.textoutputformat.separator
2015-04-08 10:06:12,971 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2015-04-08 10:06:12,971 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2015-04-08 10:06:13,072 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.CombinerOptimizer - Choosing to move algebraic foreach to combiner
2015-04-08 10:06:13,072 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.CombinerOptimizer - Choosing to move algebraic foreach to combiner
2015-04-08 10:06:13,153 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 3
2015-04-08 10:06:13,153 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 3
2015-04-08 10:06:13,154 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 3
2015-04-08 10:06:13,154 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 3
2015-04-08 10:06:13,358 [main] INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at stargate/192.168.0.11:8032
2015-04-08 10:06:13,991 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2015-04-08 10:06:13,991 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2015-04-08 10:06:14,012 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.reduce.markreset.buffer.percent is deprecated. Instead, use mapreduce.reduce.markreset.buffer.percent
2015-04-08 10:06:14,012 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2015-04-08 10:06:14,012 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2015-04-08 10:06:14,013 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-04-08 10:06:14,021 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - creating jar file Job3082232122459886865.jar
2015-04-08 10:06:14,021 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - creating jar file Job3082232122459886865.jar
2015-04-08 10:06:17,861 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - jar file Job3082232122459886865.jar created
2015-04-08 10:06:17,861 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - jar file Job3082232122459886865.jar created
2015-04-08 10:06:17,862 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.jar is deprecated. Instead, use mapreduce.job.jar
2015-04-08 10:06:17,940 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2015-04-08 10:06:17,940 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2015-04-08 10:06:18,030 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=2502
2015-04-08 10:06:18,030 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=2502
2015-04-08 10:06:18,030 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
2015-04-08 10:06:18,030 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Neither PARALLEL nor default parallelism is set for this job. Setting number of reducers to 1
2015-04-08 10:06:18,030 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Neither PARALLEL nor default parallelism is set for this job. Setting number of reducers to 1
2015-04-08 10:06:18,178 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
2015-04-08 10:06:18,178 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
2015-04-08 10:06:18,179 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker.http.address is deprecated. Instead, use mapreduce.jobtracker.http.address
2015-04-08 10:06:18,179 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-04-08 10:06:18,190 [Thread-15] INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at stargate/192.168.0.11:8032
2015-04-08 10:06:18,328 [Thread-15] INFO org.apache.hadoop.conf.Configuration.deprecation - yarn.client.max-nodemanagers-proxies is deprecated. Instead, use yarn.client.max-cached-nodemanagers-proxies
2015-04-08 10:06:18,329 [Thread-15] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2015-04-08 10:06:19,088 [Thread-15] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
2015-04-08 10:06:19,089 [Thread-15] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
2015-04-08 10:06:19,089 [Thread-15] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
2015-04-08 10:06:19,114 [Thread-15] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1
2015-04-08 10:06:19,114 [Thread-15] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1
2015-04-08 10:06:19,861 [Thread-15] INFO org.apache.hadoop.mapreduce.JobSubmitter - number of splits:1
2015-04-08 10:06:20,044 [Thread-15] INFO org.apache.hadoop.mapreduce.JobSubmitter - Submitting tokens for job: job_1428479880621_0003
2015-04-08 10:06:20,045 [Thread-15] INFO org.apache.hadoop.mapreduce.JobSubmitter - Kind: mapreduce.job, Service: job_1428479880621_0002, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@265e1006)
2015-04-08 10:06:20,046 [Thread-15] INFO org.apache.hadoop.mapreduce.JobSubmitter - Kind: RM_DELEGATION_TOKEN, Service: 192.168.0.11:8032, Ident: (owner=hduser, renewer=oozie mr token, realUser=hduser, issueDate=1428480351078, maxDate=1429085151078, sequenceNumber=6, masterKeyId=2)
2015-04-08 10:06:20,922 [Thread-15] INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted application application_1428479880621_0003
2015-04-08 10:06:21,055 [Thread-15] INFO org.apache.hadoop.mapreduce.Job - The url to track the job: http://stargate:8088/proxy/application_1428479880621_0003/
2015-04-08 10:06:21,055 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_1428479880621_0003
2015-04-08 10:06:21,055 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_1428479880621_0003
2015-04-08 10:06:21,056 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - More information at: http://stargate:50030/jobdetails.jsp?jobid=job_1428479880621_0003
2015-04-08 10:06:21,056 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - More information at: http://stargate:50030/jobdetails.jsp?jobid=job_1428479880621_0003
2015-04-08 10:06:21,165 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
2015-04-08 10:06:21,165 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
2015-04-08 10:06:30,693 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 16% complete
2015-04-08 10:06:30,693 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 16% complete
Heart beat
2015-04-08 10:06:44,406 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 33% complete
2015-04-08 10:06:44,406 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 33% complete
2015-04-08 10:06:46,537 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2015-04-08 10:06:46,537 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2015-04-08 10:06:46,540 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2015-04-08 10:06:46,540 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2015-04-08 10:06:46,542 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - creating jar file Job2683633827201366673.jar
2015-04-08 10:06:46,542 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - creating jar file Job2683633827201366673.jar
2015-04-08 10:06:49,642 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - jar file Job2683633827201366673.jar created
2015-04-08 10:06:49,642 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - jar file Job2683633827201366673.jar created
2015-04-08 10:06:49,669 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2015-04-08 10:06:49,669 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2015-04-08 10:06:49,732 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
2015-04-08 10:06:49,732 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
2015-04-08 10:06:49,733 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-04-08 10:06:49,738 [Thread-29] INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at stargate/192.168.0.11:8032
2015-04-08 10:06:49,779 [Thread-29] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2015-04-08 10:06:50,266 [Thread-29] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
2015-04-08 10:06:50,266 [Thread-29] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
2015-04-08 10:06:50,266 [Thread-29] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
2015-04-08 10:06:50,267 [Thread-29] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1
2015-04-08 10:06:50,267 [Thread-29] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1
2015-04-08 10:06:50,413 [Thread-29] INFO org.apache.hadoop.mapreduce.JobSubmitter - number of splits:1
2015-04-08 10:06:50,545 [Thread-29] INFO org.apache.hadoop.mapreduce.JobSubmitter - Submitting tokens for job: job_1428479880621_0004
2015-04-08 10:06:50,545 [Thread-29] INFO org.apache.hadoop.mapreduce.JobSubmitter - Kind: mapreduce.job, Service: job_1428479880621_0002, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@77a793df)
2015-04-08 10:06:50,546 [Thread-29] INFO org.apache.hadoop.mapreduce.JobSubmitter - Kind: RM_DELEGATION_TOKEN, Service: 192.168.0.11:8032, Ident: (owner=hduser, renewer=oozie mr token, realUser=hduser, issueDate=1428480351078, maxDate=1429085151078, sequenceNumber=6, masterKeyId=2)
2015-04-08 10:06:50,846 [Thread-29] INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted application application_1428479880621_0004
2015-04-08 10:06:50,856 [Thread-29] INFO org.apache.hadoop.mapreduce.Job - The url to track the job: http://stargate:8088/proxy/application_1428479880621_0004/
2015-04-08 10:06:50,857 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_1428479880621_0004
2015-04-08 10:06:50,857 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_1428479880621_0004
2015-04-08 10:06:50,857 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - More information at: http://stargate:50030/jobdetails.jsp?jobid=job_1428479880621_0004
2015-04-08 10:06:50,857 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - More information at: http://stargate:50030/jobdetails.jsp?jobid=job_1428479880621_0004
2015-04-08 10:07:00,615 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 50% complete
2015-04-08 10:07:00,615 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 50% complete
2015-04-08 10:07:06,849 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 66% complete
2015-04-08 10:07:06,849 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 66% complete
Heart beat
2015-04-08 10:07:11,533 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2015-04-08 10:07:11,533 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2015-04-08 10:07:11,535 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2015-04-08 10:07:11,535 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2015-04-08 10:07:11,537 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - creating jar file Job6609113403299452445.jar
2015-04-08 10:07:11,537 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - creating jar file Job6609113403299452445.jar
2015-04-08 10:07:14,599 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - jar file Job6609113403299452445.jar created
2015-04-08 10:07:14,599 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - jar file Job6609113403299452445.jar created
2015-04-08 10:07:14,608 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2015-04-08 10:07:14,608 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2015-04-08 10:07:14,658 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
2015-04-08 10:07:14,658 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
2015-04-08 10:07:14,658 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-04-08 10:07:14,662 [Thread-44] INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at stargate/192.168.0.11:8032
2015-04-08 10:07:14,697 [Thread-44] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2015-04-08 10:07:15,112 [Thread-44] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
2015-04-08 10:07:15,112 [Thread-44] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
2015-04-08 10:07:15,112 [Thread-44] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
2015-04-08 10:07:15,113 [Thread-44] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1
2015-04-08 10:07:15,113 [Thread-44] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1
2015-04-08 10:07:15,253 [Thread-44] INFO org.apache.hadoop.mapreduce.JobSubmitter - number of splits:1
2015-04-08 10:07:15,370 [Thread-44] INFO org.apache.hadoop.mapreduce.JobSubmitter - Submitting tokens for job: job_1428479880621_0005
2015-04-08 10:07:15,370 [Thread-44] INFO org.apache.hadoop.mapreduce.JobSubmitter - Kind: mapreduce.job, Service: job_1428479880621_0002, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@4448ca69)
2015-04-08 10:07:15,370 [Thread-44] INFO org.apache.hadoop.mapreduce.JobSubmitter - Kind: RM_DELEGATION_TOKEN, Service: 192.168.0.11:8032, Ident: (owner=hduser, renewer=oozie mr token, realUser=hduser, issueDate=1428480351078, maxDate=1429085151078, sequenceNumber=6, masterKeyId=2)
2015-04-08 10:07:15,608 [Thread-44] INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted application application_1428479880621_0005
2015-04-08 10:07:15,616 [Thread-44] INFO org.apache.hadoop.mapreduce.Job - The url to track the job: http://stargate:8088/proxy/application_1428479880621_0005/
2015-04-08 10:07:15,617 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_1428479880621_0005
2015-04-08 10:07:15,617 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_1428479880621_0005
2015-04-08 10:07:15,617 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - More information at: http://stargate:50030/jobdetails.jsp?jobid=job_1428479880621_0005
2015-04-08 10:07:15,617 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - More information at: http://stargate:50030/jobdetails.jsp?jobid=job_1428479880621_0005
2015-04-08 10:07:25,239 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 83% complete
2015-04-08 10:07:25,239 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 83% complete
Heart beat
2015-04-08 10:07:46,040 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
2015-04-08 10:07:46,040 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
2015-04-08 10:07:46,051 [main] INFO org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:

HadoopVersion PigVersion UserId StartedAt FinishedAt Features
2.6.0 0.10.1 hduser 2015-04-08 10:06:13 2015-04-08 10:07:46 GROUP_BY,ORDER_BY,FILTER

Success!

Job Stats (time in seconds):
JobId Maps Reduces MaxMapTime MinMapTIme AvgMapTime MaxReduceTime MinReduceTime AvgReduceTime Alias Feature Outputs
job_1428479880621_0003 1 1 3 3 3 10 10 10 filtered_words,input_lines,word_count,word_groups,words GROUP_BY,COMBINER
job_1428479880621_0004 1 1 2 2 2 3 3 3 ordered_word_count SAMPLER
job_1428479880621_0005 1 1 3 3 3 14 14 14 ordered_word_count ORDER_BY /usr/hadoop/numberwords,

Input(s):
Successfully read 10 records (2907 bytes) from: "/usr/hadoop/compare_journal_216781270000000_20141014_154039.output"

Output(s):
Successfully stored 11 records (102 bytes) in: "/usr/hadoop/numberwords"

Counters:
Total records written : 11
Total bytes written : 102
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0

Job DAG:
job_1428479880621_0003 -> job_1428479880621_0004,
job_1428479880621_0004 -> job_1428479880621_0005,
job_1428479880621_0005


2015-04-08 10:07:46,051 [main] INFO org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:

HadoopVersion PigVersion UserId StartedAt FinishedAt Features
2.6.0 0.10.1 hduser 2015-04-08 10:06:13 2015-04-08 10:07:46 GROUP_BY,ORDER_BY,FILTER

Success!

Job Stats (time in seconds):
JobId Maps Reduces MaxMapTime MinMapTIme AvgMapTime MaxReduceTime MinReduceTime AvgReduceTime Alias Feature Outputs
job_1428479880621_0003 1 1 3 3 3 10 10 10 filtered_words,input_lines,word_count,word_groups,words GROUP_BY,COMBINER
job_1428479880621_0004 1 1 2 2 2 3 3 3 ordered_word_count SAMPLER
job_1428479880621_0005 1 1 3 3 3 14 14 14 ordered_word_count ORDER_BY /usr/hadoop/numberwords,

Input(s):
Successfully read 10 records (2907 bytes) from: "/usr/hadoop/compare_journal_216781270000000_20141014_154039.output"

Output(s):
Successfully stored 11 records (102 bytes) in: "/usr/hadoop/numberwords"

Counters:
Total records written : 11
Total bytes written : 102
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0

Job DAG:
job_1428479880621_0003 -> job_1428479880621_0004,
job_1428479880621_0004 -> job_1428479880621_0005,
job_1428479880621_0005


2015-04-08 10:07:47,057 [main] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: stargate/192.168.0.11:42269. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1000 MILLISECONDS)
2015-04-08 10:07:48,059 [main] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: stargate/192.168.0.11:42269. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1000 MILLISECONDS)
2015-04-08 10:07:49,060 [main] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: stargate/192.168.0.11:42269. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1000 MILLISECONDS)
2015-04-08 10:07:49,179 [main] INFO org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2015-04-08 10:07:52,427 [main] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: stargate/192.168.0.11:36625. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1000 MILLISECONDS)
2015-04-08 10:07:53,428 [main] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: stargate/192.168.0.11:36625. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1000 MILLISECONDS)
2015-04-08 10:07:54,429 [main] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: stargate/192.168.0.11:36625. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1000 MILLISECONDS)
2015-04-08 10:07:54,535 [main] INFO org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2015-04-08 10:07:55,254 [main] INFO org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
 
Jp bordi
Greenhorn
Posts: 13
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

after analysing logs, i remark one thing, if i launch with the gui hue the oozie task / pig

if the NM deploy the container at first on Master and after on slave, that work fine

if the NM try to deploy container on slave at first it hang for ever on default_container_executor.sh
and i must kill job



job Succes, NM log master
2015-04-08 10:05:53,601 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash 1="/tmp/hadoop-hduser-hue/nm-local-dir/usercache/hduser/appcache/application_1428479880621_0002/container_1428479880621_0002_01_000001/default_container_executor.sh" language=","][/bash]
2015-04-08 10:05:56,287 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_1428479880621_0002_01_000001
2015-04-08 10:05:56,340 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 8655 for container-id container_1428479880621_0002_01_000001: 167.8 MB of 2 GB physical memory used; 1.6 GB of 4.2 GB virtual memory used
1 by user hduser
2015-04-08 10:06:21,137 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1428479880621_0003
2015-04-08 10:06:21,137 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hduser IP=192.168.0.11 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1428479880621_0003 CONTAINERID=container_1428479880621_0003_01_000001
2

job FAIL, NM Log slave, hang for ever
015-04-08 11:13:08,846 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1428482388587_0011_02_000001 transitioned from LOCALIZED to RUNNING
2015-04-08 11:13:08,956 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash 1="/tmp/hadoop-hduser-hue/nm-local-dir/usercache/hduser/appcache/application_1428482388587_0011/container_1428482388587_0011_02_000001/default_container_executor.sh" language=","][/bash]
2015-04-08 11:13:11,598 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_1428482388587_0011_02_000001
2015-04-08 11:13:11,599 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_1428482388587_0011_01_000001
2015-04-08 11:13:11,628 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 7293 for container-id container_1428482388587_0011_02_000001: 94.4 MB of 2 GB physical memory used; 1.6 GB of 4.2 GB virtual memory used
 
Jp bordi
Greenhorn
Posts: 13
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
i identify partialy the problem, when oozie job launch pig container on master, it is working fine,

but when it try to deploy the container on slave node, i get retrying connection to resourcemanager.scheduler address 0.0.0.0:8030 in the pig task,

The problem the yarn-site.xml and is correctly define on master and the 2 node, they are same file for all nodes. that include also the
resourcemanager.hostname with my server ip master

<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>stargate</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>${yarn.resourcemanager.hostname}:8030</value>
</property>

but in container lauch (slave node) it fail on retrying connection and wait until it get, that mean never and it take long time for timeout

2015-04-08 12:23:19,333 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2015-04-08 12:23:19,333 INFO [IPC Server listener on 34611] org.apache.hadoop.ipc.Server: IPC Server listener on 34611: starting
2015-04-08 12:23:19,380 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: nodeBlacklistingEnabled:true
2015-04-08 12:23:19,380 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: maxTaskFailuresPerNode is 3
2015-04-08 12:23:19,380 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: blacklistDisablePercent is 33
2015-04-08 12:23:19,542 INFO [main] org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8030
2015-04-08 12:23:20,692 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)

 
Jp bordi
Greenhorn
Posts: 13
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
similar issue on 2.3.0 hadoop

http://comments.gmane.org/gmane.comp.jakarta.lucene.hadoop.user/47539

yarn-site.xml extract

<property>
<name>yarn.resourcemanager.hostname</name>
<value>stargate</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>${yarn.resourcemanager.hostname}:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>${yarn.resourcemanager.hostname}:8025</value>
<description>ResourceManager host:port for NodeManagers. If set, overrides the hostname set in yarn.resourcemanager.hostname.</description>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>${yarn.resourcemanager.hostname}:8032</value>
</property>

call url on my cluster for check configuration, expression variable est generated

http://192.168.0.11:8088/conf

<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>stargate:8030</value>
<source>programatically</source>
</property>

<property>
<name>yarn.resourcemanager.address</name>
<value>stargate:8032</value>
<source>programatically</source>
</property>


job.xml created by container on slave node, we can see some property with $yarn.resourcemanager.hostname are not evaluate correctly and some other properties are done correctly

job.xml:<property><name>yarn.resourcemanager.scheduler.address</name><value>${yarn.resourcemanager.hostname}:8030</value><source>yarn-default.xml</source></property>


here we can see correct evaluation value

job.xml:<property><name>yarn.resourcemanager.address</name><value>stargate:8032</value><source>programatically</source></property>
 
Jp bordi
Greenhorn
Posts: 13
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
The End

ok, i get it, my cluster work fine on all datatnode

i suspect oozie 4.0.1 to be in trouble with hadoop 2.6.0, i make install oozie 4.1.0 for solve my issue, but i must recompile with the correct version hdoop

http://hadooptutorial.info/apache-oozie-installation-on-ubuntu-14-04/#comment-1470

http://mockus.in/forum/viewtopic.php?t=48

but i must install maven 3.2.1 instead of maven 2.2.1 because i get fail message oozie-hadoop is duplicate in reactor

mvn clean package assembly:single -P hadoop-2 -DskipTests

i fixe all my container resource, because i installed more memory on my master server up from 8GB to 24GB ram.

python /home/hduser/yarn-util.py -c 4 -m 12 -d 3 -k False
Using cores=4 memory=12GB disks=3 hbase=False
Profile: cores=4 memory=11264MB reserved=1GB usableMem=11GB disks=3
Num Container=6
Container Ram=1536MB
Used Ram=9GB
Unused Ram=1GB
yarn.scheduler.minimum-allocation-mb=1536
yarn.scheduler.maximum-allocation-mb=9216
yarn.nodemanager.resource.memory-mb=9216
mapreduce.map.memory.mb=1536
mapreduce.map.java.opts=-Xmx1228m
mapreduce.reduce.memory.mb=3072
mapreduce.reduce.java.opts=-Xmx2457m
yarn.app.mapreduce.am.resource.mb=3072
yarn.app.mapreduce.am.command-opts=-Xmx2457m
mapreduce.task.io.sort.mb=614

Now i am very happy and i can start to play with some toys of hadoop ecosystem (hbase,hive,mahoot,pig,…)

Issue close for me
KR
JP
 
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic