This week's book giveaway is in the Cloud/Virtualizaton forum.
We're giving away four copies of Mesos in Action and have Roger Ignazio on-line!
See this thread for details.
Win a copy of Mesos in Action this week in the Cloud/Virtualizaton forum!
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

DataNode doesnt replicate. And reduce copier fails. Can someone help me out?

 
samyukta rao
Greenhorn
Posts: 5
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Im using hadoop and terasort.
Im getting the following error.


hduser@sharan-ThinkPad:~/hadoop$ bin/hadoop jar hadoop-examples-1.1.2.jar teragen 14000000 /user/hduser/terasort-input
Warning: $HADOOP_HOME is deprecated.

Generating 14000000 using 2 maps with step of 7000000
13/06/12 12:15:23 INFO mapred.JobClient: Running job: job_201306121147_0008
13/06/12 12:15:24 INFO mapred.JobClient: map 0% reduce 0%
13/06/12 12:15:36 INFO mapred.JobClient: map 42% reduce 0%
13/06/12 12:15:38 INFO mapred.JobClient: map 56% reduce 0%
13/06/12 12:15:41 INFO mapred.JobClient: map 73% reduce 0%
13/06/12 12:15:45 INFO mapred.JobClient: map 84% reduce 0%
13/06/12 12:15:49 INFO mapred.JobClient: map 95% reduce 0%
13/06/12 12:15:53 INFO mapred.JobClient: map 100% reduce 0%
13/06/12 12:15:55 INFO mapred.JobClient: Job complete: job_201306121147_0008
13/06/12 12:15:55 INFO mapred.JobClient: Counters: 19
13/06/12 12:15:55 INFO mapred.JobClient: Job Counters
13/06/12 12:15:55 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=52881
13/06/12 12:15:55 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
13/06/12 12:15:55 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
13/06/12 12:15:55 INFO mapred.JobClient: Launched map tasks=2
13/06/12 12:15:55 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
13/06/12 12:15:55 INFO mapred.JobClient: File Input Format Counters
13/06/12 12:15:55 INFO mapred.JobClient: Bytes Read=0
13/06/12 12:15:55 INFO mapred.JobClient: File Output Format Counters
13/06/12 12:15:55 INFO mapred.JobClient: Bytes Written=1400000000
13/06/12 12:15:55 INFO mapred.JobClient: FileSystemCounters
13/06/12 12:15:55 INFO mapred.JobClient: HDFS_BYTES_READ=167
13/06/12 12:15:55 INFO mapred.JobClient: FILE_BYTES_WRITTEN=101266
13/06/12 12:15:55 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=1400000000
13/06/12 12:15:55 INFO mapred.JobClient: Map-Reduce Framework
13/06/12 12:15:55 INFO mapred.JobClient: Map input records=14000000
13/06/12 12:15:55 INFO mapred.JobClient: Physical memory (bytes) snapshot=242331648
13/06/12 12:15:55 INFO mapred.JobClient: Spilled Records=0
13/06/12 12:15:55 INFO mapred.JobClient: CPU time spent (ms)=28870
13/06/12 12:15:55 INFO mapred.JobClient: Total committed heap usage (bytes)=166133760
13/06/12 12:15:55 INFO mapred.JobClient: Virtual memory (bytes) snapshot=2319761408
13/06/12 12:15:55 INFO mapred.JobClient: Map input bytes=14000000
13/06/12 12:15:55 INFO mapred.JobClient: Map output records=14000000
13/06/12 12:15:55 INFO mapred.JobClient: SPLIT_RAW_BYTES=167




hduser@sharan-ThinkPad:~/hadoop$ bin/hadoop jar hadoop-examples-1.1.2.jar terasort /user/hduser/terasort-input /user/hduser/terasort-output
Warning: $HADOOP_HOME is deprecated.

13/06/12 12:16:18 INFO terasort.TeraSort: starting
13/06/12 12:16:19 INFO mapred.FileInputFormat: Total input paths to process : 2
13/06/12 12:16:19 INFO util.NativeCodeLoader: Loaded the native-hadoop library
13/06/12 12:16:19 WARN snappy.LoadSnappy: Snappy native library not loaded
13/06/12 12:16:19 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
13/06/12 12:16:19 INFO compress.CodecPool: Got brand-new compressor
Making 1 from 100000 records
Step size is 100000.0
13/06/12 12:16:20 INFO mapred.FileInputFormat: Total input paths to process : 2
13/06/12 12:16:20 INFO mapred.JobClient: Running job: job_201306121147_0009
13/06/12 12:16:21 INFO mapred.JobClient: map 0% reduce 0%
13/06/12 12:16:29 INFO mapred.JobClient: map 9% reduce 0%
13/06/12 12:16:35 INFO mapred.JobClient: map 18% reduce 0%
13/06/12 12:16:37 INFO mapred.JobClient: map 18% reduce 1%
13/06/12 12:16:40 INFO mapred.JobClient: map 18% reduce 6%
13/06/12 12:16:42 INFO mapred.JobClient: map 22% reduce 6%
13/06/12 12:16:45 INFO mapred.JobClient: map 27% reduce 6%
13/06/12 12:16:51 INFO mapred.JobClient: map 36% reduce 9%
13/06/12 12:16:57 INFO mapred.JobClient: map 36% reduce 10%
13/06/12 12:16:58 INFO mapred.JobClient: map 45% reduce 10%
13/06/12 12:17:00 INFO mapred.JobClient: map 45% reduce 12%
13/06/12 12:17:03 INFO mapred.JobClient: map 45% reduce 15%
13/06/12 12:17:07 INFO mapred.JobClient: map 49% reduce 15%
13/06/12 12:17:08 INFO mapred.JobClient: map 54% reduce 15%
13/06/12 12:17:17 INFO mapred.JobClient: map 54% reduce 18%
13/06/12 12:17:18 INFO mapred.JobClient: map 63% reduce 18%
13/06/12 12:17:28 INFO mapred.JobClient: map 68% reduce 18%
13/06/12 12:17:29 INFO mapred.JobClient: map 72% reduce 21%
13/06/12 12:17:36 INFO mapred.JobClient: map 77% reduce 21%
13/06/12 12:17:37 INFO mapred.JobClient: map 81% reduce 21%
13/06/12 12:17:39 INFO mapred.JobClient: map 81% reduce 24%
13/06/12 12:17:45 INFO mapred.JobClient: map 81% reduce 25%
13/06/12 12:17:46 INFO mapred.JobClient: map 86% reduce 25%
13/06/12 12:17:49 INFO mapred.JobClient: map 90% reduce 27%
13/06/12 12:17:52 INFO mapred.JobClient: map 90% reduce 30%
13/06/12 12:17:54 INFO mapred.JobClient: map 95% reduce 30%
13/06/12 12:17:56 INFO mapred.JobClient: map 100% reduce 30%
13/06/12 12:17:58 INFO mapred.JobClient: map 100% reduce 31%
13/06/12 12:18:12 INFO mapred.JobClient: map 100% reduce 33%
13/06/12 12:18:42 INFO mapred.JobClient: map 100% reduce 68%
13/06/12 12:18:45 INFO mapred.JobClient: map 100% reduce 70%
13/06/12 12:18:48 INFO mapred.JobClient: map 100% reduce 72%
13/06/12 12:18:52 INFO mapred.JobClient: map 100% reduce 74%
13/06/12 12:18:55 INFO mapred.JobClient: map 100% reduce 75%
13/06/12 12:18:58 INFO mapred.JobClient: map 100% reduce 76%
13/06/12 12:19:04 INFO mapred.JobClient: map 100% reduce 77%
13/06/12 12:19:07 INFO mapred.JobClient: map 100% reduce 78%
13/06/12 12:19:10 INFO mapred.JobClient: map 100% reduce 79%
13/06/12 12:19:16 INFO mapred.JobClient: Task Id : attempt_201306121147_0009_r_000000_0, Status : FAILED
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /user/hduser/terasort-output/_temporary/_attempt_201306121147_0009_r_000000_0/part-00000 could only be replicated to 0 nodes, instead of 1
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1639)
at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)

at org.apache.hadoop.ipc.Client.call(Client.java:1107)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
at sun.proxy.$Proxy2.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
at sun.proxy.$Proxy2.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3686)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3546)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2749)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2989)

attempt_201306121147_0009_r_000000_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.hdfs.DFSClient).
attempt_201306121147_0009_r_000000_0: log4j:WARN Please initialize the log4j system properly.
13/06/12 12:19:17 INFO mapred.JobClient: map 100% reduce 0%
13/06/12 12:19:24 INFO mapred.JobClient: map 100% reduce 3%
13/06/12 12:19:28 INFO mapred.JobClient: map 100% reduce 12%
13/06/12 12:19:35 INFO mapred.JobClient: map 100% reduce 15%
13/06/12 12:19:38 INFO mapred.JobClient: map 100% reduce 19%
13/06/12 12:19:41 INFO mapred.JobClient: map 100% reduce 24%
13/06/12 12:19:45 INFO mapred.JobClient: map 100% reduce 25%
13/06/12 12:19:48 INFO mapred.JobClient: map 100% reduce 31%
13/06/12 12:19:51 INFO mapred.JobClient: map 100% reduce 33%
^C^[[A
hduser@sharan-ThinkPad:~/hadoop$ bin/hadoop jar hadoop-examples-1.1.2.jar terasort /user/hduser/terasort-input /user/hduser/terasort-output
Warning: $HADOOP_HOME is deprecated.

^C
hduser@sharan-ThinkPad:~/hadoop$ bin/hadoop job -kill job_201306121147_0009
Warning: $HADOOP_HOME is deprecated.

Killed job job_201306121147_0009
hduser@sharan-ThinkPad:~/hadoop$ bin/hadoop job -kill job_201306121147_0008
Warning: $HADOOP_HOME is deprecated.

Killed job job_201306121147_0008
hduser@sharan-ThinkPad:~/hadoop$ stop all hadoop services
stop: Unknown job: all
hduser@sharan-ThinkPad:~/hadoop$ delete dfs/name
delete: command not found
hduser@sharan-ThinkPad:~/hadoop$ jps
17118 JobTracker
17024 SecondaryNameNode
16519 NameNode
16763 DataNode
17365 TaskTracker
28109 Jps
 
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic