aspose file tools*
The moose likes Hadoop and the fly likes DataNode doesnt replicate. And reduce copier fails.  Can someone help me out? Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Databases » Hadoop
Bookmark "DataNode doesnt replicate. And reduce copier fails.  Can someone help me out?" Watch "DataNode doesnt replicate. And reduce copier fails.  Can someone help me out?" New topic
Author

DataNode doesnt replicate. And reduce copier fails. Can someone help me out?

samyukta rao
Greenhorn

Joined: Dec 02, 2012
Posts: 5
Im using hadoop and terasort.
Im getting the following error.


hduser@sharan-ThinkPad:~/hadoop$ bin/hadoop jar hadoop-examples-1.1.2.jar teragen 14000000 /user/hduser/terasort-input
Warning: $HADOOP_HOME is deprecated.

Generating 14000000 using 2 maps with step of 7000000
13/06/12 12:15:23 INFO mapred.JobClient: Running job: job_201306121147_0008
13/06/12 12:15:24 INFO mapred.JobClient: map 0% reduce 0%
13/06/12 12:15:36 INFO mapred.JobClient: map 42% reduce 0%
13/06/12 12:15:38 INFO mapred.JobClient: map 56% reduce 0%
13/06/12 12:15:41 INFO mapred.JobClient: map 73% reduce 0%
13/06/12 12:15:45 INFO mapred.JobClient: map 84% reduce 0%
13/06/12 12:15:49 INFO mapred.JobClient: map 95% reduce 0%
13/06/12 12:15:53 INFO mapred.JobClient: map 100% reduce 0%
13/06/12 12:15:55 INFO mapred.JobClient: Job complete: job_201306121147_0008
13/06/12 12:15:55 INFO mapred.JobClient: Counters: 19
13/06/12 12:15:55 INFO mapred.JobClient: Job Counters
13/06/12 12:15:55 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=52881
13/06/12 12:15:55 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
13/06/12 12:15:55 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
13/06/12 12:15:55 INFO mapred.JobClient: Launched map tasks=2
13/06/12 12:15:55 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
13/06/12 12:15:55 INFO mapred.JobClient: File Input Format Counters
13/06/12 12:15:55 INFO mapred.JobClient: Bytes Read=0
13/06/12 12:15:55 INFO mapred.JobClient: File Output Format Counters
13/06/12 12:15:55 INFO mapred.JobClient: Bytes Written=1400000000
13/06/12 12:15:55 INFO mapred.JobClient: FileSystemCounters
13/06/12 12:15:55 INFO mapred.JobClient: HDFS_BYTES_READ=167
13/06/12 12:15:55 INFO mapred.JobClient: FILE_BYTES_WRITTEN=101266
13/06/12 12:15:55 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=1400000000
13/06/12 12:15:55 INFO mapred.JobClient: Map-Reduce Framework
13/06/12 12:15:55 INFO mapred.JobClient: Map input records=14000000
13/06/12 12:15:55 INFO mapred.JobClient: Physical memory (bytes) snapshot=242331648
13/06/12 12:15:55 INFO mapred.JobClient: Spilled Records=0
13/06/12 12:15:55 INFO mapred.JobClient: CPU time spent (ms)=28870
13/06/12 12:15:55 INFO mapred.JobClient: Total committed heap usage (bytes)=166133760
13/06/12 12:15:55 INFO mapred.JobClient: Virtual memory (bytes) snapshot=2319761408
13/06/12 12:15:55 INFO mapred.JobClient: Map input bytes=14000000
13/06/12 12:15:55 INFO mapred.JobClient: Map output records=14000000
13/06/12 12:15:55 INFO mapred.JobClient: SPLIT_RAW_BYTES=167




hduser@sharan-ThinkPad:~/hadoop$ bin/hadoop jar hadoop-examples-1.1.2.jar terasort /user/hduser/terasort-input /user/hduser/terasort-output
Warning: $HADOOP_HOME is deprecated.

13/06/12 12:16:18 INFO terasort.TeraSort: starting
13/06/12 12:16:19 INFO mapred.FileInputFormat: Total input paths to process : 2
13/06/12 12:16:19 INFO util.NativeCodeLoader: Loaded the native-hadoop library
13/06/12 12:16:19 WARN snappy.LoadSnappy: Snappy native library not loaded
13/06/12 12:16:19 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
13/06/12 12:16:19 INFO compress.CodecPool: Got brand-new compressor
Making 1 from 100000 records
Step size is 100000.0
13/06/12 12:16:20 INFO mapred.FileInputFormat: Total input paths to process : 2
13/06/12 12:16:20 INFO mapred.JobClient: Running job: job_201306121147_0009
13/06/12 12:16:21 INFO mapred.JobClient: map 0% reduce 0%
13/06/12 12:16:29 INFO mapred.JobClient: map 9% reduce 0%
13/06/12 12:16:35 INFO mapred.JobClient: map 18% reduce 0%
13/06/12 12:16:37 INFO mapred.JobClient: map 18% reduce 1%
13/06/12 12:16:40 INFO mapred.JobClient: map 18% reduce 6%
13/06/12 12:16:42 INFO mapred.JobClient: map 22% reduce 6%
13/06/12 12:16:45 INFO mapred.JobClient: map 27% reduce 6%
13/06/12 12:16:51 INFO mapred.JobClient: map 36% reduce 9%
13/06/12 12:16:57 INFO mapred.JobClient: map 36% reduce 10%
13/06/12 12:16:58 INFO mapred.JobClient: map 45% reduce 10%
13/06/12 12:17:00 INFO mapred.JobClient: map 45% reduce 12%
13/06/12 12:17:03 INFO mapred.JobClient: map 45% reduce 15%
13/06/12 12:17:07 INFO mapred.JobClient: map 49% reduce 15%
13/06/12 12:17:08 INFO mapred.JobClient: map 54% reduce 15%
13/06/12 12:17:17 INFO mapred.JobClient: map 54% reduce 18%
13/06/12 12:17:18 INFO mapred.JobClient: map 63% reduce 18%
13/06/12 12:17:28 INFO mapred.JobClient: map 68% reduce 18%
13/06/12 12:17:29 INFO mapred.JobClient: map 72% reduce 21%
13/06/12 12:17:36 INFO mapred.JobClient: map 77% reduce 21%
13/06/12 12:17:37 INFO mapred.JobClient: map 81% reduce 21%
13/06/12 12:17:39 INFO mapred.JobClient: map 81% reduce 24%
13/06/12 12:17:45 INFO mapred.JobClient: map 81% reduce 25%
13/06/12 12:17:46 INFO mapred.JobClient: map 86% reduce 25%
13/06/12 12:17:49 INFO mapred.JobClient: map 90% reduce 27%
13/06/12 12:17:52 INFO mapred.JobClient: map 90% reduce 30%
13/06/12 12:17:54 INFO mapred.JobClient: map 95% reduce 30%
13/06/12 12:17:56 INFO mapred.JobClient: map 100% reduce 30%
13/06/12 12:17:58 INFO mapred.JobClient: map 100% reduce 31%
13/06/12 12:18:12 INFO mapred.JobClient: map 100% reduce 33%
13/06/12 12:18:42 INFO mapred.JobClient: map 100% reduce 68%
13/06/12 12:18:45 INFO mapred.JobClient: map 100% reduce 70%
13/06/12 12:18:48 INFO mapred.JobClient: map 100% reduce 72%
13/06/12 12:18:52 INFO mapred.JobClient: map 100% reduce 74%
13/06/12 12:18:55 INFO mapred.JobClient: map 100% reduce 75%
13/06/12 12:18:58 INFO mapred.JobClient: map 100% reduce 76%
13/06/12 12:19:04 INFO mapred.JobClient: map 100% reduce 77%
13/06/12 12:19:07 INFO mapred.JobClient: map 100% reduce 78%
13/06/12 12:19:10 INFO mapred.JobClient: map 100% reduce 79%
13/06/12 12:19:16 INFO mapred.JobClient: Task Id : attempt_201306121147_0009_r_000000_0, Status : FAILED
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /user/hduser/terasort-output/_temporary/_attempt_201306121147_0009_r_000000_0/part-00000 could only be replicated to 0 nodes, instead of 1
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1639)
at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)

at org.apache.hadoop.ipc.Client.call(Client.java:1107)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
at sun.proxy.$Proxy2.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
at sun.proxy.$Proxy2.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3686)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3546)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2749)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2989)

attempt_201306121147_0009_r_000000_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.hdfs.DFSClient).
attempt_201306121147_0009_r_000000_0: log4j:WARN Please initialize the log4j system properly.
13/06/12 12:19:17 INFO mapred.JobClient: map 100% reduce 0%
13/06/12 12:19:24 INFO mapred.JobClient: map 100% reduce 3%
13/06/12 12:19:28 INFO mapred.JobClient: map 100% reduce 12%
13/06/12 12:19:35 INFO mapred.JobClient: map 100% reduce 15%
13/06/12 12:19:38 INFO mapred.JobClient: map 100% reduce 19%
13/06/12 12:19:41 INFO mapred.JobClient: map 100% reduce 24%
13/06/12 12:19:45 INFO mapred.JobClient: map 100% reduce 25%
13/06/12 12:19:48 INFO mapred.JobClient: map 100% reduce 31%
13/06/12 12:19:51 INFO mapred.JobClient: map 100% reduce 33%
^C^[[A
hduser@sharan-ThinkPad:~/hadoop$ bin/hadoop jar hadoop-examples-1.1.2.jar terasort /user/hduser/terasort-input /user/hduser/terasort-output
Warning: $HADOOP_HOME is deprecated.

^C
hduser@sharan-ThinkPad:~/hadoop$ bin/hadoop job -kill job_201306121147_0009
Warning: $HADOOP_HOME is deprecated.

Killed job job_201306121147_0009
hduser@sharan-ThinkPad:~/hadoop$ bin/hadoop job -kill job_201306121147_0008
Warning: $HADOOP_HOME is deprecated.

Killed job job_201306121147_0008
hduser@sharan-ThinkPad:~/hadoop$ stop all hadoop services
stop: Unknown job: all
hduser@sharan-ThinkPad:~/hadoop$ delete dfs/name
delete: command not found
hduser@sharan-ThinkPad:~/hadoop$ jps
17118 JobTracker
17024 SecondaryNameNode
16519 NameNode
16763 DataNode
17365 TaskTracker
28109 Jps
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: DataNode doesnt replicate. And reduce copier fails. Can someone help me out?