• Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

MapReduce and pipe input

 
steph ta
Greenhorn
Posts: 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi everyone, I come to you in order to try to solve a MapReduce program problem.

I have installed a Hadoop environment on a Linux machine.
With resources on the Internet, I managed to make a MapReduce program, which seems to work well with one exception.
When I launch the MR, it takes an argument with a filename:


the content of this input_data.txt file is the following one for example:


So, the MR treats correctly the 3 lines separately.

But, when I try to use a file which is a pipe like this:


in this case, the MR is waiting for the incoming stream, and in another shell I do the following thing:


after that, the MR always treats only the first line of the input stream, maybe the stream is stopped after the first '\n' or something like that

If someone has an idea, it will be great !

Thanks and regards,
Stephane
 
Jp bordi
Greenhorn
Posts: 13
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hello,

i dont know if i understand well your problem, but you cant check that if you can get some idea

i you need to use pipe or streaming, because it is not same way to manage image file and distribution node mr and flow stream from pipe ocmmand

https://developer.yahoo.com/hadoop/t...html#streaming

bin/hadoop pipes -input inputPath -output outputPath -program path/to/pipes/program/executable


https://hadoop.apache.org/docs/r2.6.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/HadoopStreaming.html
 
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic