• Post Reply Bookmark Topic Watch Topic
  • New Topic

Redirect output of sort command to a file  RSS feed

 
Sharad N Srivastava
Greenhorn
Posts: 4
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hello,

I am writing a Java program to sort the big csv file on the basis of 2nd column and redirecting the output to another file.



Everything works fine if the size of the input csv is less than 12KB.
Moment I increase the input csv size to more than 12 KB, the program acts weird. It just print the first 12KB of the input file to the output file and stops.

Can anyone please help me out. I am kind of stuck here.

Cheers
Sharad
 
Richard Tookey
Bartender
Posts: 1166
17
Java Linux Netbeans IDE
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I don't understand why you are resorting to the use of Runtime.exec() since I see nothing in your code that cannot be done in pure Java.

Having said that, addressing your immediate problem, you are not handling either the stdout or stderr streams associated with the Process objects generated by Runtime.getRuntime().exec(commandSortFileNew1) and you do not seem to be looking at the exit status code of the Process object. I know that you are not expecting anything on the Process stdout stream since you are redirecting the sort to a file but it is as well to purge it anyway.

Now why should you do all this? The stdout and stderr write to internal buffers and when either fills the Process is halted until more space becomes available BUT you are not reading from the stdout and stderr so no more space becomes available and you get a deadlock. If you do not process the Process exit code then you have no idea whether the command succeeded or failed. Now this may not be the cause of your problem but writing out the content to your console will probably give you an idea as to what is wrong.

You need to process stdout and stderr in two different threads. You can use the current execution thread for one stream and then a new thread for the other.

All this and more is explained in http://www.javaworld.com/article/2071275/core-java/when-runtime-exec---won-t.html .

The extra code required for this is quite significant and you may want to think about going back to a pure Java solution.







 
Sharad N Srivastava
Greenhorn
Posts: 4
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thank you Richard.

I am myself a big fan of Java. But here in my case, I am writing an upload service where I am required to read and sort a huge file (> 500 MB). Also, the performance matters.

The reason I have chosen to handle all the file related operation via 'Process' is because it is too fast and give the result in < 10ms for a file of 500MB size.

I agree, the streams were not handled properly in my above code.
Handling the streams and increasing the buffer size solved my problem.

Thanks for help.
 
Richard Tookey
Bartender
Posts: 1166
17
Java Linux Netbeans IDE
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Interesting. "< 10ms for a file of 500MB size" seems to me to be remarkable! I have never achieved anything like that speed .

P.S. I hope you also examined the Process exit status.
 
Campbell Ritchie
Marshal
Posts: 56578
172
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Is that sort of speed possible at all?
Are you sure the process actually completed?
 
Sharad N Srivastava
Greenhorn
Posts: 4
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Oops, sorry for confusion here I meant 10 seconds.

Also, instead of firing multiple process from Java, I created a shell script which accepts the file name as input and does all the heavy operation like SORT/SPLIT/Logic execution and returns the result.

Thanks
 
Richard Tookey
Bartender
Posts: 1166
17
Java Linux Netbeans IDE
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Sharad N Srivastava wrote:
Also, instead of firing multiple process from Java, I created a shell script which accepts the file name as input and does all the heavy operation like SORT/SPLIT/Logic execution and returns the result.


You are still using Runtime.exec() so you are still creating a Process object so you still need to handle the Process stdout and stderr streams and you still need to handle the Process exit code. You may get away without doing it properly for a time but at some point it will jump up and bite you.
 
With a little knowledge, a cast iron skillet is non-stick and lasts a lifetime.
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!