Redirect output of sort command to a file

[Sharad N Srivastava]

Hello,

I am writing a Java program to sort the big csv file on the basis of 2nd column and redirecting the output to another file.

import java.io.File;


public class SortCsv {

	private static final String filePath="/Users/sharad/EclipseWeb/workspace/Loader/file/accountUpload.csv";
	public static void main(String[] args) throws Exception {
		File file = new File(filePath);
		String fileName=file.getName();
		String tempFile=fileName.substring(0, fileName.indexOf("."))+"_TEMP"+fileName.substring(fileName.indexOf("."));
		String newFile=fileName.substring(0, fileName.indexOf("."))+"_SORTED"+fileName.substring(fileName.indexOf("."));

                // Remove the end line characters - ^M from the file if any and store the output to the temp file
		String[] commandRemoveEndLineChar = { "/bin/sh", "-c", "cat "+file.getAbsolutePath() +" | tr \"\\r\" \"\\n\" > " + file.getParent()+"/"+tempFile };
		Runtime.getRuntime().exec(commandRemoveEndLineChar);
		
                // sort the temp file and write the output to the new file
		String commandSortFile="sort -t, -k2 "+ file.getParent()+"/"+tempFile+ " > " + file.getParent()+"/"+newFile;
	
		System.out.println(commandSortFile);

		Runtime.getRuntime().exec(commandSortFileNew1);

		System.out.println(fileName);
		System.out.println(tempFile);
		System.out.println(newFile);
	}
}

Everything works fine if the size of the input csv is less than 12KB.
Moment I increase the input csv size to more than 12 KB, the program acts weird. It just print the first 12KB of the input file to the output file and stops.

Can anyone please help me out. I am kind of stuck here.

Cheers
Sharad

[Richard Tookey]

I don't understand why you are resorting to the use of Runtime.exec() since I see nothing in your code that cannot be done in pure Java.

Having said that, addressing your immediate problem, you are not handling either the stdout or stderr streams associated with the Process objects generated by Runtime.getRuntime().exec(commandSortFileNew1) and you do not seem to be looking at the exit status code of the Process object. I know that you are not expecting anything on the Process stdout stream since you are redirecting the sort to a file but it is as well to purge it anyway.

Now why should you do all this? The stdout and stderr write to internal buffers and when either fills the Process is halted until more space becomes available BUT you are not reading from the stdout and stderr so no more space becomes available and you get a deadlock. If you do not process the Process exit code then you have no idea whether the command succeeded or failed. Now this may not be the cause of your problem but writing out the content to your console will probably give you an idea as to what is wrong.

You need to process stdout and stderr in two different threads. You can use the current execution thread for one stream and then a new thread for the other.

All this and more is explained in http://www.javaworld.com/article/2071275/core-java/when-runtime-exec---won-t.html .

The extra code required for this is quite significant and you may want to think about going back to a pure Java solution.

[Sharad N Srivastava]

Thank you Richard.

I am myself a big fan of Java. But here in my case, I am writing an upload service where I am required to read and sort a huge file (> 500 MB). Also, the performance matters.

The reason I have chosen to handle all the file related operation via 'Process' is because it is too fast and give the result in < 10ms for a file of 500MB size.

I agree, the streams were not handled properly in my above code.
Handling the streams and increasing the buffer size solved my problem.

Thanks for help.

[Richard Tookey]

Interesting. "< 10ms for a file of 500MB size" seems to me to be remarkable! I have never achieved anything like that speed .

P.S. I hope you also examined the Process exit status.

[Campbell Ritchie]

Is that sort of speed possible at all?
Are you sure the process actually completed?

[Sharad N Srivastava]

Oops, sorry for confusion here I meant 10 seconds.

Also, instead of firing multiple process from Java, I created a shell script which accepts the file name as input and does all the heavy operation like SORT/SPLIT/Logic execution and returns the result.

Thanks

[Richard Tookey]

Sharad N Srivastava wrote:
Also, instead of firing multiple process from Java, I created a shell script which accepts the file name as input and does all the heavy operation like SORT/SPLIT/Logic execution and returns the result.

You are still using Runtime.exec() so you are still creating a Process object so you still need to handle the Process stdout and stderr streams and you still need to handle the Process exit code. You may get away without doing it properly for a time but at some point it will jump up and bite you.