How to use multiple threads to speed processing

[George Schrader]

I am new to Java Threads. I would like to use threads to speed the processing of the following program. This is the pojo that does the work:

import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.InputStream;
import java.math.BigInteger;
import java.security.MessageDigest;
import java.util.ArrayList;
import java.util.HashMap;

public class FileFinder {

    private File originalFileObject;
    private File fileObject;
    private HashMap<String,String> allFiles;
    private ArrayList<String> duplicateFiles;
    private int directoryCount = 0;
    private int fileCount = 0;

    public FileFinder(File fileObject)
    {
        this.originalFileObject = fileObject;
        this.fileObject = fileObject;
        this.allFiles = new HashMap<String,String>();
        this.duplicateFiles = new ArrayList<String>();
    }

    public void traverse()
    {
        recursiveTraversal(fileObject);
    }

    public void recursiveTraversal(File fileObject)
    {		
        if (fileObject.isDirectory())
        {
        	directoryCount++;
            File allFiles[] = fileObject.listFiles();
            for(File aFile : allFiles)
            {
                recursiveTraversal(aFile);
            }
        }
        else if (fileObject.isFile())
        {
            fileCount++;
        	try
        	{
	        	InputStream is = new FileInputStream(fileObject);
	        	String hashedFile = getHashedFileString(is);
	        	String filePath = fileObject.getAbsolutePath();
	        	String fileName = fileObject.getName();
	        	if (allFiles.containsKey(hashedFile))
	        	{
	        		//System.out.println("Found duplicate file!");
	        		duplicateFiles.add(filePath);
	        	}
	        	else
	        	{
	        		//System.out.println("No duplicate found.");
	        		allFiles.put(hashedFile,filePath);
        			allFiles.put(filePath,fileName);
	        	}
        	}
        	catch (FileNotFoundException e)
        	{
        		e.printStackTrace(System.err);
        	}
        }		
    }
    
    public ArrayList<String> getDuplicateFiles ()
    {
    	return duplicateFiles;
    }
    public int getDirectoryCount()
    {
    	return directoryCount;
    }
    public int getFileCount()
    {
    	return fileCount;
    }
    	private String getHashedFileString(InputStream is ) 
	{        
		String output;        
		int read;        
		byte[] buffer = new byte[8192];        
		try 
		{            
			MessageDigest digest = MessageDigest.getInstance("SHA-256"); //"MD5");            
			while ((read = is.read(buffer)) > 0) 
			{                
				digest.update(buffer, 0, read);            
			}            
			byte[] hash = digest.digest();            
			BigInteger bigInt = new BigInteger(1, hash);            
			output = bigInt.toString(16);        
		}         
		catch (Exception e) 
		{            
			e.printStackTrace( System.err );            
			return null;        
		}        
		return output;    
	}
}

The bottleneck appears to be in the getHashedFileString method. This is the main class:

import java.io.File;
import java.util.ArrayList;

public class FileFinderTest {
    public static void main(String[] args) 
    {

        long begin = System.currentTimeMillis();
        String folderPath = "C:\\Program Files (x86)";
        System.out.println("Traversing directories");
        FileFinder ff = new FileFinder(new File(folderPath));
        ff.traverse();
        ArrayList<String> identicalFiles = ff.getDuplicateFiles();
        System.out.println("Directories: "+ff.getDirectoryCount());
        System.out.println("Files : "+ff.getFileCount());
        System.out.println("Duplicate files: "+identicalFiles.size());
        long end = System.currentTimeMillis();
        long diff = (end - begin)/1000/60;
        System.out.println("Time: "+diff+" minutes");
    }
}

Is there a way to speed up the processing using threads?

Thanks in advance!

[Chris Hurst]

NIO vs IO, your currently use old style blocking IO.

I'd have a look at some of the excellent comms libs , Grizzly / MINA / Netty etc and use one of them, you may find one of them has a ready made solution for this.

There's no reason why your next read can't be occurring while your last read is being processed in parallel, how much that will help with performance I can't comment. Your probably better posting in one of the IO libs forums as well.

[Edward Harned]

Take a look at this article:
Fork-Join Development in Java SE

Ed