updating files from one repository to another

[Rish Gupta]

Hi,

I am a Java beginner and am working on a project which needs to be run once a week, what its supposed to do is updates file from one repository to another.

Lets say we have source(repository new) and destination(repository old). Now the destination needs to have all the new files copied to it that have been added to the source this week and we need to delete all the files from destination that no longer exist in source. So its basically like copying the entire source to destination but since its a large repository we need to avoid copying the files that already existed. NOTE: existing Files are never edited, only new ones are added or old ones are deleted.

I need help with the back end only for now which i want to do in java. Just to get a hang of it i tried working on a program that does the same on a local directory. this works:

import java.io.*;
import org.apache.commons.io.FileUtils;
import java.io.IOException;


public class Test {
	public static void main(String[] args) {
		
		AddOrDelete obj = new AddOrDelete();
		String[] source = {"C://filesys1"};
		String[] destination = {"C://filesys2"};
		
//---------------------------------------------------------------------		
		File f1 = new File(source[0]);
        String fileSys1[] = f1.list();
        
        File f2 = new File(destination[0]);
        String fileSys2[] = f2.list();
	
//---------------------------------------------------------------------		
        
		String[] missingFiles = obj.update(fileSys1, fileSys2);
		String[] extraFiles = obj.update(fileSys2, fileSys1);
		for(int i=0; i<missingFiles.length  && missingFiles[i]!=null; i++){
			System.out.println("add file : "+missingFiles[i]);
		}
		for(int i=0; i<extraFiles.length && extraFiles[i]!=null; i++){
			System.out.println("delete file : "+extraFiles[i]);
		}
//----------------------------------------------------------------------
		
		try{
			String orig =source[0];
			String dest = destination[0];
			for(int i=0; i<missingFiles.length && missingFiles[i]!=null; i++){
				File fOrig = new File(orig+"/"+missingFiles[i]);
				File fDest = new File(dest+"/"+missingFiles[i]);
				FileUtils.copyFile(fOrig, fDest);
			}
			
			for(int j=0; j<extraFiles.length && extraFiles[j]!=null; j++)
			{
				new File(dest+"/"+extraFiles[j]).delete();
			}
			
		}
		
		catch (IOException e){
		e.printStackTrace();
		}
			
	}
}





public class AddOrDelete {
	
	public String[] update(String[] dir1, String[] dir2){
		String[] filesToUpdate = new String[20];
		boolean copy = false;
		int k=0;
		for(int i=0; i<dir1.length; i++){
			copy=false;
			for(int j=0; j<dir2.length && !copy; j++){
				if(dir1[i].equals(dir2[j])){
					copy=true;
				}
			}
			if(copy!=true){
				filesToUpdate[k]=dir1[i];
				k++;
			}
		}
		return filesToUpdate;
	}

}

now i need to edit it for it to implement on webDAV. the problem is I am not familiar with any of the functions in java.net . can anyone tell me how to edit this to work on webDAV.

Thans.

[John Jai]

Is webDav a code maintenance tool. Can you check it it has API to support check out and check in using java?

[Tim Moores]

Why are you trying to reinvent the wheel? That's what Rsync does, not the WebDAV part, but the file syncing part.

Are both WebDAV servers in place? Do you have their URLs, usernames and passwords?

The Java class libraries do not have a WebDAV client, but searching for "java webdav library" or some such term will find several.

[Rish Gupta]

Tim Moores wrote:Why are you trying to reinvent the wheel? That's what Rsync does, not the WebDAV part, but the file syncing part.

Are both WebDAV servers in place? Do you have their URLs, usernames and passwords?

The Java class libraries do not have a WebDAV client, but searching for "java webdav library" or some such term will find several.

----------------------------------------------------------------------------------------

sorry for the late reply.
Since I am learning I kind of want to do it from scratch, and we are just working on one server where we have two folders. We just need to update one from the other and I have the user name and password. I am able to connect and get the entire file system the only part I am having trouble now is with copying a file from one folder to another. The below code works perfectly the first time but when it finds a second file that needs to copied it throws an exception.

			if(!fileExists){
					boolean flag=false;
                    //ftpFileS is of type File[] which contains listFiles from the source folder
                    //clientSrc is of type FTPClient which is working in the source directory
                    //ftpFileD is of type File[] which contains listFiles from the destination folder
                    //clientDest is of type FTPClient which is working in the destination directory
                    System.out.println("reading before inp: "+clientSrc.printWorkingDirectory()+"/"+ftpFileS.getName());
					InputStream inp=clientSrc.retrieveFileStream(ftpFileS.getName());
					System.out.println("reading after inp: "+clientSrc.printWorkingDirectory()+"/"+ftpFileS.getName());

					try{
						flag = clientDest.storeFile(ftpFileS.getName(), inp);
                         //here from what i understand it creates a new file in the destination directory with file name and contents similar to ftpFileS.getName()
						System.out.println("write success: "+flag+", "+clientDest.printWorkingDirectory()+"/"+ftpFileS.getName());
						
					}catch( IOException e){
					e.printStackTrace();
					}

					inp.close();
					}

This is the console:
reading before inp: /filesys1/Copy (2)/Copy (2) (3)/doc - Copy (2) (3) 3 - Copy.txt
reading after inp: null/doc - Copy (2) (3) 3 - Copy.txt
write success: true, /filesys2/Copy (2)/Copy (2) (3)/doc - Copy (2) (3) 3 - Copy.txt
reading before inp: /filesys1/Copy (2)/Copy (2) (3)/doc - Copy (2) (3) 4 - Copy.txt
reading after inp: null/doc - Copy (2) (3) 4 - Copy.txt
Exception in thread "main" java.lang.NullPointerException
at org.apache.commons.net.io.Util.copyStream(Util.java:100)
at org.apache.commons.net.ftp.FTPClient.__storeFile(FTPClient.java:385)
at org.apache.commons.net.ftp.FTPClient.storeFile(FTPClient.java:1379)
at FtpConnectTry.gotoDirectory(FtpConnectTry.java:12)
at FtpConnectTry.gotoDirectory(FtpConnectTry.java:89) //the only code line 89 has is a recursive call to he method
at FtpConnectTry.gotoDirectory(FtpConnectTry.java:89) //the only code line 89 has is a recursive call to he method
at FtpConnectTry.main(FtpConnectTry.java:34) //the only code line 34 has is a recursive call to he method

Also notice how it returns null in the console after the inputStream inp = new.....
Say if there are 20 files that need to be copied from source to destination my program will do exactly what it needs to if we run it 20 times, so again the problem is whenever we come into this if loop the second time it gives an error. any suggestions will be greatly appreciated.

[Campbell Ritchie]

Moving thread as too difficult for "beginning".

[Rish Gupta]

John Jai wrote:Is webDav a code maintenance tool. Can you check it it has API to support check out and check in using java?

I totally missed your comment, webdav is actually like web directory. If you access a certain url it gives you the entire directory. And there are API's for webdav, it is org.apache.webdav.ant.. But the code I wrote here is for any server to server transfer. I am probably gonna start studying the webdav API today onwards. Still, since I spent so much time on this it would be nice to know what I am doing wrong. Thanks for you interest.

[Rish Gupta]

Campbell Ritchie wrote:Moving thread as too difficult for "beginning".

Haha.. would you help me if I told you I was an expert. I just took this as project to get comfortable with java. I mean I have done calculator level programs on java before but nothing this big so far.

[Rish Gupta]

I found [FTPClient].completePendingCommand();
I am sure it has got something to do with this. If anyone knows about this please help me out. when i insert the above method after inp.close(); my program copies 2 files at a time instead of just one that it was doing earlier. I just need to figure out how to use it right.

[Winston Gutkowski]

Rish Gupta wrote:I found [FTPClient].completePendingCommand();
I am sure it has got something to do with this. If anyone knows about this please help me out. when i insert the above method after inp.close(); my program copies 2 files at a time instead of just one that it was doing earlier. I just need to figure out how to use it right.

Seems to me you're piling an awful lot of things on top of one another: directory reading, comparison, file syncing and, to top it all, FTP to do the copying.

My suggestion: back up and get each stage working before you go onto the next; and I would also use simple File I/O first and get it working with that (maybe with local directories); then only once everything else is OK, tackle the FTP part.

Also, if I were doing this, I think I'd use Files for my comparisons rather than Strings - maybe a subclass I created that can compare by names relative to a 'start directory'. That way you could use Java Sets rather than ploughing through arrays.

Winston

[Rish Gupta]

I did work with local directories. My first post was working perfectly, it did every thing.

For the FTP part I started from scratch with simply printing the directory of the source folder first.
Next step was to create blank directories which were found missing.
Then came the actual FTP file transfers into those directories and bam I hit a stone wall with no way around it.

[Rish Gupta]

import org.apache.commons.net.ftp.FTPClient;
import java.io.IOException;
import java.io.InputStream;
import java.io.PrintWriter;

import org.apache.commons.net.PrintCommandListener;
import org.apache.commons.io.FileUtils;
import org.apache.commons.net.ftp.FTPFile;
import org.apache.commons.net.ftp.FTPSClient;
import java.io.File;
import java.io.FileInputStream;
import java.net.URL;
import java.net.URLConnection;
import java.security.NoSuchAlgorithmException;

public class FtpConnectTry{
  public static void main(String[] args) {
	 
    System.out.println("Attempting...");

    try{
    	FTPClient clientSrc = new FTPClient();
    	FTPClient clientDest = new FTPClient();
    	clientSrc.connect("x.x.x.x");
    	clientDest.connect("x.x.x.x");
     	boolean loginSrc = clientSrc.login("user", "pass");
    	boolean loginDest = clientDest.login("user", "pass");
    	System.out.println("Still attempting...");
    	AddOrDelete obj = new AddOrDelete();
    	
    	clientSrc.changeWorkingDirectory("/filesys1");
    	clientDest.changeWorkingDirectory("/filesys2");
    	
    	gotoDirectory(clientSrc, clientDest);
    	        	
    	if (loginSrc) {
    		System.out.println("Login success...");
    		boolean logoutSrc = clientSrc.logout();
    		boolean logoutDest = clientDest.logout();
    		
    		if (logoutSrc) {
    			System.out.println("Logout from FTP server...src");
    		}
    		if (logoutDest) {
    			System.out.println("Logout from FTP server...dest");
    		}
    		
    	}
    	
    	else {
    		System.out.println("Login fail...");
    	}
    	
    	clientSrc.disconnect();
    }
    
    catch(IOException e){
    	e.printStackTrace();
    }
  }
  
  private static void gotoDirectory(FTPClient clientSrc, FTPClient clientDest) throws IOException {
		FTPFile[] ftpFileSrc = clientSrc.listFiles();
		FTPFile[] ftpFileDest = clientDest.listFiles();
		boolean folderExists = false;
		boolean fileExists = false;
		for (FTPFile ftpFileS : ftpFileSrc) {
			if(ftpFileS.isDirectory()){
				clientSrc.changeWorkingDirectory(ftpFileS.getName());
				for(FTPFile ftpFileD : ftpFileDest){
					System.out.println("Find directory SRC:"+ftpFileS.getName()+" || DEST:"+ftpFileD.getName());
					if((ftpFileS.getName().trim()).equals(ftpFileD.getName().trim())){
						folderExists = true;
						clientDest.changeWorkingDirectory(ftpFileS.getName());
						break;
					}
				}
				if(!folderExists){
					clientDest.makeDirectory(ftpFileS.getName());
					System.out.println("Created folder in DEST:"+ftpFileS.getName());
			    	clientDest.changeWorkingDirectory(ftpFileS.getName());
					System.out.println("Workining in dir SRC:"+clientSrc.printWorkingDirectory()+" || DEST:"+clientDest.printWorkingDirectory());
				}
				folderExists=false;
				gotoDirectory(clientSrc, clientDest);
				clientSrc.changeToParentDirectory();
				clientDest.changeToParentDirectory();
				
			}else if (ftpFileS.isFile()) {
				for(FTPFile ftpFileD : ftpFileDest){
					System.out.println("Comparing File SRC:"+ftpFileS.getName()+" || DEST:"+ftpFileD.getName());
					if((ftpFileS.getName().trim()).equals(ftpFileD.getName().trim())){
						fileExists = true;
						break;
					}
				}
				if(!fileExists){
					boolean flag=false;
					System.out.println("reading before inp: "+clientSrc.printWorkingDirectory()+"/"+ftpFileS.getName());
					InputStream inp=clientSrc.retrieveFileStream(ftpFileS.getName());
					System.out.println("read scuccess: "+clientSrc.printWorkingDirectory()+"/"+ftpFileS.getName());
					try{
						System.out.println("check for null: "+clientSrc.printWorkingDirectory()+"/"+ftpFileS.getName());

						//flag =	clientDest.storeFile(clientDest.printWorkingDirectory()+"/"+ftpFileS.getName(), inp);
						flag =	clientDest.storeFile(ftpFileS.getName(), inp);
						//flag =	clientDest.storeFile(ftpFileS.getName(), fis);
						System.out.println("write success: "+flag+", "+clientDest.printWorkingDirectory()+"/"+ftpFileS.getName());
						System.out.println("\n\n");
						
					}catch( IOException e){
					e.printStackTrace();
					}
					inp.close();
					clientSrc.completePendingCommand();
					}
				fileExists=false;
			}
		}
	}

}

Here is the entire code I am working on in case someone in the future needs to refer to some other part other than where i got stuck.

PS: the if(!fileExists) {....} part may be a lil different from what I posted earlier.