What FileReader expects

[Dan Grindstaff]

Hello All, I have the following program in which FileReader expects an argument. I am wondering why I can't get away with the full ftp path to the file.

package com.motorola.pipereader;

import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;

import au.com.bytecode.opencsv.CSVReader;

public class PipeReader {

	/**
	 * @param args
	 * @throws IOException 
	 */
	public static void main(String[] args) throws IOException {
		// TODO Auto-generated method stub
		
		CSVReader reader = null;
		try {
			reader = new CSVReader(new FileReader("ftp://<some address>/coreexport_gae.full"));
		} catch (FileNotFoundException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
	    String [] nextLine;
	    while ((nextLine = reader.readNext()) != null) {
	        // nextLine[] is an array of values from the line
	        System.out.println(nextLine[0] + nextLine[1] + "etc...");
	    }

	}

}

Any help is greatly appreciated!

[Jeff Verdegan]

FR expects a file name, that is, a path to a file on the local file system. What you gave it is a URL that requires an FTP client to resolve. FR is not an FTP client, and it's not meant for handling arbitrary URLs. You can't give it an HTTP URL either, even though that may eventually resolve to the static contents of a single file.

[Dan Grindstaff]

Hi, Thank you for the response. It sounds like I will need to import the file first? But I am wondering if there is a way around this?

[Jeff Verdegan]

Dan Grindstaff wrote:Hi, Thank you for the response. It sounds like I will need to import the file first? But I am wondering if there is a way around this?

If by "import" you mean copy it to your local file system by some other means, then, yes, that's one option.

Note also that there do exist FTP clients for Java, which you can find by googling. It's just that the FileReader class isn't one of them.

[Dan Grindstaff]

Ok, I added the following code to try and fetch the file though I would rather just erad it from where it sits.

FTPClient client = new FTPClient();
		client.connect("ftp://<some address>/coreexport_gae.full");
		client.download("coreexport_gae.full", new java.io.File("coreexport.csv"));

[Jeff Verdegan]

Dan Grindstaff wrote:Ok, I added the following code to try and fetch the file though I would rather just erad it from where it sits.

Does that FTP client offer a "spit back the contents in a stream but don't save them to a file" operation? If so, then that's what you'd use. If not, then you could see if there's some other FTP client that does so, or some other service on that host making that file available where you could do that, with a corresponding client.

For example: Apache commons FTPClient.retrieveFileStream(java.lang.String)

[Dan Grindstaff]

I have switched gears, trying to find an FTP solution and THEN I can concentrate on the CSV parser. I have tried this code for FTP:

import org.apache.commons.net.ftp.FTPClient;
import java.io.IOException;
import java.io.FileOutputStream;
 
public class FtpDownloadDemo {
    public static void main(String[] args) {
        FTPClient client = new FTPClient();
        FileOutputStream fos = null;
 
        try {
            client.connect("ftp://<some address>production");
            client.login("guest", "guest");
 
            //
            // The remote filename to be downloaded.
            //
            String filename = "coreexport_gae.full";
            fos = new FileOutputStream(filename);
 
            //
            // Download file from FTP server
            //
            client.retrieveFile("/" + filename, fos);
        } catch (IOException e) {
            e.printStackTrace();
        } finally {
            try {
                if (fos != null) {
                    fos.close();
                }
                client.disconnect();
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
 
    }
}

which returns the following error:

java.net.UnknownHostException: ftp://<some address>/production
	at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
	at java.net.InetAddress$1.lookupAllHostAddr(Unknown Source)
	at java.net.InetAddress.getAddressesFromNameService(Unknown Source)
	at java.net.InetAddress.getAllByName0(Unknown Source)
	at java.net.InetAddress.getAllByName(Unknown Source)
	at java.net.InetAddress.getAllByName(Unknown Source)
	at java.net.InetAddress.getByName(Unknown Source)
	at org.apache.commons.net.SocketClient.connect(SocketClient.java:196)
	at org.apache.commons.net.SocketClient.connect(SocketClient.java:289)
	at com.motorola.pipereader.FtpDownloadDemo.main(FtpDownloadDemo.java:13)

I am completely stuck right now.

[Jeff Verdegan]

It looks like you're trying to use "ftp://<some address>/production" as the host name or address, when in fact it should just be "<some address>".

[Dan Grindstaff]

Ok, no errors when I just have the address in there, however the file I need is still a couple directories further into the url. Here is the code:

import org.apache.commons.net.ftp.FTPClient;
import java.io.IOException;
import java.io.FileOutputStream;
 
public class FtpDownloadDemo {
    public static void main(String[] args) {
        FTPClient client = new FTPClient();
        FileOutputStream fos = null;
 
        try {
            client.connect("ct11corpfsst502.am.mot.com");
            //client.login("guest", "guest");
 
            //
            // The remote filename to be downloaded.
            //
            String filename = "/ldapfeed/production/coreexport_gae.full";
            fos = new FileOutputStream(filename);
 
            //
            // Download file from FTP server
            //
            client.retrieveFile("/" + filename, fos);
        } catch (IOException e) {
            e.printStackTrace();
        } finally {
            try {
                if (fos != null) {
                    fos.close();
                }
                client.disconnect();
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
 
    }
}

and here is the error message:

java.io.FileNotFoundException: ldapfeed\production\coreexport_gae.full (The system cannot find the path specified)
	at java.io.FileOutputStream.open(Native Method)
	at java.io.FileOutputStream.<init>(Unknown Source)
	at java.io.FileOutputStream.<init>(Unknown Source)
	at com.motorola.pipereader.FtpDownloadDemo.main(FtpDownloadDemo.java:20)

[Dan Grindstaff]

Ok, this code does not throw errors:

import org.apache.commons.net.ftp.FTPClient;
import java.io.IOException;
import java.io.FileOutputStream;
 
public class FtpDownloadDemo {
    public static void main(String[] args) {
        FTPClient client = new FTPClient();
        FileOutputStream fos = null;
 
        try {
            client.connect("ct11corpfsst502.am.mot.com");
            //client.login("guest", "guest");
 
            //
            // The remote filename to be downloaded.
            //
            String filename = "coreexport_gae.full";
            fos = new FileOutputStream(filename);
 
            //
            // Download file from FTP server
            //
            client.retrieveFile("/ldapfeed/production/" + filename, fos);
        } catch (IOException e) {
            e.printStackTrace();
        } finally {
            try {
                if (fos != null) {
                    fos.close();
                }
                client.disconnect();
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
 
    }
}

But I have a stupid question. Where am I downloading the file to?

[Jeff Verdegan]

Note that the root that the FTP server shows you will not necessarily be that hosts file system root. Typically it is not. You have to either know ahead of time what the path will be relative to wherever the FTP sets your user's root at, or you have to look for it. The FTP protocol provides a directory listing operation, and I would assume your client library provides access to it.

[Dan Grindstaff]

All Right! Making progress I found the file in the code directory but it is empty! Any ideas why it would be empty?

[Jeff Verdegan]

Dan Grindstaff wrote:But I have a stupid question. Where am I downloading the file to?

Well, you have this:

            fos = new FileOutputStream(filename);  

So it's going to the path specified by whatever you set filename to. If you didn't make it an absolute path, it will be relative to whatever the current directory was for the JVM at the time it was started. It's better not to rely on that, but to specify an absolute path instead. Or, if you have to rely on it, you can ask the File object referred to by filename what its absolute path is.

[Jeff Verdegan]

Dan Grindstaff wrote:All Right! Making progress I found the file in the code directory but it is empty! Any ideas why it would be empty?

No, I don't. I thought maybe you didn't close the FOS, but it looks like you did.

I'd start digging into the docs for the client library, see what kind of diagnostics it can give you, add some println() statements so you can see what's going on, maybe read a stream first instead of just retreiveFile, and see what's in that stream.

In other words, as with any debugging, you have to break it down into smaller and smaller pieces and to find out exactly where the problem is occurring.

[Dan Grindstaff]

Could you please give me an example of how to use RetrieveFile? Thanks Never mind I see I am already using it in one place. Duh!

[Dan Grindstaff]

Ok, I have some code here that is working:

import java.io.BufferedReader;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.URL;
import java.net.URLConnection;

public class ReadFile {
	public final static String str = "ftp://ct11corpfsst502.am.mot.com/ldapfeed/production/coreexport_gae.full";
	
	public static void main(String[] args) throws IOException {
		URL url = new URL(str);
		URLConnection con = url.openConnection();
		BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
		try {
			String inputLine;
			while ((inputLine = in.readLine()) != null)
				System.out.println(inputLine);
		} catch (FileNotFoundException e) {
			System.out.println("File not found on server.");
			System.exit(0);
		} catch (Exception e) {
			e.printStackTrace();
		} finally {
			if (in != null) in.close();
		}
		System.out.println("Read File Complete.");
	}
}

but I need a way to pass the line read from this code to the reader in :

import it.sauronsoftware.ftp4j.FTPAbortedException;
import it.sauronsoftware.ftp4j.FTPDataTransferException;
import it.sauronsoftware.ftp4j.FTPException;
import it.sauronsoftware.ftp4j.FTPIllegalReplyException;

import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;

import au.com.bytecode.opencsv.CSVReader;

public class PipeReader {

	/**
	  * @throws IOException 
	 */
	public static void main(String[] args) throws IOException   {
		// TODO Auto-generated method stub

		CSVReader reader = null;
		
		try {
			reader = new CSVReader(new FileReader("C:/Temp/coreexport_gae.csv"), '|');
		} catch (FileNotFoundException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
	    String [] nextLine;
	    while ((nextLine = reader.readNext()) != null) {
	        // nextLine[] is an array of values from the line
	    	for (int i=0; i < nextLine.length; i++ )
	        System.out.println(nextLine[0] + " " + nextLine[1] + "etc"  );
	    }

	}
}

[Dan Grindstaff]

It seems like I am getting really close with the following code:

import java.io.BufferedReader;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.net.URL;
import java.net.URLConnection;

public class ReadFile {
	public final static String str = "ftp://ct11corpfsst502.am.mot.com/ldapfeed/production/coreexport_gae.full";
	
	public static void main(String[] args) throws IOException {
		URL url = new URL(str);
		URLConnection con = url.openConnection();
		BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
		OutputStream out = new FileOutputStream(new File("C:\\temp\\newfile.csv"));
		int read = 0;
		byte bytes[] = new byte[1024];
		try {
			String inputLine;
			while((read=in.read(bytes))>0)
			//while ((inputLine = in.readLine()) != null)
				System.out.println(inputLine);
				out.write(bytes,0,read);
				out.close();
			
		} catch (FileNotFoundException e) {
			System.out.println("File not found on server.");
			System.exit(0);
		} catch (Exception e) {
			e.printStackTrace();
		} finally {
			if (in != null) in.close();
		}
		System.out.println("Read File Complete.");
	}
}

But I am mangling up something in the write section.

[Dan Grindstaff]

Ok, I finally got some code to accomplish this:

import java.io.BufferedReader;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.net.URL;
import java.net.URLConnection;

public class ReadFile {
	public final static String str = "ftp://ct11corpfsst502.am.mot.com/ldapfeed/production/coreexport_gae.full";
	
	public static void main(String[] args) throws IOException {
		URL url = new URL(str);
		URLConnection con = url.openConnection();
		InputStream reader = url.openStream();
		BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
		FileOutputStream writer = new FileOutputStream("C:\\temp\\newfile.csv");
		//OutputStream out = new FileOutputStream(new File("C:\\temp\\newfile.csv"));
		
		byte[] buffer = new byte[153600];
		int totalBytesRead = 0;
		int bytesRead = 0;
		
//		int read = 0;
//		byte bytes[] = new byte[1024];
		try {
//			String inputLine = null;
//			while((read=in.read(bytes))>0)
//			while ((inputLine = in.readLine()) != null)
//				System.out.println(inputLine);
			while ((bytesRead = reader.read(buffer)) > 0)
	        {  
	           writer.write(buffer, 0, bytesRead);
	           buffer = new byte[153600];
	           totalBytesRead += bytesRead;
	        }
	 
	        long endTime = System.currentTimeMillis();
	 
	        long startTime = 0;
			System.out.println("Done. " + (new Integer(totalBytesRead).toString()) + " bytes read (" + (new Long(endTime - startTime).toString()) + " millseconds).\n");
	        writer.close();
	        reader.close();
			
			
//				out.write(bytes,0,read);
//				out.close();
			
		} catch (FileNotFoundException e) {
			System.out.println("File not found on server.");
			System.exit(0);
		} catch (Exception e) {
			e.printStackTrace();
		} finally {
			if (in != null) in.close();
		}
		System.out.println("Read File Complete.");
	}
}

I would appreciate any help in cleaning it up and/or hints on how to further simplify. Thanks!

[Dan Grindstaff]

Dan Grindstaff wrote:Ok, I finally got some code to accomplish this:

import java.io.BufferedReader;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.net.URL;
import java.net.URLConnection;

public class ReadFile {
	public final static String str = "ftp://ct11corpfsst502.am.mot.com/ldapfeed/production/coreexport_gae.full";
	
	public static void main(String[] args) throws IOException {
		URL url = new URL(str);
		URLConnection con = url.openConnection();
		InputStream reader = url.openStream();
		BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
		FileOutputStream writer = new FileOutputStream("C:\\temp\\newfile.csv");
			
		byte[] buffer = new byte[153600];
		int totalBytesRead = 0;
		int bytesRead = 0;
		
		try {

			while ((bytesRead = reader.read(buffer)) > 0)
	        {  
	           writer.write(buffer, 0, bytesRead);
	           buffer = new byte[153600];
	           totalBytesRead += bytesRead;
	        }
	 
	        long endTime = System.currentTimeMillis();
	 
	        long startTime = 0;
			System.out.println("Done. " + (new Integer(totalBytesRead).toString()) + " bytes read (" + (new Long(endTime - startTime).toString()) + " millseconds).\n");
	        writer.close();
	        reader.close();
			
			
		} catch (FileNotFoundException e) {
			System.out.println("File not found on server.");
			System.exit(0);
		} catch (Exception e) {
			e.printStackTrace();
		} finally {
			if (in != null) in.close();
		}
		System.out.println("Read File Complete.");
	}
}

I would appreciate any help in cleaning it up and/or hints on how to further simplify. Thanks!