File copy from input stream is not complete

[murali jackson]

Hi All,
I have written a small program to copy the file from input stream and write the same to a new file.
Below is my program which works fine. But the thing is its too slow, it takes more time to complete. It was copying one byte at a time thats why it was slow.
So I have modified the program as the second program mentioned below . I have declared byte array and used that as a buffer size to read from the inputstream.
I thought it will write 1024 bytes at a time and the program would be faster, but it stops in the middle and when i open the outputfile it was just having below as content.



Please advise what is actually going wrong in my modified program... Thanks !!!

Working Program but Slow :

import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
public class InputOutputStream {
public static void main(String[] args) throws IOException {
// TODO Auto-generated method stub
FileInputStream f = new FileInputStream("c:/Document for my webservice/test1.txt");
FileOutputStream o = new FileOutputStream("c:/Document for my webservice/test.txt");
int i;
byte[] b = new byte[1];
while((i = f.read()) != -1) {
o.write(i);
System.err.println(i);
}
o.flush();
o.close();
f.close();
}
}

Not working Code :

import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
public class InputOutputStream {
public static void main(String[] args) throws IOException {
// TODO Auto-generated method stub
FileInputStream f = new FileInputStream("c:/Document for my webservice/test1.txt");
FileOutputStream o = new FileOutputStream("c:/Document for my webservice/test.txt");
int i;
byte[] b = new byte[1024];
while((i=f.read(b))!=-1) {
o.write(i);
System.err.println(i);
}
o.flush();
o.close();
f.close();
}
}

[Henry Wong]

The behavior of the read() method, of the before and after version, are different. And hence, your write() methods have to change too.

Basically, all your new code does is write the number of bytes that are read (as ASCII codes), and not the bytes that are read.

Henry

[Norm Radder]

Using proper variable names could have helped here.  Instead of the name: i use the name: nbrBytesRead
Instead of b, use bytesReadBuffer

[Campbell Ritchie]

Why are you using such low‑level code? My experiences with read() methods have been uniformly bad; I look on them as nothing but an annoyance. Use readLine() and writeLine(...) methods instead.
Also, why are you using input streams and output streams for text files? For text files use Readers and Writers. Then use a BufferedReader and a BufferedWriter.
Do you need to write flush() and close(). I thought close would call flush first.
What are you using the byte[] for?
Why are you using the close method when you could have used try‑with‑resources.
Go through the Java™ Tutorials and see how it ought to be done.

There is a slightly different method with Scanners and Formatters.

[Campbell Ritchie]

I have given you the wrong name for one of the method; sorry; it was by mistake.
Read about the read(byte[]) method carefully.

[murali jackson]

Thanks All for the advise !!!
It was very helpfull ... I will modify the code as per the suggestions provided ...

[Campbell Ritchie]

 

murali jackson wrote:Thanks All . . . It was very helpfull . . .

That's a pleasure. Do show us the improved version.

[murali jackson]

 

Campbell Ritchie wrote:That's a pleasure. Do show us the improved version.

Hi,

I have modified the code as below to use the bufferedreader and buffered writer instead of Inputstream.
The program is working but im seeing below issues . Please advise ...

1. The blank spaces in between the lines are missing in the output file.
1. File is only 98 % copied, always the final text in the file is incomplete ("Testing" in the source file is copied as "esting")

Example ..

Source file has below contents,

successfully entered into fnReadINIFile function
detail 011
sb_fixes:randomnumber#: 825832770000
sb_fixes:detail1:  
sb_fixes:detail2: CA
sb_fixes:calcdetail:before copy <CA>
About to get randomnumber
FDF HEAD IS NULL
Advance fixes detail
detail 021
sb_fixes:randomnumber#: 825832770000
sb_fixes:detail1: CA
sb_fixes:detail2: CA
sb_fixes:calcdetail:before copy <CA>
About to get randomnumber
FDF HEAD IS NULL
Advance fixes detail
Begin detail
detail 021
sb_fixes:randomnumber#: 825832840000
sb_fixes:detail1:  
sb_fixes:detail2: CA
sb_fixes:calcdetail:before copy <CA>
About to get LossLimit
FDF HEAD IS NULL
Testing

My output file is written as below

successfully entered into fnReadINIFile function
detail 011
sb_fixes:randomnumber#: 825832770000
b_fixes:detail1:  
b_fixes:detail2: CA
b_fixes:calcdetail:before copy <CA>
About to get randomnumber
FDF HEAD IS NULL
dvance fixes detail
detail 021
sb_fixes:randomnumber#: 825832770000
b_fixes:detail1: CA
b_fixes:detail2: CA
b_fixes:calcdetail:before copy <CA>
About to get randomnumber
FDF HEAD IS NULL
dvance fixes detail
egin detail
detail 021
sb_fixes:randomnumber#: 825832840000
b_fixes:detail1:  
b_fixes:detail2: CA
b_fixes:calcdetail:before copy <CA>
About to get LossLimit
FDF HEAD IS NULL
esting

But if i use fileinputstream and fileoutputstream methods, the file copying works perfectly without issues. I have posted both the programs below.
Please throw some light on this.

import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.IOException;
import java.io.Reader;
public class BufferedReaders {
   public static void main(String[] args) throws IOException {
       // TODO Auto-generated method stub
       
       
       BufferedReader reader = new BufferedReader(new FileReader("c:/MyStudies/Java file copy test/ledger.log"));
       BufferedWriter writer = new BufferedWriter(new FileWriter("c:/MyStudies/Java file copy test/ledger2.log"));
       
       while (reader.read()!=-1){
       
        String s = reader.readLine();
        writer.write(s);
        writer.newLine();
           
       
       
       }
       
       
   }
}

But the below code works perfectly in all scenarios ..

import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
public class FileCopy {
   public static void main(String[] args) throws IOException {
       // TODO Auto-generated method stub
       
       FileInputStream f = new FileInputStream("c:/MyStudies/Java file copy test/ledger.log");
       FileOutputStream o = new FileOutputStream("c:/MyStudies/Java file copy test/ledger2.log");
       
       int nbrBytesRead=0;
       byte[] bytesReadBuffer = new byte[100000];
       while((nbrBytesRead=f.read(bytesReadBuffer))!=-1) {
           
           o.write(bytesReadBuffer,0,nbrBytesRead);
           System.err.println(nbrBytesRead);
           
       }
       
       o.flush();
       o.close();
       f.close();
   }
}

[murali jackson]

Hi ,

I have understood what mistake im doing,

I modified the program to below and all is well now !!! Thanks again for the solution !!!

import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.IOException;
import java.io.Reader;
public class BufferedReaders {
   public static void main(String[] args) throws IOException {
       // TODO Auto-generated method stub
       
       
       BufferedReader reader = new BufferedReader(new FileReader("c:/MyStudies/Java file copy test/ledger.log"));
       BufferedWriter writer = new BufferedWriter(new FileWriter("c:/MyStudies/Java file copy test/ledger2.log"));
       String line;
       while ((line = reader.readLine()) != null) {
       
       
        writer.write(line);
        writer.newLine();
           
       
       
       }
       reader.close();
       writer.flush();
       writer.close();
       
   }
}

[murali jackson]

Hi,

Please advice whether my modified program above is correct or it still needs some improvement ?? Thanks !!!

[Carey Brown]

writer.close() will do a flush.

[Campbell Ritchie]

 

murali jackson wrote:. . .. . .                while (reader.read()!=-1){ . . .. . .

Nonononononononono.

What will happen is that you read the first character in the line: now are you going to read the whole of the line, or only 99% of it. That must be why you are changing “Testing” to “esting”. I told you, the read() method is too low‑level to be any good. I am not at all convinced you need the byte[]s at all. Use the high‑level methods.

. . .        while ((line = reader.readLine()) != null) { . . .

Yesyesyesyesyesyes. Well done That will work. By using the strange syntax in line 20, you make sure you only read each line once, so you get the whole of the line. I notice you have also got advance→dvance and begin→egin, so it does appear that the read() method is consuming the first letter of each line. Line 20 (as I said) makes sure you don't read two lines per loop; you would miss alternate lines and there is a risk that the last line in the output would read “null”.
Find out about try with resources as I told you yesterday, which obviates the need to close the readers and writers. If you write close(), you probably don't need to write flush() because close() calls flush() anyway.

The alternative with Scanner and Formatters would look like this:-

import java.util.Scanner;
import java.util.Formatter;
import java.nio.file.Paths;
import java.io.IOException;
/**
* Utility class for copying one text file into another
*/
class FileCopier
{
   /**
    * Main entry point for the application, to copy files. Copies the entire
    * text from the input file to the output file unchanged, except that
    * platform-specific line ends are used.
    *@param args The command-line arguments. Two file paths should be supplied,
    * otherwise "input" and "output" will be used.
    */
   public static void main(String[] args)
   {
       new FileCopier(args).copyFiles();
   }
   
   private final String inputFileName;
   private final String outputFileName;
   
   private FileCopier(String[] args)
   {
       if (args.length == 2)
       {
           inputFileName = args[0];
           outputFileName = args[1];
       }
       else
       {
           inputFileName = "input";
           outputFileName = "output";
       }
   }
   
   private void copyFiles()
   {
       try
       (Scanner fileScan = new Scanner(Paths.get(inputFileName));
        Formatter fileOutput = new Formatter(outputFileName))
       {
           while (fileScan.hasNextLine())
           {
               fileOutput.format("%s%n", fileScan.nextLine());
           }
       }
       catch (IOException exc)
       {
           exc.printStackTrace();
       }
   }
}

Note that a Scanner can take a Path as its constructor parameter, but a Formatter can't. I used the Formatter(String) constructor. More details in this old post. That class looks so simple and good that I added documentation comments and shall keep it for future reference.

Please have a look at the formatting in your post. Your line 20 would be a lot better with a bit of space between tokens, like this:-. . .                while (reader.read() != -1) { . . .The second version was correctly spaced. You have however too many empty lines in that method.

[Norm Radder]

Note: If you want an exact copy of the file, this code might change the lineend characters.  Windows will recognize several different combinations of newline and carriage return as the end of a line.
The output from this code may have a different end of line character that the input.

Using read and write with bytes will keep the output the same as the input.

[Carey Brown]

New IO (nio) has another way to copy files using channels. The idea here is to push the loop down to the operating system so that it can be run as close to the hardware as possible, ie "more efficient".

public static void copyFile( File from, File to ) throws IOException
   {
       FileChannel inChannel  = null;
       FileChannel outChannel = null;
       try
       {
           inChannel  = new FileInputStream(from).getChannel();
           outChannel = new FileOutputStream(to).getChannel();
           inChannel.transferTo( 0, inChannel.size(), outChannel );
       }
       catch( FileNotFoundException ex )
       {
        System.err.println( "ERR Copy file failed, file not found: " + from + " (" +ex.getMessage() + ")" );
        throw ex;
       }
       catch( IOException ex )
       {
        System.err.println( "ERR Copy file failed: " + from + " to " + to  + " (" +ex.getMessage() + ")" );
        ex.printStackTrace();
        throw ex;
       }
       finally
       {
           if (inChannel  != null) try { inChannel.close();  } catch( Exception e ) { }
           if (outChannel != null) try { outChannel.close(); } catch( Exception e ) { }
       }
   }

[murali jackson]

Hi All,

Thanks again for the detailed explanation !!!
It was really helpful

I have one last doubt on this topic .. what if i need to send this file to another method which is in a different class or if its in a different machine altogether like a web service.
Basically like a client server model. Client requests for a file and my server program creates the file and writes contents into the file and sends it back to client.

I have written below sample program based upon my earlier modified program. In My Server Class, I have created file and written some sample content inside it using buffered writer.
After that I have returned the buffered writer to my calling program. But after that i dont know how to process the buffered writer in my calling program to save the file.

****** Server Class ******
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.FileWriter;
import java.io.IOException;
public class FileSender {
public BufferedWriter sendfile() throws IOException {
File file = new File("C:/Document for my webservice/myfilewrite.txt");
file.createNewFile();
FileWriter writefile = new FileWriter(file);
BufferedWriter bufferedwriter = new BufferedWriter(writefile);
for (int i = 0; i < 10000; i++) {
bufferedwriter.write("I am testing this file");
bufferedwriter.newLine();
}
return bufferedwriter;
}
}
***************** CLient Class ***********************
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
public class FileReciever {
public static void main(String args[]) throws IOException {
FileSender fs = new FileSender();
BufferedWriter bw = fs.sendfile();
}
}

What im trying to learn is , my server method should be in such a way that any class can call this method and get the file and save it .
like below ... please advise...

***** Server Class *******
public class SeverProgram
{
public sendfilesassomedatatype filesender (){
** this method creates the file and writes some content into it***
**in the below statement sends the file ***
return sendfilesassomedatatype ;
}
***** Client Class *******
public class ClientProgram
{
public static void main (String args()) {
** this method calls the sendfile method
Filedatatype file = ServerProgramObject.filesender();
**** now using the above Filedatatype client needs to save the file in its machine ****
}

[Carey Brown]

You can send a file name but if your server and client are on two different machines that won't work. Otherwise you have to send all the bytes of the file. When doing this the typical approach is to load the file into a byte[] that can hold the whole file, then send the length of the array in the first four bytes followed by the buffer contents. On the client side you then read four bytes to get the file length and then loop till you've read that many bytes from the server. One way to do this is with serialization.

You can't send the file directly nor can you send the BufferedWriter, those will be meaningless to the client.?

How are the client and server supposed to communicate with each other? Sockets? HTTP? File name if they're on the same machine?

[murali jackson]

 

Carey Brown wrote:You can send a file name but if your server and client are on two different machines that won't work. Otherwise you have to send all the bytes of the file. When doing this the typical approach is to load the file into a byte[] that can hold the whole file, then send the length of the array in the first four bytes followed by the buffer contents.

Hi Carey,

This is exactly what i want to do, but dont know how to load the content of the file to  byte[] . Could you please share some sample code or tell me what class and methods i need to use to load the file to byte[].

Carey Brown wrote:

How are the client and server supposed to communicate with each other? Sockets? HTTP? File name if they're on the same machine?

I am thinking of using this method as a Webservice operation. So i thing the protocol would be SOAP.  Whenever a client makes a request to this service method.
It will send the byte[] as response and Client will process the byte[] to get the file. Thanks !!!

[Carey Brown]

 

murali jackson wrote:Could you please share some sample code or tell me what class and methods i need to use to load the file to byte[].

This is the jist. Needs some error handling.

File file = new File( "xxx" );
int len = (int)file.length();
byte[] buf = new byte[len];
int offset = 0;
int remaining = len;
BufferedInputStream in = new BufferedInputStream( new FileInputStream( file ) );
while( remaining > 0  ) {
   int numread = in.read( buf, offset, remaining );
   offset +=  numRead;
   remaining -= numRead;
}
in.close();

[murali jackson]

Thanks for providing the code Carey !!!
I will use this and post my update program here ...

[Norm Radder]

Will the file contain binary data? For example line end chararcters. Can SOAP handle binary data? The byte array could be Base64 encoded to ensure no binary data.

[Piet Souris]

You can simplify this code somewhat by using Files.readAllBytes,  see the Files api.

[murali jackson]

 

Norm Radder wrote:Will the file contain binary data? For example line end chararcters.

Hi Norm,

The file is going to be a text file or a pdf, so i guess it might have line end character.

Norm Radder wrote: Can SOAP handle binary data? The byte array could be Base64 encoded to ensure no binary data.

Sorry , I am not sure on this as im just starting to learn java webservices.

Also I have one doubt on the sample program provided...

public static void main(String args[]) throws IOException {
File file = new File( "c:/Document for my webservice/javabook.pdf" );
int len = (int)file.length();
byte[] buf = new byte[len];
int offset = 0;
int remaining = len;
System.out.println("Buffer size is  "+buf.length);
BufferedInputStream in = new BufferedInputStream( new FileInputStream( file ) );
while( remaining > 0  ) {
   int numread = in.read( buf, offset, remaining );
   offset +=  numread;
   remaining -= numread;
}
in.close();
System.outprintln("Buffer size is now "+buf.length);
}
Im getting the output as below :
Buffer size is  18098418
Buffer size is now 18098418

This is the file size of the pdf file in my hard drive. Carey told that the first four bytes will have the file length and the rest will be the actual file content. But here the entire byte array is having the length of the file.
I am finding it difficult to understand where is the first four byte which has the length of the file and the rest of the bytes which contains the data.
Sorry if my question is too silly .. I just could not visualize how the byte array is used by the program... Please explain.

[Carey Brown]

This is *a* way to do it. As mentioned you might need to use base64 which, I believe, handles the length for you(?). Serialization can also handle the length for you.

File file = new File( "xxx" );
int len = (int)file.length();
byte[] buf = new byte[len+4];
int offset = 4;
int remaining = len;
toBytes(buf,len);
BufferedInputStream in = new BufferedInputStream( new FileInputStream( file ) );
while( remaining > 0  ) {
   int numread = in.read( buf, offset, remaining );
   offset +=  numRead;
   remaining -= numRead;
}
in.close();
//...
private void toBytes( byte[] buf, int i )
{
   buf[0] = (byte)( ( i >> 24 ) & 0xFF );
   buf[1] = (byte)( ( i >> 16 ) & 0xFF );
   buf[2] = (byte)( ( i >> 8 ) & 0xFF );
   buf[3] = (byte)( i & 0xFF );
}

[murali jackson]

Hi Carey,

Sorry for the late response..
Thanks for the sample program ... im trying to visualize the working of the code.. I will let you know if i still have any doubts on it..
Thanks Again ...

[Campbell Ritchie]

What would be wrong with reading the entire text into a single String, creating a byte[] of the appropriate size, and using String#getBytes()? Apart from the problem of changing line ends, as noted in the documentation comment of my FileCopier class?

[Carey Brown]

 

Campbell Ritchie wrote:What would be wrong with reading the entire text into a single String, creating a byte[] of the appropriate size, and using String#getBytes()? Apart from the problem of changing line ends, as noted in the documentation comment of my FileCopier class?

I'm not convinced, based on Murali's comments, that we are only dealing with text files. It could be, say, an image file. Although in Muralis's code the file that is being opened is a ".txt" file, I'm assuming that was only for testing purposes.

Murali, please clarify.

[murali jackson]

Hi Carey and Campbell,

My Program can process other file types also besides text like Jar file, Image file etc.. Thanks !!!

[Norm Radder]

Can you state what you are trying to do again?  The last bits of code were for a small, specific part of the project.  Can you post a higher level description of what you are trying to do?

[murali jackson]

 

Norm Radder wrote:Can you state what you are trying to do again?  The last bits of code were for a small, specific part of the project.  Can you post a higher level description of what you are trying to do?

Hi Norm Radder,

I am trying to build a client server model where the client will request a file from the server, it can be any of the following types.

1. txt
2. word
3. pdf
4. excel
5. Jar
6. Jpeg , png.

The server will send the requested file in the form of byte[] and the client will process and save the file .

Please let me know if you need more information.

[Norm Radder]

So are these the steps the programs will take?

the client will request a file from the server
server will send the requested file
client will process
and save the file

Do you have a protocol for the requests the client sends to the server
and for how the server responds to the client?

 in the form of byte[]

Specifying how the files will be sent may be too detailed for this level of discussion.  
Ultimately the file's bytes will be sent.

[Ron McLeod]

 

Norm Radder wrote:Do you have a protocol for the requests the client sends to the server and for how the server responds to the client?

Earlier in this thread ...

Carey Brown wrote:How are the client and server supposed to communicate with each other? Sockets? HTTP? File name if they're on the same machine?

murali jackson wrote:I am thinking of using this method as a Webservice operation. So i thing the protocol would be SOAP.  Whenever a client makes a request to this service method.
It will send the byte[] as response and Client will process the byte[] to get the file. Thanks !!!

 

[Norm Radder]

Whoops, my bad.  I don't know SOAP so it didn't mean anything at the time.