OutputStream implementation

[Robert Kennedy]

There are five active threads in this applet.
ThreadOne is writing data to the os stream.
ThreadTwo lives in a wait/notify loop and wakes up (timeout on wait/notify)
every second to update a progress bar and statistics, using values stored in
a object (to which this thread has exclusive access).
The other threads are living in there own wait/notify loops and are not active unless noified.


problem scenerio:
On occasion - most of the time - ThreadOne rushes through the while loop writing data for a 2mb image in less
than a second and then waits for a longer time for a response.

ThreadTwo never has a chance to update the progress bar when ThreadOne rushes through the while loop, either because the
time span is too short or ThreadOne aggressively holds the cpu?

Between the while loop and waiting for a response code ThreadTwo gets a chance to upload the progress bar
which it updates to 100% as ThreadOne appears to have written all data to the stream in the while loop.

Then there is a long wait (for a response) while, i suspect, the data is actually being written to the stream or to disk.

Eventually the upload is completed and the same behavior may repeat for the next file.

The corresponding console output is (for each image where the 1657 and 1907 times indicates problematic behavior):
bufferSize: 8192, whileloopWriteTime: 40125, waitForResponseTime: 500
bufferSize: 8192, whileloopWriteTime: 31937, waitForResponseTime: 625
bufferSize: 8192, whileloopWriteTime: 1657, waitForResponseTime: 27843
bufferSize: 8192, whileloopWriteTime: 1907, waitForResponseTime: 33734
bufferSize: 8192, whileloopWriteTime: 48719, waitForResponseTime: 641
bufferSize: 8192, whileloopWriteTime: 34578, waitForResponseTime: 688


Could there be a problem with my implementation (OutStream or the backend use of O'Reilly's MultipartRequest)?

 
   
  private int bufferSize = 8192;
private byte[] buffer = new byte[bufferSize];
private Socket socket;
private OutputStream os;
..... etc

os = socket.getOutputStream();
..... etc
 
// write the file bytes to the socket
int bytesReadIn=-1;
long bytesSentTotal=0;
long longStart=System.currentTimeMillis();
while ( ((bytesReadIn = fis.read(buffer)) != -1) ) {
write(buffer, 0, bytesReadIn);
bytesSentTotal += bytesReadIn;
System.out.println("write file: " + bytesReadIn + " : " + bytesSentTotal);
setBytesSent(bytesSentTotal);
   }
 

write(strLFBoundaryLF);
if(socket != null) socket.shutdownOutput();
 

// read response section
String strResponse="";
String strInputLine="";
br  = new BufferedReader(new InputStreamReader(socket.getInputStream()));
while( (strInputLine = br.readLine()) != null ){
strResponse=strInputLine.trim();
if(strResponse.indexOf("RES:") != -1) break; // contains bytes
if(socket==null || !socket.isConnected()){ intError=ERROR_SOCKET; break;}
}
if(socket != null) socket.shutdownInput();
 
....etc
 
private void write (byte[] buf,int offset,int size) throws Exception {
os.write (buf,offset,size);
}

more console output normal behavior:
starting from whileloop:

ThreadOne: write file: 8192 : 8192
... etc
ThreadOne: write file: 8192 : 2351104
ThreadOne: write file: 8192 : 2359296
ThreadOne: write file: 8192 : 2367488
ThreadOne: write file: 8192 : 2375680
ThreadOne: write file: 8192 : 2383872
ThreadOne: write file: 8192 : 2392064
ThreadOne: write file: 8192 : 2400256
ThreadOne: write file: 8192 : 2408448
ThreadOne: write file: 8192 : 2416640
ThreadOne: write file: 8192 : 2424832

ThreadTwo: updating progress barandstatistics: 03A.jpg : bytes sent: 2424832

ThreadOne: write file: 8192 : 2433024
ThreadOne: write file: 8192 : 2441216
ThreadOne: write file: 8192 : 2449408
ThreadOne: write file: 8192 : 2457600
ThreadOne: write file: 8192 : 2465792
ThreadOne: write file: 8192 : 2473984
ThreadOne: write file: 8192 : 2482176
ThreadOne: write file: 8192 : 2490368
ThreadOne: write file: 8192 : 2498560
ThreadOne: write file: 8192 : 2506752
ThreadOne: write file: 8192 : 2514944
ThreadTwo: updating progress bar and statistics: 03A.jpg : bytes sent: 2514944

... etc

ThreadOne: write file: 8192 : 2621440
ThreadOne: write file: 8192 : 2629632
ThreadOne: write file: 8192 : 2637824
ThreadOne: write file: 8192 : 2646016
ThreadOne: write file: 8192 : 2654208
ThreadOne: write file: 8192 : 2662400
ThreadOne: write file: 8192 : 2670592
ThreadOne: write file: 3826 : 2674418

ThreadTwo: updating progress bar and statistics: 03A.jpg : bytes sent: 2674418


ThreadOne: write final boundry
ThreadOne: read response: (03A.jpg)
bufferSize: 8192, writeTime: 29265, responseTime: 110
ThreadOne: cleanup
============================================================================

more console output problematic behavior:

ThreadOne: write file: 8192 : 8192
... etc
ThreadOne: write file: 8192 : 2351104
ThreadOne: write file: 8192 : 2359296
ThreadOne: write file: 8192 : 2367488
ThreadOne: write file: 8192 : 2375680
ThreadOne: write file: 8192 : 2383872
ThreadOne: write file: 8192 : 2392064
ThreadOne: write file: 8192 : 2400256
ThreadOne: write file: 8192 : 2408448
ThreadOne: write file: 8192 : 2416640
ThreadOne: write file: 8192 : 2424832
ThreadOne: write file: 8192 : 2433024
ThreadOne: write file: 8192 : 2441216
ThreadOne: write file: 8192 : 2449408
ThreadOne: write file: 8192 : 2457600
ThreadOne: write file: 8192 : 2465792
ThreadOne: write file: 8192 : 2473984
ThreadOne: write file: 8192 : 2482176
ThreadOne: write file: 8192 : 2490368
ThreadOne: write file: 8192 : 2498560
ThreadOne: write file: 8192 : 2506752
ThreadOne: write file: 8192 : 2514944
ThreadOne: write file: 8192 : 2621440
ThreadOne: write file: 8192 : 2629632
ThreadOne: write file: 8192 : 2637824
ThreadOne: write file: 8192 : 2646016
ThreadOne: write file: 8192 : 2654208
ThreadOne: write file: 8192 : 2662400
ThreadOne: write file: 8192 : 2670592
ThreadOne: write file: 3826 : 2674418

ThreadTwo: updating progress bar and statistics: 04A.jpg : bytes sent: 2674418


ThreadOne: write final boundry
ThreadOne: read response: (03A.jpg)
bufferSize: 8192, writeTime: 1907, responseTime: 33734
ThreadOne: cleanup
============================================================================

[Nitesh Kant]

What i got from your post is that the progress bar updater thread wakes up every second to update the progress bar. Do you also notify the thread in between or is it always a specified time after which this thread wakes up?

I feel it is a better idea for the writer thread to keep on putting notifications(containing the bytes sent(% completion)) in a queue and the progress bar updater keeps on painting the progress bar based on these notifications.

[Robert Kennedy]

Hello,
Thanks for your response.

The writer thread puts bytes sent in a queue and the progress bar thread wakes up on it own, every second, to update the progress bar. There is no direct notification between the writer thread and the progress bar thread.

The issue appears to be that, at times (often),the writer threads writes very quickly in its while loop and then wait a long time for a response (perhaps indicating the data was not really written but somehow buffered). when the progressbar thread wakes up it updates 100% to the progress bar and the interface remains like this until a response is received by the writer thread and the next item is started.

I wonder about my implementation. Is it correct?

note: If I place a 150ms delay between each write in the while loop all behaves as expected (desired), but this is not the solution. i would prefer to resolve this issue.

Thanks!

[Nitesh Kant]

Robert:
the progress bar thread wakes up on it own, every second, to update the progress bar.

This is what looks like a problem. The progress bar thread should poll a queue for work (i.e. paint progress bar task) which directly depends on the number of bytes written.

So, as I told in my previous post, the writer thread should insert the progress information in a queue (apart from actually writing the bytes to the destination) from where the progress bar thread picks up. This will add an implicit wait-notify mechanism between the two threads via the queue. So, the progress bar will progress based on the actual progress and not after a fixed interval.

[Rob Spoor]

You could check out javax.swing.ProgressMonitor and javax.swing.ProgressMonitorInputStream for examples.

[Robert Kennedy]

Thank you for your responses.

I am not using swing, however I have seen some of the examples. The gui is in HTML but it is fed by the applet via javascript.

The progressbar thread wakes up every second and checks a number of queues
to determine if it should update to the gui. The progress bar is one of these.

The writer thread writes all its data (0-100% in the while loop) in less than a second and then waits for a response for 3 seconds.

It may be prudent to have a wait\notify relationship between these two threads. However in this problematic scenario the gui would attempt to write 50 times a second. The write thread did not really write the file in one second. i believe this is confirmed by the 3 second wait for a response. Something appears to be amiss in the write algorithm.

But i am using a very standard implementation OutputStream and OReilly. No buffer, or so it appears.