SSL/TLS Record reading in java

[Morad Ali]

Hello there,
I was making a HTTPS server in java from scratch for my school project, and I encountered an error.

I put an upload feature for users and it sends data using multipart/form-data. The problem is that any request over 16384 bytes cannot be fully read, SSLSocket.available() always return 16384, ignoring the rest of the request.

I know that the problem is that DataInputStream.read() doesn't read 'all' packets, SSL buffers the data to 16384 bytes each and sends every packet followed by ACK. Is there any read function that can read continuously from the socket so that it can read the whole packet?

[Ron McLeod]

Did you create your own HTTP client, or are you using a browser or third-party library?

I would expect that the client HTTP request would contain a either Content-Length header indicating the number of bytes of content in the message body, or a Transfer-Encoding: chunked header indicating that the content will be sent in chunks, and that the client has no more content when a chunk with a length of 0 is received.

[Morad Ali]

I use Firefox.

[Ron McLeod]

Check the HTTP headers, you should find something like:

    Content-Length: xxx
    Content-Type: multipart/form-data; boundry=----zzzzzzz

Where xxx indicates the length of the content and ----zzzzzzz defines the marker between the parts

[Matthew Bendford]

I suspect the issue to be the server code. SSL/TLS is meant to be fully transparent to the connection and sits between the TCP connection and the HTTP layer. So, first check if your code works without TLS. If so, adding TLS shouldn't change it.

Btw: DataInputStream and InputStream.available()? That already sounds wrong. Read again about how HTTP works and what InputStream.available() acutally reports back. Replace it with reading the content-length header and a counter loop.

Also: Show us the server code please. All else is randomly guessing in the blue.

[Morad Ali]

Matthew Bendford wrote:I suspect the issue to be the server code. SSL/TLS is meant to be fully transparent to the connection and sits between the TCP connection and the HTTP layer. So, first check if your code works without TLS. If so, adding TLS shouldn't change it.

Btw: DataInputStream and InputStream.available()? That already sounds wrong. Read again about how HTTP works and what InputStream.available() acutally reports back. Replace it with reading the content-length header and a counter loop.

Also: Show us the server code please. All else is randomly guessing in the blue.

Works with HTTP w/o SSL/TLS.
I use InputStream.available() to read the whole packet, I can't read content-length header before I read the packet (i guess).

[Morad Ali]

I posted my question on StackOverflow, code included.

Here

[Ron McLeod]

Morad Ali wrote:... I can't read content-length header before I read the packet

Why is that? With HTTP the headers are before the context.

[Paul Clapham]

Morad Ali wrote:I use InputStream.available() to read the whole packet, I can't read content-length header before I read the packet (i guess).

No. As Matthew already said, that's wrong. The available() method tells you how many bytes are available to be read from the input stream now. That is not necessarily the total number of bytes that will eventually be sent. You should never use that method when you want to read the entire contents of the input stream; if you do and it works, that was only accidental. So don't do it.

It isn't surprising to me that a request processed by TLS is presented differently than one not processed by TLS.

[Morad Ali]

Paul Clapham wrote:

Morad Ali wrote:I use InputStream.available() to read the whole packet, I can't read content-length header before I read the packet (i guess).

No. As Matthew already said, that's wrong. The available() method tells you how many bytes are available to be read from the input stream now. That is not necessarily the total number of bytes that will eventually be sent. You should never use that method when you want to read the entire contents of the input stream; if you do and it works, that was only accidental. So don't do it.

It isn't surprising to me that a request processed by TLS is presented differently than one not processed by TLS.

It was not accidental, TLS Records are sent 16kb each. Therefore, Any file under 16kb can be sent in one packet/record and therefore, it's not accidental.
Also, I'll try to read *without* checking s.available().

[Tim Moores]

Morad Ali wrote:It was not accidental, TLS Records are sent 16kb each. Therefore, Any file under 16kb can be sent in one packet/record and therefore, it's not accidental.
Also, I'll try to read *without* checking s.available().

can, not necessarily will. Seriously, don't rely on it. But I'm not sure what you mean by "file" (there are no files involved here) and "packet" (16KB TCP packets will go just about nowhere unfragmented, so they will be broken up, and may well arrive in discrete chunks).

See https://coderanch.com/wiki/660354 and https://coderanch.com/wiki/659924 for more information on Java I/O.

[Morad Ali]

I SOLVED IT!, here's how.

Request size: 10000b; Content-Length: 50000b; Header size: 160b;
body is stored in x.
if( (req.size-Header.size) < Content-Length) then :
  read n bytes, where n is ( Content-Length - (req.size-Header.size) + 4 "4 bytes for \r\n\r\n as they were deleted as delimeters"); store in y

Merge arrays x and y so Z = x+y;
Remove Lower Boundary from Z for multipart/form-data. write on file. close.

Code :

ArrayList<Byte> reqq = Network.read(DIS, MAX_REQ_SIZE);
			byte[] req = toPrimitives(reqq.toArray(Byte[]::new));
			List<byte[]> headerPost = tokens(req, new byte[] { 13, 10, 13, 10 }); // split using \r\n
			String o = new String(headerPost.get(0));
			ArrayList<Byte> additional = new ArrayList<Byte>();
			if(headerPost.size()>2) {
			for(int i = 0;i<headerPost.get(2).length;i++) {
				additional.add(headerPost.get(2)[i]);
			}
			if (reqTranslator(o).containsKey("Content-Length")
					&& (req.length-headerPost.get(0).length) < Integer
							.valueOf(reqTranslator(o).get("Content-Length"))) {
				// Other data available
				byte[] add = Network.ManRead(DIS, Integer.valueOf(reqTranslator(o).get("Content-Length"))
						- (headerPost.get(1).length+headerPost.get(2).length+4)); // Read n bytes
				for (int i = 0; i < add.length; i++) {
					additional.add(add[i]);
				}
			}
			}

Where headerPost has :

index0 : headers

index1 : multipart metadata

index2 : binary data (+ lower boundary if lower than 16kb)

and additional<ArrayList> is the output data. Removing lower boundary is by splitting the array according to a byte sequence then take index0.

Also, (req.length - headerPost.get(0).length) == (headerPost.get(1).length+headerPost.get(2).length+4)
Thanks so much guys ! My teacher would be delighted !

[Matthew Bendford]

That'll break the moment when content-length is either not provided or if it's wrong.
Yes, a http server should provide a correct length header - but it can be wrong or not provided at all.
Whoever gave you this task should still slap you for that bad implementation.

Also: When dealing with HTTP - it's way easier to just use URL and let the already provided implementation deal with the rest. As for writing your own http server - DON'T - use provided implementations like apache commons.