(please ignore closing streams, this code is just for demonstration)
The ServletOutputStream can at a certain point get 'committed', whatever that means. The question is - when are is the data sent to the client? Is it buffered somehow on the server, and then sent as a single http response with huge payload to the client, or something else?
Then if you see your server crash with an out-of-memory exception, that tells you one thing. If you see those bytes arriving at your client, that tells you something else.
Yes, I know I can make a practical experiment, but I was more like searching for the theory (articles, tips, maybe some RFCs) behind the practice.
The reason why I ask is that we are using a funny UI library that was imposed on my team, and it is HTTP based, but it is not a webapp framework. You can store files on the client machine, but the whole data must be first prepared on the server, and I debugged it, and it is all buffered there! I asked support about it, and they said that HTTP doesn't support streaming anyways, so they can't do anything about it. I want to answer to that, but all I have are common sense and no links to back it up. The RFC where Content-Disposition is described didn't help either.
What does the venerable tomcat do in such case?
Raf Szczypiorski wrote:But if you say there is nothing I can do to force it, there is probably also nothing that I can do to prevent it.
I didn't say that. Or at least I didn't mean to say that. What I did mean to say was, there aren't any rules in HTTP which say that the server can or may or must buffer data, and I don't believe there are any such rules in the servlet specification either.
In the Tomcat 6 Connector docs here I find:
The size (in bytes) of the buffer to be provided for socket output buffering. -1 can be specified to disable the use of a buffer. By default, a buffers of 9000 bytes will be used.
There is also
The size (in bytes) of the buffer to be provided for input streams created by this connector. By default, buffers of 2048 bytes will be provided.
If it's the latter, a word to the wise:- Do not rely on setting the buffer sizes on your container. If your TCP packets are jumping through gateways, the gateways might decide to repacket the information in the packets. TCP standard guarantees that packets will be delivered to the destination and they will be in the same order as sent from the source. It doesn't guarantee that some information will be sent in the same packet. Also, the buffers that control tomcat are the tomcat buffers. The OS has it's own buffers that it will use. So, there might be more buffering going on than you can mentally buffer.
It's ok if all you want to do is make sure tomcat uses the least amount of memory possible. However, anything beyond that you are asking to get buffered
So. I just wanted to ask what actually happens when I do that with Content-Disposition and the rest of the story. I certainly don't care whether data will be sent in the same packet, or if some gateways perform buffering - not my problem. I just find the whole story the support guys tell me really hard to believe.
If your application is mixed-bound (I invented this term ); ie; some parts of it are CPU/IO bound, and some parts are network bound. you might want to think about hosting the parts that are network bound to servers that are optimized for file sharing.
*By IO in this post, I really mean Disk IO.
Please educate me if what I say is wrong - can you provide links which describe this scenario, and also describe what streaming is and what it is not, because I obviously don't get the difference.
To get things straight - no, I was not heaving media streaming in mind, but rather the ability of the server to start transmitting data of a huge file / dynamically created bytes before it has all been read / generated, thus reducing memory footprint on the server. Does Content-Disposition allow this, is there any RFC to define this?
Paul, I know you are a bartender here, so don't get me wrong, but I consider your post to be a) a little aggressive in tone and content; and b) not really contributing to the topic, as you gave no answers, just some implications of yours. Might be just my point of view, though.
Raf Szczypiorski wrote:So you are saying that streaming == media streaming? Or rather, that whenever anybody says 'streaming', everybody else understands it as 'media streaming'?
Well, look. You're the one who used the word. And you're the one who said that various unnamed persons had told you that HTTP doesn't support "streaming". Perhaps you didn't inquire too much about what they understood when you used the word, but to me it looks like they interpreted it as meaning "media streaming". Asking what "anybody" and "everybody" think the word means isn't going to get you anywhere, because as you can see from this discussion, people can mean different things.
So really that question isn't getting you anywhere. And I find it frustrating to observe this thread: you asked how a particular product worked and I said "Try it and see what happens". But you didn't. Instead, unfortunately the thread has gone way off course into an unproductive discussion about the meanings of words (and I'm sorry that I helped it go that way). So to bring it back on course, let me repeat my suggestion to do the experiment. Seems to me that your original question could be answered by running some suitable demonstration code.
I did my experiment on Tomcat 7 and I managed to generate a few gigs of data and stream / send it to the client, although the server had only a few hundred mbs of memory. Also, sending such data as response payload requires the Content-Length to be set, which pretty much requires the whole data to be generated to be able to count the bytes.
But you are wrong saying that this simple test can prove anything - it will just prove that one of many servers (tomcat 7) is smart, for example, and it still doesn't tell me what the standard behavior is and what it is not. I was looking for some maybe general answer about how http servers must behave when data is sent with Content-Disposition: attachment. If there is no such rule that servers must obey, then be it, but I am not able to answer it on my own. Hence the question, to learn from more experienced people who might have the answer.