Hi Austen,
I doubt I am much further ahead than you, but surprisingly, I need to do exactly the same thing (send large assets via DIME from a .NET client to a Java hosted web service, using Axis 1.2).
So, we got to the point of sending files up to 75Mb reliably, over https, to our web services. We use DIME on .NET clients using Framework 1.1 and WSE 2.0. But, we could do a lot better. I apologize for the point-form below, but here's what little I know:
1. Chunking is used to help your machine's memory requirements, not to send the actual asset. The problem with large assets is, 'most' implementations, including .NET, attempt to read in the entire asset into a buffer (perhaps a byte array) and then attach it to the SOAP envelope. Needless to say, ultra-large assets like your astronomical pics are going to be huge files.
2. The 4096Kb is the default chunk size, you can try changing it later. (So if I am right, the buffer size for reading the file chunks is 4096Kb, or 4Mb).
3. DIME supports attachments up to 2^32 = 4Gb. But, and here is the rub, you can't hold a 4Gb file in any memory all at once. (Even though some Operating Systems allow memory to address a space larger than 4Gb, it'll never work). So, you need to chunk it. That means, reading chunks of the file at a time into memory and slowly attaching each piece.
4. You'll need to edit your App.config to change the maxrequest length, to something like this:
...
<microsoft.web.services2>
<messaging>
<maxRequestLength>75000</maxRequestLength>
<executionTimeout value="-1" />
</messaging>
</microsoft.web.services2>
...
The maxRequestLength is in units of Kb, I believe. You might want to play with this value, too. The file size cannot exceed this maxRequestLength. So you need to arbitrarily set a limit on file sizes.
6. Go here and copy this page to your drive, I feel it will be deleted one day and it's the best introduction to DIME I have seen on the web:
http://msdn.microsoft.com/msdnmag/issues/02/12/DIME/default.aspx 7. We also will attempt to move to XOP/MTOM one day. We have not yet tried it and so cannot comment on its performance or veracity. We have not yet tried Axis 2.0, but XFire is another SOAP engine to look at for advanced stuff like this.
8. If you try very large assets (>100Mb), you'll run into memory issues. You need to chunk to stop holding the entire file in memory all at once. To reach ultra large files (>4Gb) in theory, you'd have to leave your timeouts way open on your client and JSE, and also find a way to split the file into different subfiles, send each one in separate web service calls and reassemble them at the other end. I would suggest using SOAP headers to uniquely identify each part, and write the parts to disk. Then assemble the parts outside of all of the web service calls, in a separate
thread. This will be difficult.
With the above efforts,
you should be able to transfer files up to some limit depending on the RAM of your machine. (I have 2Gb of RAM, and can do 120Mb files easily, even though in production officially we stop at 75Mb). I am also restricted when calculating the SSL hash, for memory.
Hope this helps!
-jeff
