SOAP with attachments, 3 questions

[Tom Griffith]

Hello. if anybody has a minute, I've worked my way through SOAP with attachments using Axis but I just have three brief questions...

1. It appears the Axis web app saves each attachment sent to it in WEB-INF/attachments. Does this mean that the web service implementation class(es) are responsible for cleaning out it's attachment(s) once it is done processing them?

2. I was just fiddling around and instead of sending it as an attachment, I sent a DataHandler object reference as an implementation class method argument but it seems to have been converted/sent as an attachment instead. Would this be true for any DataHandler on any web service platform or is this something specific to Axis?

3. Is it possible to attach attributes to a DataHandler object, say the original filename, which stays with the attachment when uploading to the web service...or should attributes be sent as a method argument? I've done the latter formerly (before attachments) for single files (binary objects) or maps of multiple filenames (keys) to multiple binary objects. However, if I send multiple attachments, I don't see a way maintaining a mapping of multiple filenames to multiple attachments since the attachments are external to the xml and I don't think there's a specific order to them.

Thank you very much for reading this.
[ February 27, 2008: Message edited by: Tom Griffith ]

[Peer Reynders]

Originally posted by Tom Griffith:
Would this be true for any DataHandler on any web service platform or is this something specific to Axis?

This is true for Java web service platforms that use SAAJ because DataHandlers are responsible for the serialization to MIME and deserialization from MIME. DataHandlers already existed as part of the JavaBeans activation framework (JAF) to handle MIME media types so they were reused in SAAJ. SAAJ implements SOAP with Attachments (SwA) as recommended in the WS-I Basic Profile 1.0. Microsoft however never bothered with SwA. They already supported DIME attachments (which were never standardized) and then went straight to MTOM. So if interoperability with .Net is a concern then you need to stick with sending binary data without attachments or move straight to MTOM.

Is it possible to attach attributes to a DataHandler object, say the original filename, which stays with the attachment when uploading to the web service...or should attributes be sent as a method argument?

Theoretically you could specify all the information you want in MIME-headers specified through javax.xml.soap.AttachmentPart

However the usual SwA way is to specify "Content-Type", "Content-Transfer-Encoding" and "Content-Id" with AttachmentPart.setContentType, AttachmentPart.addMimeHeader and AttachmentPart.setContentId and then refer to the attachment in the SOAP payload through the content ID where you can associate additional information with it, such as:

<item href="cid:ID002" >
  <name>coverimage.jpg</name>
</item>

For an example see pp.141-142 of SAAJ (pdf) from Java Web Services in a Nutshell.

[Tom Griffith]

Thanks a lot Peer, will check it out.

[Peer Reynders]

Note that the Axis 2 page claims the SwA can be handled within MTOM. That may very well be - however I wouldn't rely on it until I could prove that a MTOM capable .NET client can access my SwA web service.

Originally posted by Tom Griffith:
It appears the Axis web app saves each attachment sent to it in WEB-INF/attachments. Does this mean that the web service implementation class(es) are responsible for cleaning out it's attachment(s) once it is done processing them?

As Axis 2 doesn't clean (all of) them out, so I suspect that Axis 1.x doesn't either.

NOTE : It is a must to specify a directory to temporary store the attachments. Also care should be taken to *clean that directory* from time to time.

The way it's phrased suggests that they are thinking "delete all files in the temporary directory when the server starts up". Depending on how the infrastructure works it may be difficult to determine a "right" time for the service implementation to delete the cached file.

From Fear of Attachments (PDF):

Attachments are saved to memory if they are small (under 16KB or so), and then to disk if they get larger. Axis1.1 automatically cleans up these files when the attachments are finalized, but not before. Forcing a gc on the server sporadically helps keep disk usage down and stops leakage when the server crashes.

Axis 1.0 did no cleanup whatsoever, and there was no way of getting at the files to delete by hand, other than having a housecleaner thread purge the attachment directory of old files.

[ February 27, 2008: Message edited by: Peer Reynders ]

[Tom Griffith]

Oh, wow...ok, thank you Peer. I was suspecting as much because I kinda stumbled on the attachments directory and they were all piled up over the past week that i've been messing around with this. I woulda thought server drops/restarts might have run a gc but they all remained in there.

I've been messing with AttachmentPart but i keep stumbling on a single hurdle...I am consuming the web service via a business method call...

String answer=service.processQuestion(String q);

the 'service' is a reference to the soap binding stub. This then creates it's own SOAP message (request). I'm not sure how i will be able to create a message with AttachmentParts, etc if the business method, when called on the stub, seems to create it's own SOAP message.

i'll keep messing with it...I'm just not sure how to access the actual message object in this case...thank you for everything...
[ February 27, 2008: Message edited by: Tom Griffith ]

[Peer Reynders]

Originally posted by Tom Griffith:
I'm not sure how i will be able to create a message with AttachmentParts, ...I'm just not sure how to access the actual message object in this case

That is what the Message service style is for ... however I'm not sure that it is appropriate for attachments. SAAJ (pdf) from Java Web Services in a Nutshell has the code for an SAAJ servlet that could support attachments properly. However you would have to create your own WSDL. And if you create your own WSDL then Axis' WSDL2Java might just do most of the work for you.
[ February 27, 2008: Message edited by: Peer Reynders ]

[Tom Griffith]

hello Peer. If you have a minute, I scrapped using the axis stub to build a consumer and rebuilt it using SAAJ in order to apply a contentid to each attachmentpart...as you suggest above. I determined there was no way i could do that with the axis generated stub because it encapulates the web service call inside the business method. Anyways, I'm consuming the web service with SAAJ and was successful at invoking it. however, when I sent an attachment, the consumer gets a 500 response and this error appears on the server console...

Error in MIME data stream, start boundary not found expected ------=_Part_0_12345678.1234567891234


mail.jar and activation.jar are in the classpath. Below are the snippets of code from the consumer and the service applying to the attachment...

----Consumer----

String path="c:/image.jpg";
int y =path.indexOf("/");
path = path.substring(y+1, path.length());

FileDataSource fds = new FileDataSource(path);
DataHandler dh = new DataHandler(fds);

AttachmentPart attachment = message.createAttachmentPart();

attachment.setDataHandler(dh);
attachment.setContentId(path);
attachment.setMimeHeader("Content-Type", "image/jpeg");

message.addAttachmentPart(attachment);


----Service----

Iterator iterator = m.getAttachments();

if(iterator.hasNext()) {
String id = ((AttachmentPart)iterator.next()).getContentId();
DataHandler dh = ((AttachmentPart)iterator.next()).getDataHandler();

String path="C:/zz" + id;
FileOutputStream os=new FileOutputStream(path);

dh.writeTo(os);

os.close();

}
[ February 28, 2008: Message edited by: Tom Griffith ]

[Tom Griffith]

i printed out the entire SOAP request to the console and it looks ok...the datahandler is in there and all the parts and headers look ok...

The SOAP request boundary part numbers match what the web service says it expects but can't find...

Error in MIME data stream; start boundary not found, expected ------=_Part_0_12345678.1234567891234

and meanwhile, in the SOAP request, each part is deliniated with ------=_Part_0_12345678.1234567891234

it matches what the service says it expects but cannot find...

man, don't know whats up...if anybody has a thought, I'd appreciate it. I'll keep plugging but not sure what/where to even look now. The server is getting what it expects.

[Peer Reynders]

Here are some substitutions for you to try:

----Consumer----
String path = new java.io.File("C:/image.jpg").getName();
 
java.awt.Image image = java.awt.Toolkit.getDefaultToolkit().createImage( path );
AttachmentPart attachment = message.createAttachmentPart( image, "image/jpeg" );
attachment.addMimeHeader("Content-Transfer-Encoding", "binary");
attachment.setContentId( path );
message.addAttachmentPart(attachment);
 
 
---- Service ----
if(iterator.hasNext()) {
    AttachmentPart attachment = (AttachmentPart)iterator.next();
    String id                 = attachment.getContentId();
    Object content            = attachment.getContent();
 
    if(content instanceof java.awt.Image) {
        String path="C:/zz" + id;
        // ...
    }
    // ... 
}

If you can get something like this to work start re-introducing the DataHandlers.

Note that that you must have a valid mailcap configuration for DataHandlers to work JAF 1.1 (pdf).

By default, a MailcapCommandMap searches various locations in the following order:
1. Programmatically added entries.
2. The .mailcap file in the user�s home directory.
3. The mailcap file in JAVA_HOME/lib.
4. The file or resources named META-INF/mailcap
5. The file or resource named META-INF/mailcap.default

From http://jtute.com/java6/1301.html
[ February 28, 2008: Message edited by: Peer Reynders ]

[Tom Griffith]

Hi Peer. i'll try to back the Datahandlers out and see what happens.

When I attach a DataHandler via the axis stub, the service has no problem processing the attachment, so I think the mailcap thing is ok (but will check it out). This is really frustrating...thanks for everything so far...


Well, as an addendum, i did try to do this sans DataHandlers and no matter how I spin this, I get an IllegalArgumentException: on this line in the client...

AttachmentPart attachment = message.createAttachmentPart( image, "image/jpeg" );

Ugggh. I guess it's time to reboot the cpu and see if it decides to cooperate. Somehow i think the problems may be related...
[ February 28, 2008: Message edited by: Tom Griffith ]

[Tom Griffith]

All right...I went back to DataHandlers since they go ok using the axis stub and on a whim I commented out the saveChanges() method I threw on the SOAPMessage (just kinda thought it was a good idea to call it before consuming the service). I think i lost the start boundary error as a result...thank goodness...

however...now I am dealing this phenomena...

1. If I send ~two~ attachments, the first goes through fine, the content is streamed to a new file...great...however, the second attachment seems to vanish completely inside the service...I get a successful confirmation response...no exceptions and no new second file representing the second attachment.

2. If I send a single attachment, I now receive a NoSuchElementException back from the service.

???. Thanks again Peer for your help so far. I'll keep plugging at it.

[Peer Reynders]

It is possible to force Axis 1.4 to include a Content ID if you go WSDL2Java with a WSDL written in the rpc/literal style.

<?xml version="1.0" encoding="UTF-8"?>
<definitions
    name="Axis14Attach" 
    targetNamespace="http://www.Monson-Haefel.com/ed1/SubmitBook" 
    xmlns="http://schemas.xmlsoap.org/wsdl/"
    xmlns:xsd="http://www.w3.org/2001/XMLSchema" 
    xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/"
    xmlns:mime="http://schemas.xmlsoap.org/wsdl/mime/"
    xmlns:tns="http://www.Monson-Haefel.com/ed1/SubmitBook" 
  >
    
  <message name="submission">
    <part name="title"  type="xsd:string"/>
    <part name="price"  type="xsd:float"/>
    <part name="pdfDoc" type="xsd:hexBinary"/>
  </message>
 
  <portType name="SubmitBook">
    <operation name="submit" >
      <input message="tns:submission"/>
    </operation>
  </portType>
 
  <binding name="SoapBookMimeBinding" type="tns:SubmitBook">
    <soap:binding style="rpc" transport="http://schemas.xmlsoap.org/soap/http"/>
    <operation name="submit">
      <soap:operation soapAction="" style="rpc"/>
      <input>
        <mime:multipartRelated>
          <mime:part>
            <soap:body parts="title price" use="literal" />
          </mime:part>
          <mime:part>
            <mime:content part="pdfDoc" type="application/pdf" />
          </mime:part>
        </mime:multipartRelated>
      </input>
    </operation>
  </binding>
 
  <service name="SubmitBookService">
    <port name="SubmitBookPort" binding="tns:SoapBookMimeBinding">
      <soap:address location="http://localhost:8080/axis/services/SubmitBook"/>
    </port>
  </service>
</definitions>

Client request:

POST /axis/services/SubmitBookPort HTTP/1.0
Content-Type: multipart/related; type="text/xml"; start="<DB7C8614B66A819F14D4487C00986E56>"; boundary="----=_Part_0_26644003.1204264500256"
Accept: application/soap+xml, application/dime, multipart/related, text/*
User-Agent: Axis/1.4
Host: localhost:8888
Cache-Control: no-cache
Pragma: no-cache
SOAPAction: ""
Content-Length: 12441
 

------=_Part_0_26644003.1204264500256
Content-Type: text/xml; charset=UTF-8
Content-Transfer-Encoding: binary
Content-Id: <DB7C8614B66A819F14D4487C00986E56>
 
<?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope
    xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
    xmlns:xsd="http://www.w3.org/2001/XMLSchema"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <soapenv:Body>
    <submit xmlns="">
      <title>J2EE Web services</title>
      <price>29.99</price>
      <pdfDoc href="cid:98D5324BDADA66D7C30F64152EF86F76"/>
    </submit>
  </soapenv:Body>
</soapenv:Envelope>
------=_Part_0_26644003.1204264500256
Content-Type: application/octet-stream
Content-Transfer-Encoding: binary
Content-Id: <98D5324BDADA66D7C30F64152EF86F76>
 
%PDF-1.2 
...
%%EOF
 
------=_Part_0_26644003.1204264500256--

Server Response:

HTTP/1.1 202 Accepted
Server: Apache-Coyote/1.1
Content-Length: 0
Date: Fri, 29 Feb 2008 05:55:00 GMT
Connection: close

Generate Server skeleton and client stubs:

java -cp .;%axisclasspath%;%axis_lib%/wsdl4j-1.5.1.jar org.apache.axis.wsdl.WSDL2Java -o . -d Request -s -p submitbook.ws Submitbook.wsdl

Build Service code:

javac -cp .;%axisclasspath% submitbook/ws/*.java
jar cvf submitbook.jar submitbook/ws/*.class

Deploy service code:

copy submitbook.jar %catalina_home%\webapps\axis\WEB-INF\lib
java -cp .;%axisclasspath% org.apache.axis.client.AdminClient submitbook/ws/deploy.wsdd

Test Client:

package submitbook;
 
import javax.activation.FileDataSource;
import javax.activation.DataHandler;
import submitbook.ws.SubmitBookServiceLocator;
import submitbook.ws.SubmitBook;
 
public class SubmitBookTester {
 
    public static void main(String [] args) throws Exception {
 
        SubmitBookServiceLocator  locator     = new SubmitBookServiceLocator();
        String                    tcpmon_addr = "http://localhost:8888/axis/services/SubmitBookPort";
        SubmitBook                port        = locator.getSubmitBookPort(new java.net.URL(tcpmon_addr));
        
        FileDataSource pdfSource = new FileDataSource("TestPDF.pdf");
        DataHandler handler = new DataHandler(pdfSource);
 
        port.submit( "J2EE Web services", 29.99f, handler );
 
        System.out.println( "done" );        
    }
}

Compile client code:

javac -cp .;%axisclasspath% -d .  SubmitBookTester.java

Run client code:

java -cp .;%axisclasspath%;%axis_lib%/wsdl4j-1.5.1.jar submitbook/SubmitBookTester

[Tom Griffith]

Hi Peer. I can try that but if I write the wsdl with the mime parts, doesn't that eliminate the option of adding n (ie. any number, within reason) number of attachments to the request?

I managed to corrupt the axis web app with testing and goofing with the saaj requests.

Using the axis stub, I've added 4, 5 attachments and it's smooth as silk. I'm trying to log the SOAP request through the axis stub so I can see if there are any differences with the SOAP saaj request. I'm thinking Axis might inject something like extra \r\n's in the request or whatever...but I'm trying to get something in hand to try to compare the two versions of SOAP requests.
[ February 29, 2008: Message edited by: Tom Griffith ]

[Tom Griffith]

Welp, I configured a LogHandler to try to capture the SOAP request going to the web service via the axis stub but the attachments or MIME parts of the SOAP request don't get logged. They are successfully processed. The loghandler just prints off the header and envelope contents. Without the attachment xml, I don't see a way of troubleshooting and comparing this request with saaj going into the same web service. Thank you Peer for everything, if I come up with anything, I'll post it.

[Peer Reynders]

Originally posted by Tom Griffith:
Without the attachment xml, I don't see a way of troubleshooting and comparing this request with saaj going into the same web service.

Use (Apache TCPMon(Tutorial) or java.net tcpmon).

doesn't that eliminate the option of adding n (ie. any number, within reason) number of attachments to the request?

Possibly, and with Axis a 1.4 I think likely. If you look deep enough into the specifications (SwA, Attachments Profile 1.0) there might be a way to do it - however I doubt that Axis 1.4 would support it. As it is, you are already limited to rpc/literal when you should be able to push it into document/literal (at which point Axis 1.4 becomes unable to correlate the XML element with the associated MIME content ID).

The JAX-RPC specification side steps the issue by covering the "variable number of attachments" case with the "multipart/*" MIME mediatype. That way the variable number of attachments are contained in a single "multipart/*" mime part (which is handled by javax.mail.internet.MimeMultipart). The XML element reference is then simply to the "container" MIME part - not any of the attachments within it. In the Java code you simply stuff each of the attachments into a javax.mail.internet.MimeBodyPart and then add all the MimeBodyParts into a MimeMultipart instance. Make the following replacements in the previous example WSDL

    <part name="price"       type="xsd:float"/>
    <part name="attachments" type="xsd:hexBinary"/>
...
            <mime:content part="attachments" type="multipart/*" />
 

and Axis WSDL2Java will generate the following Java interface:

package submitbook.ws;
 
public interface SubmitBook extends java.rmi.Remote {
    public void submit(
        java.lang.String title,
        float price, 
        javax.mail.internet.MimeMultipart attachments
    ) throws java.rmi.RemoteException;
}

Note however that the WSDL doesn't specify what type of attachments are supported inside the "multipart/*" attachment - so in a sense you are now facing an incomplete/open-ended web service contract.

[Tom Griffith]

Thank you Peer. I was thinking back to what you were alluding to with pulling the datahandlers out and so I messed with that angle a little and I changed the createAttachmentPart method to take the datahandler directly as an argument...

message.createAttachmentPart(dh)

as opposed to what I was doing, creating a blank (or default) attachmentpart and calling setDataHandler on it...

message.createAttachmentPart();
attachment.setDataHandler(dh);

It seems to have solved the datahandler problems...the content is successfully streamed by the service and I get an affirmative response back. The one problem that lingers, I've kinda been ignoring it to work through these content issues, is that while the attachments are added to the outgoing SOAP, the pc starts beeping (a locking "siren" sound) and although I see the SOAP message being created with attachments parts and the confirmation returns from the web service, the beeping continues. Task manager shows that the command prompt and my local server are both not responding..and the beeping continues. I terminate the command prompt, the beeping stops, and the server immediately returns to 'Running" mode. It seems like something needs terminated along the line here but I'm not sure what...all this starts up before I make the actual connection call...like i said, it only occurs with the attachments...

[Tom Griffith]

Hello Peer. It's working quirk-free now, I figured out that the printing of the SOAP input stream (with attachments) with all that binary attachment junk for troubleshooting was locking the ui. It's all working now with SAAJ. I really appreciate all of your input and valuable info...I've benefitted a ton by this exchange.