Web Services: Am I using this WADL wrong, or should I not be using a WADL at all?

[Alex Lieb]

I'm trying to make some basic requests to some web services for the sake of learning how to manipulate data with XML and make requests to web services.

I've been able to get a few of them to work using the simplest protocols I could use, but there's still a lot that's tripping me up. I'm trying to use the Flickr API because it appears to be better documented than most of the other free webservices I found, and more importantly, it allows you to make requests via REST, xml-rpc, or SOAP. So first I made a thing that called the "get related tags" method, just to verify that I was actually giving it the right parameters for tag, API Key, method name, etc...

And after some finagling it worked! That one was pretty simple to do, but I couldn't get it to do anything that required going through OAuth because I couldn't figure out how they wanted the queryString formatted. Like; you get a "base URL" which is described in different ways in multiple places, then you hash it to an MD5 String, and append it... Somewhere. In some format. I don't really know how you're supposed to append it. Trying to get it to do that was confusing as Hell because every time it failed (which was every time), I could never be sure what part I'd gotten wrong because there was no way to incrementally test it. Did I generate the base URL wrong? Did I hash it to an MD5 String wrong? Was the way I appended it to the queryString wrong? When it said "all the attributes in alphabetical order by value in name-value pairs", are the 'names' in the name-value pairs exactly what they're supposed to be? I never did get it to work, but I did verify that I could call at least one method correctly with a RESTful request.

What I really need to learn how to do in the near future though, is learn how to send data as XML validated by a schema provided by a given web service. Specifically I need to be more familiar with marshalling and unmarshalling stuff and using JAXB bindings. So I'm trying to make the exact same request I made with REST now, but by sending an XML payload. I built a schema, handed it to a new JAXB binding in my project, and told it to generate the classes, and it worked! The classes look exactly like I wanted them to, and when I make an object and marshal it into a file, the file looks exactly like I'd expect, and the values are all the same values I put into the REST request that worked, but when it tries to marshal it into the outputStream pointing at the web service endpoint, I get this as a response:

<?xml version="1.0" encoding="utf-8" ?> <methodResponse> <fault> <value> <struct> <member> <name>faultCode</name> <value><int>0</int></value> </member> <member> <name>faultString</name> <value><string>Invalid XML-RPC Method Call (root node was ns2:Request).</string></value> </member> </struct> </value> </fault> </methodResponse>

I know the API key is right, so somehow I must be sending them XML formatted in a way which is unexpected. Also, interestingly, if I take out the API Key element entirely, it just moves on to the next element in the list and gives me the same message.

This is what the WADL looks like at the part where it describes the method I'm trying to call:

<method id="tags_getRelated" name="GET"> <doc xml:lang="en">Returns a list of tags 'related' to the given tag, based on clustered usage analysis. </doc> <request> <param name="api_key" required="true" style="query" type="xsd:string"> <doc xml:lang="en">Your API application key. <a href="/services/api/misc.api_keys.html">See here</a> for more details. </doc> </param> <param name="tag" required="true" style="query" type="xsd:string"> <doc xml:lang="en">The tag to fetch related tags for. </doc> </param> <param fixed="flickr.tags.getRelated" name="method" required="true" style="query" type="xsd:string"/> <param name="auth_token" required="true" style="query" type="xsd:string"> <doc xml:lang="en">The authentication token to check. </doc> </param> </request> <response> <representation mediaType="text/xml" element="f:rsp"/> </response> </method>

This is the XSD I made for the JAXB binding:

<?xml version="1.0" encoding="UTF-8" ?> <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"> <xs:element name="Request"> <xs:complexType> <xs:sequence> <xs:element name="method" type="xs:string" maxOccurs="1"/> <xs:element name="api_key" type="xs:string" maxOccurs="1"/> <xs:element name="tag" type="xs:string" minOccurs="0" maxOccurs="1"/> </xs:sequence> </xs:complexType> </xs:element> <xs:element name="Response"> <xs:complexType> <xs:sequence> <xs:element name="tag" type="xs:string" maxOccurs="unbounded"/> </xs:sequence> </xs:complexType> </xs:element> </xs:schema>

For what it's worth I realize the "Response" element is totally wrong, and I'm kind of ok with that for the moment because until I'm actually able to *get* a valid response, I can't really tell how exactly to build the object, and it isn't particularly important either. Right now I'm just writing the inputStream to the console so I can see if it's returning anything at all, or if so, what error it's returning. Here's the code I'm using to actually send the request:

String key="ApparentlyI'mNotSupposedToShareThis.";
    String secret="OrThis.Sorry.";
    String target = "https://api.flickr.com/services/rest/?method=flickr.tags.getRelated";
    //String target = "https://api.flickr.com/services/rest/?"; I tried this target URL first; it didn't work.
    URL url = new URL(target);
    HttpURLConnection conn = (HttpURLConnection)url.openConnection();
    conn.setDoOutput(true);
    conn.setDoInput(true);
    conn.setRequestMethod("POST");
    conn.connect();
    OutputStream out = conn.getOutputStream();
    ObjectFactory of = new ObjectFactory();
    Request request = of.createRequest();
    request.setApiKey(key);
    request.setMethod("flickr.tags.getRelated");
    request.setTag(form.getParams());
    JAXBContext context = JAXBContext.newInstance(Request.class);
    Marshaller marshaller = context.createMarshaller();
    marshaller.setProperty(Marshaller.JAXB_FORMATTED_OUTPUT, Boolean.TRUE);
    marshaller.marshal(request, System.out);
    marshaller.marshal(request, out);
    if (conn.getResponseCode() != 200) {
      form.setResponse("Failed: Error Code "+conn.getResponseCode());
    } else {
      BufferedReader br = new BufferedReader(new InputStreamReader((conn.getInputStream())));
      Unmarshaller unmarshaller = context.createUnmarshaller();
      Response responseObject = Response.class.cast(unmarshaller.unmarshal(conn.getInputStream()));
      List<String> tagList = responseObject.getTag();
      String response="";
      for(int i=0;i<tagList.size();i++) {
        response=response+"\n"+tagList.get(i);
      }
      form.setResponse(response);
    }
    conn.disconnect();

I'm pretty sure the code here is right, but I can't really be 100% sure. I think there's something wrong with my XML, but I can't figure out how it should be structured in detail. The WADL tells you what parameters you need, what their names should be, and what types they should be, but I feel like I'm still missing something. And I'm actually not 100% certain I should even be using the WADL the way I am.

So I looked for something that would help me figure out how you're supposed to format XML requests to that particular web service, and I found this: https://www.flickr.com/services/api/request.xmlrpc.html

But I'm not really sure how to use this information. Also I'm confused because that request format appears to conflict with what I'm seeing in the WADL, but since there doesn't appear to be a schema or WADL or WSDL or anything specifically describing *why* the sample request looks that way, I'm not sure how to put a request for the related tags method into the same format. I feel like I'm missing something pretty major here, but I really have no idea where to start looking.

[Alex Lieb]

I rewrote my XSD. It makes a thing that looks like the example now (I think?) It's a little strange though; I wasn't sure how it would deal with having "value" in there twice, and apparently it just completely ignored the second one and made "struct" a direct descendant of "param" in the cascading mess of generated objects. Apparently every other rpc-xml request I'm seeing doesn't have that random "value" tag enclosing the struct, so I'm not sure why Flickr's example has one.

So I tried to send a request, but this time it gave me the same error. So while reformatting the thing to send the XML in this funky xml-rpc format may be necessary, apparently I'm being held up by something else that comes before that. Specifically, I think the issue is it really doesn't care what my root element is called because somehow it's getting assigned a name space prefix of "ns2". I have no idea why it would do that, but it is, and only to the root element.

Apparently JAXB just... Does that? http://stackoverflow.com/questions/16589600/remove-ns2-as-default-namespace-prefix

Sometimes? Not other times? I really do not understand the conditions under which it just decides to add a name space, but it does. And there's plenty of solutions given here for how to work around that, but I'm pretty sure it wouldn't be doing that in the first place if I'd assigned it a namespace that it thought was valid. So now my primary question is "Why does it think the root element in this case *needs* the ns2 namespace?" From what I understand, namespaces exist so that if you have two elements named the same thing from different XML documents, you'll know which one belongs to which document... But all of the elements in this document should be defined by the w3c schema referenced by the xs namespace, unless I'm misunderstanding how namespaces work? (Which is both possible and likely)

[Alex Lieb]

So I'm back. I figured out the first part (sort of).

I tried revising my schema to make the marshalled objects look like the example XML-RPC request on their site, and it failed because I couldn't figure out how to tell JAXB "Hey, I listed that 'value' element twice in the same tree because it actually does need to there." JAXB wasn't really sure how to deal with that initially. So I eventually had to figure out how to define a complex type called "value" that was able to hold *all* of the elements that all the "value" elements are able to hold in this schema -_-

It's convoluted and feels like it shouldn't be necessary, but I think because they just HAD to throw the same element into the tree twice, it is. So after fiddling with the namespaces for a while I finally found a configuration that worked. I still don't fully understand it though, so I left a note in the XML basically saying "This might not be the best way to do this! I'm still mostly guessing!"

<?xml version="1.0" encoding="UTF-8"?> <xsi:schema xmlns:xsi="http://www.w3.org/2001/XMLSchema" xmlns:jxb="http://java.sun.com/xml/ns/jaxb" jxb:version="2.1" elementFormDefault="qualified" targetNamespace="generated.ws.com" xmlns:t="generated.ws.com" > <!-- Adding the xmlns:t attribute is what finally made it able to reference the "value" type, But I'm still not 100% sure why. t isn't the default namespace; there is no default namespace. If I'm referencing the "targetNamespace" by having them be the same, and it's using that I guess that sort of makes sense? Not sure though. --> <xsi:complexType name="value"> <xsi:sequence> <xsi:element name="string" type="xsi:string" minOccurs="0" maxOccurs="1"/> <xsi:element name="struct" minOccurs="0" maxOccurs="1"> <xsi:complexType> <xsi:sequence> <xsi:element name="member" minOccurs="0" maxOccurs="unbounded"> <xsi:complexType> <xsi:sequence> <xsi:element name="name" type="xsi:string"/> <xsi:element name="value" type="t:value" minOccurs="0" maxOccurs="1"/> </xsi:sequence> </xsi:complexType> </xsi:element> </xsi:sequence> </xsi:complexType> </xsi:element> </xsi:sequence> </xsi:complexType> <xsi:complexType name="string"> <xsi:sequence> <xsi:element name="tags" minOccurs="0" maxOccurs="1"> <xsi:complexType> <xsi:sequence> <xsi:element name="tag" type="xsi:string" minOccurs="0" maxOccurs="unbounded"/> </xsi:sequence> </xsi:complexType> </xsi:element> </xsi:sequence> </xsi:complexType> <xsi:element name="methodCall"> <xsi:complexType> <xsi:sequence> <xsi:element name="methodName" type="xsi:string" minOccurs="1" maxOccurs="1"/> <xsi:element name="params" minOccurs="0" maxOccurs="1"> <xsi:complexType> <xsi:sequence> <xsi:element name="param" minOccurs="0" maxOccurs="unbounded"> <xsi:complexType> <xsi:sequence> <xsi:element name="value" type="t:value"/> </xsi:sequence> </xsi:complexType> </xsi:element> </xsi:sequence> </xsi:complexType> </xsi:element> </xsi:sequence> </xsi:complexType> </xsi:element> <xsi:element name="methodResponse"> <xsi:complexType> <xsi:sequence> <xsi:element name="params" minOccurs="0" maxOccurs="1"> <xsi:complexType> <xsi:sequence> <xsi:element name="param" minOccurs="0" maxOccurs="unbounded"> <xsi:complexType> <xsi:sequence> <xsi:element name="value" minOccurs="0" maxOccurs="1"> <xsi:complexType> <xsi:sequence> <xsi:element name="string" type="t:string" minOccurs="0" maxOccurs="1"/> </xsi:sequence> </xsi:complexType> </xsi:element> </xsi:sequence> </xsi:complexType> </xsi:element> </xsi:sequence> </xsi:complexType> </xsi:element> </xsi:sequence> </xsi:complexType> </xsi:element> </xsi:schema>

You'll notice this schema has 2 root elements though. And I know that's generally bad, but I'm not really sure how to get around that because my JAXB binding has to be able to process XML with root nodes with different names at different times. I could make another namespace for it I guess? I think? Still mostly guessing? Before I put the part in my xml schema with the methodResponse, it was returning valid XML with a list of tags, but now it breaks when I try to marshal that XML into a java object. Specifically it throws this exception:

(javax.xml.bind.UnmarshalException) javax.xml.bind.UnmarshalException: unexpected element (uri:"", local:"methodResponse"). Expected elements are <{generated.ws.com}methodCall>,<{generated.ws.com}methodResponse>

So what I'm gathering from this is it's saying "methodResponse" at uri "" is unexpected... Or methodResponse with namespace "local" is unexpected. And it's expecting "methodResponse" with a namespace of "generated.ws.com" because in my schema methodResponse has a namespace of generated.ws.com? If that's the case I'm really not sure how to tell my schema that it should expect that element to have no namespace. My second guess here would be that since there are now 2 root elements in that schema, it requires them to have namespaces declared, but that doesn't make sense because otherwise it would be giving a namespace to the methodCall element too... In which case I wouldn't be able to send method calls anymore. But those are still working; I added a bit before the unmarshalling lines to just read the response XML as text and print it to the console, and I'm still getting valid responses back with the same format I was getting before I added the methodResponse element to my schema.

The xml-to-plaintext response I'm reading in looks like this:

<?xml version="1.0" encoding="utf-8" ?> <methodResponse> <params> <param> <value> <string> <tags source="sun"> <tag>sky</tag> <tag>light</tag> <tag>sunset</tag> <tag>nature</tag> <tag>water</tag> <tag>blue</tag> <tag>clouds</tag> <tag>ocean</tag> <tag>summer</tag> <tag>beach</tag> <tag>yellow</tag> <tag>sea</tag> <tag>landscape</tag> <tag>green</tag> <tag>red</tag> <tag>sand</tag> <tag>cloud</tag> <tag>silhouette</tag> <tag>trees</tag> <tag>sunrise</tag> <tag>reflection</tag> <tag>sol</tag> <tag>canon</tag> <tag>orange</tag> <tag>cielo</tag> <tag>tree</tag> </tags> </string> </value> </param> </params> </methodResponse>

Comparing this to my response though, I'm seeing something else that could be problematic:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?> <methodCall xmlns="generated.ws.com"> <methodName>flickr.tags.getRelated</methodName> <params> <param> <value> <struct> <member> <name>api_key</name> <value> <string>blahAPIKey</string> </value> </member> <member> <name>tag</name> <value> <string>sun</string> </value> </member> </struct> </value> </param> </params> </methodCall>

My request has a namespace of ws.generated.com. If I'm getting this right, Flickr probably does not care what that is at all as long as it's not the exact same URI they use for a namespace in their schema. And somehow they're sending me back an xml element with no namespaces attached to the root element? So somehow I need to figure out how to tell my client program that the XML it gets back won't have a namespace, and that's ok, or I need to get it to tell Flickr's service that the response should have a namespace of ws.generated.com... I'm not sure how I would even begin doing either of those things, but of the two telling my program not to flip out when it sees the element without a namespace seems like the easier path and probably the more correct (and probably the only possible) path, so I'm going to start there I guess?

The unexpected thing here is I declared a new complex type called "string" in my schema, and I know I gave them both different namespaces, but I dunno.... Everything I've done up to this point has been so haphazard and guesswork'ish that even though I'm pretty sure I used namespaces correctly there to define "hey, this is the kind of string I mean when I say 'string' here", it feels like something should be breaking there... So that fact that it isn't makes me suspicious that there could be an issue with that that's getting masked by something else that's getting thrown along the way.

[Alex Lieb]

It's throwing something else now, and I think I know why it wasn't working now.

So it was looking for a thing with a namespace. I took out the target namespace and the default namespace though, and it was still expecting there to be a namespace upon unmarshalling, and I couldn't figure out why, but it appears JAXB is just really enthusiastic about figuring out namespaces, and if you don't give it a target or a default namespace, it will just assume you want a namespace of some sort on there, and for lack of a better place to get one, it expects the package name as a namespace.

So my package was named rs.generated.com, and it expected the methodResponse element to have a namespace prefix tied to a URI of "rs.generated.com." I'm not sure what the default is, but I was compounding the error by holding on to an attribute in my schema element; elementFormDefault="qualified." Apparently when JAXB sees that it adds an annotation that says "Hey, qualify the namespaces on these before you marshal or unmarshal", and that is the exact opposite of what I wanted it to do. I found a number of solutions to this exact problem involving SAX parsers, document builders, and reading in text, then converting to XML, but those all seemed awful to me and initially I couldn't quite place my finger on why. I think it's because doing that while having the xsd set to qualify the namespaces is the equivalent of telling JAXB "Hey, go over these namespaces really carefully!" then blindfolding it so it'll bypass quality control and do its job faster.

So! It's unmarshalling now, but it isn't unmarshalling correctly. It unmarshals all the elements up to "tags." After unmarshalling, I have a methodResponse object with a tree of nested objects, but "tags", the second to last element, is null. I'm not really sure why, but I have two guesses:

When I read in the XML response as text; I get this:

<?xml version="1.0" encoding="utf-8" ?> <methodResponse> <params> <param> <value> <string> <tags source="sun"> <tag>sky</tag> <tag>light</tag> <tag>sunset</tag> <tag>nature</tag> <tag>water</tag> <tag>blue</tag> <tag>clouds</tag> <tag>ocean</tag> <tag>summer</tag> <tag>beach</tag> <tag>yellow</tag> <tag>sea</tag> <tag>landscape</tag> <tag>green</tag> <tag>red</tag> <tag>sand</tag> <tag>cloud</tag> <tag>silhouette</tag> <tag>trees</tag> <tag>sunrise</tag> <tag>reflection</tag> <tag>sol</tag> <tag>canon</tag> <tag>orange</tag> <tag>cielo</tag> <tag>tree</tag> </tags> </string> </value> </param> </params> </methodResponse>

I suspect whatever is responsible for stopping it from parsing the left/right delimiters when it reads in text is also responsible for JAXB failing to unmarshal after that point. I have no idea what would cause that though. My other theory is that the attribute on the 'tags' element is messing things up somehow because JAXB doesn't expect it.

So I added a thing to the xsd, and JAXB seems to be dealing with that with no complaints. For reference, the xsd looks like this now:

<?xml version="1.0" encoding="UTF-8"?> <xsi:schema xmlns:xsi="http://www.w3.org/2001/XMLSchema" xmlns:jxb="http://java.sun.com/xml/ns/jaxb" jxb:version="2.1" elementFormDefault="unqualified" attributeFormDefault="unqualified"> <xsi:element name="methodResponse" > <xsi:complexType> <xsi:sequence> <xsi:element name="params" minOccurs="0" maxOccurs="1"> <xsi:complexType> <xsi:sequence> <xsi:element name="param" minOccurs="0" maxOccurs="unbounded"> <xsi:complexType> <xsi:sequence> <xsi:element name="value" minOccurs="0" maxOccurs="1"> <xsi:complexType> <xsi:sequence> <xsi:element name="string" minOccurs="0" maxOccurs="1"> <xsi:complexType> <xsi:sequence> <xsi:element name="tags" minOccurs="0" maxOccurs="1"> <xsi:complexType> <xsi:sequence> <xsi:element name="tag" type="xsi:string" minOccurs="0" maxOccurs="unbounded"/> </xsi:sequence> <xsi:anyAttribute id="source" /> </xsi:complexType> </xsi:element> </xsi:sequence> </xsi:complexType> </xsi:element> </xsi:sequence> </xsi:complexType> </xsi:element> </xsi:sequence> </xsi:complexType> </xsi:element> </xsi:sequence> </xsi:complexType> </xsi:element> </xsi:sequence> </xsi:complexType> </xsi:element> </xsi:schema>

It's still having the same issue with the null object, but I think I can rule out the attribute one, unless I just have my attribute tag formatted incorrectly or it doesn't actually describe the input I'm getting somehow. So I'm at an impasse again. You'll probably see me back in a week or a year or so with another slow, painful update X_X

[Alex Lieb]

Hey, so I know this is probably buried now and will never again see the light of day, but I figured it out and it's working.

So earlier I posted this:

<?xml version="1.0" encoding="utf-8" ?> <methodResponse> <params> <param> <value> <string> <tags source="sun"> <tag>sky</tag> <tag>light</tag> <tag>sunset</tag> <tag>nature</tag> <tag>water</tag> <tag>blue</tag> <tag>clouds</tag> <tag>ocean</tag> <tag>summer</tag> <tag>beach</tag> <tag>yellow</tag> <tag>sea</tag> <tag>landscape</tag> <tag>green</tag> <tag>red</tag> <tag>sand</tag> <tag>cloud</tag> <tag>silhouette</tag> <tag>trees</tag> <tag>sunrise</tag> <tag>reflection</tag> <tag>sol</tag> <tag>canon</tag> <tag>orange</tag> <tag>cielo</tag> <tag>tree</tag> </tags> </string> </value> </param> </params> </methodResponse>

That's wrong. When I bytestreamed the data into a string and printed it out, it actually looked like what's below, but I changed it before posting it here because I thought "Oh, it's just being weird because this wasn't meant to be read in as a string; SAX or something is supposed to know how to parse this." This is the string it was actually reading in:

<methodResponse> <params> <param> <value> <string> <tags source="sun"> <tag>sky</tag> <tag>light</tag> <tag>sunset</tag> <tag>nature</tag> <tag>water</tag> <tag>blue</tag> <tag>clouds</tag> <tag>ocean</tag> <tag>summer</tag> <tag>beach</tag> <tag>yellow</tag> <tag>sea</tag> <tag>landscape</tag> <tag>green</tag> <tag>red</tag> <tag>sand</tag> <tag>cloud</tag> <tag>silhouette</tag> <tag>trees</tag> <tag>sunrise</tag> <tag>reflection</tag> <tag>sol</tag> <tag>canon</tag> <tag>orange</tag> <tag>cielo</tag> <tag>tree</tag> </tags> </string> </value> </param> </params> </methodResponse>

I still do not know why it was failing to process all the escaped characters, but *only* after that point. But I did fix it; with a solution which is undoubtedly a hack, but since the error I was having was the result of faulty inputstream data, there isn't really any other way around it.

Instead of unmarshalling the raw inputstream, I uh... Well... I streamed it in as bytes and wrote it to a String, then did this:

      responseTxts=responseTxts.replaceAll("<", "<");
      responseTxts=responseTxts.replaceAll(">", ">");
      responseTxts=responseTxts.replaceAll(""", "\"");
      byte[] streamIn = responseTxts.getBytes();
      InputStream input = new ByteArrayInputStream(streamIn);
      MethodResponse response = MethodResponse.class.cast(unmarshaller.unmarshal(input));

Note: The above things aren't displaying on my browser correctly, probably because unlike my SAX parser, my browser knows how to parse &leftbracket; and &rightbracket; tags. So here's pictures for those of you who have working browsers:
http://imgur.com/a/k4U4d

If there were Programming Gods, I would be at the Temple sacrificing a cow right now and asking for forgiveness, but... IT WORKS. And furthermore, I now know that my schema is working perfectly, it's just failing to process the stuff after that node because there's a problem with the XML my program is receiving. It works now! And although I don't understand why I'm getting a bad response from a site which is pretty big and seems to be on top of its game as far as this stuff goes, I do understand why it's working now and why it wasn't working earlier.