• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

RESTlet slower than Axis2?

 
Ranch Hand
Posts: 88
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
RESTlet slower than Axis2?


Hello Guys! I've been reading about REST, and based on what people say it should be really faster than Axis2:
1) REST architeture is simpler than SOAP
2) JSON is faster to marshall and demarshall than XML
3) Axis2 generated code is way more complex than RESTlet code


It made sende, and I decided to learn REST and ultimatelly run a benchmark. It consisted on comparing local call, with SOAP call and REST call. Base rule is kinda simple:

I created a class with a method that receives an int and a double as parameter:

It creates an array of doubles and loops this array filling it with the sum of provided double and array index. Then it returns this array.

For every test, I created another loop, also based on the int, that calls operation(). It counts how many milisecs it takes for each call and for whole loop. Basically:


With first loop I can have some processing and also provide a variably big array to be marshalled and demarshalled, and with second loop I can run many calls involving many marshalls and demarshalls.

For developing the project, I used:

* For local test, it's the code above, doing direct call to operation()
* For SOAP test, I used Axis2 over Tomcat6 as server. Server has that operation, calls original operation() method, and sends its result back as response. XML processing is done by Axis2 default configs. For client I also used Axis2, with its internal HTTP client.
* For REST test, I decided to use RESTlet and Jackson, for both server and client. On server side it's also over Tomcat6, on the same war file that Axis2 is deployed. Server receives a GET request, parses int and double parameters (URL is "/operation/{size}/{starter}" resulting in http://localhost:9090/Benchmark/rest/operation/100/11.5), call the same operation() method, then encodes double array as a JSON array using Jackson and returns it.
* RESTlet client is based on org.restlet.resource.ClientResource with RESTlet internal HTTP client (I wanted to use Apache HTTP Client, but had trouble making it work, and when using it, it just hang and stopped with no error message or closing the process). When it receives JSON response, it's parsed by Jackson.

To build whole project I used Eclipse Juno. With it I was able to build Axis2 server and client from Eclipse and deploy it into a war. RESTlet Eclipse plugin didn't work and I had to do all the job manually, but it was just a matter to edit web.xml to bind it. Eclipse successifully deployed both Axis2 and RESTlet servers together on a unique war, and if I wanted I could have also added some Servlet and JSP to serve traditional HTML pages for browser users.

I will provide Eclipse project and final server war and client jar if anybody is interested, I must change package names and clean some stuff.



Well, sadly the final benchmark result was disappointing. I have previously worked on an Axi2 project where I'd query a DB for 1000 rows and send each row as an Axis2 call to a remote server, resulting in 1000 requests, and it worked fine. But when I try to do a couble hundred calls with RESTlet I start getting timeout exceptions, like:



or



These kind of Exceptions are thrown in client after many requests (a bit more than 200) and break them (of course I could add try-catch inside the loop, but meh).

And finally, if I run a small enough loop that doesn't break RESTlet, I ended up seeing that it's slower than Axis2!



Having server and client running on the same machine and a loop size=200, local call took 9 ms, all individal call having 0ms.
SOAP Axis2 calls took mostly 2 to 4 ms, a few having 20 to 40 ms delay. Total time reached 761ms!
And when it was used Jackson + RESTlet +Tomcat on server and Jackson + RESTlet + RESTlet internal HTTP client on client, most calls took 3-6ms with a few having 10-20ms delay, and a total of 1008ms!

So, what do you guys think? Anybody have some experience with RESTlet and Apache HTTP Client? Could I get better results with a better client, and not have those timeouts? Could it be some config missing? but for Axis2 I also didn't make any config and had no problem.

I tested running Axis2 with 10.000 calls and got no error at all, while RESTlet barely is able to do 200. Again, I could have added try-catch inside loop, or some refresh, or dispose and reconstruct client object, but Axis2 didn't need any of that. For Axis2 client I don't even know what it uses for HTTP client and if it shares TCP sockets for multiple calls.

My biggest concert now is that, even if I can enhance RESTlet's server or client configs, change Tomcat or internal client for more efficient pieces of s##, or whatever can be done, REST with JSON should be way faster than SOAP with XML. My best bet now would be use Jersey or another REST implementation and it work better, so that whole blame could come to RESTlet alone, but seriously I don't wanna spend time implementing it all again without real proof of big improvements.

RESTlet website is way simple, with a few FAQ and javadoc, some of its links are pointing directly to archive.org!! And although it's being activelly developed I don't see it having a good community.
 
Hikari Shidou
Ranch Hand
Posts: 88
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Above post was typed this evening, sorry for making it so big.

Indeed, for developing SOAP clients with Axis2, it's just a matter of providing it the WSDL directly from WS holding all info I need and I'm given a class, that after a minute or 2 of reading I can just create a request object with all parameters, create a stub object from which I call operation(), and transparently the SOAP RPC happens and I'm provided a response object from which I can get my result. Eclipse handles deploying a jar with all dependances inside (including XML marshaller and a GOOD HTTP client) and I'm good to go.

With RESTlet, on the other hand, I have to read about a dozen of its API classes, learn about REST architeture, understand what each URL pattern and response mean (without a standard way for server to provide this information to client) and think how to design a Java class to abstract calls to it, learn about JSON and read a couple (or more) JSON marshaller classes, and then care about a good HTTP client!

For developing a WebService server, in Axis2 it's a matter of developing an as-simple-as-possible broker class, provide it to Axis2 from withing Eclipse plugin, and I'm good to go with a war file holding full Axis2 dependances, including XML marshaller, and my SOAP server AND the automatically built WSDL that will be served to clients from withing my server AND a nice HTML page explaining to humans about available services AANNND as many Servlets and JSPs as I wanna developed, with web.xml configured for me, AAAAAAANNNND this awesome war file being ready to deploy on any Servlet container)!

For RESTlet, OMG, I have to read and learn some more classes, JSON builder, design a conversion from my services architeture to REST URL architeture, manually bind them, then manually config web.xml!

And all that complexity to have timeouts on client and the end result be slower than Axis2!! I feel like REST doesn't support cashing, it requires cashing!!



Sorry Yes I'm deeply disappointed with REST and I beg anybody to point me if I did something wrong or what could be done to enhance REST performance and simplicity. Now I know what means not being confortable to use REST! x.x

Stability/timeout problems could be solved with fixes, but performance is a bigger issue. If 200 calls can be done locally in 9ms and using Axis2 SOAP on same machine it takes 761ms, as I said it's a matter of 752ms overhead, around 3-4ms for call.

To be worh the extra complexibility and being less flexible, a final and working REST solution should take at maximum 250ms, not 1 whole second.

The issue comparing SOAP with REST is that with SOAP I can easily abstract any Java class (just be careful with dependant classes), in a way that object-to-object RPC becomes totally transparent for both sides, and by using Interfaces and Factories I can easily move system components to remote server or bring them back local. A client's code can instantiate and consume remote objects as easy as normal local ones.

REST is way more complex, because it barely supports behaviors, then intead of using Service oriented Architeture we must use Resource oriented Architeture. Sending and receiving data in REST is harder than in SOAP, while executing (requesting to be executed) more complex algorithms on it is way harder in REST, even more if we wanna remain RESTful.

SOAP is way more flexible and abstractable. I was thinking on trading simplicity for performance, but after this benchmark... It basically showed me that SOAP, XML and Axis2 aren't as heavy as I though (I previously worked on a project where each call was taking 300-400ms, but now I see it was network lag. This lag can't be solved and affects REST equally, so I'm good with the few milisecs overhead that SOAP creates), while REST+JSON (at least with tools I used)... well!



The final conclusion is that a good software architeture overcomes RPC solutions, so it's better to just use the simplest and easiest one, which can even be faster than others more complex.

By software architeture I mean, using my benchmark as example, instead of having an operation that receives a value and returns some data and may be called thousands of times, I could also serve another operation that receives an array of values and returns an array of data. That would replace those thousands of overheaded calls into a unique call, more expandable and more scalable.

In more practical examples, instead of having an operation that receives a resource ID and returns that resource or that receives a resource and persists it or that receives a resource and does some processing on it and returns it back, I can serve an operation that receives an array of IDs and returns an array of resources, or receives an array of resources and persists them all, or receives an array of resources and runs some complext transformation algorithm on them and returns them back. This kind of solution would almost eliminate RPC overhead... and it's easier to develop with SOAP than REST!
 
Hikari Shidou
Ranch Hand
Posts: 88
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I know posts are big, but I'd really like if somebody with RESTlet or other REST implementation could give me some thoughts about what I have described.

I did some more tests today, and as I can understand the cause of these timeout errors is regarding an InputStream inside org.restlet.resource.ClientResource.

I thought it was Jackson fault, because its readValue() method requires an InputStream to parse JSON, it doesn't accept a String directly. So I had to use ClientResource.get().getStream() and pass it to readValue().

I moved to Gson, that accepts String as parameter, then I could do


Way simpler... but timeout kept happening. It seems that ClientResource.get().getText() simply handles the InputStream internally and writes it into a String, so I believed the problem could be there.

As a solution, I tried client.release() in the end of the loop, and client = new ClientResource(context,endpoint) at begining. IE, instantiate a ClientResource for every iteration... and timeouts still happen. it seems to really be a RESTlet issue, its client is blocking thread to do some I/O operation and this block times out. RESTlet seems to not be able to handle multiple sequential requests, while, again, Axis2 is able to do 10K ones. After moving to Gson and do these tweaks on ClientResource, I'm not even able to do 100 requests anymore!



As we can see, in stack trace there's no reference for Gson or HTTP client, so it seems it's a internal RESTlet issue.
 
Hikari Shidou
Ranch Hand
Posts: 88
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
On a side note, I created a new server, implementing REST over raw Servlet.

Between a timeout error and another, I was able to complete a benchmark of size=100 requests, here are the results:



This new result is interesting, RESTlet server and client with Gson marshaller has similar performance to Axis2 full solution (its HTTP client and XML marshaller).

It also shows that, with the exact same RESTlet client, a Servlet-based REST server manages to be faster (exactally 33%) than Axis2 SOAP. But not fast enough to justify the extra complexity of developing it. And without a properly stable REST client, this server is burned to death.

I'm not really willing to learn Jersey and developing a new server and client solution. I will study Apache HTTP Client and develop a REST client from scratch, let's see what I'll get.
 
Hikari Shidou
Ranch Hand
Posts: 88
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
While I was trying to get response from Apache HTTP Client I discovered Apache IOUtils. It has a toString() method that can get simple responses directly from HTTP!



And nothing else!

I threw RESTlet client away, with that I was able to run benchmark against 10.000 calls! Final results:



Of course a real software wouldn't make 10K sequential calls, this really would need optimization. But for comparison purposes, RESTlet server and IOUtils clients are 2 times faster than Axis2, and when I replace RESTlet server for Servlet server I can reduce overhead in more 20%!

With these results I'm closing my benchmark tests. Architecture optimization to reduce calls is the best solution at all. And SOAP Axis2 is only 2,43 times slower then the best results I could get from Servlet-based REST server and Apache IOUtils client. Being SOAP way more flexible, powerful and also stable, I'm happily willing to pay for this overhead, even more being SOAP Axis2 easier to implement architecture optimizations.

Only in special, very simple and never changing needs (like JavaScript client), I'd use a REST WebService, and would probably implement it directly with Servlet.

Sorry for all these big posts
 
A teeny tiny vulgar attempt to get you to buy our stuff
a bit of art, as a gift, the permaculture playing cards
https://gardener-gift.com
reply
    Bookmark Topic Watch Topic
  • New Topic