In my application I have to write various web services but majority of the web service has to
query database and return back bulk data(rows>10K) through web service.
So I would like to ask what is the efficient way of transferring bulk data using web service as presently
Iam returning the dataset as xml String (using StringBuilder) from web service and consuming the same at client end.
Is there a better way to this in web service?
Front end can be in any other technology ,UI like C#
Back end : Tomcat 6 on Java 6 with Axis2
William Brogden wrote:You certainly dont want to send any bulky data as part of the SOAP body, thats what SAAJ - SOAP with attachments for Java is for.
As I recall, Axis 2 provides what you need.
(historical note - extremely poor performance with large data items in SOAP is what prompted SAAJ development)
So you are recommending me to use attachments in web service to transfer POJO (like Employee object containing
EmpName,EmpID,Dept,DOB,DOJ) , as my data consists of recordset of similar kin like the emp example.
After having reread the question, it sounds to me like a REST web service would be appropriate, provided of course that you must use SOAP web services for some reason.
There are some things that you commonly see with REST web services that may be useful in your case:
- By enclosing, for instance, a startRow and a rowCount parameter in the HTTP request string, you can limit the number of rows you retrieve.
For instance, the user is looking at some table, in which each page is 100 rows. In this case it makes no sense to retrieve all rows, but instead use the suggestion above to retrieve one page worth of rows at a time.
- By enclosing, for instance, a depth parameter in the HTTP request string, you can limit the depth of the data returned.
This is like lazy loading with Hibernate, but instead of it happening automatically, you (on the client side) have to be aware of this.
For instance, you want to retrieve a list of Customers. Each Customer may have many orders, but when looking at the customer list, you are not interested in the orders, so when retrieving the customers you set the depth to 1. This way, the orders associated to each customer are not retrieved, but only represented by an URI from which the order in question can be retrieved (in the customer representation, which can be XML, JSON etc).
Of course, the above suggestions can be implemented in a SOAP web service as well - not in the HTTP request string, though.
The drawback with REST web services the lack standardized technologies like MTOM.
It should not be too hard to implement compression of data using a servlet filter etc.