• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

How to do Bulk data transfer using Web Service

 
Greenhorn
Posts: 3
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
In my application I have to write various web services but majority of the web service has to
query database and return back bulk data(rows>10K) through web service.

So I would like to ask what is the efficient way of transferring bulk data using web service as presently
Iam returning the dataset as xml String (using StringBuilder) from web service and consuming the same at client end.
Is there a better way to this in web service?

My env:
Front end can be in any other technology ,UI like C#
Back end : Tomcat 6 on Java 6 with Axis2

Thanks in advance
 
Author and all-around good cowpoke
Posts: 13078
6
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
You certainly dont want to send any bulky data as part of the SOAP body, thats what SAAJ - SOAP with attachments for Java is for.

As I recall, Axis 2 provides what you need.

Bill
(historical note - extremely poor performance with large data items in SOAP is what prompted SAAJ development)
 
Ranch Hand
Posts: 2198
1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi!
You may also want to consider to enable MTOM, which is supported in .NET too.
Best wishes!
 
Vijit Krishnan
Greenhorn
Posts: 3
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

William Brogden wrote:You certainly dont want to send any bulky data as part of the SOAP body, thats what SAAJ - SOAP with attachments for Java is for.

As I recall, Axis 2 provides what you need.

Bill
(historical note - extremely poor performance with large data items in SOAP is what prompted SAAJ development)



So you are recommending me to use attachments in web service to transfer POJO (like Employee object containing
EmpName,EmpID,Dept,DOB,DOJ) , as my data consists of recordset of similar kin like the emp example.

Thanks in advance
 
Ivan Krizsan
Ranch Hand
Posts: 2198
1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi again!
After having reread the question, it sounds to me like a REST web service would be appropriate, provided of course that you must use SOAP web services for some reason.
There are some things that you commonly see with REST web services that may be useful in your case:
- By enclosing, for instance, a startRow and a rowCount parameter in the HTTP request string, you can limit the number of rows you retrieve.
For instance, the user is looking at some table, in which each page is 100 rows. In this case it makes no sense to retrieve all rows, but instead use the suggestion above to retrieve one page worth of rows at a time.
- By enclosing, for instance, a depth parameter in the HTTP request string, you can limit the depth of the data returned.
This is like lazy loading with Hibernate, but instead of it happening automatically, you (on the client side) have to be aware of this.
For instance, you want to retrieve a list of Customers. Each Customer may have many orders, but when looking at the customer list, you are not interested in the orders, so when retrieving the customers you set the depth to 1. This way, the orders associated to each customer are not retrieved, but only represented by an URI from which the order in question can be retrieved (in the customer representation, which can be XML, JSON etc).

Of course, the above suggestions can be implemented in a SOAP web service as well - not in the HTTP request string, though.

The drawback with REST web services the lack standardized technologies like MTOM.
It should not be too hard to implement compression of data using a servlet filter etc.
Best wishes!
 
With a little knowledge, a cast iron skillet is non-stick and lasts a lifetime.
reply
    Bookmark Topic Watch Topic
  • New Topic