• Post Reply Bookmark Topic Watch Topic
  • New Topic

Calling RestFul WS in thread  RSS feed

 
Ganesh Sundaresan
Ranch Hand
Posts: 36
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
We are calling a RESTFull webservice which gives location details on passing the longitude and latitude.
Currently we are passing the request (from datastage) to this webservice (java client) one by one for 1 milliion records (of longitude and latitude parameters) and the time it
takes to complete the processing of entire 1 million records is around 16 hours.

We tried to improve the performace by doing the following:

1. calling the WS client in Thread - 15 threads each of which processes in equal share (of 1 million)
Now the performance is : 1 hr - 25000 records


Would appreciate if there are any other good approach for this.

Thanks,
Ganesh
 
Tim Moores
Saloon Keeper
Posts: 3834
80
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
The network latency would certainly introduce some overhead. Is the web service under your control? If so, you could add another call that takes multiple long/lat pairs and returns multiple results. Start by requesting 10 at a time and see what difference that makes.
 
Deepak Bala
Bartender
Posts: 6663
5
Firefox Browser Linux MyEclipse IDE
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Another approach is to cache the data. Does the response have headers that control cache ? Etags / last modified date etc can be used to trigger a HTTP 304 by a proxy.
 
Ganesh Sundaresan
Ranch Hand
Posts: 36
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks Tim and Deepak for your suggestions.

Tim,

The WS is not under our control!!!
 
Tim Moores
Saloon Keeper
Posts: 3834
80
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Maybe you can do a deal with the folks running the WS so that they accept batch requests, or provide you with some other means of access.
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!