• Post Reply Bookmark Topic Watch Topic
  • New Topic

Efficiency of a Search Function as a Web Service

 
Andres Delrotti
Ranch Hand
Posts: 178
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi,

I am currently working on an applcation. It's a simple web application that would allow users to view employee information for the employees of a small company. It has its own specific database to store application specific data. The main database where employee information is stored is a different one. This is the database where other internal application retrieve their data. For some reasons (I don't have go into details), they won't allow us to directly access the "main employee database". They recommended us using webservices to be able to retrieve employee data. They are willing to create a webservice just for our application.

We have a requirement that every transaction should not exceed 5 seconds. Now what I'm worried is this might have an effect on our app's performance. This was the reason I came up with the idea of retrieving the data from the webservice at the start of each day then store it in memory. The problem here is when a user tries to do a "search employee" function, say like search by last name, search by first name etc. If we are to store employee data in memory, we would have to do a lot of indexing in the Hashmaps to be able to simulate the search function. Or do you think we should let them construct the search function as a web service, would that make it slower?

Is there a better and more efficient way of dealing with a problem like this? Any form of help would be appreciated.

Thanks,
Andres
 
Vilmantas Baranauskas
Ranch Hand
Posts: 89
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Regarding performing search locally: How many employees are there?

If there are not that many employees (<10.000 ?), you could possibly do search even without indexing.
If there are too many employees, you could use lucene to maintain index.

But... before implementing search on your side you should really consider several issues:
- Could it be that they have some search rules you are not aware of? E.g. "m´┐Żller" and "mueller" is the same in German. Will you be able to offer the same search algorithm?
- What about new entries just added to the system? Are you notified about them (e.g. per JMS)? I personally hate when IT systems forces you to wait till the next day before you can proceed.

Other questions to ask:
- How many searches do you have/excpect per day/hour/minute/second?
- What will be interface to the service?

If you would use SOAP to access the service, sure it would add some load but it would not be a problem unless you have many requests/sec.
 
Andres Delrotti
Ranch Hand
Posts: 178
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
^^
Thanks for the reply.

Well currently there are 300+ employee but we are anticipating it to grow to up to 500 in a year or two. I think we could say that it could possibly have a worse case of 500 requests at present.

Regarding your questions,

For the updates, we have alternatives
1. We are planning to let them create an additional webservice which would return employee records updated/added from the last time of reference.
2. If they wont allow us then maybe we would just create a running job everyday that would retrieve all employee from the employee database through the webservice and store it in our app's memory.

I think the most convenient solution is to let them expose the search function as a webservice and not the "retrieve all" employee function. You think that would be ok if the maximum number of requests that could happen is 500? with the search function needing not to exceed 5 secs?
 
Pat Farrell
Rancher
Posts: 4678
7
Linux Mac OS X VI Editor
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by Andres Delrotti:

Is there a better and more efficient way of dealing with a problem like this? Any form of help would be appreciated.


How about using a database, say MySql or postgres? Free, fast, optomized for this.
 
Andres Delrotti
Ranch Hand
Posts: 178
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
^^
You mean transfer employee data from the "employee database" and copy all of it to our own application database? We've taken that into consideration but the company won't allow us to do that. They don't like redundancy of data. They also won't allow the applciation to directly access the "employee database" through JDBC due to security issues that is why thy are recommending it to be retrieved through a web service.

My main problem here is what if I wanted to do a search employee function. Should we let it be a part of the webservice? or just retrieve all employees from the webservice, transfer it to our app memory and do the searching there.
[ April 22, 2008: Message edited by: Andres Delrotti ]
 
Doug Slattery
Ranch Hand
Posts: 294
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Depending on how much use the main employee database gets, is the 5 second requirement for 500 requests not feasible? I take it you are anticipating 500 requests throughout the day?.?. So let's say for instance, an average of 20 records per request.

Strange that if they won't let you access the database directly, why not just create a limited access stored procedure instead of a web service.

If they really are going to implement a web service, then more than likely you will get passed back a List object back instead of a result set. I suppose they could implement a (slower) way to let you iterate over the set, but I'll stay out of the politics.

But as far as performance, it would be faster to pass the entire set back from the web service, especially if it is on the same server as the database.

Even if you decide to implement an idea like Pat's, you're faced with the problem of keeping the data in sync.

I don't know if I added anything of value, :roll: but I'm wrapping up a similar project implementing web services at the DAO layer and this sparked some interest.

Ironically, we used stored procedures and returned Lists in the services.

Aloha,
Doug

-- Nothing is impossible if I'mPossible
 
Andres Delrotti
Ranch Hand
Posts: 178
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

If they really are going to implement a web service, then more than likely you will get passed back a List object back instead of a result set. I suppose they could implement a (slower) way to let you iterate over the set, but I'll stay out of the politics.

But as far as performance, it would be faster to pass the entire set back from the web service, especially if it is on the same server as the database.



The entire set from the "search function" or entire set of ALL employees? So if I could have a max of 500 concurrent users accessing the web service, it would be ok to give the web service the burden of doing the search function? rather than letting it retrieve all employees and do the searching on our part?

The disadvantage of the first is, you would always need to access the web service everytime you search while in the 2nd option, you could just do it daily.
[ April 22, 2008: Message edited by: Andres Delrotti ]
 
Vilmantas Baranauskas
Ranch Hand
Posts: 89
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
If you have just 500 entries, then there is probably the single way to do it wrongly: let the search return a list of IDs and then retrieve data for each ID by another service call WITHOUT caching on your side.

Everything else would work.

Therefore, Keep It Simple.
 
Doug Slattery
Ranch Hand
Posts: 294
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by Andres Delrotti:


The entire set from the "search function" or entire set of ALL employees? So if I could have a max of 500 concurrent users accessing the web service, it would be ok to give the web service the burden of doing the search function? rather than letting it retrieve all employees and do the searching on our part?

The set from the search function, not all employees for every query -- that's nuts!

Yes, I think the web service should shoulder the burden. That's what it's for right? Why should 500 clients have to do something that one service could do? After all, how many results are you expecting per query anyway? Maybe 5? 10? 25? That's a lot less than the 250000 (500 emp's * 500 users) when the flood gates open in the morning.

In my case, the amount of connections could potentially be pretty high (several thousand), but the sets are small (< 5).


The disadvantage of the first is, you would always need to access the web service everytime you search while in the 2nd option, you could just do it daily.

The advantage of the first is you are always assured the data is correct if a record changes later in the day. If that doesn't matter, then the 2nd is ok.

But me being me, I would still implement it in the web service. You could implement a lazy access approach where you build the list throughout the day. You could also build in a db sync'ing strategy too (hourly as an example). If it's all contained in the web service, it'll probably be easier to test and debug too.

Aloha,
Doug

-- Nothing is impossible if I'mPossible
 
Pat Farrell
Rancher
Posts: 4678
7
Linux Mac OS X VI Editor
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by Andres Delrotti:


My main problem here is what if I wanted to do a search employee function. Should we let it be a part of the webservice? or just retrieve all employees from the webservice, transfer it to our app memory and do the searching there.


Transfering the data to your application or your own database defeats the purpose of their restriction.

Employee data is sensitive, so they should provide you with an API (or web service) that does the search with their rules and security. You should just call the appropriate tools.

You could, of course, do something evil like screen scrape their forms, but that is in the totally wrong spirit, and worse, puts all of the real privacy and security responsibility on your application (and on theirs) so you have at least double the work.
 
Andres Delrotti
Ranch Hand
Posts: 178
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi thanks for all your inputs. Gonna take those into consideration. The project's design review is today so I hope everything will be ok
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!