• Post Reply Bookmark Topic Watch Topic
  • New Topic

Suggestions for an Architecture to handle 3000 requests per a second  RSS feed

 
Ransika deSilva
Ranch Hand
Posts: 524
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi all,
I want to develop an application which gets 3000 Asynchronous requests per second and I would be very glad to hear your expert ideas on this? This application is intended for the Wireless market. So far I have done some research on Mobicent application and sounds promising. But I am not very clear on this particular point; if a thread is created to handle an incoming request, we need to create 3000 threads per a second, which I am sure not very good. So what are the frameworks which I can do the research on. Thanks and regards.
 
Joseph Kampf
Greenhorn
Posts: 26
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
If your requests take 1 second then yes you will need 3000 threads to stay on top of things. But you design your application to be fairly stateless you should be able to take advantage of horizontal scaling.

Now if your response time is say, 1/2 second, you can do it with 1500 threads. That is a little bit of a smaller pill to swallow. If you can get your process time down to 1/4 second then you should easily be able to handle this. This is processing time, not round trip time.

Originally posted by Ransika deSilva:
Hi all,
I want to develop an application which gets 3000 Asynchronous requests per second and I would be very glad to hear your expert ideas on this? This application is intended for the Wireless market. So far I have done some research on Mobicent application and sounds promising. But I am not very clear on this particular point; if a thread is created to handle an incoming request, we need to create 3000 threads per a second, which I am sure not very good. So what are the frameworks which I can do the research on. Thanks and regards.
 
Gabriel Claramunt
Ranch Hand
Posts: 375
Monad Scala
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Nothing impedes you to have the threads already created waiting to be activated...
 
Ransika deSilva
Ranch Hand
Posts: 524
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi all,
Thank you so much for the replies. After I read the answers, I felt that the basic question/doubt that I had was wrong. I guess the requirement is some thing like this;
around 3000+ Asynchronous requests would be directed to my application over TCP/IP. So all these requests have to be collected with out a single or a very minimum number of requests getting discarded or ignored. At the same time I want to have data to prepare a report stating how many requests that the system facilitated and how many did get discarded.
So thinking forward, I think I need to get a very reliable Message Queuing framework which can handle this much of a load.
So my question now would be is JMS can cater this requirement or is there any other product. I am doing research on JMS now.
Your advice on this would be highly appreciated. Many thanks for your valuable time.
Regards
 
William Brogden
Author and all-around good cowpoke
Rancher
Posts: 13078
6
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
How much processing power does it take to handle one request in this system?

For HTTP requests, Tomcat can handle a remarkable number of "simultaneous" requests, and thats with all the overhead of preparing request and response objects. Of course, Tomcat uses a pool of Threads and other reusable objects. Since Thread creation is quite expensive, a pool is the way to go.

I found this interesting study of load handling.

How much of the response content is standard "boilerplate"? Is a database lookup required?

Bill
 
Stan James
(instanceof Sidekick)
Ranch Hand
Posts: 8791
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
JMS is a set of APIs on top of some provider, much like JDBC is APIs on top of Oracle or DB2 or MySQL. There are a number of choices, some may even be Open Source. I've used IBM's MQ-Series - now WebSphere MQ - and one called Fiorano MQ as providers. Both support the concept of "persistent queues" and "assured delivery" which is usually implemented by putting the messages in a database until another process picks them up. Then you'd have to ask if your JMS provider could enqueue 3000 messages per second.

Were you thinking of receiving TCP/IP messages and very quickly putting them on a JMS queue? Or asking your senders to use JMS instead of TCP/IP altogether? I like the latter approach if you can change your clients. Use the "assured delivery" language to convince them.

Either way you can get your horizontal scaling pretty easily by having a number of servers picking messages off the queue and processing them. What's your response time requirement?
 
Ransika deSilva
Ranch Hand
Posts: 524
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi,
Thanks for the replies.
Well the requirement is that SMS messages are coming over SMPP protocol and which are received through TCP/IP from the SMS Center (SMSC) and need to capture all the SMSs which are sent with out even 1 getting lost. So I thought of putting the SMS requests directly to a queue so that worker threads can grab those SMSs from the queue and process. Because of this, when the worker threads grab the SMS requests from the queue, the queue can be released/freed. Regards.
 
Peter Mayer
Greenhorn
Posts: 4
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Although this is a java forum. I have worked on apps with up to 30000 r/s and basically there is only one way to go: PHP. You may now laugh at me, however it really scales very very well, has full OO support and a lot of C extensions and a great community. If you use opcode caches like APC or something like memcached you can probably do your app using one or two servers (depending on the size of course).

Peter
 
Gabriel Claramunt
Ranch Hand
Posts: 375
Monad Scala
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I always though that PHP was well suited for Web pages.
It works also for receiving messages through TCP/IP?
 
Nicholas Jordan
Ranch Hand
Posts: 1282
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I find your requirements appealing as an avenue of thought:

It   will   require the hardware to drive such an engine coupled with well written software.

To implement Gabriel Claramunt's approach, you would allocate several thousand ( nomenclature impedes our design inquiry here ) places to put things in the constructor. The difficulty arises when we find out that a Thread object cannot be re-run.

So the approach that I took in my beginner app was to fire off thread objects in a loop, using a Hashtable to keep up with them. It would be an interesting test to find out how my approach holds up under loading of several thousand incomings. A greater knowledge of internals would be required and Java Message Service (JMS) has all the right words. I don't think your basic question is wrong - there has to be an implementation in Java that will wait for a request, then pass it to a function, which must "do as little as possible" as they say and "return"

Efficient C++: Performance Programming Techniques by Dov Bulka and David Mayhew discusses the design challenge effectively. They wrote the webserver for a Winter Olympics ~ so I can understand their informed, effective discussion of the issue in their book.

To cite the work of Thomas Wang:
This performance profile ( presented at author's site ) compares favorably with traditional fast substring scanning algorithms, such as Boyer Moore. Since Java's character is 16 bit wide, a naive implementation of Boyer Moore would require 65536 machine instructions to initialize its lookup table. This is obviously not acceptable when we are scanning any string shorter than 70000 characters long.


I am sure your requests are not 70-k, but even then an effective workup of your anticipated workload must include both software and hardware. Mr. Bulka raises this point in his book and provides a working balance for tradeoff decisons.

In general, Java implements Threading handoffs more effectively than competing technologies - so I would stay looking within Java but be demandingly critical in specifiying performance-critical robustness be identifiable by objective measure that can stand cross-attack by progenitor nephew interests. It is much more relevant to your analysis than common sense and computer science will recognize. Often it drives the entire purchase decision indepented of whether it makes any sense. Then they put it off on you when it does not work.
[ September 01, 2007: Message edited by: Nicholas Jordan ]
 
Nicholas Jordan
Ranch Hand
Posts: 1282
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by Peter Mayer:
....( horray for php )....


Then how come it runs like a stumbling Giraffa camelopardalis and makes me scared of an intrusion every time I see it come up under http ?
 
Stan James
(instanceof Sidekick)
Ranch Hand
Posts: 8791
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I like words like "queue" and "worker thread" here. You can do both all in one JVM using the thread pools in Java 5. This little snippet does just that:

Now I've run that with several requests per second. The thread pool grows to match the largest number of concurrent requests and makes good reuse of threads. I haven't had a good excuse yet, but your load would be reason enough to learn NIO which I think would use fewer threads. But all of this assumes the MessageHandler can keep up with the demand using some number of threads.

When we were talking about JMS, we could distribute this among several JVMs on several servers. You might have that very same loop, but the MessageHandler (and perhaps this whole JVM) would do nothing but put the message on a JMS queue. Then some number of servers could pull them off and do whatever needs done. You could scale to more load just by buying more worker servers out of petty cash (choke) up to the point where this loop doesn't keep up.

Can you put a load balancer in front of this? The clients hit an IP address which is really an ArrowPoint or similar box that splits traffic among some number of servers. That also helps your reliability - if one goes down the others in the group keep working.
 
Ransika deSilva
Ranch Hand
Posts: 524
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks a lot for the replies. It is no doubt that what I am planning to implement needs a lot of thinking on the Architectural aspect. I will definitely keep all the advices and suggestions given by you when I am doing the designing. Thanks a lot ones again for the replies. When I am done with the design, I will surely explain the key points which I used when designing, so that any one who wants to know how to implement an application of that scale can easily use it as prototype. Regards.
 
Gabriel Claramunt
Ranch Hand
Posts: 375
Monad Scala
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by Stan James:
Can you put a load balancer in front of this? The clients hit an IP address which is really an ArrowPoint or similar box that splits traffic among some number of servers. That also helps your reliability - if one goes down the others in the group keep working.[/QB]


Probably a little of basic queue theory would help. The arrival rate probably is not fixed but the processing time is. It would help you to find the number of "consumers" needed to keep up with the load.
 
Elihu Smails
Ranch Hand
Posts: 37
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
http://mina.apache.org
 
Ransika deSilva
Ranch Hand
Posts: 524
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi all,
MINA project sounds very promising and I will look into this in greater detail. Specially the connection types it can accept TCP/IP and Serial is some thing that I am looking forward to have in the application I am developing. Thanks a lot for the time you spent on answering the question I posted. Regards.
 
With a little knowledge, a cast iron skillet is non-stick and lasts a lifetime.
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!