• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Liutauras Vilda
  • Paul Clapham
Sheriffs:
  • paul wheaton
  • Tim Cooke
  • Henry Wong
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Piet Souris
Bartenders:
  • Mike London

Executor Framework with Spring

 
Ranch Hand
Posts: 35
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi Ranchers,
In one of our applications we are using Executor framework to send 30 SQL queries in parallel to achieve high performance, which in turn opens 30 database connections at the same time. The application might receive 300 hits in a second, which translates the total number of DB connections to be 9000 (May be little less). What should be our approach to implement a hybrid solution where some queries will run in parallel and some in sequence. Or is there any other framework that can be leveraged. We are using Spring JDBC template for fetching the results.

Thanks in Advance.
 
Ranch Hand
Posts: 47
1
  • Likes 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Well, it is hard to answer in details as your question is not very detailed.

I usually use some combination of methods marked with @Scheduled and @Async. E.g. some method is scheduled to launch every second and call several asynchronous methods (sometimes I use atomic counters of work in progress).

the total number of DB connections to be 9000


Just make sure you use connection pool properly to avoid hitting performance with reopening every single connection
 
Abhishek Kanchan
Ranch Hand
Posts: 35
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thanks Rodion for your response.

The number of DB connections is the main challenge for us.
I will try with the @Scheduled and @Async and see if I can have 5 groups of 6 queries each and use only 5 DB connections.

Is there any framework which can help to schedule a set of SQL queries in groups. (To minimize the connection and get good performance)

Thanks.
 
Rancher
Posts: 2759
32
Eclipse IDE Spring Tomcat Server
  • Likes 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
If you are using the Executor frameweork correctly, then you shouldn't be using that many database connections.

First, The executor executes the tasks on a Thread pool. The size of the Thread pool is fixed (or can be limited). So, once you have created a ThreadPool. So, by definition, your ThreadPool will limit how many tasks will execute concurrently. If you are creating a Thread Pool for every hit, then you are making a big mistake. You should have a single thread pool executor that executes all tasks from all hit on the application. You can tune the thread pool to be as large as the hardware can handle. However, don't keep instantiating and destroying thread pool. Thread are costly things, and since Thread Pools encapsulate a bunch of threads, Thread Pools are even costlier

Second, you should use a connection pool. Each task shouldn't open and close it's own connection. Instead you should have a single connection pool, and each task should borrow a connection from the connection pool, and return it back. Opening a a connection is a costly operation. A connection pool ensures that connections don;t open and close every time. Generally, speaking, your database can handle a lot more connections when you use a connection pool in the application. This is because it doesn't have to take the overhead of establishing connections

If you do what to run some queries in sequence, your simplest solution is to just have one task that execute all those queries in sequence. You might want to look at Spring batch, that allows you to sequence steps, and some steps can be run in parallel. However, Spring Batch may be overkill for you.
 
Abhishek Kanchan
Ranch Hand
Posts: 35
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi Jayesh,

Thanks for your valuable response, that gives me a couple of options to explore.
I will respond once I have a working solution.

Thanks.
 
With a little knowledge, a cast iron skillet is non-stick and lasts a lifetime.
reply
    Bookmark Topic Watch Topic
  • New Topic