• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Paul Clapham
  • Tim Cooke
  • Jeanne Boyarsky
  • Liutauras Vilda
Sheriffs:
  • Frank Carver
  • Henry Wong
  • Ron McLeod
Saloon Keepers:
  • Tim Moores
  • Frits Walraven
  • Tim Holloway
  • Stephan van Hulst
  • Carey Brown
Bartenders:
  • Al Hobbs
  • Piet Souris
  • Himai Minh

batch insert queue

 
Greenhorn
Posts: 27
1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hey Guys

I have looked around on the net and i am thinking there is another name for what i am trying to do because it must not be that uncommon a requirement.. Here is my scenario

I have a webservice that gathers some information from SQL reads and returns a JSON object to the user. Right now i write that JSON object back to a log table in the DB so i can track what the users requested and got. Currently the user waits while i write that response in but that is unnecessary and not the way i want it to work. The service may get thousands of requests per minute so what i want to accomplish is simply to send that log info model off to another class that queues up these until it hits say 1000 records or 1 minute (and allows the service to carry on returning the response) whichever comes first type thing and then write them into the table in a batch insert and remove them from the queue. I am hoping to design this right the first time so i am wondering if someone can point me in the right direction for some info to get this started, or if you had experience with something similar give me some advice perhaps?

Cheers
L
 
Bartender
Posts: 4109
72
Hibernate Fedora Chrome
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hint: Have you read about asynchronous messaging?
 
Greg Belyea
Greenhorn
Posts: 27
1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
No, but i sure will tonight and see where that leads me
 
Greg Belyea
Greenhorn
Posts: 27
1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
So after some reading i tried using a class with a static method and a FutureTask for this... I have another class that implements Callable and has a LinkedList which i pass the objects i want to log into as they come in... I have a configurable property that i can set so that when the LinkedList hits that size it does a batch insert of the objects into the database, clears the list and is ready to go again. It ended up being quite uncomplicated but i am a little wary of what the downfalls of this might be.. one being what if the application was to crash for whatever reason and i have the batch set to 1000 records and i lose 500 records. I am thinking that the benefit of not spawning a new thread for every insert (which ofcourse does not scale but would be sure to never lose much) much outweighs the risk of losing a tiny percentage of records in the case of a disaster however i'd like to know what others think if there are any opinions out there?

Cheers
 
today's feeble attempt to support the empire
the value of filler advertising in 2021
https://coderanch.com/t/730886/filler-advertising
reply
    Bookmark Topic Watch Topic
  • New Topic