Win a copy of Murach's Python Programming this week in the Jython/Python forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic

speed up data saving  RSS feed

 
krishna prasad gunasekaran
Ranch Hand
Posts: 158
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
good day,
1. i have grid that hosts more than 2000 records.
2. i get the xml of the grid.
3. parse the xml to an ArrayList.
4. call the DAO.
5. save the data in db through procedure.

but in db, i employ a for loop to iterate through the arrayList. so it's taking almost 45 seconds to store. is there a way to speedup the saving process.

instead of parsing the xml in step 3, can we parse xml in procedure and save?
 
Joe Ess
Bartender
Posts: 9406
12
Linux Mac OS X Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I take it this has something to do with the web application you describe here? You have left out a lot of vital information (Tell The Details), so it's difficult to make concrete suggestions. If your grid is in a Java GUI, you may want to avoid serializing it to XML and back again. Is your app multithreaded? If it isn't, it should be.
If you have an HTML/DHTML GUI, you may want to look at using AJAX to free your GUI from waiting for the update request to return.
Look at your hardware. If the CPU or memory is getting maxed out, no amount of code optimization will make your app go fast.
 
krishna prasad gunasekaran
Ranch Hand
Posts: 158
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thankyou Joe,

sorry for not giving enough information you require.
1. it's an html grid.

after clicking save button
1. take the xml i got from grid to server.
2. parse the xml to an ArrayList of records.
3. take the ArrayList to DAO.
4. iterate through the ArrayList and store records in db.

iteration is what consuming the time.
that's why i asked is there a way to transfer data from xml to db directly.

sorry if i sounded with superiority complex.
but you could have avoided this quoting

"blabbing like a narcissistic fool with a superiority complex" ~ N.A.


because a person with superiority complex never requests for an answer from forum.

good day, Joe.
 
Joe Ess
Bartender
Posts: 9406
12
Linux Mac OS X Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by krisp prasad:

sorry if i sounded with superiority complex.
but you could have avoided this quoting


The bit about the superiority complex is my signature. Someone on this forum once said that about me.
As for your problem, there are many different database products. Unfortunately, you do not share which one you are using so I cannot say if they support XML. Tell The Details.
In any case, the database would have to process a fair amount of XML and that would probably take some time. I still think your best bet is to look into an AJAX-type strategy to decouple your user interface from the update.
 
krishna prasad gunasekaran
Ranch Hand
Posts: 158
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
thankyou Joe,
i am using oracle 9i that supports xml parsing. java parser or oracle parser, which one is efficient?

suppose that i have an employee table with three columns which are to be filled by data from html grid which too has three columns. in my case there is a minimum of 2000 records of employees.

which of the following options is efficient

option 1 java parsing
1. parse the xml to an ArrayList in Action file.
2. send the ArrayList to DAO.
3. iterate through the ArrayList and save individual records in employee
table(i have employed a for loop, miniumum of 2000 iterations).

option 2 Oracle parsing
1. take the xml to Oracle.
2. parse the xml and save individual records in employee table.
can Oracle XML DB solve my problem.

option 3 ajax
i am not clear about this thing you told me. we have used ajax. but even with ajax, you still have to parse the xml to fetch individual column data.

good day joe,
 
Joe Ess
Bartender
Posts: 9406
12
Linux Mac OS X Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by krisp prasad:

option 3 ajax
i am not clear about this thing you told me. we have used ajax. but even with ajax, you still have to parse the xml to fetch individual column data.


The same way one can use threads to make a responsive Swing GUI, one can use AJAX to make a web interface do updates asynchronously. I think your other options above are just moving the work around and will not actually improve your response time (big generalization, but 2000 updates are going to take time no matter what - there is no substitute for doing the benchmarks yourself).
 
krishna prasad gunasekaran
Ranch Hand
Posts: 158
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
thankyou Joe,
i have decided to use ajax. one final question. parsing xml in java or oracle, which one is efficient?
 
Joe Ess
Bartender
Posts: 9406
12
Linux Mac OS X Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by krisp prasad:
parsing xml in java or oracle, which one is efficient?


I have no idea, and the answer is probably dependent on the nature of your application and data. Again, there is no substitute for doing your own benchmarks.
One more thing I thought of: why send the entire grid of 2000 entries? Why not only send the rows/entries that changed?
Have you checked your hardware to make sure it is up to the task?
 
krishna prasad gunasekaran
Ranch Hand
Posts: 158
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
thankyou joe
After every addition, update or removal of a record, use ajax to save or delete that particular record in db. is this what you are trying to say. then won't the server be hit by high frequency of ajax request? won't this affect server performance? Please bear with my questions if you find me technically naive.

good day joe
 
Joe Ess
Bartender
Posts: 9406
12
Linux Mac OS X Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I was thinking of keeping track of what records changed and only uploading the changed ones. Your idea (upload add/delete/edit when the user commits it) is probably better for most applications*.
As for server load, doing small updates is probably easier on the server than doing 2000 records at a time*. Remember AJAX is built upon standard HTTP. Web servers are optimized to handle LOTS of traffic (with the proper hardware!). Now this is within reason. If you were posting each keystroke as it happens for 10,000 users, that would not scale well either. That said, your gui using this strategy will be far more responsive and your user won't have to type in 1000 records only to find his computer crashed and he has to start from scratch.

*Again, this all speculation on my part and is something that you would have to benchmark for your particular application. Exactly what constitutes the sweet spot for performance will vary from app to app.
 
It is sorta covered in the JavaRanch Style Guide.
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!