• Post Reply Bookmark Topic Watch Topic
  • New Topic

loading resources from a JAR file  RSS feed

 
Jason Barraclough
Greenhorn
Posts: 5
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hiya folks!

I suspect this is a really simple answer, but a bit of background first...

I have a collection of 1,500,000 surnames and 150,000 firstnames that I have created a JTextField "auto-completion/suggestion" field for. As the user types letter by letter, the result is filtered based on likelehood ratios and limited by a maximum number of completions.
I tested a database for this (DerbyDB) but was unimpressed with the results, as even with a non-blocking UI, the suggestions lagged for each letter typed (I suspect that it was due to repeated connections and closures of the database).

I have found that iteration through an ArrayList seems to do a great job with queries completed in milliseconds (and with a non-blocking UI approach, seems to have instant suggestions despite the massive list). The problem is, reading these in from a text file in to an ArrayList takes 130minutes initially(!!!).
Serialising this ArrayList to data.dat for example seems to RAPIDLY speed this up (less than 15seconds to load it back into the memory... kinda bizarre the speed difference).

Here are my questions.
1) why the huge difference in speed when loading text in to the arraylist c/w serialising - especially considering that the serialised array list is actually human readable text...
2) I want to distribute the application as a neat "package". Having a heap of serialised resources looks "unclean" with a jar packager in MacOS for example. Is there a way to pack the serialised ArrayList in to a jar file and then load it directly from the jar at runtime? How is this done?

Many thanks folks!
 
Paul Clapham
Sheriff
Posts: 22835
43
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Jason Barraclough wrote:I tested a database for this (DerbyDB) but was unimpressed with the results, as even with a non-blocking UI, the suggestions lagged for each letter typed (I suspect that it was due to repeated connections and closures of the database).


Well, for sure. So I would start by fixing your code so that it doesn't repeatedly create database connections. Just open a connection when your application starts up and leave it open.
 
Jason Barraclough
Greenhorn
Posts: 5
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Tried that too, but eventually it just "drops" the connection and fires an exception (which has to be handled etc... further lags).
Additionally, the LIMIT function that appears in MySQL doesn't work well with derby, hence the reason I dropped it completely. Since my list is probability based, and the iteration breaks as soon as 5 matches are found, the majority of the time it spends very little effort iterating. And the code is a heck of a lot simpler than the database approach...

Should I really persist with a database? What would the advantage be if the intended solution is an embedded datastructure?
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!