B* Tree overflow problem

[David Gospel]

I am facing a major memory problem on a research project that I am working on right now. The project is designing a graphmaster for an Artificial Intelligent chatterbot. The AI bot requires patterns to be loaded in memory for answering questions posed to it by users.

So for efficient pattern searching I used the B* Tree to hash patterns and store the answers to those patterns at those nodes.

Now comes the problem. The entire pattern database is around 25 MB. So I need to load the whole database in memory. However I am able to load only around 4 MB data. After that the JVM crashes complaining about insufficient memory. I tried to perform a lot of optimizations relating to the structure of the B* tree, but without success. Any idea what can be done in this particular scenario. Without loading those patterns in memory the AI bot cannot work..

Any help would be appreciated.
Thanks in advance

[Johnson Gable]

You should consider decreasing the hashing tree size.. Maybe that could solve the problem.

[William Brogden]

Have you looked into increasing the amount of memory available to Java as controlled by command line parameters? See the tools.html file in your tooldocs (you did download and install the Java Docs, right?)
That 25MB of text is going to need a LOT more memory since
1) Java strings use 2bytes/char
2) you are creating lots of objects
Bill

[Stan James]

Can you challenge that "in memory" requirement? What user cares if it's in memory? Well, maybe a professor making an assignment. Real people care about rapid access to up-to-date information. Can you load only n levels of the tree below the current node? That would give very fast access to the next couple branches. Then maybe discard parts and load lower parts on another thread? Think of a sliding window in memory over the whole tree on disk. Not saying that's a great idea ... just saying think outside the box (or the memory in the box) and see where you go.