Hot to deal with OutOfMemoryError

[Gaurav Chikara]

Whenever I am building my hashMap from a short text file then it runs ok
But when I try to do the same from a 781 KB file then I get this error
Can any one tell me remedy for it
Thanks in advance

[Maulin Vasavada]

Hi Gaurav
It depends on lot of factors.
- You can try to write the code to consider less memory usage. e.g. instead using String s = s1 + s2 100000 times use StringBuffer and all
- and some other little things to lessen your memory requirements
- run JVM with -Xms option to increase maximum heap size
BUT if you really have very large dataset then at somepoint you MAY run out options
Please post your code if possbile to see if there is anything easy to do before increasing memory size or claim that "its too much data" ..
Regards
Maulin

[Gaurav Chikara]

import java.io.*;
import java.util.regex.*;
import java.util.*;
public class DFADictionary {
  // Has not been used so far
  private String inputStr;
  // This String represents the each word read on new line
  private String regEx;
  // To get current user directory
  private String path = System.getProperty("user.dir")+File.separator;
  //Utility Class for pattern Matching
  private Pattern pattern;
  private Matcher matcher;
  // Used for pattern matching
  private boolean found;
  // To track the case of reading first word and subsequent words
  private boolean flag;
  // Contains the values of characters with keys as prev state and values are
  // character along with next state of character
  // Tracks only those words which are distinct and are not found amongst any patterns
  private int charScanned;
  // Used to be used in CallHashMapSubstringSearch
  private int prevState;
  //Used for ArrayList items are added only if they are distinct
  private static boolean matchFound;
  private ArrayList wordsProcessed = new ArrayList();
  private int wordScanned;
  private HashMap dictionary = new HashMap();
  public DFADictionary() {
    try {
   BufferedReader  br = new BufferedReader(new FileReader(path+"regex.txt"));
    while(br.ready())
    {
      regEx = br.readLine();
      matchFound=false;
      processInput(regEx);
      showOutput();
    }
    br.close();
  }
 catch (FileNotFoundException fnfe) {
     System.out.println("Cannot locate input file! "+fnfe.getMessage());
     System.exit(0);
 }
 catch (IOException fnfe) {
      System.out.println("Cannot locate input file! "+fnfe.getMessage());
      System.exit(0);
 }
  }
  public void processInput(String inputStr){
    if(flag==false) // First word is read
    {
      flag=true;             // so that subsequent entries always go to else part
     // This variables holds the value of characters scanned so far
      charScanned+=inputStr.length();
      dictionary.put(inputStr,wordScanned+"");
    }
    else
    {
      searchAndProcess(inputStr);
     // This method will first search for occurence of this word
    }
    //The each unmatched word processed is added to ArrayList for future comparisons

     wordsProcessed.add(inputStr);
     wordScanned+=inputStr.length();
  }
  private void searchAndProcess(String regEx){
           // Check whether the pattern of this word already exists
              // by comparing it with each of previous words in ArrayList
              if(!wordsProcessed.contains(regEx))
              dictionary.put(regEx,wordScanned+"");
     for(int i=0; i<wordsProcessed.size();i++)
     {
       compareWord(regEx,wordsProcessed.get(i).toString(),i);
     }
  }

 public void compareWord(String regEx,String input,int i){
   String prevValue = "-1";
   if(regEx.length()<=input.length())  //To check if input Pattern already exists in some previous
   {       // entries
     try {
       pattern = Pattern.compile(regEx);
       matcher = pattern.matcher(input);
     }
     catch(PatternSyntaxException pex){
       System.out.println("Pattern "+pex.getMessage());
     }
     if(matcher.find()) {
       // If the word is not being matched from first word
       if(i!=0){
         int start = getCharScanned(i-1)+matcher.start();
         int end = getCharScanned(i-1)+matcher.end();
         System.out.println("I found the substring \"" + matcher.group() +
                            "\" starting at index " +start +
                            " and ending at index " + end + ".");
         String value= dictionary.get(regEx).toString();
         dictionary.put(regEx,value+","+start);
//         searchInHashMap(regEx, true,end);
       }
       else
       {
         System.out.println("I found the substring \"" + matcher.group() +
                   "\" starting at index " + matcher.start() +
                   " and ending at index " + matcher.end() + ".");
         try {
           prevValue = dictionary.get(regEx).toString();
         }
         catch(NullPointerException nex) {
           prevValue = "-1";
         }
         if(prevValue!=null)
         dictionary.put(regEx,prevValue+","+matcher.start()+"");
       }
     }
     else
     {
       searchInHashMap(regEx, false, -1);
     }
   }
 }
 public int getCharScanned(int index){
   int count=0;
  for(int i=0;i<=index;i++)
   count = count+wordsProcessed.get(i).toString().length();
   return count;
 }
 public static void main(String g[])
 {
   new DFADictionary();
 }
}

This was my code but only diff is that inpur file regex.txt is of size 780 K and contains lot of gibberish characters.I am trying to make a search engine which will store all entries of a particular keyword in a rextr file
in HashMap in form of key value pairs where key is keyword and value is all occureneces of it separated by comma
Thanks in advance
[ Jess added UBB [code] tags to preserve whitespace, check 'em out! ]
[ April 20, 2004: Message edited by: Jessica Sant ]

[C. Nimo]

Hi.
Well, as stated before, you ARE using many Strings and String operations. It will be worthwhile to see what you can change to StringBuffers.
Also, as stated before, I'd check the JVM's maximum heap size flag, which is, btw, -Xmx not -Xms.
Nimo.

[Mogens Nidding]

The problem does not lie in the Java options, nor does it have anything to do with the fact that you use many string operations - you can safely do that. The problem lies in the design. If you have a good development environment, try to run the program in a debugger and see what happens. To give an example:
readLine() reads random input lines, but the program treats them as if they were words. Actually, it is far worse: The program treats them as if they were regular expressions. If the program reads e.g. a line containing only an "e", it will try to match "e" against every input line. And every "e" will match even if it is in the middle of a word. That's a lot of matches. Now imagine how many matches will occur if the program reads a blank line.
The explosive number of matches combined with the fact that the program stores every match in the directionary causes the OutOfMemoryException.
To say it in another way, what the program does is this:

1) For each line l in the file, do this:

1--a) Store the line in the dictionary

1--b) For each previously read line m in the dictionary, do this:

1--b--x) If m matches (~= contains) l, record that l was seen at the current file position

What I guess it should do is this:

1) For each word w in the file, do this:

1--a) Record that w was seen at the current file position

With this simpler structure, the program will be shorter, faster, and will no doubt be able to organize files of several megabytes without needing special java options.
I assume the first step you need to take is find a clean way to break the input into words. You can use a StringTokenizer for this purpose. You can also use regexps for this, but if you choose to do it, you should only need one generic regular expression that will match any word.
Hope this helps!
[ April 21, 2004: Message edited by: Nicky Bodentien ]