Problem with processing data files of size larger than 350 MB

[amit bose]

Hi All,

Please find below the details of my query.

Problem: I need to process a huge(350 MB size) data file in Java.The data file is is basically a concatenation of multiple XMLs together.

What I need to do is..
(a) check if there are some unwanted characters in bewteen the XML tags
(b) If Yes, remove the tags
After the validation stage above, I need to write the file back to disc.

E.g. Input Data file sample (D1)
<?xml version="1.0" encoding="UTF-8"><books><!--- Books1. xml - some more tags go here --></books>some junk here
<?xml version="1.0" encoding="UTF-8"><books><!--- Books2. xml - some more tags go here --></books>
<?xml version="1.0" encoding="UTF-8"><books><!--- Books3. xml - some more tags go here --></books>more junk
<?xml version="1.0" encoding="UTF-8"><books><!--- Books4. xml - some more tags go here --></books>

(Please note that the content in input data file above will appear in a single line; For better readability I have shown indentation of the XMLs)


E.g. Output Data file sample (D2)
<?xml version="1.0" encoding="UTF-8"><books><!--- Books1. xml - some more tags go here --></books>
<?xml version="1.0" encoding="UTF-8"><books><!--- Books2. xml - some more tags go here --></books>
<?xml version="1.0" encoding="UTF-8"><books><!--- Books3. xml - some more tags go here --></books>
<?xml version="1.0" encoding="UTF-8"><books><!--- Books4. xml - some more tags go here --></books>

(The text 'some junk here' and 'more junk' have been removed in D2)


Earlier Solution:

I have shared my code below:

    
            //Read the file from disc           
            String filePath = in.getAbsolutePath();
            StringBuffer sbfContent = new StringBuffer(new Long(in.length()).intValue());
            try{
                
                  FileInputStream fstream = new FileInputStream(filePath);            
                  BufferedReader br = new BufferedReader(new InputStreamReader(fstream, "UTF-8"));            
                  String strLine;
                
                  //Read File Line By Line
                  while ((strLine = br.readLine()) != null)   {
                  sbfContent = sbfContent.append(strLine);
                  System.gc();
                  strLine = null;                 
                  }
                
                  //Close all the input streams
                  fstream.close();
                  br.close();
                  System.gc();
                      
                    //Logic to remove hanging elements 
                    String delim = "\\<\\?xml version=\"1.0\" encoding=\"UTF-8\"\\?\\>";
                    String xmlDecl = "<?xml version=\"1.0\" encoding=\"UTF-8\"?>";
                    String strContent = sbfContent.toString();
      
                    //Split the input file using delimiter 
                    String[] result = strContent.split(delim);
                    StringBuffer sbfValidatedContent = new StringBuffer(strContent.length());
                    int i = result.length;
                                     
                    //Process only if there are more than one input XML
                    String strTemp = null;
                    for (int x=0; x<i; x++) {       
                    strTemp =  result[x].trim();
                    if(!"".equals(strTemp)) {
                       
                        //Get the last character in the doc
                        int j = strTemp.length();           
                       String strLast = strTemp.substring(j-1);
                       
                       //Compare the last character with '>'; to verify correct output
                       if (!">".equals(strLast) ) {
                              int k = strTemp.lastIndexOf(">");
                              if(k != -1) {
                                    sbfValidatedContent.append(xmlDecl)
                                                               .append(strTemp.substring(0, k+1));                              
                              } else {
                                          //If the split file does not contain any XML declaration, then append blank
                                          sbfValidatedContent.append("");                                       
                              }
                       } else{
                              sbfValidatedContent.append(xmlDecl)
                                                         .append(strTemp);                     
                       }                
                      }
                    strTemp = null;
                    }

                
                //Write the file back to disc       
          	BufferedWriter out = new BufferedWriter(new OutputStreamWriter(new FileOutputStream(filePath), "UTF-8"));
          	out.write(sbfValidatedContent.toString());            
          	out.close();

The above code thrown OutOfMemory Error for file size more than 100 MB and it happens when I am trying to read the file.

The next thing that I tried was using buffers to read the file than line-by-line:

    	    BufferedReader br = new BufferedReader(new InputStreamReader(fstream, "UTF-8"));
	    String strLine;
	    String readData = null;
	    char[] buf = new char[4096];
	    int numRead = 0;
	    while ((numRead = br.read(buf)) != -1) {
	  	System.gc();
		readData = String.valueOf(buf, 0, numRead);                       
	  	sbfContent.append(readData);
	  
	   	//Give up resources
	   	buf = null;
	  	readData = null;
	  	System.gc();
		buf = new char[4096];
	  }

The above code worked fine till the data file size was 200 MB or less. However, now I have a data file of 350 MB size and it keeps giving the out of memory error.
Increasing the buffer size does not sound like a good option.

Let me know if there are any pointers for this problem.


Thanks,
Amit

[Somnath Mallick]

I think, since you are getting an out of memory error, it would help if you increase your JVM heap size.

java -Xms64m -Xmx256m classname

[William Brogden]

It looks to me like there is only one pass through the file.

Why don't you write chunks of valid data as they are accumulated?

Bill

[Carey Brown]

Your best bet is to process the XML in a serial fashion, in that way you have no memory problems. You could use either the STAX or SAX libraries for this.

Secondly, you are keeping 4 copies of the data in memory: sbfContent(twice), result, and sbfValidatedContent.

sbfContent should be emptied before trying to append to it again.

result and sbfValidatedContent should be released before trying to re-read the file.

[amit bose]

Somnath Mallick wrote:I think, since you are getting an out of memory error, it would help if you increase your JVM heap size.

java -Xms64m -Xmx256m classname

Thanks for the pointer Somnath.

However, I am already using a large heap size as below:

    java -Xms32m -Xmx1024m 

[amit bose]

William Brogden wrote:It looks to me like there is only one pass through the file.

Why don't you write chunks of valid data as they are accumulated?

Bill

Thanks for the pointer Bill.

Actually, I wanted to write chunks of valid data as they are accumulated but firstly, I need to read the input data file where the code fails. The input is also not a XML file that could be processes easily but rather multiple XMLs concatenated together.

[amit bose]

Carey Brown wrote:Your best bet is to process the XML in a serial fashion, in that way you have no memory problems. You could use either the STAX or SAX libraries for this.

Secondly, you are keeping 4 copies of the data in memory: sbfContent(twice), result, and sbfValidatedContent.

sbfContent should be emptied before trying to append to it again.

result and sbfValidatedContent should be released before trying to re-read the file.

Thanks for the pointer Carey.

I was going through the webpage but it seems STAX API allows to stream XML data. As my input is not a XML file but rather multiple XMLs concatenated together, I am not sure if I can use this. Please correct me if I am wrong.

Also, regarding the duplicacy of data in memory: I will remove the duplicacy but the code fails prior to reaching the duplicated content (i.e. sbfValidatedContent etc.)

[Somnath Mallick]

Since you say that the code is failing at the reading part, I think the sbfContent is becoming too big for the JVM to handle! Could you debug the code and tell us where exactly (which line) the code is failing.