Process Huge XML documents using VTD XML parser

[Rajesh MadhanGopal]

Hi,

we came across the VTD XML Parser which seems to be very cool to parse XML files and we decided to use the VTD-XML Huge API to parse an xml file which is 3GB in size. we followed the documentation given in the below link

http://vtd-xml.sourceforge.net/codeSample/cs12.html

For some reason the VTD Huge API seems to fail while parsing and doesnt seem to be able to complete parsing the file, we were told that there could be some problem with the API as it was still in beta version. If there is anyone who was in fact able to use the API for parsing huge xml files, could you please let me know in case of any known tweaks or issues.

[Paul Clapham]

Seems to me your first step should be to find out if you actually have a problem. You said

For some reason the VTD Huge API seems to fail while parsing and doesnt seem to be able to complete parsing the file...

which suggests that you don't quite know whether the code worked or not. So first find out whether it's working or not. Does it produce the output you expected? Or does it throw an exception? If there's a problem, it may be as simple as not setting your classpath correctly, so asking people to provide solutions for problems which may not be your problem is really a waste of time for all concerned.

[Rajesh MadhanGopal]

Hi,

May be i wasnt clear enough...we tried the normal VTD API for a file which was 600MB size and it seemed to work fine, since the VTD HUGE parser was meant for files bigger than 2 GB - we tried a file which was 3GB and it did not work fine. We even tried a smaller file (500 MB) with VTD Huge and it did not seem to work fine. The code was updated correctly for the VTD HUGE parser and also the Jars required for VTD HUGE were included in the classpath, may be i am missing something which iam unable to figure out as yet..

But the question was : If there was a known issue with VTD Huge parser atall as some forums hinted the same....

[Karthik Shiraly]

Rajesh MadhanGopal wrote:...600MB size and it seemed to work fine....3GB and it did not work fine

Hi,

The post is rather vague. Are you getting an exception? Please TellTheDetails because ItDoesntWorkIsUseless.