Tool for reading big text files

[sachin yadav]

Hi All,

I have a problem where my java code has to read big log files(around 4GB) containing XML and parser those XML to populate data. I can easily parse data using JAXB but i need to know if there is any open source tool provided to read those log files? I have to read these log files to identify a XML request chunk and process it continually till end of file.

I can have java code to read using buffered reader but performance will be an issue then for the reading a 4GB file along with processing data chunk. Can someone please suggest a tool of any design pattern/architectural ideas?

[Ulf Dittmer]

The SAX API should be able to handle XML files of just about any size, and is likely to be faster than using JAXB or other high-level APIs.

[sachin yadav]

Will SAX be able to read and parser XML from a text log file of 4 GB?

[Ulf Dittmer]

Thart's what i said, and it is, of course, easy enough to test.

[sachin yadav]

Thanks Mr. Dittmer, i will try it.

[Rob Spoor]

SAX and StAX don't store the entire file in memory, only the separate elements and their text. After it's been processed, SAX / StAX discards it (your code may still keep a reference though). Unless your XML file has elements with millions of attributes, or has enormous pieces of text, SAX and StAX should be able to handle the file just perfectly.