BufferedReader Problem

[susha susha]

Hello everbody,
I am doing a statistics program which reads a very big log file which goes up to GB.i am using bufferedreader's readline method.but its throwing out of memory exception.
Which other method should i have to use ??? but i wanat to read only line by line...
can i use datainputstream's readline method or randomaccessfile's readline method.. can i use

[William Brogden]

Your problem must lie elsewhere. BufferedReader only uses a small amount of memory. Probably you are hanging on to data (Strings?) you don't need to keep.
Bill

[Ernest Friedman-Hill]

Moving to Java in General (Intermediate); followups there, please.

[Jim Yingst]

Are you sure the log file has line separators in it? If not, then readLine() will try to put the whole file into a single String, whcih would explain your problem. Otherwise the most likely explanation is what Bill said - you're probably holding onto Strings in memory somehow. Are you adding anything into a List or other memory structure each time you process a line?

[Nicholas Cheung]

Please also note that the *line seperator*.
If the log file is created in MS Window while your program is run on Unix based system, the line seperator is different!
In MS Windows, it is \r\n, but in Unix based system, it is \n only. Thus, you should also make sure the log file matches the property.
Otherwise, the readline() method may reads the whole file into memory and OutOfMemoryException may then be thrown.
Nick

[Jim Yingst]

Nicholas - well no, actually the readLine() method of BufferedReader is pretty flexible about this. See the API. It does not use System.getProperty("line.separator") at all. Either a lone \r, or a lone \n, or a \r\n are interpreted as a single line separator. (Meaning \n\r would be two line separators.) In some cases this means readLine() may detect an extra line separator where none was intended, but never the other way around.

[susha susha]

dear all,
thank you for your valuable information..
but,
Since i am using buffered reader.. its storing in a buffer area so out of memory error is araised...
but if use the readline() method of randomaccess file or datainputstream does this error will be raised..
thanx in advance
rekha

[Maulin Vasavada]

Hi susha
As you mentioned, your file could be in GBs. So probably JVM is running out of memory while reading the data and putting into the string buffer you have. So may be the problem is just the so much data.
If that is then you can try by increasing the size of maximum memory available to JVM by -Xms options if you are using SUN JDK. For other JDK's I am not sure how we can increase memory as have never worked with them.
so you would do something like,
c:\>java -X30M yourMainClass yourInputs
where M indicates Megabytes...
do this and try to see if the data you can read is more as you increase memory (even if at some point you run out memory and get the exception). You know what I mean? This procedure will ensure if the problem is the huge data size...
Regards
Maulin

[susha susha]

dear all,
hi Jim Yingst,
For ur information there is no line separator.
Actually i am reading log file line by line...
in each i am taking string tokens like... date,ip addresses,hours etc and adding onto the separate vectors...
will it be a probs...
Thanx for the suggestion for increasing the jvm size..
but i dont understand the syntax ..
what is that input i should give..
thanx
susha

[Jeroen Wenting]

In case there's no line separator you can't use readLine at all.
You have to determine your record length and read the bytes manually by that number at a time.
And of course don't hang onto the data once you're done with it because that would lead to memory being used too much.
Use StringBuffers everywhere instead of Strings to reduce memory usage and object creation.

[Jim Yingst]

For ur information there is no line separator.
Actually i am reading log file line by line...
If there are no line separators, then "I am reading log file line by line" means "I am reading one really, really long line." Which is why you're running out of memory. As Jeroen says, you'll need to look for some other record separator. Or you need to know something about how the file is organized. It's hard to say anything else here - who would ever write a 1 GB "log file" with no line separators?

[Max Habibi]

Hi susha,
What I would suggest is this following.
1. create a File object that points to your log.
2. get the size of that File object, and divide by 100.
3. loop through file, and read 1/100 of it at a time.
HTH,
M

[Stefan Wagner]

Originally posted by susha susha:

For ur information there is no line separator.
Actually i am reading log file line by line...
in each i am taking string tokens like... date,ip addresses,hours etc and adding onto the separate vectors...

I guess you're not familiar with the term 'line separator' and expect something very special here.
If you read it line by line there MUST be line-separators, because lines are separated by theses magic line-separators.
But if you read a 1-GB logfile and store date, IPs and hours in Vectors, they might need about 1GB in memory.
Of course, a timestamp and an IP may be represented in a more or less memory-consuming format (more: String, less: IP: 4-Bytes) - but you need the -X switch though.

java -Xms1024M YourApp 

will launch the jvm with 1024 MB ramsize.
Of course your machine should have these 1024 MB ram (real + swap).

 java -X 

gives a very brief overview.

[susha susha]

dear Stefan Wagner,Jim Yingst
Readline method do not expect line separator...
Kindly look into the API doc.It reads the file line by line
with out any line separators..
Hi,
Habib,
This could the best suggetion..
I will do that..
Thanx
Susha

[susha susha]

dear Stefan Wagner,Jim Yingst
Readline method do not expect line separator...
Kindly look into the API doc.It reads the file line by line
with out any line separators..
Hi,
Habib,
This could the best suggetion..
I will do that..
Thanx
Susha

[Jim Yingst]

Susha, it seems that somhow we're just not communicating here. Let me try another approach:
How many lines are there in your log file?
How long is the first line in your log file?
Can you show us what the first 3 lines in your log file look like (if they're not too big)?

[Stefan Wagner]

susu:

Readline method do not expect line separator...
Kindly look into the API doc.It reads the file line by line
with out any line separators..

I guess you misinterpreted something.
If you have a sourcefile

abcd efgh 1223\n
asdf jkl 017

with \n as lineterminator (-separator).
When you perform

s = readLine ();

s will be "abcd efgh 1223" and not "abcd efgh 1223\n", which means, the lineseparator is removed by reading the line, but it depends vitaly on the existance of lineseparators.
If you don't believe: Spend me a cent for every LS you find in the 1 GB logfile, and I stop insisting on my opinion