• Post Reply Bookmark Topic Watch Topic
  • New Topic

Reading big txt file  RSS feed

 
Hernan Tavella
Ranch Hand
Posts: 42
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hello, i'm trying to read a big txt file (8gb) is a log from a server. This is the code that i use to read it, but my question is if any better and fast way to read the file.

 
K. Tsang
Bartender
Posts: 3648
16
Firefox Browser Java Mac OS X
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
you should look into NIO (the new IO) API.

Using the traditional IO API, may take longer but what are you doing with the read data? If you are just printing out to console, why not just open the log file as is. If you are searching for particular error code or text, then just define this search while the program read the file.
 
Hernan Tavella
Ranch Hand
Posts: 42
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
The idea would to split the file in smaller pieces. But first i need to read it or not?
 
Richard Tookey
Bartender
Posts: 1166
17
Java Linux Netbeans IDE
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Since you want to read the file as characters you should wrap the input stream in an InputStreamReader (specfiying the character encoding of the file)and wrap that in a BufferedReader. You can then read lines or characters.

Note - your original code is flawed. Your creation of a String assumes that the read() method reads the whole of a buffer each time but this is not guaranteed. Also, it assumes the default character encoding has one byte per character since it cannot guarantee to read an exact a whole number of characters otherwise.
 
Debal Guha
Greenhorn
Posts: 9
Eclipse IDE Spring Tomcat Server
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Since it's a very big log file, isn't it better to use unix commands/shell script/perl script itself to piece it, instead of using java? Just curious guys......my point is .....Is java really a good choice to handle such big files?? Won't it be faster with unix utilities??? Unless there are very complex logic involved.
 
William Brogden
Author and all-around good cowpoke
Rancher
Posts: 13078
6
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
It is my understanding that converting bytes to UNICODE characters in a Java String is rather expensive.



If your log is entirely in ASCII characters, leaving them as bytes is indicated.

Exactly what do you have to do with the log? There is a big difference between processing every line and scanning for an unusual character sequence.

Bill
 
Consider Paul's rocket mass heater.
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!