Win a copy of Kotlin in Action this week in the Kotlin forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic

How to input big big ascii files  RSS feed

 
Jimmy Chen
Ranch Hand
Posts: 54
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I want to input a very big txt file(about 500m) and analysis it.

Unless I read it all, I can't analysis it. But it's so big, I'm afraid
java can't deal with it. Can I use a array like 50000*50000? Is it a big
problem to jvm?

thanks!
 
William A White
Ranch Hand
Posts: 35
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Depending on the kind of analysis you need to do, you probably don't need to load the whole thing into memory. Just read it line by line, discarding each line as it is read and analyzed.
 
Joe Ess
Bartender
Posts: 9428
12
Linux Mac OS X Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by Yu Liang:
I'm afraid java can't deal with it.


It isn't Java you have to worry about. It's your hardware. Do you have over 500Mb of ram? You should take a look at
Memory-mapped files.
 
Steven Bell
Ranch Hand
Posts: 1071
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
You may want to look at the FileChannel and MappedByteBuffer classes.
 
Jimmy Chen
Ranch Hand
Posts: 54
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks you all. I will try.
 
With a little knowledge, a cast iron skillet is non-stick and lasts a lifetime.
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!