Win a copy of Murach's Python Programming this week in the Jython/Python forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic

maximum size in memory file  RSS feed

 
jay vas
Ranch Hand
Posts: 407
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi guys :

Two really important questions for an app that must be deployed soon, so please help ASAP...
1) What is the maximum size of a random access file in java ? Is it limited to the size of the JVM ?

2) Is there a way to allocate an unlimited amount of memory and heap space to the JVM, that is, allow the JVM to use swap space on the hard drive when the maximum JVM maximum memory is used up ?

Thanks

J
 
Ulf Dittmer
Rancher
Posts: 42970
73
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I would be surprised if there is an upper limit on the size of a random access file Java can use. The file contents are not kept in memory, after all.

It should be easy to test if this is true, though.
 
jay vas
Ranch Hand
Posts: 407
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Why arent the file contents in memory ? I thought that was the point

Basically, I need a file that I can rapidly write to. Im finding that writing to a regular file is a little slower then what i need, and im assuming a RandomAccessFile object will speed things along.
 
jay vas
Ranch Hand
Posts: 407
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Why arent the file contents in memory ? I thought that was the point

Basically, I need a file that I can rapidly write to. Im finding that writing to a regular file is a little slower then what i need, and im assuming a RandomAccessFile object will speed things along.
 
Ulf Dittmer
Rancher
Posts: 42970
73
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Why arent the file contents in memory ? I thought that was the point

No, the point is to provide random access (which FileInputStream and FileOutputStream don't). You might want to investigate the NIO package, especially FileChannel and MappedByteBuffer, for memory-mapped files.

Any I/O involving files of a size where you're concerned about maximum memory is bound to be not very fast.
 
jay vas
Ranch Hand
Posts: 407
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks again... Im still lost here though and want to ask you two more questions, if you dont mind !

1) Could you please elaborate on the statement at the end "an i/o on files where youre concerned about memory is bound not to be fast"?

2) Will an the NIO solutions you discussed be limited by JVM memory, or will they be managed by the OS so that they may be arbitrarily large ?
 
Jim Yingst
Wanderer
Sheriff
Posts: 18671
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I'm not Ulf, but...

[Jay]: 1) Could you please elaborate on the statement at the end "an i/o on files where youre concerned about memory is bound not to be fast"?

You're evidently working with very big files, and big files take time to read or write. Furthermore any time you attempt to use more memory than is physically available using swap files, that slows things down. Real RAM is faster than reading and writing files. But if you use more RAM than your machine has, you're not using real RAM, you're using virtual RAM, which means you're really reading and writing files. There is no reason to think that using virtual memory should be faster than files, because it really is using files.

[Jay]: 2) Will an the NIO solutions you discussed be limited by JVM memory, or will they be managed by the OS so that they may be arbitrarily large ?

In general I believe they're limited by JVM memory, but this may depend on the operating system. The specifications are vague about this, so the best thing to do is simply test to find out. If you can create a MappedByteBuffer which is bigger than your JVM memory, then you'll know. I suspect that you won't be able to use this to access files bigger than memory.

Have you tried using a profiler to discover which operations are taking the most time? IO is probably your bottleneck, but it can still be worthwhile to know which specific operations are the problem. There may still be ways to speed up the code, but it depends what you're doing.
 
Ulf Dittmer
Rancher
Posts: 42970
73
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Could you please elaborate on the statement at the end "an i/o on files where youre concerned about memory is bound not to be fast"?

JVMs are typically run with at least 256 MB of memory on the server. If that is not sufficient for holding all file data you must be dealing with very large files, certainly more than 100MB. Performing write access with that amount of I/O is just not going to be very fast no matter how you do it, just by its size.
 
Don't get me started about those stupid light bulbs.
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!