• Post Reply Bookmark Topic Watch Topic
  • New Topic

Java Read/Write on huge file system  RSS feed

 
Karthik Karnan
Greenhorn
Posts: 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I am working on a J2EE application where the user can upload/read/delete files. There is going to be a separate file server (SAN drive) for this.

Here is the worst case.

File Server size = 100TB
Maximum allowed size per file = 5MB
This leads to the maximum number of files = 20971520


The files will be stored in a specific folder structure but I don't know how much it will impact the performance of file search/read/write/delete in my java program for this large number of files. I thought of splitting the 100TB into multiple drives but still I have no idea what are the problems I am going to face at the worst case.

I appreciate any kind of suggestions. Thanks a lot in advance.
 
Tim Holloway
Bartender
Posts: 18531
61
Android Eclipse IDE Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Welcome to the Ranch, Karthik!

Since the files themselves aren't all that big, Java itself probably isn't going to be the limiting factor. More important is going to be the directory structure and filesystem that you use.

Not all filesystems even support multi-Terabytes of data. Some do, but should be considered carefully. I once worked with one that was projected to require multiple days to repair itself in the event of a major crash.

Different filesystems also vary in how they treat directories. Some limit the total number of files in a directory. Some don't technically limit them, but have problems with limitations in the support utilities. Some can handle many files, but the time required to open a file is dependent on the speed of a linear search of a directory. Others may use hashed directories, which generally makes them faster at opening files... provided that you don't end up with too many hash collisions and long overflow chains.

Your Mileage May Vary, in other words. I recommend setting up a pilot study.
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!