• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

Which way is better to store objects on my disk?

 
Ranch Hand
Posts: 271
6
Eclipse IDE MySQL Database Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hello everyone

So I am working on a project which requires a lot of data to be stored on the disk now I have to ways (but feel free to recommend other ways) to store that data and extract them. Please keep in mind there will be no less than 2000 Objects to be saved

Method 1:
Save each object in its own .dat file and organize them into folders based on their type like in the pic below. The .dat files names will be set in the following format "[Name]_[Unique ID].dat"



This way the computer when displaying a list will just display the following format "[Name] [Unique ID]" saving precious memory and when a user needs to see the details of the object the method readFromFile(String fileName) would be called.

Me personally I prefer this method over the other one I am about to mention for the following reasons:
If one Objects is corrupted it will not corrupt the rest along with it since each item is independant from the other.
Easier to extract and filter them because they are all split up in folders accordingly.
Would be easy to modify an object since all i need to do is simply delete the folder and create a new one in its place.


Method 2:
Just the whole list of the objects saved in .dat file of the super type so I would have the following .dat files only.

Attachments
Weapons
Cloths
Soldiers
Vehicles
Mags

Now this method is a million times easier to setup but if one file gets corrupted I will have hundreds or more objects that would need to be re-created.

If there are other ways I am all eyes and hopefully brains to understand them

Cheers
 
Sheriff
Posts: 5555
326
IntelliJ IDE Python Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hmm, how to store data on a disk... this feels like a problem that's already been solved... oh that's right, ever heard of a database? Why not use one of those?
 
Marshal
Posts: 79151
377
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Yosuf Ibrahim wrote:. . . Save each object in its own .dat file and organize them into folders based on their type . . .

That looks like a maintenance headache You will have no end of difficulty linking the individual files to your objects. That is why file management systems have largely been abandoned.

Just the whole list of the objects saved in .dat file of the super type  . . .

That looks like a recipe for poor performance as you will find yourself reading the same file repeatedly and discarding much of the information read.

You may be able to serialise the entire application into an object file, but there must be a better way to store information. Maybe it has already been mentioned in replies to your query.
 
Marshal
Posts: 28175
95
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I'd agree with Tim that a database would be the preferable way to go. To me a database is an SQL database, but your objects may not fit well into a small collection of tables. If that's the case then you could look at NoSQL, of which there are a lot of potential candidates depending on how much of their advertised features you actually need. (And which I know very little about.)

It would also be helpful if you had some idea of how often you planned to save and retrieve those objects and whether groups of objects might be combined. I made good use of XML serialization to save objects which were configuration for a JMX server, so the objects were rarely read or written. That's a simple thing to do but it might not match your requirements.
 
Saloon Keeper
Posts: 10687
85
Eclipse IDE Firefox Browser MySQL Database VI Editor Java Windows ChatGPT
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Yosuf Ibrahim wrote:I do not think databases would fit my needs because think of the objects I am saving as words in a dictionary, the user will be only reading that data will not modify it or add more to it, all the data will be added during the development of the application.


The data has to be put into persistent storage at some point so that the user can read it, even if that is by the developer. Database, flat file, CSV file, XML file, JSON file, properties file, serialized files, etc., are a variety of ways to persist storage of data. For the initial loading of data you might need a utility program to enter the data a persist it in you choice of approach, or a text editor if you choose a flat file.

What is the "unique id" used for? Can you have multiple weapons of the same type?
 
Yosuf Ibrahim
Ranch Hand
Posts: 271
6
Eclipse IDE MySQL Database Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Carey Brown wrote:

Yosuf Ibrahim wrote:
What is the "unique id" used for? Can you have multiple weapons of the same type?



The unique ID aka classname is how the game engine knows the weapon the developer is trying to equip a soldier, crate etc.

 
Carey Brown
Saloon Keeper
Posts: 10687
85
Eclipse IDE Firefox Browser MySQL Database VI Editor Java Windows ChatGPT
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Yosuf Ibrahim wrote:The unique ID aka classname is how the game engine knows the weapon the developer is trying to equip a soldier, crate etc.


So, you can only have one gun?
 
Yosuf Ibrahim
Ranch Hand
Posts: 271
6
Eclipse IDE MySQL Database Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Carey Brown wrote:

Yosuf Ibrahim wrote:The unique ID aka classname is how the game engine knows the weapon the developer is trying to equip a soldier, crate etc.


So, you can only have one gun?



Of every type yeah, for example:

Name: RPG-42 Alamut
ClassName: launch_RPG32_F
 
Carey Brown
Saloon Keeper
Posts: 10687
85
Eclipse IDE Firefox Browser MySQL Database VI Editor Java Windows ChatGPT
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Yosuf Ibrahim wrote:

Carey Brown wrote:So, you can only have one gun?


Of every type yeah, for example:

Name: RPG-42 Alamut
ClassName: launch_RPG32_F


It sound like the name itself is unique if you can only have one, so why would you also need a unique id?
 
Yosuf Ibrahim
Ranch Hand
Posts: 271
6
Eclipse IDE MySQL Database Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Carey Brown wrote:

Yosuf Ibrahim wrote:

Carey Brown wrote:So, you can only have one gun?


Of every type yeah, for example:

Name: RPG-42 Alamut
ClassName: launch_RPG32_F


It sound like the name itself is unique if you can only have one, so why would you also need a unique id?



It is the way the game has been made, the name is displayed for the players the classname is used by developers\modders to communicate with the engine. This is why I need them all
 
Carey Brown
Saloon Keeper
Posts: 10687
85
Eclipse IDE Firefox Browser MySQL Database VI Editor Java Windows ChatGPT
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
You could use a single CSV file to contain all of the weapons and other classes. Example:
 
Tim Cooke
Sheriff
Posts: 5555
326
IntelliJ IDE Python Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
The application I work on at work has a very large static data set that is read into memory on startup and never modified at run time. That data set lives in a single file and is modified as and when new items are required, which is rarely. The new data is then loaded the next time the application is started up.
 
Yosuf Ibrahim
Ranch Hand
Posts: 271
6
Eclipse IDE MySQL Database Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Tim Cooke wrote:The application I work on at work has a very large static data set that is read into memory on startup and never modified at run time. That data set lives in a single file and is modified as and when new items are required, which is rarely. The new data is then loaded the next time the application is started up.



Isn't that bad programming, having all the data read on startup using up maybe a GB of ram if not more that is not required??
 
Carey Brown
Saloon Keeper
Posts: 10687
85
Eclipse IDE Firefox Browser MySQL Database VI Editor Java Windows ChatGPT
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Tim Cooke wrote:The application I work on at work has a very large static data set that is read into memory on startup and never modified at run time. That data set lives in a single file and is modified as and when new items are required, which is rarely. The new data is then loaded the next time the application is started up.

Now I'm confused. If you already have a large single file why are you looking to make lots of little .dat files?
[edit]Sorry, thought Tim was the OP.
 
Tim Cooke
Sheriff
Posts: 5555
326
IntelliJ IDE Python Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Yosuf Ibrahim wrote:Isn't that bad programming, having all the data read on startup using up maybe a GB of ram if not more that is not required??


It's a design choice. The key requirement of this particular application is speed, speed, and more speed and that data set is queried a lot, so having it in memory is the right solution for us. Hardware cost is a secondary concern, in fact it's almost of no concern as long as it's fast as heck.
 
Carey Brown
Saloon Keeper
Posts: 10687
85
Eclipse IDE Firefox Browser MySQL Database VI Editor Java Windows ChatGPT
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Yosuf Ibrahim wrote:Isn't that bad programming, having all the data read on startup using up maybe a GB of ram if not more that is not required??


Not necessarily. How many game-objects do you anticipate? What would be the approx. number of bytes needed to store the average object? Some data might take less room in memory than in your file. For example, the word "weapon" might be stored in memory as an enum which would take up less space. If space is really an issue then a database is probably your best bet. It wouldn't need to be a complicated database and perhaps a single table might do the trick depending on how you implement it.

I noticed in your other thread that a weapon can have an attachment(s) and/or magazine(s). Do you have some sort of object relationships stored in your huge file?

When you say ClassName, are we talking about a Java class or some game-object class?

 
Tim Cooke
Sheriff
Posts: 5555
326
IntelliJ IDE Python Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
What we're all kinda getting at here is that there are trade offs with every solution and it's up to you to decide which works for you the best. For example, the application I mentioned needed fast read times and spending $$$ on server memory is no problem. Perhaps you need quite fast read times too but can't afford to throw memory at the server, in which case a database might be a good solution. You can still have a static data set in the database, say you have a one shot data load process that can be done by a database admin person, then your application needs only to support read operations on the database. Perhaps speed is of no importance and the file based system you originally proposed might work out after all.

Know your problem before reaching for a solution.
 
reply
    Bookmark Topic Watch Topic
  • New Topic