Win a copy of Reactive Streams in Java: Concurrency with RxJava, Reactor, and Akka Streams this week in the Reactive Progamming forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Liutauras Vilda
  • Junilu Lacar
  • Jeanne Boyarsky
  • Bear Bibeault
Sheriffs:
  • Knute Snortum
  • Tim Cooke
  • Devaka Cooray
Saloon Keepers:
  • Ron McLeod
  • Stephan van Hulst
  • Tim Moores
  • Tim Holloway
  • Carey Brown
Bartenders:
  • Piet Souris
  • Frits Walraven
  • Ganesh Patekar

Mega File I/O Efficiency Strategies?

 
Greenhorn
Posts: 19
Eclipse IDE VI Editor Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Let's first say that a database would be the best solution but unfortunately is not an option in this situation for reasons outside of my control.

You have an application that checks the status of a widget every 10 minutes. Let's say there are 2000 unique widgets to check.

The requirement is to keep the last 5 status results of a widget and it has to be stored in a file (widget12345.stat).

The application uses a ScheduledExecutorService to run every 10 minutes and a thread pool to do the work of checking the status of each widget and storing in a file.

Academically speaking, I hate using files, but have no choice. Is there any way around the File I/O bottleneck of updating 2000 files every 10 minutes?

Or there is not much that can be done here?
 
Saloon Keeper
Posts: 6225
58
Eclipse IDE Firefox Browser MySQL Database VI Editor Java Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Have you done any sort of performance evaluation? It seems to me that updating 2000 files with only 5 lines in then every 10 minutes shouldn't be an issue, but I could be wrong.

Could you use a single random access file with fixed length records? That would only involve 2000*1 record writes every 10 minutes.
 
Bartender
Posts: 1868
81
Android IntelliJ IDE MySQL Database Chrome Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Have you looked into server less databases, maybe something like SQLite (http://sqlite.org/)?
Using something like that may be more efficient then using just files and you can name these files whatever you want.

Edit Reason: Typo fixed
 
Seymour Jones
Greenhorn
Posts: 19
Eclipse IDE VI Editor Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
SQLite is not an option.

I mis-spoke on the widget files. Some will have 5 lines some may have up to 15 lines.
 
Carey Brown
Saloon Keeper
Posts: 6225
58
Eclipse IDE Firefox Browser MySQL Database VI Editor Java Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Seymour Jones wrote:SQLite is not an option.


In order for us to suggest useful alternatives it would help us to know what in your requirements disallows SQLite. I'm still leaning towards a random access file but perhaps the issue that causes you to reject SQLite  will also eliminate a random access file.
 
Carey Brown
Saloon Keeper
Posts: 6225
58
Eclipse IDE Firefox Browser MySQL Database VI Editor Java Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
What kind of data will need to be written? Does every widget have the same field(s)? Do you have a DAO class that represents a widget's record? Can all the widgets make use of one DAO class? Once the data is written to a file, how will it be accessed? How often? Concurrently with writing?
 
Seymour Jones
Greenhorn
Posts: 19
Eclipse IDE VI Editor Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I am operating in an environment where there is a list of approved standards that I must adhere to and SQLite is not on the list.

A sample widget file would look something like this....

cat widget12345.stat

 
Rancher
Posts: 3439
33
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
More clarification on the requirements:

keep the last 5 status results of a widget

Does that say there is a circular queue or list that holds the last 5 result packets.  Each result packet can have from 5 to 15 lines.
What is the max size in bytes of the largest result packet?
 
Seymour Jones
Greenhorn
Posts: 19
Eclipse IDE VI Editor Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Norm Radder wrote:More clarification on the requirements:

keep the last 5 status results of a widget

Does that say there is a circular queue or list that holds the last 5 result packets.  Each result packet can have from 5 to 15 lines.



Each result packet can have up to 15 items (app1 - app15).

Each item has a "status" in order from oldest to most current and it will keep up to 5. Once you reach 5, the oldest status will be bumped off.
 
Norm Radder
Rancher
Posts: 3439
33
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
What is the max size in bytes of the largest result packet?

My objective it to allocate a fixed size slot in a random access file for each packet for each widget and have a pointer to key points in that slot to keep from having to move bytes around.
If 2000*maxSlotSize is reasonable
 
Carey Brown
Saloon Keeper
Posts: 6225
58
Eclipse IDE Firefox Browser MySQL Database VI Editor Java Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
The requirements in your original post don't leave any wiggle room to suggest alternatives.

Writing one file per widget is inefficient. Could you have a new file every 10 minutes whose file name is the timestamp? Write all widget data for that 10 minute interval to a single file and then delete files older than some older timestamp.
 
Carey Brown
Saloon Keeper
Posts: 6225
58
Eclipse IDE Firefox Browser MySQL Database VI Editor Java Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Here's a simple test for seeing how long it would take to write 2000 files. On my machine it only took about 4 seconds which is trivial for 10 minute intervals.
 
Bartender
Posts: 3509
150
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
if the status of such a widget has not changed in the last 10 minutes, must you still update the file?
And must you update the (changed?) files every 10 minutes, or can you keep them in memory and do the writes only once in every, say, hour?
 
Carey Brown
Saloon Keeper
Posts: 6225
58
Eclipse IDE Firefox Browser MySQL Database VI Editor Java Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Be careful to avoid premature optimization. I don't think that you've established that file I/O is your bottle neck.
 
I didn't like the taste of tongue and it didn't like the taste of me. I will now try this tiny ad:
Java file APIs (DOC, XLS, PDF, and many more)
https://products.aspose.com/total/java
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!