• Post Reply Bookmark Topic Watch Topic
  • New Topic

Reading File  RSS feed

 
muhsin sabeer
Ranch Hand
Posts: 35
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I am devloping an application which get the input from user and search in the ascii file and display back to the user. The ascii file is very big(18mb atleast). What is the best way to do this? I guess using vector and array cost more memory loading.
Pls advice me.
 
Stan James
(instanceof Sidekick)
Ranch Hand
Posts: 8791
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Here are a few things to think about ... I'm sure there are more options

If you have fixed length records that are sorted on a key field (big ifs) you can use RandomAccessFile to do a binary search and home in on the record you want.

Try reading the whole file into memory. If the records have a key field, like SSN to identify a person, you could put them into a Map with a key. Otherwise a List of some type is better than Vector. Use JConsole with JDK 5 or some tool to see how much memory is used before & after loading, decide if you can spare that much.

To save memory, read the whole file but only save keys, byte offsets into the file and maybe record length in memory. Then when you need a record use the key to get the offset and use RandomAccessFile to read the record. Again see how much memory you use.

Use a database.

Any of those sound promising?
 
Jeff Albertson
Ranch Hand
Posts: 1780
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
>=18MB!!! Why no database?
 
muhsin sabeer
Ranch Hand
Posts: 35
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
The text file is generated from database(oracle) server. This file is archive records from the database. Current application doesn't has function to query the archive data. So, the DBA helps to copy the archive data ( by requested day). Each day got huge amount of data. Currently running the simple perl script to search for the record. But I want to build nice GUI to do this task.

Please advice me the design issues here. Anyway??
 
Joe Ess
Bartender
Posts: 9426
12
Linux Mac OS X Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by muhsin sabeer:
What is the best way to do this?


That really depends on what your requirements are. Why are you replacing the Perl script? Just to have a pretty interface? Why not make a pretty interface that calls the Perl script?
If you are concerned about query speed, there's always comprimises between the speed of a query, the amount of memory required and the amount of work you have to do. Is the linear search through the data too slow? You could load it into an in-memory structure but that would cost lots of memory. 18 megs of text is going to be at least twice that big in-memory, plus whatever data structures you use to make an index or hash of the raw data. Does the computer you are running on have tons of memory? If yes, maybe that's your answer.
Sorting the data would make your query go faster but there's an upfront cost. You could build an index of the file on-disk (a file with rows of two fields each: key field and index into the 18 meg file, sort the index by the key field and you can do a binary search of the index). It would be VERY fast, not a big memory cost, but again, there's the upfront sort and index build cost.
Anything more complex than that (i.e. searching on more than one field, combinations of values, ranges) and you're better off loading the data back into a database.
Again, you have to define your requirements. THEN you can ask if there is a "best way" to solve them.
 
muhsin sabeer
Ranch Hand
Posts: 35
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi Joe Ess,

What are the ways to call from Swing GUI to perl script?
Are there any way to search a record in text file without opening or loading into main memory (like in Microsoft VBA).
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!