• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Ron McLeod
  • Rob Spoor
  • Tim Cooke
  • Junilu Lacar
Sheriffs:
  • Henry Wong
  • Liutauras Vilda
  • Jeanne Boyarsky
Saloon Keepers:
  • Jesse Silverman
  • Tim Holloway
  • Stephan van Hulst
  • Tim Moores
  • Carey Brown
Bartenders:
  • Al Hobbs
  • Mikalai Zaikin
  • Piet Souris

Looking for a design pattern to process18000 records

 
Greenhorn
Posts: 25
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi, A question regarding what pattern to use. I have about 18000 records that I need to validate and process before I can apply the business rules. I sure don't want to read 18000 records at one time but rather read them in chunks. is there any pattern out there that I can use. thank you.
 
Bartender
Posts: 1205
22
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I think the most obvious of the canonical patterns to use would be the Proxy?

The client object would instantiate the LotsOfDataProxy object and access it as though all the data were actually in memory. The LotsOfDataProxy object, which would implement perhaps the Collection or Enumeration interface, would then manage the reading of records in chunks at a time. In this case you'd be making a Proxy for the collection of records, not for the records themselves.

If it turns out that not every records will need to be considered in the business logic, LotsOfDataProxy might read the data into a byte array and then use a ByteArrayInputStream wrapped in/decorated with an ObjectInputStream to instantiate only the Record objects that are actually needed. Or maybe a BufferedInputStream makes more sense.

So I'd say Proxy is the pattern, but there are still a bunch of application specific details not dictated by the pattern that you'll have to work out.

What else...
If there is a small number of types of records and all the records of any one type share quite a bit of common information, you could look at the Flyweight pattern. This might cut down on the amount of memory needed to load 18000 records into memory.

Ryan
 
Zul Chewanabas
Greenhorn
Posts: 25
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
thank you Ryan. I will look into that pattern. I appreciate it!
 
author
Posts: 14112
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Well, why wouldn't you just read and validate them in a loop?
 
Zul Chewanabas
Greenhorn
Posts: 25
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
yes, the loop will be used in the implementation. I was looking towards designing it first, i.e loading 18000 into a loop at one time is probably not feasible - ryan answered that question with the pattern so I can chunk the data appropriately. thx
 
Ryan McGuire
Bartender
Posts: 1205
22
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I know back when I was writing C code it was MUCH more efficient to read in 10,000 bytes from a file into a buffer and then parse through it in memory than to read 200 structs @ 50 bytes each in a loop.

I have not tried the same comparison in Java, but i wouldn't be surprised if the same type of tradeoff still holds.

Ryan
 
Ilja Preuss
author
Posts: 14112
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Originally posted by Zul Chewanabas:
yes, the loop will be used in the implementation. I was looking towards designing it first, i.e loading 18000 into a loop at one time is probably not feasible - ryan answered that question with the pattern so I can chunk the data appropriately. thx



I don't understand what you mean by "loadind into a loop". What I was referring to was something along the lines of



I don't see any need for a more advanced design pattern yet.
 
Ilja Preuss
author
Posts: 14112
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Originally posted by Ryan McGuire:
I know back when I was writing C code it was MUCH more efficient to read in 10,000 bytes from a file into a buffer and then parse through it in memory than to read 200 structs @ 50 bytes each in a loop.

I have not tried the same comparison in Java, but i wouldn't be surprised if the same type of tradeoff still holds.



It holds in many cases. That's what BufferedInputStream is good for!
 
Zul Chewanabas
Greenhorn
Posts: 25
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
"Out of clutter, find simplicity" ~ Einstein. thanks guys

here is related article - not really related to my batch job but more like for multiple accesses.

http://www.javaworld.com/javaworld/jw-07-2001/jw-0720-cache.html

ltr
 
reply
    Bookmark Topic Watch Topic
  • New Topic