Granny's Programming Pearls
"inside of every large program is a small program struggling to get out"
JavaRanch.com/granny.jsp
Win a copy of Cross-Platform Desktop Applications: Using Node, Electron, and NW.js this week in the JavaScript forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic

What is the fastest way to loop in java in huge data structure  RSS feed

 
Meir Yan
Ranch Hand
Posts: 599
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
He all
i have data structure that is list that in every element there is hashtable that contains key value pairs, there is no limit on the list size
and it can be 500 elements long , now the thing is i need to out put this structure content as string , to do simple loop on the list and drill
to each hashtable and get the key value as string will take very long time .
how can i optimize this task? and make this action with minimum cpu ?
 
Peter Chase
Ranch Hand
Posts: 1970
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
If your requirement is to output the whole data structure (is it?), then surely you cannot avoid traversing the whole list and every hash map. You can use an Iterator or an enhanced "for" loop to do so.

If you're building up a huge String from the data, do use StringBuilder. This is better than StringBuffer because it avoids unnecessary synchronisation. Both are much better than String concatenation using "+" as that creates a lot of unnecessary temporary objects.
 
Ilja Preuss
author
Sheriff
Posts: 14112
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by Meir Yan:

i have data structure that is list that in every element there is hashtable that contains key value pairs, there is no limit on the list size
and it can be 500 elements long , now the thing is i need to out put this structure content as string , to do simple loop on the list and drill
to each hashtable and get the key value as string will take very long time .
how can i optimize this task? and make this action with minimum cpu ?


What do you mean by "very long"? How much time did it take when you tried?

500 elements aren't particularly many - it shouldn't take longer than a few milliseconds to iterate through them, even when using a non-optimized approach.

Moving to our Performance forum...
 
William Brogden
Author and all-around good cowpoke
Rancher
Posts: 13078
6
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
You are looking in the wrong place for performance.
Generally speaking, loop iterations are very fast, it is String manipulation operations that are time and memory consuming.
To minimize String manipulation write to a file or an in memory equivalent CharArrayWriter created with a large initial size.

Bill
 
Peter Chase
Ranch Hand
Posts: 1970
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I agree that the precise type of loop makes little difference, providing the same number of objects are visited.

I also agree that String manipulation can take significant time, if the Strings are extremely long or there are very many manipulations.

However, I find the idea that writing to a file could be faster than StringBuilder decidedly unlikely; in-memory operations are usually faster, providing there is enough memory available.

And isn't CharArrayWriter rather similar to StringBuilder, but with less text manipulation functionality and some extraneous stream-related stuff? Why would CharArrayWriter be faster?
 
Stan James
(instanceof Sidekick)
Ranch Hand
Posts: 8791
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I agree with all the above that the string handling will be more of an issue than the loop. Avoid the + operator in that kind of loop even though it might take many thousands of iterations to measure any difference.

One smart thing you can do while looping through everything in a Map is to get the entrySet() and iterate those. It's a little easier to read and to execute than getting the keySet() and then getting each value.
 
Ilja Preuss
author
Sheriff
Posts: 14112
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by William Brogden:
Generally speaking, loop iterations are very fast, it is String manipulation operations that are time and memory consuming.


Generally speaking, that's true, but for 500 iterations? I find it rather unlikely that string manipulation is the issue, either. Which doesn't mean that I would fully rule it out - just that I wouldn't optimize it before I see in a profiler that it's actually the problem. (Just two weeks ago, we spent days optimizing a specific part of our system - with not much luck. After some analysis with the help of JProfiler, we wrapped a Stream into a BufferedInputStream and got a 90 percent performance boost. All of our guesses on where the problem is likely to be were *so* just wrong... )
 
Peter Chase
Ranch Hand
Posts: 1970
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
As I understand it, there are about 500 entries in the outer list, but each list entry contains a hash table. We aren't told how many entries there typically are in each hash table, but it seems likely that we are dealing with thousands, not hundreds, of strings in total.
 
William Brogden
Author and all-around good cowpoke
Rancher
Posts: 13078
6
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
However, I find the idea that writing to a file could be faster than StringBuilder decidedly unlikely; in-memory operations are usually faster, providing there is enough memory available.


Certainly true - I just wanted to mention the file option in case the object was to minimize memory use. Note that with StringBuilder or similar in memory approach, you really want the creation to use a size that will be sufficient for the whole thing because otherwise the automatic buffer expansion will end up using quite a lot of memory.

Whichever route you go, I suggest you try to avoid all formatting in your print or write or append or whatever statements. Code like:

write( a + ":" + b + "\n" )

is going to cause extra object creation.

Bill
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!