• Post Reply Bookmark Topic Watch Topic
  • New Topic

Partioning with String based key  RSS feed

 
Jehan Jaleel
Ranch Hand
Posts: 196
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi,

I am trying to implement parallel processing for Spring Batch. Basically I have a job that reads from a DB and processes that data. The challenge for me now is to partition that data into different sets so that each thread has a separate partition to work on. Following is some code I found to create the partition...



The problem is that this assumes the column to partition is an Integer, but in my case it is a String (alpha numeric). The values in my column are still unique, they are codes like "AV21" or "BC33" but the above code would not work for it.

Any suggestions on how to re write this to partition for String key?

BTW, I posted this in the Java General forum and not Spring forum because it is not really a Spring question. It is really a plain Java question.

Thanks in advance for any help.
 
Winston Gutkowski
Bartender
Posts: 10575
66
Eclipse IDE Hibernate Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Jehan Jaleel wrote:The problem is that this assumes the column to partition is an Integer, but in my case it is a String (alpha numeric). The values in my column are still unique, they are codes like "AV21" or "BC33" but the above code would not work for it.

I'm still not sure exactly what it is you need; an alphanumeric string can be converted to a number simply by encoding it as base 36. The only problem is that a long can only hold 12 such digits.

Any suggestions on how to re write this to partition for String key?

Well, the normal way would be to take the total number of objects, divide by however many threads you think you need, and get the keys for each breakpoint, but maybe I'm missing something?

]BTW, I posted this in the Java General forum and not Spring forum because it is not really a Spring question. It is really a plain Java question.

Thanks for pointing that out, but I'm not sure that this actually IS a Java question. Seems to me that what you want to know is how to partition a set of values that you want to process. All the database stuff is just hooey.

Be happy to help more if/when you can provide a bit more info - and by that I mean info about the data and how you want to split it; not what you've implemented (presumably, in error) in Java.

Winston

PS: PLEASE take out that line connect at the end of the 'min' defintiion. It isn't necessary, and it drives the window software here mad because it makes the line so long.
 
Jehan Jaleel
Ranch Hand
Posts: 196
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks for the response Winston. I guess what I was really asking is how to convert an alphanumeric string to an int (or long) and you answered this question that to use base 36 encoding. Now in my code I am converting to long where needed and then converting back to String...

 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!