Fine grained vs Coarse grained Parallelism and Java

[Abhijeet Ravankar]

Dear Sergey,

We have two paradigms for parallel programming. Coarse grained (grid, cluster) model and Fine grained model
(multi core processors like GPU, CELL/BE etc). While Java has a proven track for grid computing model, I find its
performance to be unsatisfactory in multicore programming (like JCUDA for GPGPU). And there are very few and
complex tools for multi-core programming. And we expect to see the cores increase every year.

1) I would like to know your views about fine grained vs coarse grained programming model. And role of Java in it.

2) In distant future when the number of cores of processors will be more than 1000s (its already 200+ in GPU), what
do you think is going to be the programming model.

3) With Moore's law, memory & frequency wall, power consumption, multi core processors is the trend. And to get most
performance out of them, parallel programming is the way. Do you feel that in near future, "serial programming will be
obsolete and just a special case of parallel programming"?

Thanks.

[Sergey Babkin]

Abhijeet Ravankar wrote:Dear Sergey,

We have two paradigms for parallel programming. Coarse grained (grid, cluster) model and Fine grained model
(multi core processors like GPU, CELL/BE etc). While Java has a proven track for grid computing model, I find its
performance to be unsatisfactory in multicore programming (like JCUDA for GPGPU). And there are very few and
complex tools for multi-core programming. And we expect to see the cores increase every year.

1) I would like to know your views about fine grained vs coarse grained programming model. And role of Java in it.

I haven's actually programmed GPUs myself (other than some logic in the SCSI controllers). For what I've read about them, they are very specialized processors with a strangely shaped instruction set and limited memory. As such I doublt that they would be very suitable for implmenetation of a complex interpreted language. They are a natural target for compilation of the comptutation-intensive algorithms. And Java has a lot of interpreted logic in its semantics, and relies on a big runtime support. However the arithmetic expressions in it can be compiled. So I guess there will be a convergence, the special-case compiled subsets of Java may be developed, and over time the GPUs will become more like the fully functional CPUs and gain more memory, becoming more suitable for Java interpretation.

2) In distant future when the number of cores of processors will be more than 1000s (its already 200+ in GPU), what
do you think is going to be the programming model.

I think the problem with GPUs (and with things like FPGAs) is that they aren't very convenient to control in the common time-shared model. The context switches are expensive, so you can't time-slice them easily between different processes. One process has to assume the full control over a pool of GPUs and feed them all with its data. I think the key to making these large numbers of cores work is by finding some ways to make the synchronization with them in general and context switches in particular cheap. Until then, they would have to be treated as sort of "pocket clusters".

3) With Moore's law, memory & frequency wall, power consumption, multi core processors is the trend. And to get most
performance out of them, parallel programming is the way. Do you feel that in near future, "serial programming will be
obsolete and just a special case of parallel programming"?

I think that there still are a lot of tasks that are not easily parallelizable. Of course, even those have parallelizable bits and pieces. So they may end up as a serial "control program", periodically branching out and rejoining in the library routines to perform portions of calculations in parallel.

[Abhijeet Ravankar]

Dear Sergey,

Thank you very much for sharing your insight about coarse and fine grained parallelism. Yes, as you
wrote, GPUs and FPGAs are extremely difficult to control. I understood the problem of interpreted
language like Java used in complex architectures of GPUs etc. Perhaps the future version of Java /
future subsets of Java will have this capability, as you said.

Really nice having shared your thoughts. Thank you once more.