Integrating Erlang with other languages

[Cristina Yenyxe]

I'm currently attending a M.Sc. at university about parallel and distributed computing, so Erlang is one of the topics that I feel interested in.

They have teached us that when programming in C, MPI and OpenMP could be mixed, so I'm curious if Erlang could be easily integrated with C in order to distribute the data using Erlang and make the calculations using OpenMP. Would it be the optimum solution or an Erlang-only one would be faster in most of the cases?

Thanks in advance, and congratulations for the book, I've read the sample chapters and they look very interesting!

[Richard Carlsson]

I'd say that Erlang makes the "glue" and supervision code enormously easier to write, but if you really have to do massively parallel scientific computing, you'd need to make the parts of the code that are closest to the hardware as efficient as possible. At a minimum, run the actual computations in a separate C program on each computer, and use Erlang to distribute the workload and collect results. Erlang was not made for number crunching.

Whether it would be worth involving OpenMP or MPI as well, I'm not sure - but maybe if you have a wide-area cluster of individual massively multicore machines (or local clusters connected with high-speed data links), you'd like to try to max out by running OpenMP locally, and then use Erlang to synchronize the whole thing over the campus-wide or global network.

So that you don't have to write this sort of thing on your own, there is already a very nice project by Nokia called Disco: http://discoproject.org/ (Basically, Disco is Hadoop in less than 6000 lines of Erlang code, and lets you script the local jobs in Python.)

[Cristina Yenyxe]

Richard Carlsson wrote:Whether it would be worth involving OpenMP or MPI as well, I'm not sure - but maybe if you have a wide-area cluster of individual massively multicore machines (or local clusters connected with high-speed data links), you'd like to try to max out by running OpenMP locally, and then use Erlang to synchronize the whole thing over the campus-wide or global network.

Yeah, I was considering that last idea, MPI doesn't seem to fit very well in this situation.

Richard Carlsson wrote:So that you don't have to write this sort of thing on your own, there is already a very nice project by Nokia called Disco: http://discoproject.org/ (Basically, Disco is Hadoop in less than 6000 lines of Erlang code, and lets you script the local jobs in Python.)

I love Python, and Disco looks great! I have sooo many thing to learn about this field... Thanks for the info.