Scala for Java developers: scala for big data

[Janeice DelVecchio]

I have used java mostly for big data (spark) development. That said, I know that spark is natively written in scala. Is there any benefit to using one over the other?

Are there ever benefits to using Java over scala?

[Campbell Ritchie]

Good to see you back

[Shawn Franks]

Other than knowing java already, not really.  Unlike python or groovy, Scala and Java are both strict types.

Benefits to using scala:

You do not need to declare types for variables, but you might want to for practical reasons.

Spark was built using scala, so you have more libraries and full access to spark libraries you won't have in java.

It takes less code than it would in Java.

Simpler more functional coding.

Import aliases and the ability to import entire classes as locally defined.  Sort of like python.

Cons to using scala:

You are basically learning a new language even know it is based off java.

Some things can be very frustrating that should be simple.

There are many formats for writing scala, it can make it difficult to read code from others if you do not know all the variences.

For example you don't a user could write abc.>6 where the . Can be confusing.   There are more complex variations too which can be difficult.



These are just my personal opinions.  I've gone from Java > Groovy > Python > Scala.

Groovy being a favorite because it is extremely flexible and allows the least code, but has overhead.

Scala/Python are my next favorites.

Scala I really hate the strict types it makes it difficult for writing UDF. While scala has an EitherOr type spark doesn't support the type even know you return StringType for example.

Groovy doesn't care what type you return, it will find the best match if you don't declare.

Imo iterating dynamic json is impossible in scala as well.  In groovy or python you don't need to know the types of key names but in scala you need to know.  There are some libraries but none are great.


Overall I'd say go to Scala.

Python is not yet supported with newer Kafka libraries, and has less support than java, which has less support than Scala.

[Janeice DelVecchio]

I see no benefit to python if I know java. All the spark libraries do everything I need, particularly with the spark 2 apis. To me, python is for folks who want to crunch something together without knowing a lot of syntax pomp and circumstance. But that's just my religion.

Is there any detriment to mixing some java classes with scala in an app?

[Janeice DelVecchio]

Campbell Ritchie wrote:Good to see you back

Lovely to see you as well.

[Shawn Franks]

No, but you in most cases you wouldn't want to unless something needs a java type specifically.  For example, lists , longs, dates, some times you need java.  In a lot of cases you can add .toJava or toScala to convert from one to another.

[Janeice DelVecchio]

Thank you!

Yes the converting is something I've had to mess with when I was working with spark. I can see where if you have a mix it would be more painful.

Enjoy the rest of the week here for your book promo!

[Shawn Franks]

Btw check out http://javatoscala.com.

Also for things like date a d long I'll use aliases.

import java.util.{Date => jDate}

This way it's I know what's java and scala.  That's just me though.

[Janeice DelVecchio]

You've been super helpful.

I didn't even know aliases existed! That's neat!

[T Weston]

I'd just chip in and say that although there is interop between Scala and Java, it's generally not very straight forward. Specifically, if you convert from a Java collection to Scala or the other way round, there will be a lot of noise/boilerplate and if you don't bother, you won't be taking advantage of the language features (for example, you'll end up calling flatmap and not flatMap when using Java collections in Scala and won't be able to use for-comprehension).