• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Tim Cooke
  • paul wheaton
  • Paul Clapham
  • Ron McLeod
Sheriffs:
  • Jeanne Boyarsky
  • Liutauras Vilda
Saloon Keepers:
  • Tim Holloway
  • Carey Brown
  • Roland Mueller
  • Piet Souris
Bartenders:

Scala for Java developers: scala for big data

 
Bartender
Posts: 1849
15
Eclipse IDE Spring VI Editor Java Linux Windows
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I have used java mostly for big data (spark) development. That said, I know that spark is natively written in scala. Is there any benefit to using one over the other?

Are there ever benefits to using Java over scala?
 
Marshal
Posts: 80874
506
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Good to see you back
 
Ranch Hand
Posts: 30
1
  • Likes 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Other than knowing java already, not really.  Unlike python or groovy, Scala and Java are both strict types.

Benefits to using scala:

You do not need to declare types for variables, but you might want to for practical reasons.

Spark was built using scala, so you have more libraries and full access to spark libraries you won't have in java.

It takes less code than it would in Java.

Simpler more functional coding.

Import aliases and the ability to import entire classes as locally defined.  Sort of like python.

Cons to using scala:

You are basically learning a new language even know it is based off java.

Some things can be very frustrating that should be simple.

There are many formats for writing scala, it can make it difficult to read code from others if you do not know all the variences.

For example you don't a user could write abc.>6 where the . Can be confusing.   There are more complex variations too which can be difficult.



These are just my personal opinions.  I've gone from Java > Groovy > Python > Scala.

Groovy being a favorite because it is extremely flexible and allows the least code, but has overhead.

Scala/Python are my next favorites.

Scala I really hate the strict types it makes it difficult for writing UDF. While scala has an EitherOr type spark doesn't support the type even know you return StringType for example.

Groovy doesn't care what type you return, it will find the best match if you don't declare.

Imo iterating dynamic json is impossible in scala as well.  In groovy or python you don't need to know the types of key names but in scala you need to know.  There are some libraries but none are great.


Overall I'd say go to Scala.

Python is not yet supported with newer Kafka libraries, and has less support than java, which has less support than Scala.
 
Janeice DelVecchio
Bartender
Posts: 1849
15
Eclipse IDE Spring VI Editor Java Linux Windows
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I see no benefit to python if I know java. All the spark libraries do everything I need, particularly with the spark 2 apis. To me, python is for folks who want to crunch something together without knowing a lot of syntax pomp and circumstance. But that's just my religion.

Is there any detriment to mixing some java classes with scala in an app?
 
Janeice DelVecchio
Bartender
Posts: 1849
15
Eclipse IDE Spring VI Editor Java Linux Windows
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Campbell Ritchie wrote:Good to see you back


Lovely to see you as well.
 
Shawn Franks
Ranch Hand
Posts: 30
1
  • Likes 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
No, but you in most cases you wouldn't want to unless something needs a java type specifically.  For example, lists , longs, dates, some times you need java.  In a lot of cases you can add .toJava or toScala to convert from one to another.
 
Janeice DelVecchio
Bartender
Posts: 1849
15
Eclipse IDE Spring VI Editor Java Linux Windows
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thank you!

Yes the converting is something I've had to mess with when I was working with spark. I can see where if you have a mix it would be more painful.

Enjoy the rest of the week here for your book promo!
 
Shawn Franks
Ranch Hand
Posts: 30
1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Btw check out http://javatoscala.com.

Also for things like date a d long I'll use aliases.

import java.util.{Date => jDate}

This way it's I know what's java and scala.  That's just me though.
 
Janeice DelVecchio
Bartender
Posts: 1849
15
Eclipse IDE Spring VI Editor Java Linux Windows
  • Likes 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
You've been super helpful.

I didn't even know aliases existed! That's neat!
 
author
Posts: 23
5
  • Likes 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I'd just chip in and say that although there is interop between Scala and Java, it's generally not very straight forward. Specifically, if you convert from a Java collection to Scala or the other way round, there will be a lot of noise/boilerplate and if you don't bother, you won't be taking advantage of the language features (for example, you'll end up calling flatmap and not flatMap when using Java collections in Scala and won't be able to use for-comprehension).
reply
    Bookmark Topic Watch Topic
  • New Topic