• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Paul Clapham
  • Ron McLeod
  • Bear Bibeault
  • Liutauras Vilda
Sheriffs:
  • Jeanne Boyarsky
  • Tim Cooke
  • Junilu Lacar
Saloon Keepers:
  • Tim Moores
  • Tim Holloway
  • Stephan van Hulst
  • Jj Roberts
  • Carey Brown
Bartenders:
  • salvin francis
  • Frits Walraven
  • Piet Souris

How come transformation methods of Spark like map, reduce, filter also in Scala (without Spark)?

 
Ranch Foreman
Posts: 2339
12
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
In Spark, there are transformation methods like map, reduce, filter. These methods are specific to Spark and are  called Spark transformations. Howcome these methods are present in Scala programming language (without Spark).

Thanks
 
Saloon Keeper
Posts: 12605
273
  • Likes 2
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Mapping, filtering and reducing are some of the most essential operations in any functional language. Scala needs them regardless of whether you use Spark or not.

A better question is, why does Spark for Scala feel that it needs separate versions of these functions?
 
Monica Shiralkar
Ranch Foreman
Posts: 2339
12
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I think the answer to the question you said as the better question  to ask is that since Spark requires these methods to do the work they are intended to on a cluster of machines  so they require a different version of these methods. That is what I think. I may be wrong.
 
Ranch Hand
Posts: 376
Scala Monad
  • Likes 2
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Spark got inspired for  it's API from Scala's collection API that has those methods (filter, map, flatMap, reduce, etc). When invoked in a Scala collection, they run locally, returning a new collection, when using Spark, the methods will be invoked on the RDD API, and will return a transformed RDD that will run in the Spark cluster.
reply
    Bookmark Topic Watch Topic
  • New Topic