• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Paul Clapham
  • Ron McLeod
  • Liutauras Vilda
  • Bear Bibeault
Sheriffs:
  • Jeanne Boyarsky
  • Tim Cooke
  • Devaka Cooray
Saloon Keepers:
  • Tim Moores
  • Tim Holloway
  • Piet Souris
  • salvin francis
  • Stephan van Hulst
Bartenders:
  • Frits Walraven
  • Carey Brown
  • Jj Roberts

What is file system used for when using Spark on hadoop cluster versus using spark without hadoop

 
Ranch Foreman
Posts: 2085
12
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
We can use spark on hadoop cluster or without hadoop. When we use it on hadoop cluster it uses hdfs file system. What does it require HDFS for in that case? And if not using on hadoop cluster what file system it may use and why is that required to be used?

thanks
 
Ranch Hand
Posts: 31
3
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Spark is a distributed processing engine.  If you run it on Hadoop you can tell it to use the HDFS cluster as its data store.  Or you can tell it to read/write data from other sources if these are available.

If you run Spark stand-alone, it can read from your local file system, or another source if one is available.

It will store temporary data where you tell it to e.g. on HDFS if you are on Hadoop, or on your local file system if you are running Spark on your laptop.
 
Monica Shiralkar
Ranch Foreman
Posts: 2085
12
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks. And if I am using it for say reading Streaming data from Kafka, then does it require a file system. If so , for what?
 
I'm not dead! I feel happy! I'd like to go for a walk! I'll even read a tiny ad:
the value of filler advertising in 2020
https://coderanch.com/t/730886/filler-advertising
reply
    Bookmark Topic Watch Topic
  • New Topic