Spark In Action: Languages

[Will Myers]

Hi,
I looked at Spark a long time ago and it seemed it was a part of a larger ecosystem, can you use it on its own or does it need a Hadoop cluster to run (or some other framework)?

[Sharma Ashutosh]

Does it is going to cover Spark integration with various other technologies(reading data from data lake, diverse data sets, diverse real time streams, Snowflake and other NoSQL DBs and Apache Kafka)?
Are the code examples correct and working fine OOTB?

[Christopher Webster]

@Will:

Spark can run on your laptop reading/writing data to flat-files locally, or it can run in cluster mode on various distributed platforms (HDFS, Hadoop, Kubernetes, Cassandra, ...), as well as being available as a service from major cloud providers like Google (Dataproc), Azure, Amazon.

So you can learn or experiment with Spark on your laptop, develop code there, then deploy the same code to run on a cluster in the cloud.

Databricks is the company behind Spark (as well as the proprietary Databricks data platform), so check out their website for tutorials and background info on Spark:

https://databricks.com/spark/about

There are also lots of interesting talks - about what you can do with Spark and how people are using it in the real world - available online from the various conferences e.g. https://databricks.com/sparkaisummit/north-america-2020