Search within Big Table
Register / Login
this forum made possible by our volunteer staff, including ...
Stephan van Hulst
Why are we allowed to call only UDFs and not functions from pyspark dataframe operations
posted 2 weeks ago
Number of slices to send:
Optional 'thank-you' note:
Suppose we have to populate value of a new column myCol based on some computation value returned from some function say myFunc
Instead of calling myFunc like in below code which is not allowed
df =df.withColumn('myCol', myFunc())
We will have to create a UDF for myFunc and then call it.That will work.
What is the reason spark doesn't allow us to call function like this but allows only UDFs at this place ?
Boost this thread!
Migrating from Spark 1.6 to newer version
When to use Pandas dataframe instead of PySpark dataframe?
How does lazy evaluation happen in dataframes in Spark which do not have actions unlike RDDs
How to get index of DataFrame?