• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Ron McLeod
  • Rob Spoor
  • Tim Cooke
  • Junilu Lacar
Sheriffs:
  • Henry Wong
  • Liutauras Vilda
  • Jeanne Boyarsky
Saloon Keepers:
  • Jesse Silverman
  • Tim Holloway
  • Stephan van Hulst
  • Tim Moores
  • Carey Brown
Bartenders:
  • Al Hobbs
  • Mikalai Zaikin
  • Piet Souris

Why are we allowed to call only UDFs and not functions from pyspark dataframe operations

 
Ranch Hand
Posts: 2511
13
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Suppose we have to populate value of a new column myCol based on some computation value returned from some function say myFunc

Instead of calling myFunc like in below code which is not allowed


We will have to create a UDF for myFunc and then call it.That will work.

What is the reason spark doesn't allow us to call function like this but allows only UDFs at this place ?


Thanks
 
reply
    Bookmark Topic Watch Topic
  • New Topic