This week's book giveaway is in the Python forum.
We're giving away four copies of High Performance Python for Data Analytics and have Tiago Rodrigues Antao on-line!
See this thread for details.
Win a copy of High Performance Python for Data Analytics this week in the Python forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Paul Clapham
  • Ron McLeod
  • Bear Bibeault
  • Liutauras Vilda
Sheriffs:
  • Jeanne Boyarsky
  • Tim Cooke
  • Junilu Lacar
Saloon Keepers:
  • Tim Moores
  • Tim Holloway
  • Stephan van Hulst
  • Jj Roberts
  • Carey Brown
Bartenders:
  • salvin francis
  • Frits Walraven
  • Piet Souris

Azure Data Engineering - Question for Vlad Riscutia

 
Greenhorn
Posts: 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Which criteria should be applied while choosing the specific language out of the supported ones for the data analysis scripts on Databricks? Does one provide benefits over the others?
 
Greenhorn
Posts: 4
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Is it possible to combine and leverage k8s amazing orchestration mechanisms with hdfs for maintaining our hadoop components and use cases up to date?
 
Author
Posts: 16
5
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

G Mukherjee wrote:Which criteria should be applied while choosing the specific language out of the supported ones for the data analysis scripts on Databricks? Does one provide benefits over the others?



I believe the main thing to think about is who is going to use this language and what is the knowledge gap. Are data scientists going to target Databricks, or is it only engineers? Data scientists might be more familiar with Python, R, SQL while software engineers might prefer Java or Scala. Are people willing to standardize on a language and learn it if they are not familiar with it? From my experience, this is the main challenge in a data team, as different disciplines come with different language knowledge.

In terms of mechanics, I believe all Databricks languages target the same underlying API, so there shouldn't be any major differences in capabilities. So it is mostly a people/standardization problem: Having multiple languages makes things harder to maintain. Standardizing on one require some training for part of the organization.
 
Vlad Riscutia
Author
Posts: 16
5
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

PatrickAlex Silva wrote:Is it possible to combine and leverage k8s amazing orchestration mechanisms with hdfs for maintaining our hadoop components and use cases up to date?



It probably is possible but in our team we tried to stay away from k8s as much as possible. This is a great solution for managing micro-services but I still count it as infrastructure you need to manage. We tried as much as possible to leverage "higher-level" PaaS solutions which don't require as much maintenance. I'm covering some of these in my book: using Data Factory for ETL, using Azure Machine Learning for ML workloads etc. You can do a lot with the right combination of services without having to orchestrate containers.
 
Greenhorn
Posts: 8
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi Vlad,

Again, congratulations on your new book!

I'm new to DevOps using Azure and it's a big paradigm shift for me in terms of build integration and faster to release into production in good quality.
My question is, do you think DevOps using Azure will change the development cycle in terms of continuous integration and maintaining the quality of product releases?

Regards,
Ryan
 
Vlad Riscutia
Author
Posts: 16
5
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Ryan Jumawan wrote:Hi Vlad,

Again, congratulations on your new book!

I'm new to DevOps using Azure and it's a big paradigm shift for me in terms of build integration and faster to release into production in good quality.
My question is, do you think DevOps using Azure will change the development cycle in terms of continuous integration and maintaining the quality of product releases?

Regards,
Ryan



Thanks Ryan!

To answer your question: yes, I strongly believe DevOps changes the development cycle. Actually this is a main focus of my book. I view data engineering as bringing software engineering discipline (source control, security, reliability, scale etc.) to the realm of data.
 
reply
    Bookmark Topic Watch Topic
  • New Topic