Azure Data Engineering - Question for Vlad Riscutia

[G Mukherjee]

Which criteria should be applied while choosing the specific language out of the supported ones for the data analysis scripts on Databricks? Does one provide benefits over the others?

[PatrickAlex Silva]

Is it possible to combine and leverage k8s amazing orchestration mechanisms with hdfs for maintaining our hadoop components and use cases up to date?

[Vlad Riscutia]

G Mukherjee wrote:Which criteria should be applied while choosing the specific language out of the supported ones for the data analysis scripts on Databricks? Does one provide benefits over the others?

I believe the main thing to think about is who is going to use this language and what is the knowledge gap. Are data scientists going to target Databricks, or is it only engineers? Data scientists might be more familiar with Python, R, SQL while software engineers might prefer Java or Scala. Are people willing to standardize on a language and learn it if they are not familiar with it? From my experience, this is the main challenge in a data team, as different disciplines come with different language knowledge.

In terms of mechanics, I believe all Databricks languages target the same underlying API, so there shouldn't be any major differences in capabilities. So it is mostly a people/standardization problem: Having multiple languages makes things harder to maintain. Standardizing on one require some training for part of the organization.

[Vlad Riscutia]

PatrickAlex Silva wrote:Is it possible to combine and leverage k8s amazing orchestration mechanisms with hdfs for maintaining our hadoop components and use cases up to date?

It probably is possible but in our team we tried to stay away from k8s as much as possible. This is a great solution for managing micro-services but I still count it as infrastructure you need to manage. We tried as much as possible to leverage "higher-level" PaaS solutions which don't require as much maintenance. I'm covering some of these in my book: using Data Factory for ETL, using Azure Machine Learning for ML workloads etc. You can do a lot with the right combination of services without having to orchestrate containers.

[Ryan Jumawan]

Hi Vlad,

Again, congratulations on your new book!

I'm new to DevOps using Azure and it's a big paradigm shift for me in terms of build integration and faster to release into production in good quality.
My question is, do you think DevOps using Azure will change the development cycle in terms of continuous integration and maintaining the quality of product releases?

Regards,
Ryan

[Vlad Riscutia]

Ryan Jumawan wrote:Hi Vlad,

Again, congratulations on your new book!

I'm new to DevOps using Azure and it's a big paradigm shift for me in terms of build integration and faster to release into production in good quality.
My question is, do you think DevOps using Azure will change the development cycle in terms of continuous integration and maintaining the quality of product releases?

Regards,
Ryan

Thanks Ryan!

To answer your question: yes, I strongly believe DevOps changes the development cycle. Actually this is a main focus of my book. I view data engineering as bringing software engineering discipline (source control, security, reliability, scale etc.) to the realm of data.