Vlad Riscutia

Jose Samonte wrote:Good day,

What are the factors to consider to help management decide to use Azure or an on-premise server/s to do data analytics and/or machine learning?
Thank you.

Regards,
Joey

Hi Joey,

This is the main value prop of the cloud: if you run your analytics and ML in the cloud, you save a lot of engineering time spent on managing on-prem infrastructure. Current cloud offerings are mature enough that you can achieve quite a lot without having to write any code. One of the principles I outline in my book is "codeless infrastructure" and a focus on PaaS offerings. To name a few: Azure Data Factory provides cloud scale ETL, Azure Purview provides metadata management/governance, Azure Machine Learning handles all infrastructure concerns related to running ML. Most of these services are pay-per-use, and once you add up the cost of on-prem hardware, IT, + custom solutions built on top of that it should be clear that the cloud ends up being much cheaper. Not to mention the reliability aspect - better let Microsoft ensure infra is always up and running rather than spend engineering resources to do the same on-prem.

paul nisset wrote:Hi,
I am thinking about trying out Azure for some pet/non  commercial  projects . Does your book cover how to minimize costs to try out database driven ,Java based , Web Services with Azure?
I tried out AWS on the promise of a free 1 year trial below a certain level of activity. Then they billed me the first month  for a small  web site that had no traffic.
I just felt scammed by Amazon .
thank you.

You can start here: https://azure.microsoft.com/en-us/free/.

The Azure trial offering includes $200 credit plus many services that are free. The page lists very clearly what is offered for free as part of the trial, what is always free and so on.

Minimizing costs is very specific to what you are trying to implement, though through the book I made sure examples can be run within the free trial. For example, for Azure Data Explorer, in the book I'm provisioning a Dev instance (cheapest SKU) and recommend stopping the cluster when not in use.

Hope this helps.

Lanny Gilbert wrote:I am being charged with setting up a runtime for a vendor product that requires a (in my mind anyway) pretty large Cassandra instance to operate properly. Does your book cover how to use Cassandra (Cosmos) within Azure, with regards to sizing, monitoring, best practices (over and above standard Cassandra best practices), etc.? Thanks!

My book doesn't specifically cover Cassandra. We have so many options for data storage and compute that it is impossible to cover them all in one book. I do talk through a reference implementation but in general I'm trying to focus on best practices and patterns that can be applied regardless of the specific implementation. For example: DevOps (Git, automatic deployment), monitoring, access control, governance and so on.

Ryan Jumawan wrote:Hi Vlad,

Again, congratulations on your new book!

I'm new to DevOps using Azure and it's a big paradigm shift for me in terms of build integration and faster to release into production in good quality.
My question is, do you think DevOps using Azure will change the development cycle in terms of continuous integration and maintaining the quality of product releases?

Regards,
Ryan

Thanks Ryan!

To answer your question: yes, I strongly believe DevOps changes the development cycle. Actually this is a main focus of my book. I view data engineering as bringing software engineering discipline (source control, security, reliability, scale etc.) to the realm of data.

Eric Weinberg wrote:How did you end up using Azure? Is it something you were familiar with from your 'day job', or did you see some specific advantages in the data engineering space that Azure had over the other cloud providers?

Well, I work at Microsoft

PatrickAlex Silva wrote:Is it possible to combine and leverage k8s amazing orchestration mechanisms with hdfs for maintaining our hadoop components and use cases up to date?

It probably is possible but in our team we tried to stay away from k8s as much as possible. This is a great solution for managing micro-services but I still count it as infrastructure you need to manage. We tried as much as possible to leverage "higher-level" PaaS solutions which don't require as much maintenance. I'm covering some of these in my book: using Data Factory for ETL, using Azure Machine Learning for ML workloads etc. You can do a lot with the right combination of services without having to orchestrate containers.

G Mukherjee wrote:Which criteria should be applied while choosing the specific language out of the supported ones for the data analysis scripts on Databricks? Does one provide benefits over the others?

I believe the main thing to think about is who is going to use this language and what is the knowledge gap. Are data scientists going to target Databricks, or is it only engineers? Data scientists might be more familiar with Python, R, SQL while software engineers might prefer Java or Scala. Are people willing to standardize on a language and learn it if they are not familiar with it? From my experience, this is the main challenge in a data team, as different disciplines come with different language knowledge.

In terms of mechanics, I believe all Databricks languages target the same underlying API, so there shouldn't be any major differences in capabilities. So it is mostly a people/standardization problem: Having multiple languages makes things harder to maintain. Standardizing on one require some training for part of the organization.

Hi everyone! Glad to be here and I'm happy to answer questions to the best of my ability.

Looking forward to the promo and answering your questions!

Yes, it does. It also covers generic programming.

paul nisset wrote:This begs a follow up of why did you choose TypeScript as opposed to Java or something else for the examples for the book?
thanks,
Paul

TypeScript has cleaner syntax for things like sum types and generics, that's the main reason. Originally I was thinking C++ but the C++ template syntax is a mess and I felt it would distract from the message, which is pretty language-agnostic.

I very much agree with what you are saying in that is what I also mention in the book. Code is read many more times than it is written, so I wouldn't save on typing a type name even if the compiler can easily infer it. It makes it much clearer to whoever reads the code what is going on.

No Angular knowledge needed The book doesn't rely on Angular.

Divya Shiv wrote:Dear Author,

Would like to know  about the main features of TypeScript used for solving the complex /modern programming  paradigms?

My book goes over many features of modern programming languages which are useful for solving complex problems: type composition, functional programming, OOP, and generic programming to name a few. If you have more specific questions I'll be happy to answer.

David Sachdev wrote:I've got years of Java and I've worked with various UI technologies, but I haven't spend much time with Angular  - and I need to rectify this. I've been busy in the DevOps world, but don't want to get too far away from application code.  Is this book a decent place for me to jump in, or would it be a better second read after spending some time with an Angular and Typescript book (or site).

Thanks
David

Hi David,

The book is not about Angular, it's more of a software engineering book that shows how to leverage the type system of a programming language to write good code. While the code examples are TypeScript, it is not tied to that particular language either and should benefit developers who use other languages like Java, C# etc.