Win a copy of High Performance Python for Data Analytics this week in the Python forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Paul Clapham
  • Ron McLeod
  • Bear Bibeault
  • Liutauras Vilda
Sheriffs:
  • Jeanne Boyarsky
  • Tim Cooke
  • Junilu Lacar
Saloon Keepers:
  • Tim Moores
  • Tim Holloway
  • Stephan van Hulst
  • Jj Roberts
  • Carey Brown
Bartenders:
  • salvin francis
  • Frits Walraven
  • Piet Souris

Kafka vs Traditional ETL/Batch

 
Ranch Hand
Posts: 81
Spring Java Ubuntu
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi Bill,

Excited to see your upcoming book on latest sensation in enterprise technology area.

I attended an overview session on Kafka recently and I have couple of questions on which I want to request your opinion -

How do you differentiate the traditional Enterprise service buses with ETL combined with Kafka. As an introduction, Kafka is doing real time fast processing of streams but we have other systems like Tibco FTL and IBM MQ which have achieved lightening speed of millions transactions per second. How does kafka provides upper-hand with existing products ?

Another question is do you think that kafka will be able to replace existing batch systems. With scenarios with high availability , kafka may process transactional data quickly but overnight batch may still become bottle neck. What are the few practices you want to share with audiences here about integrating kafka with exiting systems.

Thank You again for bringing the detailed book.
 
Saloon Keeper
Posts: 23070
157
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I just installed the latest version of Hitachi Pentaho.

You may be interested to know that the Pentaho DI engine (Kettle) now runs under Kafka.
 
Ranch Hand
Posts: 66
1
IntelliJ IDE Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Tim Holloway wrote:I just installed the latest version of Hitachi Pentaho.

You may be interested to know that the Pentaho DI engine (Kettle) now runs under Kafka.



Pentaho DI does run Kafka out of the box, but it's far from perfect (It really only works for generic AVRO messages + it doesn't have Kerberos support). I had to write my own custom step to successfully solve the requirements we had in our project.
 
this is supposed to be a surprise, but it smells like a tiny ad:
Building a Better World in your Backyard by Paul Wheaton and Shawn Klassen-Koop
https://coderanch.com/wiki/718759/books/Building-World-Backyard-Paul-Wheaton
reply
    Bookmark Topic Watch Topic
  • New Topic