• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Tim Cooke
  • paul wheaton
  • Jeanne Boyarsky
  • Ron McLeod
Sheriffs:
  • Paul Clapham
  • Liutauras Vilda
  • Devaka Cooray
Saloon Keepers:
  • Tim Holloway
  • Roland Mueller
Bartenders:

Real-time Hadoop

 
Ranch Hand
Posts: 119
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
What about Hadoop in the context of real-time requirements for some systems like those exist in mobile networks?
and if it is not supported, what are the alternatives in particularly for the real-time case?

Regards,
Mohamed El-Refaey
 
author
Posts: 15
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
So true realtime isn't a core Hadoop strength at this point. The processing model really doesn't lend itself to those sort of requirements.

Looking ahead though two interesting developments are Yarn and Tez. The former treats the Hadoop platform as a more generic processing framework and Tez (just starting) will be building processing models that differ significantly from what we are used to in MapReduce today. Those seem like areas of major innovation to me and I suspect we'll see a whole raft of different types of processing happening on Hadoop in the future.

Garry
 
Mohamed El-Refaey
Ranch Hand
Posts: 119
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I see, but it looks that tez is trying to increase processing speed but not really a real-time solution ..
Thanks for referring to yarn and tez, wonderful framework


Regards,
Mohamed
 
Ranch Hand
Posts: 221
Scala Python Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Real Time in a Hadoop cluster is one of the many things possible when you use MapR Enterprise Grade Distribution compared to Apache Hadoop or other distributions thanks to the many enhancements and features of MapR-FS, which is a read/write file system among others.

See this use case documented here http://www.mapr.com/blog/twitter-feed-fuels-real-time-hadoop-with-storm-and-maprr-at-the-strata-conference.

If you want to accomplish this using other Hadoop distributions at least you would need 2 clusters and a lot of more hardware, therefore way more expensive.


HTH,


Carlos.
 
Mohamed El-Refaey
Ranch Hand
Posts: 119
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Interesting use case ...it also mentioned that it deals with real time sensors' data, in deployment such as IoT.
Thanks Carlos!
 
Enjoy the full beauty of the english language. Embedded in this tiny ad:
Smokeless wood heat with a rocket mass heater
https://woodheat.net
reply
    Bookmark Topic Watch Topic
  • New Topic