Real-time stream processing: Are we doing it wrong?

Humans to Machine – Shift of data source

Data has been growing exponentially. We have more data streaming through the wire than we can keep them on disk from both value and volume perspective. These data are being created by everything we deal with on daily basis. When humans were the dominant creator of data, we naturally used to have fewer amounts of data to deal with and at the same time value used to persist for a longer period. This, in fact, holds true now as well, if humans are the creator of the data.

However, humans are no longer the dominant creator of the data. Machines, sensors, devices etc. have taken over long time back. These data, created by machines with humongous speed, is so much that in last two years we had 90% of the data created since the dawn of civilization. These data tend to have limited shelf life as far as value is concerned. The value of data decreases rapidly with time. If the data is not processed as soon as possible then it may not be very useful for ongoing businesses and operations. Naturally, we need to have different thought process and approach to deal with …

Read More on Datafloq

Comments are closed, but trackbacks and pingbacks are open.