Map - Reduce - Already happening...

Following up on the previous post [http://architectguy.blogspot.com/2008/11/map-reduce-relevance-to-analytics.html], and after some more quality time with google, it's clear that there is more activity around the extension of map-reduce to large scale analytics than I originally thought or knew about.

Joe Hellerstein has published in O'Reilly Radar an interesting post on this exact subject [http://radar.oreilly.com/2008/11/the-commoditization-of-massive.html]. It is very interesting reading.

I think that beyond "massive data analysis" we will see the application of map-reduce to "massive event correlation". The same way the amount of data available for (and crying for) analytics processing is staggering and continues growing at staggering speed, the number and complexity of events that need to be processed to find the patterns and correlations that make them relevant to business apps processing is staggering and continues growing.
RFID, all sorts of devices connected to the cloud, exchanging information / pushing events related in many ways, all sorts of formal or informal transactions, connections and disconnections, ...

Joe points out that we will get to some convergence between traditional data management environments (your SQL) and the map-reduce approaches. He even points out to offerings put forward by innovative players - so what I thought was bound to happen in my previous post is happening even faster.

To paraphrase (in a slightly reworded fashion) what I wrote earlier,
- event pattern identification and correlation are ripe for this type of approaches, and will require it
- new algorithms will be made possible that leverage the approaches to provide results we do not think are possible today

This is one of those typical cycles: the abundance of data and events pushes the creation of new techniques, which in turn enable new applications which produce more data and events.

Very interesting times.