Predictions and Surprises

WSJ has this interesting series demystifying - or at least discussing - issues around numeracy, probabilities, and the corresponding impact.

I just read the following: http://blogs.wsj.com/numbersguy/the-crash-calculations-621/

A couple of key things that I feel are not really covered deeply enough:
- the quality, and ultimately the validity of predictions in a given context are a direct function of the relevance to that context of the explicit and implicit assumptions in the modeling process used to create the prediction
- decisions should never be made only on predictions obtained through models, they should include scenario based simulation and impact analysis

The reason we build models is precisely to create abstractions that we can manipulate with the tools of our mind and our technology. Tools that allow us to get out of the immediate sensor-driven reaction mode, and forecast. I believe modeling is essential to forecasting - I know some will say just crunching numbers with no a priori model is the path of the future (a Wired article I read one day) but that's a fallacy: as soon as you use the result of the crunching, you are using a model, maybe an implicit one, but you are using a model.

But who says abstraction says context-dependent simplification. And that is key. Understanding the context the abstraction assumes, and the sensitivity of the resulting model and predictions to variations in that context is paramount to being able to leverage these predictions.

That gets lost. Because it's complicated and because of the multiple psychological aspects that make understanding and leveraging statistical and probabilistic results very difficult for the average person.

See the points made in the EDM blog by Carole-Ann (http://www.edmblog.com/).

Map - Reduce - Already happening...

Following up on the previous post [http://architectguy.blogspot.com/2008/11/map-reduce-relevance-to-analytics.html], and after some more quality time with google, it's clear that there is more activity around the extension of map-reduce to large scale analytics than I originally thought or knew about.

Joe Hellerstein has published in O'Reilly Radar an interesting post on this exact subject [http://radar.oreilly.com/2008/11/the-commoditization-of-massive.html]. It is very interesting reading.

I think that beyond "massive data analysis" we will see the application of map-reduce to "massive event correlation". The same way the amount of data available for (and crying for) analytics processing is staggering and continues growing at staggering speed, the number and complexity of events that need to be processed to find the patterns and correlations that make them relevant to business apps processing is staggering and continues growing.
RFID, all sorts of devices connected to the cloud, exchanging information / pushing events related in many ways, all sorts of formal or informal transactions, connections and disconnections, ...

Joe points out that we will get to some convergence between traditional data management environments (your SQL) and the map-reduce approaches. He even points out to offerings put forward by innovative players - so what I thought was bound to happen in my previous post is happening even faster.

To paraphrase (in a slightly reworded fashion) what I wrote earlier,
- event pattern identification and correlation are ripe for this type of approaches, and will require it
- new algorithms will be made possible that leverage the approaches to provide results we do not think are possible today

This is one of those typical cycles: the abundance of data and events pushes the creation of new techniques, which in turn enable new applications which produce more data and events.

Very interesting times.

Map - Reduce - relevance to analytics?

One of the key problems we face when dealing with vast numbers of events and/or vast amounts of data is how to efficiently parallelize the processing so as to accelerate the analytic development cycles. This is not just a question of efficiency (minimize the downtime of high cost PhD resources, ...) but also a question of feasibility (make the development of the analytics fit in business driven time-lines, such as the time to detect and react to new fraud mechanisms).

My experience, coming from the real time embedded systems days, is that the best ways to achieve massive parallelization result from highly simplified low level programming models. These models are designed to do only very simple operations but with nothing in them that prevents systematic parallelization.

Google has made popular (at least in the technosphere) the map-reduce approach derived from the functional programming world [http://en.wikipedia.org/wiki/MapReduce]. Google officially applies the approach to its search core capabilities, but it's very likely it uses it for much more than that.
Its formal definition is trivial, but map-reduce does provide a solid basis for massive parallelization of list processing. Which is very much what a lot - a lot - of analytics development work is. It may very well end up being a key component that will allow the implementation of complex analytics processing of amounts of data well beyond what we tackle today, enabling applications that are beyond reach today.
Today, map-reduce is essentially used for search (processing text), network processing (as in social networks), etc. The Hadoop open source effort [http://hadoop.apache.org/core/] has a list of current applications to which its map-reduce implementation is applied [http://wiki.apache.org/hadoop/PoweredBy]: that makes a very interesting reading.

Its applicability to machine learning / predictive analytics building is illustrated by the Mahout effort [http://lucene.apache.org/mahout/] which seeks to leverage Hadoop to specific implementations of traditional machine learning algorithms [http://www.cs.stanford.edu/people/ang//papers/nips06-mapreducemulticore.pdf]. I see immediate applications to both including this approaches to CEP (highly parallel processing of events to achieve fast correlation) and predictive analytics development (highly parallel processing of data to find patterns - neural-net like; highly parallel implementations of genetic approaches, etc.).

I would be curious to know what the reader thinks about this.

I also believe a few developments are bound to happen, and I expect to see more happening around them in the coming few years:
- development of hybrid systems combining more traditional processing (SQL, SAS datasets,...) with map-reduce, potentially introduced by the vendors themselves but more likely by minor innovative players first
- development of new algorithms in event processing / data processing / machine learning that leverage map-reduce
- introduction of other approaches similar to map-reduce as the corresponding results.

I am particularly intrigued by the last two possibilities.