Predictions and Surprises

WSJ has this interesting series demystifying - or at least discussing - issues around numeracy, probabilities, and the corresponding impact.

I just read the following: http://blogs.wsj.com/numbersguy/the-crash-calculations-621/

A couple of key things that I feel are not really covered deeply enough:
- the quality, and ultimately the validity of predictions in a given context are a direct function of the relevance to that context of the explicit and implicit assumptions in the modeling process used to create the prediction
- decisions should never be made only on predictions obtained through models, they should include scenario based simulation and impact analysis

The reason we build models is precisely to create abstractions that we can manipulate with the tools of our mind and our technology. Tools that allow us to get out of the immediate sensor-driven reaction mode, and forecast. I believe modeling is essential to forecasting - I know some will say just crunching numbers with no a priori model is the path of the future (a Wired article I read one day) but that's a fallacy: as soon as you use the result of the crunching, you are using a model, maybe an implicit one, but you are using a model.

But who says abstraction says context-dependent simplification. And that is key. Understanding the context the abstraction assumes, and the sensitivity of the resulting model and predictions to variations in that context is paramount to being able to leverage these predictions.

That gets lost. Because it's complicated and because of the multiple psychological aspects that make understanding and leveraging statistical and probabilistic results very difficult for the average person.

See the points made in the EDM blog by Carole-Ann (http://www.edmblog.com/).