Financial instruments, systems analysis and decision management

I have been reading a little about what is being presented as the causes for the current worldwide financial crisis. Nothing really surprising in the descriptions, of course, we’ve all heard or read these experts who failed to predict anything going to great details in explaining a posteriori what happened, and provide supposedly valuable insight into what would be coming.

What really amazes me is how each one of the human branches of activities likes living in its own ivory tower. Each industry learns its own lessons from its own failures, or attempts to, but rarely does one industry draw the lessons from another industry’s failures. I am certain some of that has to do with the fact that industries are frequently out of sync in their cycles, but also from a lack of abstraction and high level view on the nature of the industry, its processes and decisions.

A key factor is in this mess is simply the fact that the financial industry created all sorts of financial instruments working at ever higher levels of abstraction, more and more removed from the reality of the underlying relevant physical or legal entities without really understanding their systems behavior.

Derivatives, derivatives on derivatives, swaps, etc: sure, each and every one of them has a solid – and here, sorry for my friends in that space, I will qualify solid as “seemingly” solid – mathematical support. Sure, the bright mathematicians and physicist Wall Street hired by the plane load out of Europe (Russia, France, etc.) are very capable in their domain. They can understand business problems, abstract them into sophisticated models, even understand the conditions under which these models apply, provide all sorts of sensitivity analysis to take into account potential environmental changes, etc. I will not question that.
But I will say that there is no evidence whatsoever that the financial industry applied – or applies – serious systems analysis to the combination of these instruments as they are applied. The obvious aspects that systems analysts pay attention to have been totally ignored: no real systems analysis of applicability, analysis of error / exception propagation, exception management, etc.
As a consequence, each model may very well be well understood, inducing a false sense of security in that this is all under control.

When systems break, they most often break in the interface between components (the famous Mars Climate orbiter issue: http://mars.jpl.nasa.gov/msp98/news/mco990930.html), or in exception handling (the famous Ariane rocket explosion: read the report from one of my teachers, Jacques Louis Lions, http://esamultimedia.esa.int/docs/esa-x-1819eng.pdf), or in the actual decision making in particular when it involves humans (see http://architectguy.blogspot.com/2008/12/precision-speed-and-mistakes.html below, this crisis).
The financial industry – driven by its purely speculative focus, and totally blinded to the fact it no longer understands its instruments – failed and continues failing on all three counts.
For all their intellect and knowledge, the Nobel Prize winner creators of LTCM failed. The industry around it failed more than a decade ago, but little was learned from there. It went back to the same (maybe less obvious) hubris and disregard for proper risk management. With the same consequences…

Frequently enough these failures have simple root causes, overlooked only because of the overall complexity of the system and incredibly cheap to repair compared to the huge cost of these failures. Imagine if the verification systems of Société Générale had actually worked (http://en.wikipedia.org/wiki/January_2008_Soci%C3%A9t%C3%A9_G%C3%A9n%C3%A9rale_trading_loss_incident) : Kerviel would be unknown, and the company would not have lost billions of dollars (and this was limited thanks to the intervention of the company’s crack team and the help from the French government)

Other industries have learned and built into their practices both simplification of components and management of the systems to reach a level in which, when a black swan does appear, the consequences can be assessed and remedial action can be taken. The financial industry should do the same.

Decision Management can help as it increases its focus on scenario-based risk assessment. Modeling decisions and managing around them scenarios that enable the systems analysis of the business processes should allow much better understanding and control of the consequences of the decisions in changing conditions. A lot of the energy spent building predictive models based on well bounded simplifications should be shifted towards modeling the decisions that are made, the business outcomes, their dependencies with respect tot the environment, and managing vast and ever enriched portfolios of scenarios around them to keep risk to black swans in check.

This will require discipline, as it basically emphasizes analyzing highly unlikely but consequence heavy scenarios in times when things are going seemingly well.
But I believe it will be needed.

The Decision Management discipline can help, leverage it.

On a side note on all this,
- Read Nassim Taleb (http://en.wikipedia.org/wiki/Nassim_Nicholas_Taleb).
- Watch him discuss the crisis on TV (http://paul.kedrosky.com/archives/2008/10/12/nassim_taleb_ge.html)
- Watch him and Benoit Mandelbrot (http://en.wikipedia.org/wiki/Benoit_Mandelbrot) debate the crisis: http://uk.youtube.com/watch?v=DLFkQdiXPbo&NR=1

CouchDB for Enterprise Applications?

A little late in noticing this: CouchDB has become a first class citizen project of Apache [http://couchdb.apache.org/].

This is an interesting development. For us, enterprise application folks, it takes a while to get rid of the relational paradigm and bring ourselves to consider a document-oriented database such as this one. But I have to say that years of squeezing documents into the relational model have slowly driven me to appreciate the virtue of something different.
Full blown document management systems have always been something that feels too much "document publisher" oriented. The preoccupations seem to be largely around supporting the management of documents humans read, and, as a consequence, some of the enterprise considerations are not really there.
CouchDB may be the beginning of something different.

Here is what I like about CouchDB
- It's schema-free. The documents it manages are not tied a predefined format and support both attachments and versioning.
- Views are manipulated in "scripting" language
- Indexing, incremental replication support
- Views leverage a scalable map-reduce approach
- Full REST api, client language independence
- Architected from the ground up for massive scalability and high availability deployments
- Architected from the ground up for smart replication
- Integration with Lucene for full text search (upcoming I think)

Here is what I do not like about CouchDB
- Javascript everywhere as the default. Javascript has evolved quite a lot since I first dealt with it, but I still have difficulties accepting its permissiveness.
Of course, this can be changed - in particular for the construction of views. This alleviates my concern.

I am ambivalent on the usage of JSON as the storage format. While I like the principles of JSON, it lacks some of the higher level features that enterprise systems in general care about.
I also have not a clear view yet on how transactional behavior will work. ACID-compliance on single nodes seems to be the focus.
I also do not have a clear view yet on how security will be managed, but I trust those issues are either sorted out or will be sorted out.

That being said, the REST/HTTP - JSON - JavaScript combination will be appealing to most of the Web 2.0 / Enterprise 2.0 community. Adobe and others have done a lot to popularize this combination and it does offer a high degree of flexibility.
This popularization, combined with the up-front massive scalability built into much of the architecture, leads me to believe that performance for document based interactions will meet the requirements of most enterprise applications. I have, however, not seen independent data around this.

There have been other efforts along the same lines. But I do like the usage of Erlang [http://www.erlang.org/] for the implementation language. Erlang is an elegant language for developing scalable server code and satisfies most of the constraints I have seen been mandatory to satisfy in high performance distributed systems.

I am curious to know what the Enterprise Software community thinks of this. We all have a number of cases in which we would trade some relational features for the flexibility this document-centric approach brings. I can see a few for the products I am responsible for...

Are there other efforts similar to this you would recommend looking at? I have looked (a little) at Hypertable on Hadoop. But it's not the same thing...

More search questions

As usual, tough to drop a subject once you start thinking about it.

A couple of things I did not go into in my previous post on search [http://architectguy.blogspot.com/2008/12/search-questions.html] that I think are relevant to the discussion:

- The reputation system needs to be made contextual (yes, here it comes again). The computation of the reputation for a given document needs to take into account the context / motivation the search is taking place in. For example, a document on operating system may have very little reputation overall, but it talks about a microkernel architecture it is highly relevant for those searching for information on software architecture. The reputation of that document should be much higher in the context of a search for information on software architecture than it is in context-less searches.
What that means is essentially that the current focus on just hits is not enough. Google has of course realized that (everything I say they probably have thought of for decades) and has introduced tons of tweaks to the algorithm, including giving more weight to certain referrers. But the key issue remains: the reputation is uni-dimensional and too much information gets lost in that compression.
I think a system that takes into account semantic information provided by the sites (explicit or calculated) may help create multi-dimensional reputation scores that could be leveraged by searches for which the context or motivation is available (again explicitly - which I believe in - or explicitly - which I find tough to imagine at this point).

- Part of the context is "time". When we make searches, the recency of the documents returned by the search is frequently not well managed. The algorithms are tweaked to manage recency, but there is contextual information that should allow the search to be smarter on how it orders what it presents. Again, I believe that is part of the context.

- An issue with personalization is, again, the problem that the motivation for a search is contextual to the point where taking a past search into account to decide on the answers for the future ones may be counterproductive.
This is not new, it has been commented on many times as part of the feedback on large commercial recommendation capabilities (Amazon, TiVo): make a few searches for an exceptional reason (buying a present to a friend with very different tastes to yours, gathering information on a given subject to write a paper you will never look at again, etc..), and mis-targeted recommendations will start coming. These systems cannot guess the context, and hence cannot know when to remember about your interactions, when to forget about them. And both learning and forgetting are key.
With no context on the motivation for the search, only minimal personalization is possible. I have two Amazon accounts - one personal, one professional, precisely to avoid the pollution of the recommendations for one with that for the other.

I am certain lots of people more educated than me on these issues have already explored these issues and provided their opinion on how relevant they are.

I am certain the technical challenges are significant, but I believe the combination of verticalized searches (those for which you have an easy way of specifying the motivation / context for the search), semantically tagged documents, and semantically aware reputation systems will enable better searches, and more targeted monetizable ads.

This has been fun, moving on to other preoccupations.

Search questions

Virtually attending LeWeb08, I got to listen to a presentation by Marissa Meyer on the future of search [http://www.techcrunch.com/2008/12/10/marissa-mayer-at-le-web-the-almost-complete-interview/].
Of course, I know very little about search, but somehow, I find the idea that personalization will be core to the future of search controversial, in a good way.

First, I doubt the personalization of search results will be based on what general search engine knows about the searcher. The reason is that, contrary to many other activities you would do on the Web, searching is fully horizontal. The motivations for searching are different at different points in time, and the same keywords may be used to look for information for very different purposes, resulting in very different documents being relevant.
How is that context captured in an horizontal search? I find it very difficult to imagine that the context can be built up from an analysis of the interactions the user has with the general search engine. Constructing the context will require many other sources, and it's of course possible that Google (Merissa Meyer's employer) is thinking about levering all their assets - including mail, blogs, instant messaging, etc.
Part of the missing context is related to the fact that a lot of the interactions the user has with the environment takes place outside those assets themselves. Reading the resulting documents, following links provided by those documents. It's frequent that in my long web sessions I only spend small time with the generic search and lots of time following paths maybe originating with the search results but mostly happening outside search. Yes, I know the search engines know about those links, but they do not know what paths the users follow through them.

But second, and this is key, generic searches do not have access to the most important contextual information: why is the user making a given search at any given point in time, what is his/her the motivation? This is much more important than we may originally think. Generic search engines, lacking that information, will provide at best the 'most popular' (by some measure) interpretation of the search - and that may or not correspond to what the user is looking for.

Why is this important? Alain Rappaport who founded Medstory [http://www.medstory.com/] and sold it to Microsoft once explained to me his view - which I find compelling - that there is a sweet spot of high value searches that correspond to 'motivated searches', searches that are made for a specific purpose, which, if identified, allows to organize the search and the results in significantly more relevant ways. For example, using Medstory, users can look for information related to medical issues in a system that establishes semantic links to connected issues and provides you with highly relevant information, even though it may not occur to the user that the connections existed. The user types "Heart attack" and gets, in addition to a Google-style list of results, a top bar that provides semantically related search refinement capabilities, etc. The site no longer provides ads (it has not been updated otherwise since the acquisition a couple of years ago), but you can imagine how targeted the ads could be.
I believe in this. I believe that in the short term we will see the emergence of verticalized search engines that will follow Google's model but will set up a context for the search and will be able to provide much more relevant results than generic search engines.

The next step may be semantic search and semantic analysis ad management (such as what Peer39 is trying to do - [http://peer39.com/]). That may be, but I continue thinking that extracting the search context will be too complicated and error-prone. It is probably much more useful (and achievable) to provide explicit verticalized searches and make them as easy to use and relevant as possible.

Maybe that's what Marissa Meyer meant, and Google's (and Microsoft's) forays into the health world may indicate so.

Precision, Speed and Mistakes

One of the benefits of being part of the Twitter craze is that you see real time what interests people you find interesting. Similar principle as other older services, but for some reason I can still not figure out, more effective.

My attention was drawn to a fascinating article written for Vanity Fair by William Langewiesche. Langewiesche is an interesting writer/journalist. I had previously read a few books he wrote: "The Outlaw Sea" (a chilling description of the ocean as the last out-of-legal-reach frontier - something the pirates in Somalia, and the seaborne Mumbai attacks remind us of), "Sahara Unveiled" (an interesting journey in a region of the world I spent quite some time in, and that he describes and analyses quite well) and "Inside the sky" (a beautiful account of the experience of flying)

The article [http://www.vanityfair.com/magazine/2009/01/air_crash200901?printable=true&currentPage=all] focuses on the circumstances that led to the mid-air collision between a private Embraer business jet and a GOL 737 over the Amazon a few years ago.
The key conclusion of the article: the speed and precision that modern technology enables magnifies the effects of mistakes.
In this case, both jets were flying on intersecting fly paths, flown by ultra-precise auto-pilots. Mistakes were made in communication between the pilots and flight control (language was a barrier), and the transponder and TCAS on the business jet were accidentally turned off.
In the past, the natural imprecision of human-controlled flight would have reduced the risk of the collision - but in this case, the jets intersected with surgical precision, the business jet surviving the collision but the 737 being lost.

This is an incredibly vivid illustration of the need of performing overall systems analysis in any application that embeds highly precise technology that changes significantly the way operations are conducted. Such an analysis must focus on:
- where in the system are the points at which changes of protocol, translation, etc., present opportunities for miscommunication or failures?
- how does the system cope with these failures?
- how does the system provide effective monitoring so that these failures are detected as soon as possible - at a speed which is compatible with the speed at which the application is proceeding (you need to move fast to prevent an issue that is occurring fast)?
- how quickly can the application change operation mode to cope with the failure?
- etc.

This is directly related to the preoccupations of Enterprise Decision Management (EDM). EDM-centric applications need to monitor the decisions being made, the outcomes of those decisions, the business value (safety for example in the case above) of the decisions, detect deviance, etc. These applications need to identify the decision metrics to monitor, the conditions under which these decision metrics are indicative of problems, the mechanisms to change the operation mode (move to human processing for example) to cope with some of those problems, etc.

The technologies and approaches used are similar to those involved in the "improvement loop" side of decision management.

IBM's BEM announcement

Here we go: http://www.informationweek.com/news/business_intelligence/bpm/showArticle.jhtml?articleID=212200753&cid=RSSfeed_IWK_Business_Intelligence

IBM launches an attempt to define the "Business Event Management" space. This plays directly into all the controversies around CEP and its overlap and positioning with respect to other spaces such as "Business Process Management" or "Business Rules Management". You can find some of the discussions in Carole-Ann's blog [http://www.edmblog.com/weblog/2008/11/an-attempt-at-demystifying-cep-bpm-and-brms.html], Tim's blog [http://www.thecepblog.com/] and in my own blog in previous postings [http://architectguy.blogspot.com/2008/11/more-on-cep.html and http://architectguy.blogspot.com/2008/11/state-events-time-confusion-around-cep.html].

Is this move by IBM good or bad?

Let's read a little the details:

a) "IBM is staking out a position in complex event processing as applied to business processes"
Ok, so the focus is around events as it pertains to their relevance to business processes. But we should probably not read too much into it given that IBM approaches this whole space from the business process perspective.
b) [it] " is an ability to define a discrete software event, then monitor it over time, set parameters around it, and trigger alerts if the rules governing it are violated"
Ok, fairly standard. But it does refer to "rules governing" alerts, I assume. To use terminology used before, this would be event correlation and pattern matching to translate the "ambient" events to business events.
c) Which is supported by the quote made of Luckham: "What is a complex event? It is an event that could only happen if lots of other events happened"
So far, pretty simple.

But then:
d) "By being able to combine views of software events and business events in one set of tools, IBM gains the prospect of "allowing business users to more easily change the logic in the business event processes," Reeves said. And they should be able to do so in business language, not programming languages"
This is interesting.

The content of the announcement seems to confirm two key things:
- First, event management is primarily about the identification and generation of business events from the ambient events. Similar to what Carole-Ann and I had written in previous posts.
- Second, IBM wants to introduce high level EPLs to express the logic for that processing that are business-centric, something very similar to what Business Rules Languages and approaches are in the business rules management area.

Of course, it is interesting that IBM is making this positioning public so early after the acquisition of Ilog, and long before any true integration with ApSoft.

This may be a way for IBM to establish the "Business Event Management" denomination early on.

I am ambivalent on the approach. While it's probably clear to the reader that I am not a fanatic of the "CEP" acronym, I think I need to read / learn more than a press release on what IBM really intends to cover with their new old TLA.
I also need to understand what IBM thinks of the relative roles of BEM and BRMS. I think that, as pointed out in some of the blogs referred to earlier, the roles are clear, but I am not sure what I call clear is what others would call acceptable.

What's your take?