I have been reading a little about what is being presented as the causes for the current worldwide financial crisis. Nothing really surprising in the descriptions, of course, we’ve all heard or read these experts who failed to predict anything going to great details in explaining a posteriori what happened, and provide supposedly valuable insight into what would be coming.
What really amazes me is how each one of the human branches of activities likes living in its own ivory tower. Each industry learns its own lessons from its own failures, or attempts to, but rarely does one industry draw the lessons from another industry’s failures. I am certain some of that has to do with the fact that industries are frequently out of sync in their cycles, but also from a lack of abstraction and high level view on the nature of the industry, its processes and decisions.
A key factor is in this mess is simply the fact that the financial industry created all sorts of financial instruments working at ever higher levels of abstraction, more and more removed from the reality of the underlying relevant physical or legal entities without really understanding their systems behavior.
Derivatives, derivatives on derivatives, swaps, etc: sure, each and every one of them has a solid – and here, sorry for my friends in that space, I will qualify solid as “seemingly” solid – mathematical support. Sure, the bright mathematicians and physicist Wall Street hired by the plane load out of Europe (Russia, France, etc.) are very capable in their domain. They can understand business problems, abstract them into sophisticated models, even understand the conditions under which these models apply, provide all sorts of sensitivity analysis to take into account potential environmental changes, etc. I will not question that.
But I will say that there is no evidence whatsoever that the financial industry applied – or applies – serious systems analysis to the combination of these instruments as they are applied. The obvious aspects that systems analysts pay attention to have been totally ignored: no real systems analysis of applicability, analysis of error / exception propagation, exception management, etc.
As a consequence, each model may very well be well understood, inducing a false sense of security in that this is all under control.
When systems break, they most often break in the interface between components (the famous Mars Climate orbiter issue: http://mars.jpl.nasa.gov/msp98/news/mco990930.html), or in exception handling (the famous Ariane rocket explosion: read the report from one of my teachers, Jacques Louis Lions, http://esamultimedia.esa.int/docs/esa-x-1819eng.pdf), or in the actual decision making in particular when it involves humans (see http://architectguy.blogspot.com/2008/12/precision-speed-and-mistakes.html below, this crisis).
The financial industry – driven by its purely speculative focus, and totally blinded to the fact it no longer understands its instruments – failed and continues failing on all three counts.
For all their intellect and knowledge, the Nobel Prize winner creators of LTCM failed. The industry around it failed more than a decade ago, but little was learned from there. It went back to the same (maybe less obvious) hubris and disregard for proper risk management. With the same consequences…
Frequently enough these failures have simple root causes, overlooked only because of the overall complexity of the system and incredibly cheap to repair compared to the huge cost of these failures. Imagine if the verification systems of Société Générale had actually worked (http://en.wikipedia.org/wiki/January_2008_Soci%C3%A9t%C3%A9_G%C3%A9n%C3%A9rale_trading_loss_incident) : Kerviel would be unknown, and the company would not have lost billions of dollars (and this was limited thanks to the intervention of the company’s crack team and the help from the French government)
Other industries have learned and built into their practices both simplification of components and management of the systems to reach a level in which, when a black swan does appear, the consequences can be assessed and remedial action can be taken. The financial industry should do the same.
Decision Management can help as it increases its focus on scenario-based risk assessment. Modeling decisions and managing around them scenarios that enable the systems analysis of the business processes should allow much better understanding and control of the consequences of the decisions in changing conditions. A lot of the energy spent building predictive models based on well bounded simplifications should be shifted towards modeling the decisions that are made, the business outcomes, their dependencies with respect tot the environment, and managing vast and ever enriched portfolios of scenarios around them to keep risk to black swans in check.
This will require discipline, as it basically emphasizes analyzing highly unlikely but consequence heavy scenarios in times when things are going seemingly well.
But I believe it will be needed.
The Decision Management discipline can help, leverage it.
On a side note on all this,
- Read Nassim Taleb (http://en.wikipedia.org/wiki/Nassim_Nicholas_Taleb).
- Watch him discuss the crisis on TV (http://paul.kedrosky.com/archives/2008/10/12/nassim_taleb_ge.html)
- Watch him and Benoit Mandelbrot (http://en.wikipedia.org/wiki/Benoit_Mandelbrot) debate the crisis: http://uk.youtube.com/watch?v=DLFkQdiXPbo&NR=1
Monday, December 22, 2008
Monday, December 15, 2008
CouchDB for Enterprise Applications?
A little late in noticing this: CouchDB has become a first class citizen project of Apache [http://couchdb.apache.org/].
This is an interesting development. For us, enterprise application folks, it takes a while to get rid of the relational paradigm and bring ourselves to consider a document-oriented database such as this one. But I have to say that years of squeezing documents into the relational model have slowly driven me to appreciate the virtue of something different.
Full blown document management systems have always been something that feels too much "document publisher" oriented. The preoccupations seem to be largely around supporting the management of documents humans read, and, as a consequence, some of the enterprise considerations are not really there.
CouchDB may be the beginning of something different.
Here is what I like about CouchDB
- It's schema-free. The documents it manages are not tied a predefined format and support both attachments and versioning.
- Views are manipulated in "scripting" language
- Indexing, incremental replication support
- Views leverage a scalable map-reduce approach
- Full REST api, client language independence
- Architected from the ground up for massive scalability and high availability deployments
- Architected from the ground up for smart replication
- Integration with Lucene for full text search (upcoming I think)
Here is what I do not like about CouchDB
- Javascript everywhere as the default. Javascript has evolved quite a lot since I first dealt with it, but I still have difficulties accepting its permissiveness.
Of course, this can be changed - in particular for the construction of views. This alleviates my concern.
I am ambivalent on the usage of JSON as the storage format. While I like the principles of JSON, it lacks some of the higher level features that enterprise systems in general care about.
I also have not a clear view yet on how transactional behavior will work. ACID-compliance on single nodes seems to be the focus.
I also do not have a clear view yet on how security will be managed, but I trust those issues are either sorted out or will be sorted out.
That being said, the REST/HTTP - JSON - JavaScript combination will be appealing to most of the Web 2.0 / Enterprise 2.0 community. Adobe and others have done a lot to popularize this combination and it does offer a high degree of flexibility.
This popularization, combined with the up-front massive scalability built into much of the architecture, leads me to believe that performance for document based interactions will meet the requirements of most enterprise applications. I have, however, not seen independent data around this.
There have been other efforts along the same lines. But I do like the usage of Erlang [http://www.erlang.org/] for the implementation language. Erlang is an elegant language for developing scalable server code and satisfies most of the constraints I have seen been mandatory to satisfy in high performance distributed systems.
I am curious to know what the Enterprise Software community thinks of this. We all have a number of cases in which we would trade some relational features for the flexibility this document-centric approach brings. I can see a few for the products I am responsible for...
Are there other efforts similar to this you would recommend looking at? I have looked (a little) at Hypertable on Hadoop. But it's not the same thing...
This is an interesting development. For us, enterprise application folks, it takes a while to get rid of the relational paradigm and bring ourselves to consider a document-oriented database such as this one. But I have to say that years of squeezing documents into the relational model have slowly driven me to appreciate the virtue of something different.
Full blown document management systems have always been something that feels too much "document publisher" oriented. The preoccupations seem to be largely around supporting the management of documents humans read, and, as a consequence, some of the enterprise considerations are not really there.
CouchDB may be the beginning of something different.
Here is what I like about CouchDB
- It's schema-free. The documents it manages are not tied a predefined format and support both attachments and versioning.
- Views are manipulated in "scripting" language
- Indexing, incremental replication support
- Views leverage a scalable map-reduce approach
- Full REST api, client language independence
- Architected from the ground up for massive scalability and high availability deployments
- Architected from the ground up for smart replication
- Integration with Lucene for full text search (upcoming I think)
Here is what I do not like about CouchDB
- Javascript everywhere as the default. Javascript has evolved quite a lot since I first dealt with it, but I still have difficulties accepting its permissiveness.
Of course, this can be changed - in particular for the construction of views. This alleviates my concern.
I am ambivalent on the usage of JSON as the storage format. While I like the principles of JSON, it lacks some of the higher level features that enterprise systems in general care about.
I also have not a clear view yet on how transactional behavior will work. ACID-compliance on single nodes seems to be the focus.
I also do not have a clear view yet on how security will be managed, but I trust those issues are either sorted out or will be sorted out.
That being said, the REST/HTTP - JSON - JavaScript combination will be appealing to most of the Web 2.0 / Enterprise 2.0 community. Adobe and others have done a lot to popularize this combination and it does offer a high degree of flexibility.
This popularization, combined with the up-front massive scalability built into much of the architecture, leads me to believe that performance for document based interactions will meet the requirements of most enterprise applications. I have, however, not seen independent data around this.
There have been other efforts along the same lines. But I do like the usage of Erlang [http://www.erlang.org/] for the implementation language. Erlang is an elegant language for developing scalable server code and satisfies most of the constraints I have seen been mandatory to satisfy in high performance distributed systems.
I am curious to know what the Enterprise Software community thinks of this. We all have a number of cases in which we would trade some relational features for the flexibility this document-centric approach brings. I can see a few for the products I am responsible for...
Are there other efforts similar to this you would recommend looking at? I have looked (a little) at Hypertable on Hadoop. But it's not the same thing...
Sunday, December 14, 2008
More search questions
As usual, tough to drop a subject once you start thinking about it.
A couple of things I did not go into in my previous post on search [http://architectguy.blogspot.com/2008/12/search-questions.html] that I think are relevant to the discussion:
- The reputation system needs to be made contextual (yes, here it comes again). The computation of the reputation for a given document needs to take into account the context / motivation the search is taking place in. For example, a document on operating system may have very little reputation overall, but it talks about a microkernel architecture it is highly relevant for those searching for information on software architecture. The reputation of that document should be much higher in the context of a search for information on software architecture than it is in context-less searches.
What that means is essentially that the current focus on just hits is not enough. Google has of course realized that (everything I say they probably have thought of for decades) and has introduced tons of tweaks to the algorithm, including giving more weight to certain referrers. But the key issue remains: the reputation is uni-dimensional and too much information gets lost in that compression.
I think a system that takes into account semantic information provided by the sites (explicit or calculated) may help create multi-dimensional reputation scores that could be leveraged by searches for which the context or motivation is available (again explicitly - which I believe in - or explicitly - which I find tough to imagine at this point).
- Part of the context is "time". When we make searches, the recency of the documents returned by the search is frequently not well managed. The algorithms are tweaked to manage recency, but there is contextual information that should allow the search to be smarter on how it orders what it presents. Again, I believe that is part of the context.
- An issue with personalization is, again, the problem that the motivation for a search is contextual to the point where taking a past search into account to decide on the answers for the future ones may be counterproductive.
This is not new, it has been commented on many times as part of the feedback on large commercial recommendation capabilities (Amazon, TiVo): make a few searches for an exceptional reason (buying a present to a friend with very different tastes to yours, gathering information on a given subject to write a paper you will never look at again, etc..), and mis-targeted recommendations will start coming. These systems cannot guess the context, and hence cannot know when to remember about your interactions, when to forget about them. And both learning and forgetting are key.
With no context on the motivation for the search, only minimal personalization is possible. I have two Amazon accounts - one personal, one professional, precisely to avoid the pollution of the recommendations for one with that for the other.
I am certain lots of people more educated than me on these issues have already explored these issues and provided their opinion on how relevant they are.
I am certain the technical challenges are significant, but I believe the combination of verticalized searches (those for which you have an easy way of specifying the motivation / context for the search), semantically tagged documents, and semantically aware reputation systems will enable better searches, and more targeted monetizable ads.
This has been fun, moving on to other preoccupations.
A couple of things I did not go into in my previous post on search [http://architectguy.blogspot.com/2008/12/search-questions.html] that I think are relevant to the discussion:
- The reputation system needs to be made contextual (yes, here it comes again). The computation of the reputation for a given document needs to take into account the context / motivation the search is taking place in. For example, a document on operating system may have very little reputation overall, but it talks about a microkernel architecture it is highly relevant for those searching for information on software architecture. The reputation of that document should be much higher in the context of a search for information on software architecture than it is in context-less searches.
What that means is essentially that the current focus on just hits is not enough. Google has of course realized that (everything I say they probably have thought of for decades) and has introduced tons of tweaks to the algorithm, including giving more weight to certain referrers. But the key issue remains: the reputation is uni-dimensional and too much information gets lost in that compression.
I think a system that takes into account semantic information provided by the sites (explicit or calculated) may help create multi-dimensional reputation scores that could be leveraged by searches for which the context or motivation is available (again explicitly - which I believe in - or explicitly - which I find tough to imagine at this point).
- Part of the context is "time". When we make searches, the recency of the documents returned by the search is frequently not well managed. The algorithms are tweaked to manage recency, but there is contextual information that should allow the search to be smarter on how it orders what it presents. Again, I believe that is part of the context.
- An issue with personalization is, again, the problem that the motivation for a search is contextual to the point where taking a past search into account to decide on the answers for the future ones may be counterproductive.
This is not new, it has been commented on many times as part of the feedback on large commercial recommendation capabilities (Amazon, TiVo): make a few searches for an exceptional reason (buying a present to a friend with very different tastes to yours, gathering information on a given subject to write a paper you will never look at again, etc..), and mis-targeted recommendations will start coming. These systems cannot guess the context, and hence cannot know when to remember about your interactions, when to forget about them. And both learning and forgetting are key.
With no context on the motivation for the search, only minimal personalization is possible. I have two Amazon accounts - one personal, one professional, precisely to avoid the pollution of the recommendations for one with that for the other.
I am certain lots of people more educated than me on these issues have already explored these issues and provided their opinion on how relevant they are.
I am certain the technical challenges are significant, but I believe the combination of verticalized searches (those for which you have an easy way of specifying the motivation / context for the search), semantically tagged documents, and semantically aware reputation systems will enable better searches, and more targeted monetizable ads.
This has been fun, moving on to other preoccupations.
Search questions
Virtually attending LeWeb08, I got to listen to a presentation by Marissa Meyer on the future of search [http://www.techcrunch.com/2008/12/10/marissa-mayer-at-le-web-the-almost-complete-interview/].
Of course, I know very little about search, but somehow, I find the idea that personalization will be core to the future of search controversial, in a good way.
First, I doubt the personalization of search results will be based on what general search engine knows about the searcher. The reason is that, contrary to many other activities you would do on the Web, searching is fully horizontal. The motivations for searching are different at different points in time, and the same keywords may be used to look for information for very different purposes, resulting in very different documents being relevant.
How is that context captured in an horizontal search? I find it very difficult to imagine that the context can be built up from an analysis of the interactions the user has with the general search engine. Constructing the context will require many other sources, and it's of course possible that Google (Merissa Meyer's employer) is thinking about levering all their assets - including mail, blogs, instant messaging, etc.
Part of the missing context is related to the fact that a lot of the interactions the user has with the environment takes place outside those assets themselves. Reading the resulting documents, following links provided by those documents. It's frequent that in my long web sessions I only spend small time with the generic search and lots of time following paths maybe originating with the search results but mostly happening outside search. Yes, I know the search engines know about those links, but they do not know what paths the users follow through them.
But second, and this is key, generic searches do not have access to the most important contextual information: why is the user making a given search at any given point in time, what is his/her the motivation? This is much more important than we may originally think. Generic search engines, lacking that information, will provide at best the 'most popular' (by some measure) interpretation of the search - and that may or not correspond to what the user is looking for.
Why is this important? Alain Rappaport who founded Medstory [http://www.medstory.com/] and sold it to Microsoft once explained to me his view - which I find compelling - that there is a sweet spot of high value searches that correspond to 'motivated searches', searches that are made for a specific purpose, which, if identified, allows to organize the search and the results in significantly more relevant ways. For example, using Medstory, users can look for information related to medical issues in a system that establishes semantic links to connected issues and provides you with highly relevant information, even though it may not occur to the user that the connections existed. The user types "Heart attack" and gets, in addition to a Google-style list of results, a top bar that provides semantically related search refinement capabilities, etc. The site no longer provides ads (it has not been updated otherwise since the acquisition a couple of years ago), but you can imagine how targeted the ads could be.
I believe in this. I believe that in the short term we will see the emergence of verticalized search engines that will follow Google's model but will set up a context for the search and will be able to provide much more relevant results than generic search engines.
The next step may be semantic search and semantic analysis ad management (such as what Peer39 is trying to do - [http://peer39.com/]). That may be, but I continue thinking that extracting the search context will be too complicated and error-prone. It is probably much more useful (and achievable) to provide explicit verticalized searches and make them as easy to use and relevant as possible.
Maybe that's what Marissa Meyer meant, and Google's (and Microsoft's) forays into the health world may indicate so.
Of course, I know very little about search, but somehow, I find the idea that personalization will be core to the future of search controversial, in a good way.
First, I doubt the personalization of search results will be based on what general search engine knows about the searcher. The reason is that, contrary to many other activities you would do on the Web, searching is fully horizontal. The motivations for searching are different at different points in time, and the same keywords may be used to look for information for very different purposes, resulting in very different documents being relevant.
How is that context captured in an horizontal search? I find it very difficult to imagine that the context can be built up from an analysis of the interactions the user has with the general search engine. Constructing the context will require many other sources, and it's of course possible that Google (Merissa Meyer's employer) is thinking about levering all their assets - including mail, blogs, instant messaging, etc.
Part of the missing context is related to the fact that a lot of the interactions the user has with the environment takes place outside those assets themselves. Reading the resulting documents, following links provided by those documents. It's frequent that in my long web sessions I only spend small time with the generic search and lots of time following paths maybe originating with the search results but mostly happening outside search. Yes, I know the search engines know about those links, but they do not know what paths the users follow through them.
But second, and this is key, generic searches do not have access to the most important contextual information: why is the user making a given search at any given point in time, what is his/her the motivation? This is much more important than we may originally think. Generic search engines, lacking that information, will provide at best the 'most popular' (by some measure) interpretation of the search - and that may or not correspond to what the user is looking for.
Why is this important? Alain Rappaport who founded Medstory [http://www.medstory.com/] and sold it to Microsoft once explained to me his view - which I find compelling - that there is a sweet spot of high value searches that correspond to 'motivated searches', searches that are made for a specific purpose, which, if identified, allows to organize the search and the results in significantly more relevant ways. For example, using Medstory, users can look for information related to medical issues in a system that establishes semantic links to connected issues and provides you with highly relevant information, even though it may not occur to the user that the connections existed. The user types "Heart attack" and gets, in addition to a Google-style list of results, a top bar that provides semantically related search refinement capabilities, etc. The site no longer provides ads (it has not been updated otherwise since the acquisition a couple of years ago), but you can imagine how targeted the ads could be.
I believe in this. I believe that in the short term we will see the emergence of verticalized search engines that will follow Google's model but will set up a context for the search and will be able to provide much more relevant results than generic search engines.
The next step may be semantic search and semantic analysis ad management (such as what Peer39 is trying to do - [http://peer39.com/]). That may be, but I continue thinking that extracting the search context will be too complicated and error-prone. It is probably much more useful (and achievable) to provide explicit verticalized searches and make them as easy to use and relevant as possible.
Maybe that's what Marissa Meyer meant, and Google's (and Microsoft's) forays into the health world may indicate so.
Friday, December 5, 2008
Precision, Speed and Mistakes
One of the benefits of being part of the Twitter craze is that you see real time what interests people you find interesting. Similar principle as other older services, but for some reason I can still not figure out, more effective.
My attention was drawn to a fascinating article written for Vanity Fair by William Langewiesche. Langewiesche is an interesting writer/journalist. I had previously read a few books he wrote: "The Outlaw Sea" (a chilling description of the ocean as the last out-of-legal-reach frontier - something the pirates in Somalia, and the seaborne Mumbai attacks remind us of), "Sahara Unveiled" (an interesting journey in a region of the world I spent quite some time in, and that he describes and analyses quite well) and "Inside the sky" (a beautiful account of the experience of flying)
The article [http://www.vanityfair.com/magazine/2009/01/air_crash200901?printable=true¤tPage=all] focuses on the circumstances that led to the mid-air collision between a private Embraer business jet and a GOL 737 over the Amazon a few years ago.
The key conclusion of the article: the speed and precision that modern technology enables magnifies the effects of mistakes.
In this case, both jets were flying on intersecting fly paths, flown by ultra-precise auto-pilots. Mistakes were made in communication between the pilots and flight control (language was a barrier), and the transponder and TCAS on the business jet were accidentally turned off.
In the past, the natural imprecision of human-controlled flight would have reduced the risk of the collision - but in this case, the jets intersected with surgical precision, the business jet surviving the collision but the 737 being lost.
This is an incredibly vivid illustration of the need of performing overall systems analysis in any application that embeds highly precise technology that changes significantly the way operations are conducted. Such an analysis must focus on:
- where in the system are the points at which changes of protocol, translation, etc., present opportunities for miscommunication or failures?
- how does the system cope with these failures?
- how does the system provide effective monitoring so that these failures are detected as soon as possible - at a speed which is compatible with the speed at which the application is proceeding (you need to move fast to prevent an issue that is occurring fast)?
- how quickly can the application change operation mode to cope with the failure?
- etc.
This is directly related to the preoccupations of Enterprise Decision Management (EDM). EDM-centric applications need to monitor the decisions being made, the outcomes of those decisions, the business value (safety for example in the case above) of the decisions, detect deviance, etc. These applications need to identify the decision metrics to monitor, the conditions under which these decision metrics are indicative of problems, the mechanisms to change the operation mode (move to human processing for example) to cope with some of those problems, etc.
The technologies and approaches used are similar to those involved in the "improvement loop" side of decision management.
My attention was drawn to a fascinating article written for Vanity Fair by William Langewiesche. Langewiesche is an interesting writer/journalist. I had previously read a few books he wrote: "The Outlaw Sea" (a chilling description of the ocean as the last out-of-legal-reach frontier - something the pirates in Somalia, and the seaborne Mumbai attacks remind us of), "Sahara Unveiled" (an interesting journey in a region of the world I spent quite some time in, and that he describes and analyses quite well) and "Inside the sky" (a beautiful account of the experience of flying)
The article [http://www.vanityfair.com/magazine/2009/01/air_crash200901?printable=true¤tPage=all] focuses on the circumstances that led to the mid-air collision between a private Embraer business jet and a GOL 737 over the Amazon a few years ago.
The key conclusion of the article: the speed and precision that modern technology enables magnifies the effects of mistakes.
In this case, both jets were flying on intersecting fly paths, flown by ultra-precise auto-pilots. Mistakes were made in communication between the pilots and flight control (language was a barrier), and the transponder and TCAS on the business jet were accidentally turned off.
In the past, the natural imprecision of human-controlled flight would have reduced the risk of the collision - but in this case, the jets intersected with surgical precision, the business jet surviving the collision but the 737 being lost.
This is an incredibly vivid illustration of the need of performing overall systems analysis in any application that embeds highly precise technology that changes significantly the way operations are conducted. Such an analysis must focus on:
- where in the system are the points at which changes of protocol, translation, etc., present opportunities for miscommunication or failures?
- how does the system cope with these failures?
- how does the system provide effective monitoring so that these failures are detected as soon as possible - at a speed which is compatible with the speed at which the application is proceeding (you need to move fast to prevent an issue that is occurring fast)?
- how quickly can the application change operation mode to cope with the failure?
- etc.
This is directly related to the preoccupations of Enterprise Decision Management (EDM). EDM-centric applications need to monitor the decisions being made, the outcomes of those decisions, the business value (safety for example in the case above) of the decisions, detect deviance, etc. These applications need to identify the decision metrics to monitor, the conditions under which these decision metrics are indicative of problems, the mechanisms to change the operation mode (move to human processing for example) to cope with some of those problems, etc.
The technologies and approaches used are similar to those involved in the "improvement loop" side of decision management.
Monday, December 1, 2008
IBM's BEM announcement
Here we go: http://www.informationweek.com/news/business_intelligence/bpm/showArticle.jhtml?articleID=212200753&cid=RSSfeed_IWK_Business_Intelligence
IBM launches an attempt to define the "Business Event Management" space. This plays directly into all the controversies around CEP and its overlap and positioning with respect to other spaces such as "Business Process Management" or "Business Rules Management". You can find some of the discussions in Carole-Ann's blog [http://www.edmblog.com/weblog/2008/11/an-attempt-at-demystifying-cep-bpm-and-brms.html], Tim's blog [http://www.thecepblog.com/] and in my own blog in previous postings [http://architectguy.blogspot.com/2008/11/more-on-cep.html and http://architectguy.blogspot.com/2008/11/state-events-time-confusion-around-cep.html].
Is this move by IBM good or bad?
Let's read a little the details:
a) "IBM is staking out a position in complex event processing as applied to business processes"
Ok, so the focus is around events as it pertains to their relevance to business processes. But we should probably not read too much into it given that IBM approaches this whole space from the business process perspective.
b) [it] " is an ability to define a discrete software event, then monitor it over time, set parameters around it, and trigger alerts if the rules governing it are violated"
Ok, fairly standard. But it does refer to "rules governing" alerts, I assume. To use terminology used before, this would be event correlation and pattern matching to translate the "ambient" events to business events.
c) Which is supported by the quote made of Luckham: "What is a complex event? It is an event that could only happen if lots of other events happened"
So far, pretty simple.
But then:
d) "By being able to combine views of software events and business events in one set of tools, IBM gains the prospect of "allowing business users to more easily change the logic in the business event processes," Reeves said. And they should be able to do so in business language, not programming languages"
This is interesting.
The content of the announcement seems to confirm two key things:
- First, event management is primarily about the identification and generation of business events from the ambient events. Similar to what Carole-Ann and I had written in previous posts.
- Second, IBM wants to introduce high level EPLs to express the logic for that processing that are business-centric, something very similar to what Business Rules Languages and approaches are in the business rules management area.
Of course, it is interesting that IBM is making this positioning public so early after the acquisition of Ilog, and long before any true integration with ApSoft.
This may be a way for IBM to establish the "Business Event Management" denomination early on.
I am ambivalent on the approach. While it's probably clear to the reader that I am not a fanatic of the "CEP" acronym, I think I need to read / learn more than a press release on what IBM really intends to cover with their new old TLA.
I also need to understand what IBM thinks of the relative roles of BEM and BRMS. I think that, as pointed out in some of the blogs referred to earlier, the roles are clear, but I am not sure what I call clear is what others would call acceptable.
What's your take?
IBM launches an attempt to define the "Business Event Management" space. This plays directly into all the controversies around CEP and its overlap and positioning with respect to other spaces such as "Business Process Management" or "Business Rules Management". You can find some of the discussions in Carole-Ann's blog [http://www.edmblog.com/weblog/2008/11/an-attempt-at-demystifying-cep-bpm-and-brms.html], Tim's blog [http://www.thecepblog.com/] and in my own blog in previous postings [http://architectguy.blogspot.com/2008/11/more-on-cep.html and http://architectguy.blogspot.com/2008/11/state-events-time-confusion-around-cep.html].
Is this move by IBM good or bad?
Let's read a little the details:
a) "IBM is staking out a position in complex event processing as applied to business processes"
Ok, so the focus is around events as it pertains to their relevance to business processes. But we should probably not read too much into it given that IBM approaches this whole space from the business process perspective.
b) [it] " is an ability to define a discrete software event, then monitor it over time, set parameters around it, and trigger alerts if the rules governing it are violated"
Ok, fairly standard. But it does refer to "rules governing" alerts, I assume. To use terminology used before, this would be event correlation and pattern matching to translate the "ambient" events to business events.
c) Which is supported by the quote made of Luckham: "What is a complex event? It is an event that could only happen if lots of other events happened"
So far, pretty simple.
But then:
d) "By being able to combine views of software events and business events in one set of tools, IBM gains the prospect of "allowing business users to more easily change the logic in the business event processes," Reeves said. And they should be able to do so in business language, not programming languages"
This is interesting.
The content of the announcement seems to confirm two key things:
- First, event management is primarily about the identification and generation of business events from the ambient events. Similar to what Carole-Ann and I had written in previous posts.
- Second, IBM wants to introduce high level EPLs to express the logic for that processing that are business-centric, something very similar to what Business Rules Languages and approaches are in the business rules management area.
Of course, it is interesting that IBM is making this positioning public so early after the acquisition of Ilog, and long before any true integration with ApSoft.
This may be a way for IBM to establish the "Business Event Management" denomination early on.
I am ambivalent on the approach. While it's probably clear to the reader that I am not a fanatic of the "CEP" acronym, I think I need to read / learn more than a press release on what IBM really intends to cover with their new old TLA.
I also need to understand what IBM thinks of the relative roles of BEM and BRMS. I think that, as pointed out in some of the blogs referred to earlier, the roles are clear, but I am not sure what I call clear is what others would call acceptable.
What's your take?
Monday, November 24, 2008
Map - Reduce - Already happening...
Following up on the previous post [http://architectguy.blogspot.com/2008/11/map-reduce-relevance-to-analytics.html], and after some more quality time with google, it's clear that there is more activity around the extension of map-reduce to large scale analytics than I originally thought or knew about.
Joe Hellerstein has published in O'Reilly Radar an interesting post on this exact subject [http://radar.oreilly.com/2008/11/the-commoditization-of-massive.html]. It is very interesting reading.
I think that beyond "massive data analysis" we will see the application of map-reduce to "massive event correlation". The same way the amount of data available for (and crying for) analytics processing is staggering and continues growing at staggering speed, the number and complexity of events that need to be processed to find the patterns and correlations that make them relevant to business apps processing is staggering and continues growing.
RFID, all sorts of devices connected to the cloud, exchanging information / pushing events related in many ways, all sorts of formal or informal transactions, connections and disconnections, ...
Joe points out that we will get to some convergence between traditional data management environments (your SQL) and the map-reduce approaches. He even points out to offerings put forward by innovative players - so what I thought was bound to happen in my previous post is happening even faster.
To paraphrase (in a slightly reworded fashion) what I wrote earlier,
- event pattern identification and correlation are ripe for this type of approaches, and will require it
- new algorithms will be made possible that leverage the approaches to provide results we do not think are possible today
This is one of those typical cycles: the abundance of data and events pushes the creation of new techniques, which in turn enable new applications which produce more data and events.
Very interesting times.
Joe Hellerstein has published in O'Reilly Radar an interesting post on this exact subject [http://radar.oreilly.com/2008/11/the-commoditization-of-massive.html]. It is very interesting reading.
I think that beyond "massive data analysis" we will see the application of map-reduce to "massive event correlation". The same way the amount of data available for (and crying for) analytics processing is staggering and continues growing at staggering speed, the number and complexity of events that need to be processed to find the patterns and correlations that make them relevant to business apps processing is staggering and continues growing.
RFID, all sorts of devices connected to the cloud, exchanging information / pushing events related in many ways, all sorts of formal or informal transactions, connections and disconnections, ...
Joe points out that we will get to some convergence between traditional data management environments (your SQL) and the map-reduce approaches. He even points out to offerings put forward by innovative players - so what I thought was bound to happen in my previous post is happening even faster.
To paraphrase (in a slightly reworded fashion) what I wrote earlier,
- event pattern identification and correlation are ripe for this type of approaches, and will require it
- new algorithms will be made possible that leverage the approaches to provide results we do not think are possible today
This is one of those typical cycles: the abundance of data and events pushes the creation of new techniques, which in turn enable new applications which produce more data and events.
Very interesting times.
Sunday, November 23, 2008
Map - Reduce - relevance to analytics?
One of the key problems we face when dealing with vast numbers of events and/or vast amounts of data is how to efficiently parallelize the processing so as to accelerate the analytic development cycles. This is not just a question of efficiency (minimize the downtime of high cost PhD resources, ...) but also a question of feasibility (make the development of the analytics fit in business driven time-lines, such as the time to detect and react to new fraud mechanisms).
My experience, coming from the real time embedded systems days, is that the best ways to achieve massive parallelization result from highly simplified low level programming models. These models are designed to do only very simple operations but with nothing in them that prevents systematic parallelization.
Google has made popular (at least in the technosphere) the map-reduce approach derived from the functional programming world [http://en.wikipedia.org/wiki/MapReduce]. Google officially applies the approach to its search core capabilities, but it's very likely it uses it for much more than that.
Its formal definition is trivial, but map-reduce does provide a solid basis for massive parallelization of list processing. Which is very much what a lot - a lot - of analytics development work is. It may very well end up being a key component that will allow the implementation of complex analytics processing of amounts of data well beyond what we tackle today, enabling applications that are beyond reach today.
Today, map-reduce is essentially used for search (processing text), network processing (as in social networks), etc. The Hadoop open source effort [http://hadoop.apache.org/core/] has a list of current applications to which its map-reduce implementation is applied [http://wiki.apache.org/hadoop/PoweredBy]: that makes a very interesting reading.
Its applicability to machine learning / predictive analytics building is illustrated by the Mahout effort [http://lucene.apache.org/mahout/] which seeks to leverage Hadoop to specific implementations of traditional machine learning algorithms [http://www.cs.stanford.edu/people/ang//papers/nips06-mapreducemulticore.pdf]. I see immediate applications to both including this approaches to CEP (highly parallel processing of events to achieve fast correlation) and predictive analytics development (highly parallel processing of data to find patterns - neural-net like; highly parallel implementations of genetic approaches, etc.).
I would be curious to know what the reader thinks about this.
I also believe a few developments are bound to happen, and I expect to see more happening around them in the coming few years:
- development of hybrid systems combining more traditional processing (SQL, SAS datasets,...) with map-reduce, potentially introduced by the vendors themselves but more likely by minor innovative players first
- development of new algorithms in event processing / data processing / machine learning that leverage map-reduce
- introduction of other approaches similar to map-reduce as the corresponding results.
I am particularly intrigued by the last two possibilities.
My experience, coming from the real time embedded systems days, is that the best ways to achieve massive parallelization result from highly simplified low level programming models. These models are designed to do only very simple operations but with nothing in them that prevents systematic parallelization.
Google has made popular (at least in the technosphere) the map-reduce approach derived from the functional programming world [http://en.wikipedia.org/wiki/MapReduce]. Google officially applies the approach to its search core capabilities, but it's very likely it uses it for much more than that.
Its formal definition is trivial, but map-reduce does provide a solid basis for massive parallelization of list processing. Which is very much what a lot - a lot - of analytics development work is. It may very well end up being a key component that will allow the implementation of complex analytics processing of amounts of data well beyond what we tackle today, enabling applications that are beyond reach today.
Today, map-reduce is essentially used for search (processing text), network processing (as in social networks), etc. The Hadoop open source effort [http://hadoop.apache.org/core/] has a list of current applications to which its map-reduce implementation is applied [http://wiki.apache.org/hadoop/PoweredBy]: that makes a very interesting reading.
Its applicability to machine learning / predictive analytics building is illustrated by the Mahout effort [http://lucene.apache.org/mahout/] which seeks to leverage Hadoop to specific implementations of traditional machine learning algorithms [http://www.cs.stanford.edu/people/ang//papers/nips06-mapreducemulticore.pdf]. I see immediate applications to both including this approaches to CEP (highly parallel processing of events to achieve fast correlation) and predictive analytics development (highly parallel processing of data to find patterns - neural-net like; highly parallel implementations of genetic approaches, etc.).
I would be curious to know what the reader thinks about this.
I also believe a few developments are bound to happen, and I expect to see more happening around them in the coming few years:
- development of hybrid systems combining more traditional processing (SQL, SAS datasets,...) with map-reduce, potentially introduced by the vendors themselves but more likely by minor innovative players first
- development of new algorithms in event processing / data processing / machine learning that leverage map-reduce
- introduction of other approaches similar to map-reduce as the corresponding results.
I am particularly intrigued by the last two possibilities.
Friday, November 21, 2008
Something different - Lapointe and Perec
Not being able to sleep much has its small rewards... such as having time to spend in seemingly futile brain teasers, mind games and other "exercises" that, maybe because of the general exhaustion, I find quite amusing.
I was listening to a few old recordings by Boby Lapointe, a 50s-60s French singer, master of clever alliterations and plays on words. For some reason the lyrics of "Ta Katie t'a quitte" reminded me that I had read somewhere that he had introduced a bi-bi-binary system that would make manipulating large numbers a pleasure.
His system was basicaly an hexadecimal system (base 16 = base 2 ^ 2 ^ 2 ... no wonder Lapointe would focus on base 16 given his love for alliterations...), for which he defined a complete symbol set and corresponding pronunciation.
With this system, something like "119" + "137" = "256" would become "bibi" + "koka" = "hahoho" (thanks Google).
Much better...
Following hyperlinks reading that stuff, I stumbled upon references to other language games. In particular the famous (in the French speaking world) book written by Perec (La Disparition) that not only fully avoids using the letter "e" in a 300+ pages work, but also is largely self referential, discussing as part of its plot the disappearance of said letter [http://www.amazon.fr/Disparition-Georges-Perec/dp/207071523X/ref=sr_1_3?ie=UTF8&s=books&qid=1227261348&sr=8-3].
This is a brilliant technical achievement: in French, around 85% of the words that you would ever write will have the letter "e" in it (with or without accentuation).
Furthermore, the result is a book you would read: an interesting plot, constructed characters, etc. I read it many years ago, after a teacher gave me a copy of the book after challenging me to write a formal essay with the same challenge (I could not go beyond three pages in one hour, and barely articulated a very limited argumentation).
Georges Perec loved these formal challenges, very similar in nature to what Boby Lapointe did in songs.
The same teacher mentioned earlier also introduced me to the French poetry from the Baroque period - a lot of the same alliterations and plays on words. I still have in my brain somewhere the first verses of Pierre de Marbeuf's "La mer et l'amour ont l'amer pour partage".
In any case, I love this stuff.
I would love to find the equivalent kind of work in English. Recommendations?
I was listening to a few old recordings by Boby Lapointe, a 50s-60s French singer, master of clever alliterations and plays on words. For some reason the lyrics of "Ta Katie t'a quitte" reminded me that I had read somewhere that he had introduced a bi-bi-binary system that would make manipulating large numbers a pleasure.
His system was basicaly an hexadecimal system (base 16 = base 2 ^ 2 ^ 2 ... no wonder Lapointe would focus on base 16 given his love for alliterations...), for which he defined a complete symbol set and corresponding pronunciation.
With this system, something like "119" + "137" = "256" would become "bibi" + "koka" = "hahoho" (thanks Google).
Much better...
Following hyperlinks reading that stuff, I stumbled upon references to other language games. In particular the famous (in the French speaking world) book written by Perec (La Disparition) that not only fully avoids using the letter "e" in a 300+ pages work, but also is largely self referential, discussing as part of its plot the disappearance of said letter [http://www.amazon.fr/Disparition-Georges-Perec/dp/207071523X/ref=sr_1_3?ie=UTF8&s=books&qid=1227261348&sr=8-3].
This is a brilliant technical achievement: in French, around 85% of the words that you would ever write will have the letter "e" in it (with or without accentuation).
Furthermore, the result is a book you would read: an interesting plot, constructed characters, etc. I read it many years ago, after a teacher gave me a copy of the book after challenging me to write a formal essay with the same challenge (I could not go beyond three pages in one hour, and barely articulated a very limited argumentation).
Georges Perec loved these formal challenges, very similar in nature to what Boby Lapointe did in songs.
The same teacher mentioned earlier also introduced me to the French poetry from the Baroque period - a lot of the same alliterations and plays on words. I still have in my brain somewhere the first verses of Pierre de Marbeuf's "La mer et l'amour ont l'amer pour partage".
In any case, I love this stuff.
I would love to find the equivalent kind of work in English. Recommendations?
Thursday, November 20, 2008
More on CEP
Obviously, the CEP / BRMS / BPM / boundaries debate remains hot and continues provoking soul searching.
My previous post on the this was a long (and complicated in the way it was phrased) ranting around the fact that CEP attempts to cover too many areas of the EDM picture, and at the same time does not do a deep enough job at solving the challenges around its core contribution - events as first class citizens in EDM apps. I do think the points I tried to make there remain valid.
I did also contribute a post in another blog as a reaction to a thread on the same subject.
Here is the essence of what I wrote there:
In this debate, we are essentially dealing with three core notions: events, decisions, processes. There is of course much more than that, but I think the confusion we are seeing results from lack of solid boundaries between responsibilities around these three notions. Getting to basics helps.
- Events are all about detecting what decision needs to be taken in a complex temporal context (essentially event correlation, converting monitored ‘ambient’ events into relevant business events that require reaction)
- Decisions (and I will use that term instead of rules) are all about deciding what needs to be done as a reaction to those business events
- Processes are all about executing the decision taken
In a very simplistic analogy,
- Dealing with events in an enterprise application is akin to the sensory system absorbing the information from all the sensors and sources it is connected to, and constructing a view for further decisioning and action, with or without immediate reaction, and communicating it through the nervous system.
- Dealing with decisions in an enterprise application is akin to the multi-step highly collaborative decision making the brain engages in, resulting in events and situational analysis, inclusion of further data, inferences, deductions, etc. and leading to conclusions on further reactions or proactive actions.
- Dealing with processes in an enterprise application is akin to the body executing the plan elaborated by the brain including the input from the nervous system.
CEP should address primarily the first, BRMS and other decision management technologies the second, business process the latter.
What I just outlined is centered operational aspects - as in the operational / improvement aspects distinction presented by Carole-Ann in her talks about the EDM vision at ORF and BRF [http://www.edmblog.com].
The improvement aspects cover how to leverage the information gathered by the operational system - as well as expertise - in order to improve the decisions. It starts by gathering the information and understanding it. All aspects above are to be covered: Am I capturing the right events? What if I decided to capture something slightly different: does that make my business events more relevant? Why did I capture this event at that point in time? What other events are captured within a given contextual distance from this one? Ditto for the decisions (the rules). Ditto for the execution (the processes).Why not just call these things "Event Analytics", "Decision Analytics", "Process Analytics"?Yes, they are connected. But different.
This is simplistic of course, but it has the virtue of being clear. Maybe we can try to reframe the discussions on these issues in that context?
My previous post on the this was a long (and complicated in the way it was phrased) ranting around the fact that CEP attempts to cover too many areas of the EDM picture, and at the same time does not do a deep enough job at solving the challenges around its core contribution - events as first class citizens in EDM apps. I do think the points I tried to make there remain valid.
I did also contribute a post in another blog as a reaction to a thread on the same subject.
Here is the essence of what I wrote there:
In this debate, we are essentially dealing with three core notions: events, decisions, processes. There is of course much more than that, but I think the confusion we are seeing results from lack of solid boundaries between responsibilities around these three notions. Getting to basics helps.
- Events are all about detecting what decision needs to be taken in a complex temporal context (essentially event correlation, converting monitored ‘ambient’ events into relevant business events that require reaction)
- Decisions (and I will use that term instead of rules) are all about deciding what needs to be done as a reaction to those business events
- Processes are all about executing the decision taken
In a very simplistic analogy,
- Dealing with events in an enterprise application is akin to the sensory system absorbing the information from all the sensors and sources it is connected to, and constructing a view for further decisioning and action, with or without immediate reaction, and communicating it through the nervous system.
- Dealing with decisions in an enterprise application is akin to the multi-step highly collaborative decision making the brain engages in, resulting in events and situational analysis, inclusion of further data, inferences, deductions, etc. and leading to conclusions on further reactions or proactive actions.
- Dealing with processes in an enterprise application is akin to the body executing the plan elaborated by the brain including the input from the nervous system.
CEP should address primarily the first, BRMS and other decision management technologies the second, business process the latter.
What I just outlined is centered operational aspects - as in the operational / improvement aspects distinction presented by Carole-Ann in her talks about the EDM vision at ORF and BRF [http://www.edmblog.com].
The improvement aspects cover how to leverage the information gathered by the operational system - as well as expertise - in order to improve the decisions. It starts by gathering the information and understanding it. All aspects above are to be covered: Am I capturing the right events? What if I decided to capture something slightly different: does that make my business events more relevant? Why did I capture this event at that point in time? What other events are captured within a given contextual distance from this one? Ditto for the decisions (the rules). Ditto for the execution (the processes).Why not just call these things "Event Analytics", "Decision Analytics", "Process Analytics"?Yes, they are connected. But different.
This is simplistic of course, but it has the virtue of being clear. Maybe we can try to reframe the discussions on these issues in that context?
Tuesday, November 11, 2008
State, events, time - a view on the confusion around CEP
CEP
Over the recent few months, the EDM world I work in has seen a lot of noise generated by the arrival of CEP - "Complex Event Processing" - and the impact it has had in terms of provoking soul searching in the BRMS and, to a lesser extent the EDA, ESP and other E(x)(x) worlds.
The "E" in these E(x)(x) is "events" which are of course at the core of CEP.
But so is "complex" and "processing", both of which lead more to the area of EDM or BRMS.
This is one of these situations in which a technology addressing at its core a valid set of concerns gets dropped in the middle of a complex soup of acronyms, and confusion ensues.
Of course, the CEP specialists should bear with me for the duration of the blog. I am aware that CEP has been around for a long time, etc., but it's also clear that it is undergoing a renewal through its adoption by the big platform vendors (IBM and Oracle acquisitions) and the innovative ones (Tibco, JBoss).
Why am I writing this blog? Essentially because I believe that the confusion is self-inflicted because as an industry (enterprise applications) we have not been careful to focus the usage of the terminology to its key area - events - and we've spent too much energy trying to justify the complex and processing aspect.
While some of the points below touch on semantic confusion (big words), I will try to remain pragmatic. Let's take "event", "complex processing" in turn.
"Events"
It is true that "events" and their semantics are quite important in systems that need to make decisions.
This is not new, it has always been the case. So why are we - the enterprise software world - only recently starting to give to "events" the preeminence they already have in other worlds - such as the real-time systems world I started in?
Well, simply because the enterprise world has a way to cope with a lot of the value of events by translating events into state.
Traditionally, the occurrence of events, and in sophisticated systems, even their sequencing and timing, are encoded in the state of the system, transforming the "event" management problem into the dual "state" management problem.
Take an example: a fraud detection system is of course keenly interested in knowing what happened where when in which order. It's not a surprise that most CEP vendors use fraud detection as a key example. Well, guess what? Fair Isaac's Falcon Fraud Manager, by far the most used credit card transaction fraud detection system, as well as a host of other fraud detection systems, do not use CEP as defined by the current vendors. They translate events into profile (read "state") information, and they enrich at run time the profiles using sophisticated business rules (not even production systems). This profile encodes variable values that are highly targeted precisely to capture the essence of the business semantics of the events: "number of purchases of gas using the same credit card charging less than a certain amount within a window of time and a geographic region".
You could say that what they have is a sophisticated hugely scalable (90%+ of all credit card transactions in the US go through Falcon) stateful decision management system, supported by a powerful "cache" of variables computed through complex business rules.
And there are many cases like that.
The reality is that the "enterprise" world has been dealing with events for ever. They have simply not needed to resort to any E(x)(x) notion / stack / etc.
That being said, I believe there is an important piece in "event" centric expression of logic - and that is the separation of the event processing logic from the rest, and the resulting clarity, elegance, maintainability and ultimately all those scalability, ability to audit, robustness, ..., qualities that result from clean concepts, clean designs, clean architectures.
Which brings me to the following first opinion on CEP:
(1) CEP should stop worrying about the origin of the events and focus on the events themselves. It does not matter how the events originate, and the issues of ESB, EDA, ESP, in-database generation, etc..., are all orthogonal / independent on how events-dependant logic is managed.
And to the second opinion on CEP:
(2) CEP should stop worrying about caching. Yes, caching is important but irrelevant to the power of the approach - as the fact that among the largest and most scalable event-driven enterprise apps, many handle the issue with no need to couple the event management piece from the management of caches. Right now, there are efforts to extend the Rete structures and to adapt the algorithm to build this cache in a more efficient way for the corresponding type of rules processing. Great usage of the technology, but that will not make the potential power of event centric approaches any more compelling.
Maybe it's time to be a little more constructive.
(3) CEP would do wonders in enterprise systems if it focused all its attention to the "event" part: the semantics of events, the richness of the event context (time, location, referential, ...), the clean semantics of the key contextual notions (time operations including referential management, etc...), etc.
The events semantics question is not innocent, and is absolutely not a settled question - witness the numerous exchanges between clever people than me on this subject.
I had a discussion with Paul Haley once on this subject and we went into the "what is an event" question that has the virtue of quickly getting people upset. It's a valid question: the definition of "event = observed state transition" has the bad taste of defining events in terms of state, but its key issue in my eyes is that it supposes observation of a state and lacks content.
The value of events is that they have intrinsic context that are not naturally contained in state systems: they occur at a point in time with respect to a given referential - or more generally, they occur at a contextual point (could be time+location, etc.) with respect to a given referential. Different events used within the same system may have their intrinsic context expressed with respect to different referentials - and that will be the default case in any distributed system.
Events occur, they are atomic, immutable. Their only existence is their occurrence. We may track the fact that they occurred, but an event instance only happens once and is instantaneous. An event does not last a duration. That is not logical to me - its effects or the state change it triggers may last a duration, but the event in itself is instantaneous.
Which leads to the fact that there are natural event correlations you want to express (not just discover): an event creates a state transition, a correlated event will create another state transition that will bring the state back to the original one.
This is just my opinion - but if you talk to more than one real specialist, you will get more than one view. Not a good sign of maturity of the concepts.
Clean ontologies / semantics / etc. needed.
With the clarification of the semantics of events and their intrinsic referential-dependent context are clarified, we need to focus on what we want to express about events - and for that, we need to bring on the enterprise application experts.
There is a lot to learn from the real-time systems experts - refer to the very old but incredibly good insights from Parnas' work. There is a lot to get back from the original event correlation systems - many built with systems such as Ilog's rules engine. These could even be said to be the purest predecessors to what CEP attempts to do.
What this will end up doing is giving us - the decision management world - a very powerful tool to "naturally" express logic on events, with their referential-dependant context, and to do so in a way that enables true management (things like verification of the logic included), powerful optimizations, etc.
I honestly do not think we are there, and I would really like to see the standardization world - the OMGs and others - help us get there; but I do think we need the enterprise business app drivers. We had that, in real-time systems: the military and transportation apps.
I would love the specialists to prove me wrong and to show me we are there.
"Complex processing"
This will be shorter.
As stated above, I believe that:
- CEP should narrow its processing ambitions. One approach is to focus the purpose of its processing to clear outcomes - such as what the correlation engines did 20 years ago. For example, we could say that CEP's processing is about processing / correlating events to generate higher order events: transaction events to generate a potentially fraudulent transaction event. I will call these "ambient events" and "business events": the CEP processing goal is to translate ambient events into business events
- CEP should focus the complexity of its processing to the corresponding revised ambitions.
- CEP should leave all issues related to event streaming, event transport, event communication, etc. to other layers.
I may be a purist, but I see a simple picture:
- leave anything related to transport, communication to other layers
- use this revised CEP to express and execute event-relevant logic, the purpose of which is to translate the ambient events into relevant business events
- have these business events trigger business processes (however lightweight you want to make them)
- have these business processes invoke decision services implemented through decision management to decide what they should be doing at every step
- have the business processes invoke action services to execute the actions decided by the decision services
- all the while generating business events or ambient events
- etc.
As such, CEP will include a semantically sound event-with-intrinsic-referential-dependent-context model, a corresponding language (EPL or vocabulary) to express logic, algorithms to efficiently execute (wide open field - tons of people doing analytics, Bayesian, rules, ...), techniques to verify (wide open - and fairly empty: I only know of the real-time folks), etc.
And there, the value of CEP is clear. Of course, it is lower than what CEP vendors would like, but significant anyway.
I am hoping this is controversial enough to bring on flames...
Over the recent few months, the EDM world I work in has seen a lot of noise generated by the arrival of CEP - "Complex Event Processing" - and the impact it has had in terms of provoking soul searching in the BRMS and, to a lesser extent the EDA, ESP and other E(x)(x) worlds.
The "E" in these E(x)(x) is "events" which are of course at the core of CEP.
But so is "complex" and "processing", both of which lead more to the area of EDM or BRMS.
This is one of these situations in which a technology addressing at its core a valid set of concerns gets dropped in the middle of a complex soup of acronyms, and confusion ensues.
Of course, the CEP specialists should bear with me for the duration of the blog. I am aware that CEP has been around for a long time, etc., but it's also clear that it is undergoing a renewal through its adoption by the big platform vendors (IBM and Oracle acquisitions) and the innovative ones (Tibco, JBoss).
Why am I writing this blog? Essentially because I believe that the confusion is self-inflicted because as an industry (enterprise applications) we have not been careful to focus the usage of the terminology to its key area - events - and we've spent too much energy trying to justify the complex and processing aspect.
While some of the points below touch on semantic confusion (big words), I will try to remain pragmatic. Let's take "event", "complex processing" in turn.
"Events"
It is true that "events" and their semantics are quite important in systems that need to make decisions.
This is not new, it has always been the case. So why are we - the enterprise software world - only recently starting to give to "events" the preeminence they already have in other worlds - such as the real-time systems world I started in?
Well, simply because the enterprise world has a way to cope with a lot of the value of events by translating events into state.
Traditionally, the occurrence of events, and in sophisticated systems, even their sequencing and timing, are encoded in the state of the system, transforming the "event" management problem into the dual "state" management problem.
Take an example: a fraud detection system is of course keenly interested in knowing what happened where when in which order. It's not a surprise that most CEP vendors use fraud detection as a key example. Well, guess what? Fair Isaac's Falcon Fraud Manager, by far the most used credit card transaction fraud detection system, as well as a host of other fraud detection systems, do not use CEP as defined by the current vendors. They translate events into profile (read "state") information, and they enrich at run time the profiles using sophisticated business rules (not even production systems). This profile encodes variable values that are highly targeted precisely to capture the essence of the business semantics of the events: "number of purchases of gas using the same credit card charging less than a certain amount within a window of time and a geographic region".
You could say that what they have is a sophisticated hugely scalable (90%+ of all credit card transactions in the US go through Falcon) stateful decision management system, supported by a powerful "cache" of variables computed through complex business rules.
And there are many cases like that.
The reality is that the "enterprise" world has been dealing with events for ever. They have simply not needed to resort to any E(x)(x) notion / stack / etc.
That being said, I believe there is an important piece in "event" centric expression of logic - and that is the separation of the event processing logic from the rest, and the resulting clarity, elegance, maintainability and ultimately all those scalability, ability to audit, robustness, ..., qualities that result from clean concepts, clean designs, clean architectures.
Which brings me to the following first opinion on CEP:
(1) CEP should stop worrying about the origin of the events and focus on the events themselves. It does not matter how the events originate, and the issues of ESB, EDA, ESP, in-database generation, etc..., are all orthogonal / independent on how events-dependant logic is managed.
And to the second opinion on CEP:
(2) CEP should stop worrying about caching. Yes, caching is important but irrelevant to the power of the approach - as the fact that among the largest and most scalable event-driven enterprise apps, many handle the issue with no need to couple the event management piece from the management of caches. Right now, there are efforts to extend the Rete structures and to adapt the algorithm to build this cache in a more efficient way for the corresponding type of rules processing. Great usage of the technology, but that will not make the potential power of event centric approaches any more compelling.
Maybe it's time to be a little more constructive.
(3) CEP would do wonders in enterprise systems if it focused all its attention to the "event" part: the semantics of events, the richness of the event context (time, location, referential, ...), the clean semantics of the key contextual notions (time operations including referential management, etc...), etc.
The events semantics question is not innocent, and is absolutely not a settled question - witness the numerous exchanges between clever people than me on this subject.
I had a discussion with Paul Haley once on this subject and we went into the "what is an event" question that has the virtue of quickly getting people upset. It's a valid question: the definition of "event = observed state transition" has the bad taste of defining events in terms of state, but its key issue in my eyes is that it supposes observation of a state and lacks content.
The value of events is that they have intrinsic context that are not naturally contained in state systems: they occur at a point in time with respect to a given referential - or more generally, they occur at a contextual point (could be time+location, etc.) with respect to a given referential. Different events used within the same system may have their intrinsic context expressed with respect to different referentials - and that will be the default case in any distributed system.
Events occur, they are atomic, immutable. Their only existence is their occurrence. We may track the fact that they occurred, but an event instance only happens once and is instantaneous. An event does not last a duration. That is not logical to me - its effects or the state change it triggers may last a duration, but the event in itself is instantaneous.
Which leads to the fact that there are natural event correlations you want to express (not just discover): an event creates a state transition, a correlated event will create another state transition that will bring the state back to the original one.
This is just my opinion - but if you talk to more than one real specialist, you will get more than one view. Not a good sign of maturity of the concepts.
Clean ontologies / semantics / etc. needed.
With the clarification of the semantics of events and their intrinsic referential-dependent context are clarified, we need to focus on what we want to express about events - and for that, we need to bring on the enterprise application experts.
There is a lot to learn from the real-time systems experts - refer to the very old but incredibly good insights from Parnas' work. There is a lot to get back from the original event correlation systems - many built with systems such as Ilog's rules engine. These could even be said to be the purest predecessors to what CEP attempts to do.
What this will end up doing is giving us - the decision management world - a very powerful tool to "naturally" express logic on events, with their referential-dependant context, and to do so in a way that enables true management (things like verification of the logic included), powerful optimizations, etc.
I honestly do not think we are there, and I would really like to see the standardization world - the OMGs and others - help us get there; but I do think we need the enterprise business app drivers. We had that, in real-time systems: the military and transportation apps.
I would love the specialists to prove me wrong and to show me we are there.
"Complex processing"
This will be shorter.
As stated above, I believe that:
- CEP should narrow its processing ambitions. One approach is to focus the purpose of its processing to clear outcomes - such as what the correlation engines did 20 years ago. For example, we could say that CEP's processing is about processing / correlating events to generate higher order events: transaction events to generate a potentially fraudulent transaction event. I will call these "ambient events" and "business events": the CEP processing goal is to translate ambient events into business events
- CEP should focus the complexity of its processing to the corresponding revised ambitions.
- CEP should leave all issues related to event streaming, event transport, event communication, etc. to other layers.
I may be a purist, but I see a simple picture:
- leave anything related to transport, communication to other layers
- use this revised CEP to express and execute event-relevant logic, the purpose of which is to translate the ambient events into relevant business events
- have these business events trigger business processes (however lightweight you want to make them)
- have these business processes invoke decision services implemented through decision management to decide what they should be doing at every step
- have the business processes invoke action services to execute the actions decided by the decision services
- all the while generating business events or ambient events
- etc.
As such, CEP will include a semantically sound event-with-intrinsic-referential-dependent-context model, a corresponding language (EPL or vocabulary) to express logic, algorithms to efficiently execute (wide open field - tons of people doing analytics, Bayesian, rules, ...), techniques to verify (wide open - and fairly empty: I only know of the real-time folks), etc.
And there, the value of CEP is clear. Of course, it is lower than what CEP vendors would like, but significant anyway.
I am hoping this is controversial enough to bring on flames...
Thursday, November 6, 2008
Enterprise Decision Management? How about State Decision Management?
This is interesting: http://www.nytimes.com/2008/03/28/world/americas/28cybersyn.html?_r=1&oref=slogin&ref=americas&pagewanted=print and http://www.guardian.co.uk/technology/2003/sep/08/sciencenews.chile
Very interesting for me since I work in decision management, and Chile is where I come from. I had seen mentions of this in the past (hence my sudden recollection of it during this sleepless night) in Andreseen's blog. Wikipedia has a summary (http://en.wikipedia.org/wiki/Project_Cybersyn) that mentions usage of Bayesian filtering although does not insist on the learning and adaptive aspects.
It also refers to this: http://www.williambowles.info/sa/FanfareforEffectiveFreedom.pdf - which I really do not know what to make of. It brings me back to my early days in Engineering school when I was studying what we used to call "Automatique" and was all about adaptive control. But applied to social and economic matters at the scale of a country!?
On the other hand, this lecture on cybernetics by Beer is intriguing: http://lispmeister.com/downloads/Stafford-Beer-forty-years-of-cybernetics-lecture.wma (pretty big)
So it turns out that Chile does not just do wine and pisco (sorry Peru).
Very interesting for me since I work in decision management, and Chile is where I come from. I had seen mentions of this in the past (hence my sudden recollection of it during this sleepless night) in Andreseen's blog. Wikipedia has a summary (http://en.wikipedia.org/wiki/Project_Cybersyn) that mentions usage of Bayesian filtering although does not insist on the learning and adaptive aspects.
It also refers to this: http://www.williambowles.info/sa/FanfareforEffectiveFreedom.pdf - which I really do not know what to make of. It brings me back to my early days in Engineering school when I was studying what we used to call "Automatique" and was all about adaptive control. But applied to social and economic matters at the scale of a country!?
On the other hand, this lecture on cybernetics by Beer is intriguing: http://lispmeister.com/downloads/Stafford-Beer-forty-years-of-cybernetics-lecture.wma (pretty big)
So it turns out that Chile does not just do wine and pisco (sorry Peru).
Wednesday, November 5, 2008
The new programming languages discussions
I am spending time getting a hold of the new approaches on programming languages.
Many years ago, I discovered Bentley's Programming Pearls (http://www.amazon.com/Programming-Pearls-2nd-ACM-Press/dp/0201657880/ref=sr_11_1/176-0790589-8960217?ie=UTF8&qid=1225956297&sr=11-1). I have true love for well defined "little languages" that have a clear purpose and deliver it with elegance, concision.
I was lucky to work around a language for a business rules engine, and I remember with fondness the long discussions around language design and pretty important things that are now obvious but back then, in my youth and the earlier days of OO, seemed subject for debate: value versus object, object identity versus object state, etc... How the Java JDK got it wrong in a number of classes in java.util.
And the sempiternal debate around strong vs other typing systems.
Since then I discovered (through Paul Ford's excellent ftrain.com site) Processing (http://ftrain.com/ProcessingProcessing.html), a very nice language. And then many others.
However my work diverted my energy into less fun stuff, repositories, server side infrastructure, etc.. And management (see other posts...)
But now I am back to more fundamental thinking about all this, and it's now starting to make sense again. DSLs (graphical, syntactic) constructed using tooling provided by dynamic languages leveraging DLR support resting on language neutral VMs for which the low level core code is written in strongly-typed Java like languages. "Mais bon sang, mais c'est bien sur".
The low level core code needs scalability, performance, compile time checking for mission critical code. Java gives you that - and yes, Java does give you that, hotspot compilation is doing wonders this day.
The dynamic, sometimes reflective, languages in the middle (and I will call dynamic even those that do 'duck typing' - 'if it looks like a duck, it sounds like a duck, it smells like a duck, then it's most likely a duck') give you productivity - and Python and Ruby have proven they can deliver and be safe. The 'Da Vinci' VM, even more so than the DLR effort on .NET, promise to combine dynamic language interoperability (making it easier to free DSLs from the tyranny of the underlying supporting programming language) and performance (enabling hotspot compilation) (read John Rose's very instructive blog: http://blogs.sun.com/jrose/).
These dynamic and reflective languages enable the easy creation of maintainable DSLs.
This is powerful.
Makes me rethink a lot of things.
Then there is the question of 'model driven' versus 'dsl driven'. For me, at a high level, DSLs are there to express the models; and, in the other direction, models define the boundaries for the DSLs. The key is that DSLs should not express more than the corresponding models (lest they become just other horizontal programming languages), but they should express them fully, with all the required flexibility, with elegance and in a concise fashion.
Back to the beginning of this post and Bentley's pearls.
But this is a big exciting subject. It will keep me awake at night. Complicated, but in a good way ;)
Many years ago, I discovered Bentley's Programming Pearls (http://www.amazon.com/Programming-Pearls-2nd-ACM-Press/dp/0201657880/ref=sr_11_1/176-0790589-8960217?ie=UTF8&qid=1225956297&sr=11-1). I have true love for well defined "little languages" that have a clear purpose and deliver it with elegance, concision.
I was lucky to work around a language for a business rules engine, and I remember with fondness the long discussions around language design and pretty important things that are now obvious but back then, in my youth and the earlier days of OO, seemed subject for debate: value versus object, object identity versus object state, etc... How the Java JDK got it wrong in a number of classes in java.util.
And the sempiternal debate around strong vs other typing systems.
Since then I discovered (through Paul Ford's excellent ftrain.com site) Processing (http://ftrain.com/ProcessingProcessing.html), a very nice language. And then many others.
However my work diverted my energy into less fun stuff, repositories, server side infrastructure, etc.. And management (see other posts...)
But now I am back to more fundamental thinking about all this, and it's now starting to make sense again. DSLs (graphical, syntactic) constructed using tooling provided by dynamic languages leveraging DLR support resting on language neutral VMs for which the low level core code is written in strongly-typed Java like languages. "Mais bon sang, mais c'est bien sur".
The low level core code needs scalability, performance, compile time checking for mission critical code. Java gives you that - and yes, Java does give you that, hotspot compilation is doing wonders this day.
The dynamic, sometimes reflective, languages in the middle (and I will call dynamic even those that do 'duck typing' - 'if it looks like a duck, it sounds like a duck, it smells like a duck, then it's most likely a duck') give you productivity - and Python and Ruby have proven they can deliver and be safe. The 'Da Vinci' VM, even more so than the DLR effort on .NET, promise to combine dynamic language interoperability (making it easier to free DSLs from the tyranny of the underlying supporting programming language) and performance (enabling hotspot compilation) (read John Rose's very instructive blog: http://blogs.sun.com/jrose/).
These dynamic and reflective languages enable the easy creation of maintainable DSLs.
This is powerful.
Makes me rethink a lot of things.
Then there is the question of 'model driven' versus 'dsl driven'. For me, at a high level, DSLs are there to express the models; and, in the other direction, models define the boundaries for the DSLs. The key is that DSLs should not express more than the corresponding models (lest they become just other horizontal programming languages), but they should express them fully, with all the required flexibility, with elegance and in a concise fashion.
Back to the beginning of this post and Bentley's pearls.
But this is a big exciting subject. It will keep me awake at night. Complicated, but in a good way ;)
Tuesday, November 4, 2008
Back...
After a long absence from the blog, I am back to it.
I have spent a couple of months heads down focusing on management issues, most of them consequences of the way my current employer has chosen to face the current crisis. Typical 'protect the EPS' reaction.
I posted earlier on the counter-cyclists, and the reasons why it makes a lot of sense to manage good times better so that we can manage bad times in a strategic way and benefit from the exit from downturns rather than play catch up. It looks simple on paper - I am certain it's more difficult than that - but I have to say my growing experience with upper management does not give me the feeling this requires anything else than thinking strategically rather than tactical. Where this fails is when upper management is mercenary upper management: on board for 2 to 5 years and not interested in long term growth.
This is important for us software architects - not just because of the obvious impact on our continued employment, but also because architecture is strategic, and gets sacrificed at the altar of short term tactical moves. This is obvious to say, again, and obvious to understand, but it still amazes me how selective corporate perception is, and how little care is given to preserving corporate memory.
I will for a while - hopefully - stop blogging about this kind of thing. And go back to more technical stuff - however difficult it is to do in abstract of the overall situation...
I have spent a couple of months heads down focusing on management issues, most of them consequences of the way my current employer has chosen to face the current crisis. Typical 'protect the EPS' reaction.
I posted earlier on the counter-cyclists, and the reasons why it makes a lot of sense to manage good times better so that we can manage bad times in a strategic way and benefit from the exit from downturns rather than play catch up. It looks simple on paper - I am certain it's more difficult than that - but I have to say my growing experience with upper management does not give me the feeling this requires anything else than thinking strategically rather than tactical. Where this fails is when upper management is mercenary upper management: on board for 2 to 5 years and not interested in long term growth.
This is important for us software architects - not just because of the obvious impact on our continued employment, but also because architecture is strategic, and gets sacrificed at the altar of short term tactical moves. This is obvious to say, again, and obvious to understand, but it still amazes me how selective corporate perception is, and how little care is given to preserving corporate memory.
I will for a while - hopefully - stop blogging about this kind of thing. And go back to more technical stuff - however difficult it is to do in abstract of the overall situation...
Thursday, August 28, 2008
Lem
I had never been a SciFi person in the past - but over the past two-three years a person who is very close to me have me discover it, and I have since been reading some of the best work there with my eldest son.
I just finished Solaris and Fiasco by S. Lem (http://en.wikipedia.org/wiki/Stanislaw_Lem). These are not as much SciFi as they are philosophical essays. Impressive work, and very good reading, highly recommended.
I am not surprised at all Lem did not see eye to eye with his peers (http://www.lem.pl/cyberiadinfo/english/faq/faq.htm#sfwa)...
Of course, it is ironic that the person who initiated me to this hates Solaris... and possibly would hate Fiasco. But that goes to prove that you can embrace and extend ;)!
I just finished Solaris and Fiasco by S. Lem (http://en.wikipedia.org/wiki/Stanislaw_Lem). These are not as much SciFi as they are philosophical essays. Impressive work, and very good reading, highly recommended.
I am not surprised at all Lem did not see eye to eye with his peers (http://www.lem.pl/cyberiadinfo/english/faq/faq.htm#sfwa)...
Of course, it is ironic that the person who initiated me to this hates Solaris... and possibly would hate Fiasco. But that goes to prove that you can embrace and extend ;)!
Friday, August 8, 2008
Context, context...
As I was writing the previous post, it struck me that I was wondering about precisely what Borges covers in "Pierre Menard, author of the Quixote".
One message Borges delivers: the meaning of the literary text is more a function of the context (historical, cultural, etc) than it is of the text itself. Which means each reader creates the meaning of what he/she reads starting from what the author wrote - each one of us, in a personal way.
Another message he delivers: the meaning of the literary text is more a function of the process involved in creating it than in the result.
What I find more attractive than the messages (I do not want to want into discussions that involve the Barthes and the Baudrillards of the world... too far from what I can grasp) is the way Borges uses his immense wit to convey them in a tiny fraction of what these so-called scientists do, while provoking in the reader the pleasure that comes with stumbling on unexpected beautiful gems.
Enjoy (text in spanish):
"
Es una revelación cotejar el Don Quijote de Menard con el de Cervantes. Éste, por ejemplo, escribió (Don Quijote, primera parte, noveno capítulo):
También es vívido el contraste de los estilos. El estilo arcaizante de Menard —extranjero al fin— adolece de alguna afectación. No así el del precursor, que maneja con desenfado el español corriente de su época.
"
One message Borges delivers: the meaning of the literary text is more a function of the context (historical, cultural, etc) than it is of the text itself. Which means each reader creates the meaning of what he/she reads starting from what the author wrote - each one of us, in a personal way.
Another message he delivers: the meaning of the literary text is more a function of the process involved in creating it than in the result.
What I find more attractive than the messages (I do not want to want into discussions that involve the Barthes and the Baudrillards of the world... too far from what I can grasp) is the way Borges uses his immense wit to convey them in a tiny fraction of what these so-called scientists do, while provoking in the reader the pleasure that comes with stumbling on unexpected beautiful gems.
Enjoy (text in spanish):
"
Es una revelación cotejar el Don Quijote de Menard con el de Cervantes. Éste, por ejemplo, escribió (Don Quijote, primera parte, noveno capítulo):
- ... la verdad, cuya madre es la historia, émula del tiempo, depósito de las acciones, testigo de lo pasado, ejemplo y aviso de lo presente, advertencia de lo por venir.
- ... la verdad, cuya madre es la historia, émula del tiempo, depósito de las acciones, testigo de lo pasado, ejemplo y aviso de lo presente, advertencia de lo por venir.
También es vívido el contraste de los estilos. El estilo arcaizante de Menard —extranjero al fin— adolece de alguna afectación. No así el del precursor, que maneja con desenfado el español corriente de su época.
"
Thursday, August 7, 2008
A now something totally different...
I don't know what to make of this: [http://orwelldiaries.wordpress.com/].
I have genuine respect and admiration for Orwell. Through his life, involvement and writings, Orwell represents a amalgam of many of those people on the left of the political spectrum who are genuinely touched by injustice, unfairness, have a realistic understanding of the world and are not swayed by rhetoric, and hold their friends to the same standards as their "enemies".
I am part of a diaspora of sorts caused by political turmoil. I know how simple things are on one hand (how can you not fight against regimes that kill and torture?) and how complicated they can be on the other hand (do I have the right to bring my whole family into exile because of my ideals? Is it right for me to risk the welfare of my kids because of my belief system?). But while it's true that the real choices are significantly less obvious than idealists would like, there are some clear universal principles that are worth standing up for.
What I read from Orwell, what I read about him (read Hitchens [http://www.amazon.com/Why-Orwell-Matters-Christopher-Hitchens/dp/0465030491]and [http://www.amazon.com/Unacknowledged-Legislation-Writers-Public-Sphere/dp/1859843832/ref=sr_1_29?ie=UTF8&s=books&qid=1218157821&sr=1-29]), increases my admiration for the writer.
His lucidity and his ability to convey reality clearly dazzles me, and I am not easily impressed. Read his essay on how the language is perverted for political purposes in everyday discourse: [http://www.mtholyoke.edu/acad/intrel/orwell46.htm].
But my respect and admiration are not blind - what about Orwell's delivery of a list of crypto-communists to the Foreign Office towards the end of his life? Well, as I wrote earlier, things are never as simple as they seem. This whole story has been hotly debated, and I am in no way qualified to argue but am ready to believe he did draft such a letter. That list reflects essentially his fierce anti-communism, his realistic choice of one evil versus another (I am not necessarily agreeing, I am stating a fact), and his public statements. While the list did name names, its purposes was to identify cultural agents that could not be trusted with anti-communism propaganda. Still a vile list, and it does not make me like Orwell-the-man better. But participating to those activities is the kind of decisions many brilliant and otherwise admirable people end up taking - something that I would really like to read more about. Borges is another such case.
Back to the blog that will open. According to what I understand, it's an interesting exercise: an attempt to blog through a few years using material from Orwell's journal in chronological order, starting 8/9/1939. This means the blog will go on from now to 2012.
I am glad we will get to know more about the writings - the raw writings. I understand some of what will be published is not-yet-published material.
And I wonder whether the new format, the blog, will somehow alter the message - make it resonate more, or less. I am certain the observations Orwell makes remain relevant - but will the blog medium used to deliver them change how they are received?
In any case, a very interesting project. Check it out!
I have genuine respect and admiration for Orwell. Through his life, involvement and writings, Orwell represents a amalgam of many of those people on the left of the political spectrum who are genuinely touched by injustice, unfairness, have a realistic understanding of the world and are not swayed by rhetoric, and hold their friends to the same standards as their "enemies".
I am part of a diaspora of sorts caused by political turmoil. I know how simple things are on one hand (how can you not fight against regimes that kill and torture?) and how complicated they can be on the other hand (do I have the right to bring my whole family into exile because of my ideals? Is it right for me to risk the welfare of my kids because of my belief system?). But while it's true that the real choices are significantly less obvious than idealists would like, there are some clear universal principles that are worth standing up for.
What I read from Orwell, what I read about him (read Hitchens [http://www.amazon.com/Why-Orwell-Matters-Christopher-Hitchens/dp/0465030491]and [http://www.amazon.com/Unacknowledged-Legislation-Writers-Public-Sphere/dp/1859843832/ref=sr_1_29?ie=UTF8&s=books&qid=1218157821&sr=1-29]), increases my admiration for the writer.
His lucidity and his ability to convey reality clearly dazzles me, and I am not easily impressed. Read his essay on how the language is perverted for political purposes in everyday discourse: [http://www.mtholyoke.edu/acad/intrel/orwell46.htm].
But my respect and admiration are not blind - what about Orwell's delivery of a list of crypto-communists to the Foreign Office towards the end of his life? Well, as I wrote earlier, things are never as simple as they seem. This whole story has been hotly debated, and I am in no way qualified to argue but am ready to believe he did draft such a letter. That list reflects essentially his fierce anti-communism, his realistic choice of one evil versus another (I am not necessarily agreeing, I am stating a fact), and his public statements. While the list did name names, its purposes was to identify cultural agents that could not be trusted with anti-communism propaganda. Still a vile list, and it does not make me like Orwell-the-man better. But participating to those activities is the kind of decisions many brilliant and otherwise admirable people end up taking - something that I would really like to read more about. Borges is another such case.
Back to the blog that will open. According to what I understand, it's an interesting exercise: an attempt to blog through a few years using material from Orwell's journal in chronological order, starting 8/9/1939. This means the blog will go on from now to 2012.
I am glad we will get to know more about the writings - the raw writings. I understand some of what will be published is not-yet-published material.
And I wonder whether the new format, the blog, will somehow alter the message - make it resonate more, or less. I am certain the observations Orwell makes remain relevant - but will the blog medium used to deliver them change how they are received?
In any case, a very interesting project. Check it out!
Wednesday, August 6, 2008
Counter-Cyclists
Sometimes you stumble on an article which for some reason just speaks to you. Just when you thought you had the perfect articulation of what you want to express in your head, you read how somebody has done it so much better...
As mentioned earlier, my work is at an enterprise software vendor. Hit pretty hard by the current conditions. We are going, like many of our public peers, through the phase in which we need to adapt to new temporary market conditions, and keep our key indicators (EPS, ...) in check. And, as many of our public peers, we react by contracting after having expanded in better times. Tightening our belt when the conditions are poor. Both blaming a black swan (the mortgage crisis) and taking credit for having seen it early (because it does not reflect well not seeing what we should have seen, or - although not this time - because our brains do not let us treat pure randomness as such, we must see order - read Taleb's books [see http://www.fooledbyrandomness.com/]).
The well understood risk is reduced ability to deliver, reduced ability to support our customers, reduced ability to deliver growth, etc. The EPS will look ok, but the risk is big that our compromised abilities will render us obsolete when the environment changes.
I have been through these cycles many times. And I have seen what separates those who risk their survival at every crisis from those who turn the threat into opportunity: the latter are counter-cyclists. I did not know how to express it simply and clearly, but that is what they are.
This article from Wharton [see http://www.whartonsp.com/articles/article.asp?p=429379] describes that very nicely.
Counter-cyclists keep control over the expenses and growth through up turns. They do not suffocate innovation and growth - they apply the same caution in up turns as in down turns, being selective, being focused. That may seem like a defensive strategy, but is not.
Because counter-cyclists leverage their control over these expenses and growth to avoid over-reacting in down turns. They can afford to increase their expenses and enable growth generating activities during the down turn, precisely when the competition is weaker, and putting them in a premium position to take advantage of the following recovery.
They can cherry pick the best employees - those that the others "have" to let go to protect EPS. They can cherry pick their investments - those that the others "have" to let go to protect the numbers, to focus on core values, etc...
they can choose the moment they divest, where they divest, etc.
This is really not rocket-science - and is obviously taught in MBA and various management classes, illustrated by a number of examples. We can even find Latin phrases that express the core of this.
As mentioned earlier, my work is at an enterprise software vendor. Hit pretty hard by the current conditions. We are going, like many of our public peers, through the phase in which we need to adapt to new temporary market conditions, and keep our key indicators (EPS, ...) in check. And, as many of our public peers, we react by contracting after having expanded in better times. Tightening our belt when the conditions are poor. Both blaming a black swan (the mortgage crisis) and taking credit for having seen it early (because it does not reflect well not seeing what we should have seen, or - although not this time - because our brains do not let us treat pure randomness as such, we must see order - read Taleb's books [see http://www.fooledbyrandomness.com/]).
The well understood risk is reduced ability to deliver, reduced ability to support our customers, reduced ability to deliver growth, etc. The EPS will look ok, but the risk is big that our compromised abilities will render us obsolete when the environment changes.
I have been through these cycles many times. And I have seen what separates those who risk their survival at every crisis from those who turn the threat into opportunity: the latter are counter-cyclists. I did not know how to express it simply and clearly, but that is what they are.
This article from Wharton [see http://www.whartonsp.com/articles/article.asp?p=429379] describes that very nicely.
Counter-cyclists keep control over the expenses and growth through up turns. They do not suffocate innovation and growth - they apply the same caution in up turns as in down turns, being selective, being focused. That may seem like a defensive strategy, but is not.
Because counter-cyclists leverage their control over these expenses and growth to avoid over-reacting in down turns. They can afford to increase their expenses and enable growth generating activities during the down turn, precisely when the competition is weaker, and putting them in a premium position to take advantage of the following recovery.
They can cherry pick the best employees - those that the others "have" to let go to protect EPS. They can cherry pick their investments - those that the others "have" to let go to protect the numbers, to focus on core values, etc...
they can choose the moment they divest, where they divest, etc.
This is really not rocket-science - and is obviously taught in MBA and various management classes, illustrated by a number of examples. We can even find Latin phrases that express the core of this.
Enterprise software and usability
The long debate...
A lot has been said about this.
Of course - usability is important. It reduces learning curves, it increases effectiveness, it increases sexiness. It increases appeal and reduces cost of ownership.
But reality is more complicated than that, of course.
- Those who buy are not those who use, and the enterprise market builds for those who buy, guided by those who say what should be looked at - analysts and big vendors offering the convenience of the 'no fault' purchase ("nobody has been fired for buying
", " is in the top-right quadrant").
The set of criteria that ends up being applied at the end of the day is functionality-based - even though users frequently do not need more functionality.Is this a good thing? Not from the perspective of the end user, but can the software vendor be blamed for increasing value for its shareholders and employees by submitting to the tyranny of the existing approach? Maybe in the long run - but debatable.
You could as well point out that those who buy have a responsibility to their users, the value they create and ultimately their shareholders. As long as the connection between the benefits to the actual users and the value of these benefits to the shareholders do not trump the other real or perceived priorities of the buyer, I see little chance of this situation changing fundamentally. - Building software is hard. Building enterprise class software is hard, and must take into account constraints and/or requirement classes that are difficult to reconcile nicely. The product managers and developers of enterprise software have to contend with difficult scalability issues, security issues, compliance issues, legacy support issues (both technical and human), long and complex release cycles, lack of acceptance by large customers of bleeding edge solutions, etc...
None of these precludes building usable software - it just makes it harder, and when you combine this point with the previous one, it's clear that usability will frequently be partially sacrificed. - Usability is relative to users, essentially cultural entities. Highly dependant on context. I have the utmost respect for well designed usability - but I represent one cultural perspective, and I know that many things that make me efficient are not efficient for my colleagues.
This is of course why we end up with the seemingly impossible to exhaust list of configuration options in so many systems that have to cater to more than one function and more than one user type.
Is this unavoidable? Tough to say, again. Certainly more effort could go into avoiding resorting to the "let's give them options" syndrome, but we do not live in a theoretical world - moving the responsibility to end-users themselves (or their proxies in the IT organizations) may very well represent the rational compromise between cost and benefit from the perspective of both seller and buyers.
Fighting this is hard. But some do... You'll notice, however, that most of the "oh how usable this is" comments tend to be around software users *want* to use rather than those, even painstakingly well designed - at least from some aspects, that users *must* use.
A lot more can be said about all these subjects and much more articulate people than me have weighed in. But the fundamental reasons are above.
Is usability a lost cause in enterprise software? I don't think so. I think we are on the verge of major changes.
- The enterprise 2.0 revolution (sorry - buzzword, let's assume there is one and it fits what the pundits claim it is) will move the center of gravity to the users by reducing the role of intermediaries.That will change the distribution of power described above, and make the actual users both more relevant and more responsible in the purchasing and renewal decisions.
- As interoperability increases, out-of-the-box integration will cease being a major issue and stickiness will become one.Reducing the switching cost will empower the end user with the same effect.
Of course, I am not naive enough to think that will happen fast and software vendors will make it easy on their own. It will happen under the pressure of the disruptors. - As software-as-a-service turns to reality in more parts of the enterprise, the reduction in perceived, felt cost by buyers and users will result in more emphasis in the other areas neglected so far.
- As marketing goes "2.0" and customer acquisition becomes more viral, etc...
So I do think it will happen. But it will be tied to the empowering of the software user that will result from the "2.0" disruptive technologies and approaches. Don't expect it from those that are not threatened.
Ok, enough buzzwords for one night.
Tuesday, August 5, 2008
First post
Well, this is it. First steps in the blogosphere, which probably makes me a late-comer these days. Not the last thing I will be a late-comer to...
This blog is intended to cover topics which attract my attention, for better or for worse, and which I think will be interesting to others. It will revolve mostly around technology and product issues - but it will sometimes bleed into more personal, and potentially slippery subjects.
As you have probably guessed, I work in the software business. The enterprise software business. Not a pretty space right now - an odd mix of high complexity, low prices, market turmoil, and slim business and professional growth.
A lot of what I will debate here will revolve around the issues that we wrestle with every day. Not just the technical issues, but also the human issues.
Let's start...
P.S. My blog-editing software complains about "blogosphere" not been spelled properly... The irony - language always gives you away, as G Carlin used to say.
This blog is intended to cover topics which attract my attention, for better or for worse, and which I think will be interesting to others. It will revolve mostly around technology and product issues - but it will sometimes bleed into more personal, and potentially slippery subjects.
As you have probably guessed, I work in the software business. The enterprise software business. Not a pretty space right now - an odd mix of high complexity, low prices, market turmoil, and slim business and professional growth.
A lot of what I will debate here will revolve around the issues that we wrestle with every day. Not just the technical issues, but also the human issues.
Let's start...
P.S. My blog-editing software complains about "blogosphere" not been spelled properly... The irony - language always gives you away, as G Carlin used to say.
Subscribe to:
Posts (Atom)
Posts
