Agile Software Development - debate?

There has been a significant upsurge of discussions around the "agile" software development approach (as ill-defined as that may be). A backlash of sorts… Not surprising - I have yet to meet a software development approach that does not go through the typical cycle of discovery, dismissal, early adoption, enthusiastic adoption, boredom, discussion, rejection, obsolescence. The proponents of these approaches will happily quote Gandhi at various stages in the cycle: "First they ignore you, then they laugh at you, then they fight you, and then you win".

But we are talking about development approaches, which offer the interesting couple of challenges:

• the perfect solution costs nothing, presents no risk, and requires no revision
• the perfect solution solves the problems as stated, as understood by all stakeholders when stated and later, as discovered by new stakeholders after the delivery

And this results in the following challenge to any approach: what is the true measure of success it is being judged against?

I have had my share of software development responsibilities, as a developer, as an architect, as a development manager and even as a product manager. I will state the obvious - it's understanding and managing what you are measuring success against that counts, not how you get there. It's so easy to get bogged down into discussing the fine details of the processes and deliverables of a software development approach, and not understand what its motivation is, what it attempts to solve, and how to measure its effectiveness. I have seen more times than I care to count the adoption of a formal approach presented as the solution to problems not understood, invariably with dismal results, leading to the adoption of new approaches with the same lack of success.

Software development is difficult - it is not reducible to pure "mechanical" processes because the amount of creativity involved in it is such at this stage of its maturity that the "human" aspects play a huge role in its implementation. It's impossible to consider software developers as essentially interchangeable for any tasks of consequence. It's impossible to ignore human psychology when assigning tasks, problem solving, emergency triage, etc... It's impossible to set aside the fact that requirements do evolve, that customers do discover new needs, that they want them satisfied quickly or they move on, etc.

Recognizing this, and incorporating it in the approach, is what has made the "agile" approaches ("agile" is umbrella term for so many variations) valuable. There is nothing magical there, and nothing that was not done in many places before. I distill it down to two essential motivations:

• it's easy to lose focus of the true objectives, and the objectives vary frequently, leading to disasters: let's make the objectives those that ultimately matter, keep them transparent, manageable in size and in time, and let's make sure they are reviewed frequently
• it's easy to forget about human dynamics and the social aspects of software development, leading to disasters: let's have effective coordination, let's have common sense, participant-adapted processes and metrics, let's foster collaboration across developers and functions

Keep the focus on your transparent measurable and evolving objectives. Take into account the social aspects of software development. All the processes, deliverables, and recipes in the "agile" approaches are there to support these two key motivations.

Why do I say all this? Simply because the debates on the details of which "agile" method is better than others are noise. The key questions are: are these two motivations important to you? Do you recognize in them the purpose of removing problems you are facing? If so, any of the "agile" methods, adapted to your realities will fit the bill.

Decision Management is connected to all this. Its key goal is to infuse applications with the ability to respond in an agile way to regulatory, market and eco-system changes. The key question is then whether the approach to develop and manage the decisions is truly agile, and how you do assess that.

Carole-Ann has started writing about it, and will be soon presenting at RulesFest 2011 on this very subject.

Interesting stuff indeed.

Unstructured flows

I have not blogged for a while… Too much work, too much involvement in too many decisions with too little time and information. Pretty mind-numbing work.

But it’s time for me to make some of my neurons and synapses to work.

During last couple of weeks, Carole-Ann (www.edmblog.com, www.twitter.com/cmatignon) attended the Gartner BPM summit. One of the key things she conveyed was a fair amount of discussion around the issue of “unstructured flows” and how the industry is addressing them. Besides the fact that there is no unanimity around what to name these flows, there is of course no real agreement on how important they are, and how relevant to problems they are.

I will try to give a first reaction to this in terms of the implications to decision management.

I will assume a very simple distinction between “structured flows” and “unstructured flows”
- “Structured flows” are those that can relatively easily be described in control flow diagrams, with explicit exception management – and by “easily” I mean in a way that can be explained based on a diagram in a way that does not require writing down additional details or having exceptions described through another formalism (something that tends to happen with business exceptions).
- “Unstructured flows” are those that cannot be described that way. They tend to be composed of micro-flows (pre-defined or constructed on the fly) that are stitched together at run time through the recognition of patterns in events.
This may or not correspond to what the rest of the industry sees as distinctions. If not, then just consider these definitions to be specific to this blog.

While I did not attend the conference, I am confronted on a daily basis with this exact issue. Among other things, I am currently responsible for the Enterprise Architecture group at my company, and involved in the architecture and implementation of large enterprise applications – most of which involve both “structured flows” and “unstructured flows”.

One key characteristic I see is the following:
- “Structured flows” tend to cover a large part of the automation of these enterprise applications – but focus on essentially those flows that are fairly clear, require little human intervention, and by virtue of being easy to automate, end up becoming a “must-have” but no longer a differentiator.
- “Unstructured flows” tend to focus on those difficult cases that are – at that point of maturity of the application – not fully automated, and where the interplay between humans (or at least not predictable events) and flows presents the big differentiator in the application – in terms of risk and/or value.

We can take many examples where that is the case.
Take fraud management:
- Automated “structured flows” capture the essence of the known or highly predictable fraud – and catch a large part of the fraud attempts
- But it takes “unstructured flows” to have humans intervene in helping qualify the complicated cases (high value customer, high amounts, etc…) and in identifying new fraud modes
Take insurance underwriting:
- Automated “structured flows” cover anywhere in the 60%-85% range of applications – and almost everybody has it
- But it takes humans involved in “unstructured flows” to deal with the “referrals” which is where the delicate dealing of special cases can help maximize the value/risk ratio


What is the implication of all this for decision management?

Decision management has already been largely involved in improving the relevance of “structured flows” to the business needs and constraints. Up to a large extent, the success of BPM in large enterprise applications can be traced to its ability to isolate the key business decision points, and automate the execution of these decisions in a repeatable and efficient way.
Business Rules Management Systems provide that key mechanism to separate from the flow logic the decision logic in a way that is manageable by the business and controllable by IT. They allow the “structure flows” to cope with the complexities of policies, procedures and practices specific to industries, sectors, enterprises, departments, etc.. And they are at the core at the success of many large scale enterprise applications.

In this kind of applications, the roles are clearly differentiated (even though BPM vendors will argue they handle decision management – they don’t): typically, BRMS handles the decisions, “structure flows” handle the execution of those decisions.
Generalizing it – and referring to a number of on-going discussions around CEP, BPM and BRMS (http://www.edmblog.com/weblog/2008/11/an-attempt-at-demystifying-cep-bpm-and-brms.html and http://architectguy.blogspot.com/2008/11/more-on-cep.html):
- CEP detects business events from the flow of system and application events
- Business events trigger “structured flows”
- Which delegate decisions to BRMS
- And then carry on the execution of those decisions leveraging various integration capabilities

Bread and butter stuff.

Dealing with “unstructured flows” introduces both challenges and opportunities in terms of decision management. Some of these – and I do not intend to be complete here:
- Decisions are taken by clearly outlined “decision services” – implemented through BRMS etc – as well as by less formal (in the sense of software-codified) services – humans in particular; either separately or in conjunction.
- Decisions and actions are stitched together through micro-flows that are triggered through complex event inter-play.
- Since decisions will take into account more informal steps, understanding them and managing their performance becomes significantly more difficult

The last two points are essential.

True decision management for “unstructured flows” will require:
- Understanding events, understanding event correlation and the translation of system/application events into real business events – this is what event management (I hate to try to use the CEP term) should cover.
- Making every effort to understand decisions at large: both the parts codified in BRMS, translated from predictive analytics or simply extracted from policies, procedures, as well as the parts not yet codified there and less formal.
- Including significant collaboration aspects as part of the context of decisions.
- Tracking the performance of the decisions been made to identify both potential for further automation of the informal parts, and to improve the usage made of the corresponding high cost resources.
- Simulating the decisions – including putting to work knowledge gained through the tracking of the decisions made in the informal parts of the decisions.
- Progressively optimizing the unstructured flows through experiments (champion / challenger)

Counter-intuitively (maybe), “unstructured flows” will provide more challenges and more opportunity for decision management technologies and products.

Predictions and Surprises

WSJ has this interesting series demystifying - or at least discussing - issues around numeracy, probabilities, and the corresponding impact.

I just read the following: http://blogs.wsj.com/numbersguy/the-crash-calculations-621/

A couple of key things that I feel are not really covered deeply enough:
- the quality, and ultimately the validity of predictions in a given context are a direct function of the relevance to that context of the explicit and implicit assumptions in the modeling process used to create the prediction
- decisions should never be made only on predictions obtained through models, they should include scenario based simulation and impact analysis

The reason we build models is precisely to create abstractions that we can manipulate with the tools of our mind and our technology. Tools that allow us to get out of the immediate sensor-driven reaction mode, and forecast. I believe modeling is essential to forecasting - I know some will say just crunching numbers with no a priori model is the path of the future (a Wired article I read one day) but that's a fallacy: as soon as you use the result of the crunching, you are using a model, maybe an implicit one, but you are using a model.

But who says abstraction says context-dependent simplification. And that is key. Understanding the context the abstraction assumes, and the sensitivity of the resulting model and predictions to variations in that context is paramount to being able to leverage these predictions.

That gets lost. Because it's complicated and because of the multiple psychological aspects that make understanding and leveraging statistical and probabilistic results very difficult for the average person.

See the points made in the EDM blog by Carole-Ann (http://www.edmblog.com/).

Netbooks accelerate The Cloud?

The move to "netbooks" and other always connected devices seems to finally be here to stay - it's been talked for a while, but the convergence of technology and economics to make it a reality is now with us.
There is a very interesting article in Wired (if I remember well) that chronicles the "strange" history of the new generation of "netbooks": how something that was created to address the needs of those who cannot afford the current laptops ended up filling a real need even amongst those who are the target of these current laptops. Essentially, we have reached the point in which these tools are becoming appliances, judged more by the function they actually fulfil than by the performances they can exhibit regardless of our need for them.

Combined with the emergence of The Cloud, this shift promises a change in the way all of us, consumers and enterprises, will interact with applications. It seems obvious, but the trend is accelerating. Applications will be at least partially on The Cloud, they will be multi-device, multi-access, asynchronous.

That will bring a series of challenges to us, software architects. We will need to make The Cloud ensure the viability of the model for complex, long lived, IP rich, confidential, secure flows. We will need to make the asynchronous models work well. We will need to make the user interfaces multi-device and secure, etc...
One proof this is moving forward, and fast, is that companies such as Microsoft, which do not create trends but make them mainstream, are getting there. The Azure effort (http://www.microsoft.com/azure/) is well known, but they are thinking across the board on user interfaces, on new browser (the window to The Cloud) architecture approaches (http://research.microsoft.com/pubs/79655/gazelle.pdf).

It will also bring challenges to the business models for software products, of course. The current models are not adapted, and it will take time before a norm gets established.
It will also bring new security needs: big cloud centers will need to be protected against all sorts of attacks, cyber and physical, etc.

Interesting disruption.

The cloud matters

Well, this is certainly going to be "duh"-obvious, but it's clear that we are entering the year of The Cloud. The rapid-fire succession of announcements from major players of one or another form of cloud offering is not just the result of flock mentality or boredom early in the year: it's a symptom of the relevance the cloud has reached.

Why now?

- A first reason is the current crisis. On each and every project, controlling cost has become the core preoccupation. And I really mean control as in exerting control over it, predicting it, managing it, not just containing it.
- A second reason is the availability of manageable technical solutions to support cloud-based platforms and applications. We are beyond cloud-supported data or document storage, we are now in the days of cloud-supported services and netbooks.
- And a third reason is the emergence of cloud providers with enough clout to allow significant early adopters to take the rational bet of overcoming the remaining complexities and confusions and start leveraging the offerings. Traditional platform vendors, major Web Commerce players, innovative newcomers.

The confluence of these three trends is - in my neophyte opinion - the key for the unavoidable success of The Cloud in 2009.

How will that happen?

- The economic drivers make The Cloud more relevant. Projects will focus on the management of cost, and The Cloud's inherent support for controllable scalability will make it the most attractive platform to work on.

- The innovative development community - including software vendors as well as system integrators - will flock to The Cloud, will support the development of the corresponding tooling (IDEs, platforms/middleware, ...) and will ensure the success of the various cloud-based or cloud-supported platforms (PaaS) on which service-based (SaaS) applications will be created and/or composed.

- The support for dedicated governance and management provided by cloud-based or cloud-supported platforms will prove to be very attractive to businesses having been burnt by previous attempts to create on-premise major SOA applications.

- Governance, life cycle management, etc, will become sources of differentiation between cloud offerings, and that will lead to significant innovation and momentum in that very key domain.

- Ubiquity and better governance and management will enable more customer/collaboration/social-centric (rather than process-centric) applications, shifting the focus from typical B2B/B2C to Business-to-Community (B2Comm?).

- Which will create further need for differentiation, better quality services, easier to assemble into high value evolving adaptive applications taking as part of their essence the various communities (users, analysts, customers, etc..)

- Development will be permanent, deployment frequent, adaption constant. New cultural points of view will be included in the applications.

- Etc.

All this, all this enabled by the fact that The Cloud makes the applications cost manageable. It becomes something that can be throttled. Just think about it: how can we compare the cost manageability of a major cloud-supported enterprise application versus a major J2EE enterprise application (if you want to suffer, say Websphere)?


What to be careful about?

Applications, in particular the "enterprise" applications I deal with, live and die through the quality of the decisions they support. An application can be very beautiful and execute very fast and in a secure way, but it fails to generate the business value that is expected, it's growth - or even it's survival - is questionnable.

Enterprise Decision Management (EDM) or Decision Management (DM) addresses that. In the pre-cloud days.

The challenge is now to think about how to approach EDM / DM for The Cloud.

Interesting times.

A little more on risk mismanagement

The NYT just published an interesting analysis of the role played by modern financial models in the current meltdown [http://www.nytimes.com/2009/01/04/magazine/04risk-t.html?_r=1&partner=permalink&exprod=permalink&pagewanted=all]. The article highlights the particular role played by VaR models and the institutionalized and improper reliance on that kind of models.

I have blogged about this in the past. It's common knowledge - although also commonly ignored - that mathematical models are just that - models. They operate under sets of assumptions that need to be understood to be applied.

But as pointed out earlier [http://architectguy.blogspot.com/2008/12/financial-instruments-systems-analysis.html], the key problem from a technical standpoint is that no real system analysis has been attempted on the complex combination of financial instruments being put to work. No exception or error propagation analysis, no interface consistency analysis, etc. These are all words that are foreign to the daily practice of these instruments. Quants are not system analysts.

To the industry's credit, it may well be that the sheer complexity that the systems analysis entails is such that it is practically unfeasible. If that ends up being the case, regulation-based restrictions on these instruments in the largest financial markets may help reign in the complexity - at the cost of reduced creativity. The current meltdown can explain how it is that such a move would probably be positive in the short to medium term - and would avoid having the tax payers foot a huge bill to bail out an already highly rewarded and ultimately irresponsible industry.

If the financial industry wants to avoid over-regulation, it will have to prove it can control its risk, through a combination of better overall systems analysis and better understanding of the decisions made.

Decision management is gradually including the relevant aspect of systems analysis, and will turn to be an unavoidable piece of the core processes that use the complex financial industries. You need to be prepared to deal with the unexpected and understand the impact of the decisions you make.

Financial instruments, systems analysis and decision management

I have been reading a little about what is being presented as the causes for the current worldwide financial crisis. Nothing really surprising in the descriptions, of course, we’ve all heard or read these experts who failed to predict anything going to great details in explaining a posteriori what happened, and provide supposedly valuable insight into what would be coming.

What really amazes me is how each one of the human branches of activities likes living in its own ivory tower. Each industry learns its own lessons from its own failures, or attempts to, but rarely does one industry draw the lessons from another industry’s failures. I am certain some of that has to do with the fact that industries are frequently out of sync in their cycles, but also from a lack of abstraction and high level view on the nature of the industry, its processes and decisions.

A key factor is in this mess is simply the fact that the financial industry created all sorts of financial instruments working at ever higher levels of abstraction, more and more removed from the reality of the underlying relevant physical or legal entities without really understanding their systems behavior.

Derivatives, derivatives on derivatives, swaps, etc: sure, each and every one of them has a solid – and here, sorry for my friends in that space, I will qualify solid as “seemingly” solid – mathematical support. Sure, the bright mathematicians and physicist Wall Street hired by the plane load out of Europe (Russia, France, etc.) are very capable in their domain. They can understand business problems, abstract them into sophisticated models, even understand the conditions under which these models apply, provide all sorts of sensitivity analysis to take into account potential environmental changes, etc. I will not question that.
But I will say that there is no evidence whatsoever that the financial industry applied – or applies – serious systems analysis to the combination of these instruments as they are applied. The obvious aspects that systems analysts pay attention to have been totally ignored: no real systems analysis of applicability, analysis of error / exception propagation, exception management, etc.
As a consequence, each model may very well be well understood, inducing a false sense of security in that this is all under control.

When systems break, they most often break in the interface between components (the famous Mars Climate orbiter issue: http://mars.jpl.nasa.gov/msp98/news/mco990930.html), or in exception handling (the famous Ariane rocket explosion: read the report from one of my teachers, Jacques Louis Lions, http://esamultimedia.esa.int/docs/esa-x-1819eng.pdf), or in the actual decision making in particular when it involves humans (see http://architectguy.blogspot.com/2008/12/precision-speed-and-mistakes.html below, this crisis).
The financial industry – driven by its purely speculative focus, and totally blinded to the fact it no longer understands its instruments – failed and continues failing on all three counts.
For all their intellect and knowledge, the Nobel Prize winner creators of LTCM failed. The industry around it failed more than a decade ago, but little was learned from there. It went back to the same (maybe less obvious) hubris and disregard for proper risk management. With the same consequences…

Frequently enough these failures have simple root causes, overlooked only because of the overall complexity of the system and incredibly cheap to repair compared to the huge cost of these failures. Imagine if the verification systems of Société Générale had actually worked (http://en.wikipedia.org/wiki/January_2008_Soci%C3%A9t%C3%A9_G%C3%A9n%C3%A9rale_trading_loss_incident) : Kerviel would be unknown, and the company would not have lost billions of dollars (and this was limited thanks to the intervention of the company’s crack team and the help from the French government)

Other industries have learned and built into their practices both simplification of components and management of the systems to reach a level in which, when a black swan does appear, the consequences can be assessed and remedial action can be taken. The financial industry should do the same.

Decision Management can help as it increases its focus on scenario-based risk assessment. Modeling decisions and managing around them scenarios that enable the systems analysis of the business processes should allow much better understanding and control of the consequences of the decisions in changing conditions. A lot of the energy spent building predictive models based on well bounded simplifications should be shifted towards modeling the decisions that are made, the business outcomes, their dependencies with respect tot the environment, and managing vast and ever enriched portfolios of scenarios around them to keep risk to black swans in check.

This will require discipline, as it basically emphasizes analyzing highly unlikely but consequence heavy scenarios in times when things are going seemingly well.
But I believe it will be needed.

The Decision Management discipline can help, leverage it.

On a side note on all this,
- Read Nassim Taleb (http://en.wikipedia.org/wiki/Nassim_Nicholas_Taleb).
- Watch him discuss the crisis on TV (http://paul.kedrosky.com/archives/2008/10/12/nassim_taleb_ge.html)
- Watch him and Benoit Mandelbrot (http://en.wikipedia.org/wiki/Benoit_Mandelbrot) debate the crisis: http://uk.youtube.com/watch?v=DLFkQdiXPbo&NR=1

IBM's BEM announcement

Here we go: http://www.informationweek.com/news/business_intelligence/bpm/showArticle.jhtml?articleID=212200753&cid=RSSfeed_IWK_Business_Intelligence

IBM launches an attempt to define the "Business Event Management" space. This plays directly into all the controversies around CEP and its overlap and positioning with respect to other spaces such as "Business Process Management" or "Business Rules Management". You can find some of the discussions in Carole-Ann's blog [http://www.edmblog.com/weblog/2008/11/an-attempt-at-demystifying-cep-bpm-and-brms.html], Tim's blog [http://www.thecepblog.com/] and in my own blog in previous postings [http://architectguy.blogspot.com/2008/11/more-on-cep.html and http://architectguy.blogspot.com/2008/11/state-events-time-confusion-around-cep.html].

Is this move by IBM good or bad?

Let's read a little the details:

a) "IBM is staking out a position in complex event processing as applied to business processes"
Ok, so the focus is around events as it pertains to their relevance to business processes. But we should probably not read too much into it given that IBM approaches this whole space from the business process perspective.
b) [it] " is an ability to define a discrete software event, then monitor it over time, set parameters around it, and trigger alerts if the rules governing it are violated"
Ok, fairly standard. But it does refer to "rules governing" alerts, I assume. To use terminology used before, this would be event correlation and pattern matching to translate the "ambient" events to business events.
c) Which is supported by the quote made of Luckham: "What is a complex event? It is an event that could only happen if lots of other events happened"
So far, pretty simple.

But then:
d) "By being able to combine views of software events and business events in one set of tools, IBM gains the prospect of "allowing business users to more easily change the logic in the business event processes," Reeves said. And they should be able to do so in business language, not programming languages"
This is interesting.

The content of the announcement seems to confirm two key things:
- First, event management is primarily about the identification and generation of business events from the ambient events. Similar to what Carole-Ann and I had written in previous posts.
- Second, IBM wants to introduce high level EPLs to express the logic for that processing that are business-centric, something very similar to what Business Rules Languages and approaches are in the business rules management area.

Of course, it is interesting that IBM is making this positioning public so early after the acquisition of Ilog, and long before any true integration with ApSoft.

This may be a way for IBM to establish the "Business Event Management" denomination early on.

I am ambivalent on the approach. While it's probably clear to the reader that I am not a fanatic of the "CEP" acronym, I think I need to read / learn more than a press release on what IBM really intends to cover with their new old TLA.
I also need to understand what IBM thinks of the relative roles of BEM and BRMS. I think that, as pointed out in some of the blogs referred to earlier, the roles are clear, but I am not sure what I call clear is what others would call acceptable.

What's your take?

Map - Reduce - Already happening...

Following up on the previous post [http://architectguy.blogspot.com/2008/11/map-reduce-relevance-to-analytics.html], and after some more quality time with google, it's clear that there is more activity around the extension of map-reduce to large scale analytics than I originally thought or knew about.

Joe Hellerstein has published in O'Reilly Radar an interesting post on this exact subject [http://radar.oreilly.com/2008/11/the-commoditization-of-massive.html]. It is very interesting reading.

I think that beyond "massive data analysis" we will see the application of map-reduce to "massive event correlation". The same way the amount of data available for (and crying for) analytics processing is staggering and continues growing at staggering speed, the number and complexity of events that need to be processed to find the patterns and correlations that make them relevant to business apps processing is staggering and continues growing.
RFID, all sorts of devices connected to the cloud, exchanging information / pushing events related in many ways, all sorts of formal or informal transactions, connections and disconnections, ...

Joe points out that we will get to some convergence between traditional data management environments (your SQL) and the map-reduce approaches. He even points out to offerings put forward by innovative players - so what I thought was bound to happen in my previous post is happening even faster.

To paraphrase (in a slightly reworded fashion) what I wrote earlier,
- event pattern identification and correlation are ripe for this type of approaches, and will require it
- new algorithms will be made possible that leverage the approaches to provide results we do not think are possible today

This is one of those typical cycles: the abundance of data and events pushes the creation of new techniques, which in turn enable new applications which produce more data and events.

Very interesting times.

Map - Reduce - relevance to analytics?

One of the key problems we face when dealing with vast numbers of events and/or vast amounts of data is how to efficiently parallelize the processing so as to accelerate the analytic development cycles. This is not just a question of efficiency (minimize the downtime of high cost PhD resources, ...) but also a question of feasibility (make the development of the analytics fit in business driven time-lines, such as the time to detect and react to new fraud mechanisms).

My experience, coming from the real time embedded systems days, is that the best ways to achieve massive parallelization result from highly simplified low level programming models. These models are designed to do only very simple operations but with nothing in them that prevents systematic parallelization.

Google has made popular (at least in the technosphere) the map-reduce approach derived from the functional programming world [http://en.wikipedia.org/wiki/MapReduce]. Google officially applies the approach to its search core capabilities, but it's very likely it uses it for much more than that.
Its formal definition is trivial, but map-reduce does provide a solid basis for massive parallelization of list processing. Which is very much what a lot - a lot - of analytics development work is. It may very well end up being a key component that will allow the implementation of complex analytics processing of amounts of data well beyond what we tackle today, enabling applications that are beyond reach today.
Today, map-reduce is essentially used for search (processing text), network processing (as in social networks), etc. The Hadoop open source effort [http://hadoop.apache.org/core/] has a list of current applications to which its map-reduce implementation is applied [http://wiki.apache.org/hadoop/PoweredBy]: that makes a very interesting reading.

Its applicability to machine learning / predictive analytics building is illustrated by the Mahout effort [http://lucene.apache.org/mahout/] which seeks to leverage Hadoop to specific implementations of traditional machine learning algorithms [http://www.cs.stanford.edu/people/ang//papers/nips06-mapreducemulticore.pdf]. I see immediate applications to both including this approaches to CEP (highly parallel processing of events to achieve fast correlation) and predictive analytics development (highly parallel processing of data to find patterns - neural-net like; highly parallel implementations of genetic approaches, etc.).

I would be curious to know what the reader thinks about this.

I also believe a few developments are bound to happen, and I expect to see more happening around them in the coming few years:
- development of hybrid systems combining more traditional processing (SQL, SAS datasets,...) with map-reduce, potentially introduced by the vendors themselves but more likely by minor innovative players first
- development of new algorithms in event processing / data processing / machine learning that leverage map-reduce
- introduction of other approaches similar to map-reduce as the corresponding results.

I am particularly intrigued by the last two possibilities.

More on CEP

Obviously, the CEP / BRMS / BPM / boundaries debate remains hot and continues provoking soul searching.

My previous post on the this was a long (and complicated in the way it was phrased) ranting around the fact that CEP attempts to cover too many areas of the EDM picture, and at the same time does not do a deep enough job at solving the challenges around its core contribution - events as first class citizens in EDM apps. I do think the points I tried to make there remain valid.

I did also contribute a post in another blog as a reaction to a thread on the same subject.
Here is the essence of what I wrote there:

In this debate, we are essentially dealing with three core notions: events, decisions, processes. There is of course much more than that, but I think the confusion we are seeing results from lack of solid boundaries between responsibilities around these three notions. Getting to basics helps.

- Events are all about detecting what decision needs to be taken in a complex temporal context (essentially event correlation, converting monitored ‘ambient’ events into relevant business events that require reaction)
- Decisions (and I will use that term instead of rules) are all about deciding what needs to be done as a reaction to those business events
- Processes are all about executing the decision taken

In a very simplistic analogy,
- Dealing with events in an enterprise application is akin to the sensory system absorbing the information from all the sensors and sources it is connected to, and constructing a view for further decisioning and action, with or without immediate reaction, and communicating it through the nervous system.
- Dealing with decisions in an enterprise application is akin to the multi-step highly collaborative decision making the brain engages in, resulting in events and situational analysis, inclusion of further data, inferences, deductions, etc. and leading to conclusions on further reactions or proactive actions.
- Dealing with processes in an enterprise application is akin to the body executing the plan elaborated by the brain including the input from the nervous system.

CEP should address primarily the first, BRMS and other decision management technologies the second, business process the latter.

What I just outlined is centered operational aspects - as in the operational / improvement aspects distinction presented by Carole-Ann in her talks about the EDM vision at ORF and BRF [http://www.edmblog.com].

The improvement aspects cover how to leverage the information gathered by the operational system - as well as expertise - in order to improve the decisions. It starts by gathering the information and understanding it. All aspects above are to be covered: Am I capturing the right events? What if I decided to capture something slightly different: does that make my business events more relevant? Why did I capture this event at that point in time? What other events are captured within a given contextual distance from this one? Ditto for the decisions (the rules). Ditto for the execution (the processes).Why not just call these things "Event Analytics", "Decision Analytics", "Process Analytics"?Yes, they are connected. But different.

This is simplistic of course, but it has the virtue of being clear. Maybe we can try to reframe the discussions on these issues in that context?

State, events, time - a view on the confusion around CEP

CEP

Over the recent few months, the EDM world I work in has seen a lot of noise generated by the arrival of CEP - "Complex Event Processing" - and the impact it has had in terms of provoking soul searching in the BRMS and, to a lesser extent the EDA, ESP and other E(x)(x) worlds.
The "E" in these E(x)(x) is "events" which are of course at the core of CEP.
But so is "complex" and "processing", both of which lead more to the area of EDM or BRMS.

This is one of these situations in which a technology addressing at its core a valid set of concerns gets dropped in the middle of a complex soup of acronyms, and confusion ensues.

Of course, the CEP specialists should bear with me for the duration of the blog. I am aware that CEP has been around for a long time, etc., but it's also clear that it is undergoing a renewal through its adoption by the big platform vendors (IBM and Oracle acquisitions) and the innovative ones (Tibco, JBoss).

Why am I writing this blog? Essentially because I believe that the confusion is self-inflicted because as an industry (enterprise applications) we have not been careful to focus the usage of the terminology to its key area - events - and we've spent too much energy trying to justify the complex and processing aspect.

While some of the points below touch on semantic confusion (big words), I will try to remain pragmatic. Let's take "event", "complex processing" in turn.

"Events"

It is true that "events" and their semantics are quite important in systems that need to make decisions.

This is not new, it has always been the case. So why are we - the enterprise software world - only recently starting to give to "events" the preeminence they already have in other worlds - such as the real-time systems world I started in?

Well, simply because the enterprise world has a way to cope with a lot of the value of events by translating events into state.
Traditionally, the occurrence of events, and in sophisticated systems, even their sequencing and timing, are encoded in the state of the system, transforming the "event" management problem into the dual "state" management problem.
Take an example: a fraud detection system is of course keenly interested in knowing what happened where when in which order. It's not a surprise that most CEP vendors use fraud detection as a key example. Well, guess what? Fair Isaac's Falcon Fraud Manager, by far the most used credit card transaction fraud detection system, as well as a host of other fraud detection systems, do not use CEP as defined by the current vendors. They translate events into profile (read "state") information, and they enrich at run time the profiles using sophisticated business rules (not even production systems). This profile encodes variable values that are highly targeted precisely to capture the essence of the business semantics of the events: "number of purchases of gas using the same credit card charging less than a certain amount within a window of time and a geographic region".
You could say that what they have is a sophisticated hugely scalable (90%+ of all credit card transactions in the US go through Falcon) stateful decision management system, supported by a powerful "cache" of variables computed through complex business rules.
And there are many cases like that.
The reality is that the "enterprise" world has been dealing with events for ever. They have simply not needed to resort to any E(x)(x) notion / stack / etc.

That being said, I believe there is an important piece in "event" centric expression of logic - and that is the separation of the event processing logic from the rest, and the resulting clarity, elegance, maintainability and ultimately all those scalability, ability to audit, robustness, ..., qualities that result from clean concepts, clean designs, clean architectures.

Which brings me to the following first opinion on CEP:
(1) CEP should stop worrying about the origin of the events and focus on the events themselves. It does not matter how the events originate, and the issues of ESB, EDA, ESP, in-database generation, etc..., are all orthogonal / independent on how events-dependant logic is managed.
And to the second opinion on CEP:
(2) CEP should stop worrying about caching. Yes, caching is important but irrelevant to the power of the approach - as the fact that among the largest and most scalable event-driven enterprise apps, many handle the issue with no need to couple the event management piece from the management of caches. Right now, there are efforts to extend the Rete structures and to adapt the algorithm to build this cache in a more efficient way for the corresponding type of rules processing. Great usage of the technology, but that will not make the potential power of event centric approaches any more compelling.

Maybe it's time to be a little more constructive.

(3) CEP would do wonders in enterprise systems if it focused all its attention to the "event" part: the semantics of events, the richness of the event context (time, location, referential, ...), the clean semantics of the key contextual notions (time operations including referential management, etc...), etc.

The events semantics question is not innocent, and is absolutely not a settled question - witness the numerous exchanges between clever people than me on this subject.

I had a discussion with Paul Haley once on this subject and we went into the "what is an event" question that has the virtue of quickly getting people upset. It's a valid question: the definition of "event = observed state transition" has the bad taste of defining events in terms of state, but its key issue in my eyes is that it supposes observation of a state and lacks content.
The value of events is that they have intrinsic context that are not naturally contained in state systems: they occur at a point in time with respect to a given referential - or more generally, they occur at a contextual point (could be time+location, etc.) with respect to a given referential. Different events used within the same system may have their intrinsic context expressed with respect to different referentials - and that will be the default case in any distributed system.
Events occur, they are atomic, immutable. Their only existence is their occurrence. We may track the fact that they occurred, but an event instance only happens once and is instantaneous. An event does not last a duration. That is not logical to me - its effects or the state change it triggers may last a duration, but the event in itself is instantaneous.
Which leads to the fact that there are natural event correlations you want to express (not just discover): an event creates a state transition, a correlated event will create another state transition that will bring the state back to the original one.
This is just my opinion - but if you talk to more than one real specialist, you will get more than one view. Not a good sign of maturity of the concepts.

Clean ontologies / semantics / etc. needed.

With the clarification of the semantics of events and their intrinsic referential-dependent context are clarified, we need to focus on what we want to express about events - and for that, we need to bring on the enterprise application experts.
There is a lot to learn from the real-time systems experts - refer to the very old but incredibly good insights from Parnas' work. There is a lot to get back from the original event correlation systems - many built with systems such as Ilog's rules engine. These could even be said to be the purest predecessors to what CEP attempts to do.

What this will end up doing is giving us - the decision management world - a very powerful tool to "naturally" express logic on events, with their referential-dependant context, and to do so in a way that enables true management (things like verification of the logic included), powerful optimizations, etc.


I honestly do not think we are there, and I would really like to see the standardization world - the OMGs and others - help us get there; but I do think we need the enterprise business app drivers. We had that, in real-time systems: the military and transportation apps.

I would love the specialists to prove me wrong and to show me we are there.


"Complex processing"

This will be shorter.

As stated above, I believe that:
- CEP should narrow its processing ambitions. One approach is to focus the purpose of its processing to clear outcomes - such as what the correlation engines did 20 years ago. For example, we could say that CEP's processing is about processing / correlating events to generate higher order events: transaction events to generate a potentially fraudulent transaction event. I will call these "ambient events" and "business events": the CEP processing goal is to translate ambient events into business events
- CEP should focus the complexity of its processing to the corresponding revised ambitions.
- CEP should leave all issues related to event streaming, event transport, event communication, etc. to other layers.

I may be a purist, but I see a simple picture:
- leave anything related to transport, communication to other layers
- use this revised CEP to express and execute event-relevant logic, the purpose of which is to translate the ambient events into relevant business events
- have these business events trigger business processes (however lightweight you want to make them)
- have these business processes invoke decision services implemented through decision management to decide what they should be doing at every step
- have the business processes invoke action services to execute the actions decided by the decision services
- all the while generating business events or ambient events
- etc.

As such, CEP will include a semantically sound event-with-intrinsic-referential-dependent-context model, a corresponding language (EPL or vocabulary) to express logic, algorithms to efficiently execute (wide open field - tons of people doing analytics, Bayesian, rules, ...), techniques to verify (wide open - and fairly empty: I only know of the real-time folks), etc.

And there, the value of CEP is clear. Of course, it is lower than what CEP vendors would like, but significant anyway.

I am hoping this is controversial enough to bring on flames...

Enterprise Decision Management? How about State Decision Management?

This is interesting: http://www.nytimes.com/2008/03/28/world/americas/28cybersyn.html?_r=1&oref=slogin&ref=americas&pagewanted=print and http://www.guardian.co.uk/technology/2003/sep/08/sciencenews.chile

Very interesting for me since I work in decision management, and Chile is where I come from. I had seen mentions of this in the past (hence my sudden recollection of it during this sleepless night) in Andreseen's blog. Wikipedia has a summary (http://en.wikipedia.org/wiki/Project_Cybersyn) that mentions usage of Bayesian filtering although does not insist on the learning and adaptive aspects.

It also refers to this: http://www.williambowles.info/sa/FanfareforEffectiveFreedom.pdf - which I really do not know what to make of. It brings me back to my early days in Engineering school when I was studying what we used to call "Automatique" and was all about adaptive control. But applied to social and economic matters at the scale of a country!?

On the other hand, this lecture on cybernetics by Beer is intriguing: http://lispmeister.com/downloads/Stafford-Beer-forty-years-of-cybernetics-lecture.wma (pretty big)

So it turns out that Chile does not just do wine and pisco (sorry Peru).

Back...

After a long absence from the blog, I am back to it.

I have spent a couple of months heads down focusing on management issues, most of them consequences of the way my current employer has chosen to face the current crisis. Typical 'protect the EPS' reaction.

I posted earlier on the counter-cyclists, and the reasons why it makes a lot of sense to manage good times better so that we can manage bad times in a strategic way and benefit from the exit from downturns rather than play catch up. It looks simple on paper - I am certain it's more difficult than that - but I have to say my growing experience with upper management does not give me the feeling this requires anything else than thinking strategically rather than tactical. Where this fails is when upper management is mercenary upper management: on board for 2 to 5 years and not interested in long term growth.

This is important for us software architects - not just because of the obvious impact on our continued employment, but also because architecture is strategic, and gets sacrificed at the altar of short term tactical moves. This is obvious to say, again, and obvious to understand, but it still amazes me how selective corporate perception is, and how little care is given to preserving corporate memory.

I will for a while - hopefully - stop blogging about this kind of thing. And go back to more technical stuff - however difficult it is to do in abstract of the overall situation...

Counter-Cyclists

Sometimes you stumble on an article which for some reason just speaks to you. Just when you thought you had the perfect articulation of what you want to express in your head, you read how somebody has done it so much better...

As mentioned earlier, my work is at an enterprise software vendor. Hit pretty hard by the current conditions. We are going, like many of our public peers, through the phase in which we need to adapt to new temporary market conditions, and keep our key indicators (EPS, ...) in check. And, as many of our public peers, we react by contracting after having expanded in better times. Tightening our belt when the conditions are poor. Both blaming a black swan (the mortgage crisis) and taking credit for having seen it early (because it does not reflect well not seeing what we should have seen, or - although not this time - because our brains do not let us treat pure randomness as such, we must see order - read Taleb's books [see http://www.fooledbyrandomness.com/]).
The well understood risk is reduced ability to deliver, reduced ability to support our customers, reduced ability to deliver growth, etc. The EPS will look ok, but the risk is big that our compromised abilities will render us obsolete when the environment changes.

I have been through these cycles many times. And I have seen what separates those who risk their survival at every crisis from those who turn the threat into opportunity: the latter are counter-cyclists. I did not know how to express it simply and clearly, but that is what they are.
This article from Wharton [see http://www.whartonsp.com/articles/article.asp?p=429379] describes that very nicely.

Counter-cyclists keep control over the expenses and growth through up turns. They do not suffocate innovation and growth - they apply the same caution in up turns as in down turns, being selective, being focused. That may seem like a defensive strategy, but is not.
Because counter-cyclists leverage their control over these expenses and growth to avoid over-reacting in down turns. They can afford to increase their expenses and enable growth generating activities during the down turn, precisely when the competition is weaker, and putting them in a premium position to take advantage of the following recovery.
They can cherry pick the best employees - those that the others "have" to let go to protect EPS. They can cherry pick their investments - those that the others "have" to let go to protect the numbers, to focus on core values, etc...
they can choose the moment they divest, where they divest, etc.

This is really not rocket-science - and is obviously taught in MBA and various management classes, illustrated by a number of examples. We can even find Latin phrases that express the core of this.

Enterprise software and usability

The long debate...

A lot has been said about this.

Of course - usability is important. It reduces learning curves, it increases effectiveness, it increases sexiness. It increases appeal and reduces cost of ownership.

But reality is more complicated than that, of course.

• Those who buy are not those who use, and the enterprise market builds for those who buy, guided by those who say what should be looked at - analysts and big vendors offering the convenience of the 'no fault' purchase ("nobody has been fired for buying ", " is in the top-right quadrant").
  The set of criteria that ends up being applied at the end of the day is functionality-based - even though users frequently do not need more functionality.Is this a good thing? Not from the perspective of the end user, but can the software vendor be blamed for increasing value for its shareholders and employees by submitting to the tyranny of the existing approach? Maybe in the long run - but debatable.
  You could as well point out that those who buy have a responsibility to their users, the value they create and ultimately their shareholders. As long as the connection between the benefits to the actual users and the value of these benefits to the shareholders do not trump the other real or perceived priorities of the buyer, I see little chance of this situation changing fundamentally.
  
• Building software is hard. Building enterprise class software is hard, and must take into account constraints and/or requirement classes that are difficult to reconcile nicely. The product managers and developers of enterprise software have to contend with difficult scalability issues, security issues, compliance issues, legacy support issues (both technical and human), long and complex release cycles, lack of acceptance by large customers of bleeding edge solutions, etc...
  None of these precludes building usable software - it just makes it harder, and when you combine this point with the previous one, it's clear that usability will frequently be partially sacrificed.
  
• Usability is relative to users, essentially cultural entities. Highly dependant on context. I have the utmost respect for well designed usability - but I represent one cultural perspective, and I know that many things that make me efficient are not efficient for my colleagues.
  This is of course why we end up with the seemingly impossible to exhaust list of configuration options in so many systems that have to cater to more than one function and more than one user type.
  Is this unavoidable? Tough to say, again. Certainly more effort could go into avoiding resorting to the "let's give them options" syndrome, but we do not live in a theoretical world - moving the responsibility to end-users themselves (or their proxies in the IT organizations) may very well represent the rational compromise between cost and benefit from the perspective of both seller and buyers.
  Fighting this is hard. But some do... You'll notice, however, that most of the "oh how usable this is" comments tend to be around software users *want* to use rather than those, even painstakingly well designed - at least from some aspects, that users *must* use.

A lot more can be said about all these subjects and much more articulate people than me have weighed in. But the fundamental reasons are above.

Is usability a lost cause in enterprise software? I don't think so. I think we are on the verge of major changes.

• The enterprise 2.0 revolution (sorry - buzzword, let's assume there is one and it fits what the pundits claim it is) will move the center of gravity to the users by reducing the role of intermediaries.That will change the distribution of power described above, and make the actual users both more relevant and more responsible in the purchasing and renewal decisions.
  
• As interoperability increases, out-of-the-box integration will cease being a major issue and stickiness will become one.Reducing the switching cost will empower the end user with the same effect.
  Of course, I am not naive enough to think that will happen fast and software vendors will make it easy on their own. It will happen under the pressure of the disruptors.
  
• As software-as-a-service turns to reality in more parts of the enterprise, the reduction in perceived, felt cost by buyers and users will result in more emphasis in the other areas neglected so far.
  
• As marketing goes "2.0" and customer acquisition becomes more viral, etc...

So I do think it will happen. But it will be tied to the empowering of the software user that will result from the "2.0" disruptive technologies and approaches. Don't expect it from those that are not threatened.

Ok, enough buzzwords for one night.