Agile Software Development - debate?

There has been a significant upsurge of discussions around the "agile" software development approach (as ill-defined as that may be). A backlash of sorts… Not surprising - I have yet to meet a software development approach that does not go through the typical cycle of discovery, dismissal, early adoption, enthusiastic adoption, boredom, discussion, rejection, obsolescence. The proponents of these approaches will happily quote Gandhi at various stages in the cycle: "First they ignore you, then they laugh at you, then they fight you, and then you win".

But we are talking about development approaches, which offer the interesting couple of challenges:

• the perfect solution costs nothing, presents no risk, and requires no revision
• the perfect solution solves the problems as stated, as understood by all stakeholders when stated and later, as discovered by new stakeholders after the delivery

And this results in the following challenge to any approach: what is the true measure of success it is being judged against?

I have had my share of software development responsibilities, as a developer, as an architect, as a development manager and even as a product manager. I will state the obvious - it's understanding and managing what you are measuring success against that counts, not how you get there. It's so easy to get bogged down into discussing the fine details of the processes and deliverables of a software development approach, and not understand what its motivation is, what it attempts to solve, and how to measure its effectiveness. I have seen more times than I care to count the adoption of a formal approach presented as the solution to problems not understood, invariably with dismal results, leading to the adoption of new approaches with the same lack of success.

Software development is difficult - it is not reducible to pure "mechanical" processes because the amount of creativity involved in it is such at this stage of its maturity that the "human" aspects play a huge role in its implementation. It's impossible to consider software developers as essentially interchangeable for any tasks of consequence. It's impossible to ignore human psychology when assigning tasks, problem solving, emergency triage, etc... It's impossible to set aside the fact that requirements do evolve, that customers do discover new needs, that they want them satisfied quickly or they move on, etc.

Recognizing this, and incorporating it in the approach, is what has made the "agile" approaches ("agile" is umbrella term for so many variations) valuable. There is nothing magical there, and nothing that was not done in many places before. I distill it down to two essential motivations:

• it's easy to lose focus of the true objectives, and the objectives vary frequently, leading to disasters: let's make the objectives those that ultimately matter, keep them transparent, manageable in size and in time, and let's make sure they are reviewed frequently
• it's easy to forget about human dynamics and the social aspects of software development, leading to disasters: let's have effective coordination, let's have common sense, participant-adapted processes and metrics, let's foster collaboration across developers and functions

Keep the focus on your transparent measurable and evolving objectives. Take into account the social aspects of software development. All the processes, deliverables, and recipes in the "agile" approaches are there to support these two key motivations.

Why do I say all this? Simply because the debates on the details of which "agile" method is better than others are noise. The key questions are: are these two motivations important to you? Do you recognize in them the purpose of removing problems you are facing? If so, any of the "agile" methods, adapted to your realities will fit the bill.

Decision Management is connected to all this. Its key goal is to infuse applications with the ability to respond in an agile way to regulatory, market and eco-system changes. The key question is then whether the approach to develop and manage the decisions is truly agile, and how you do assess that.

Carole-Ann has started writing about it, and will be soon presenting at RulesFest 2011 on this very subject.

Interesting stuff indeed.

Financial instruments, systems analysis and decision management

I have been reading a little about what is being presented as the causes for the current worldwide financial crisis. Nothing really surprising in the descriptions, of course, we’ve all heard or read these experts who failed to predict anything going to great details in explaining a posteriori what happened, and provide supposedly valuable insight into what would be coming.

What really amazes me is how each one of the human branches of activities likes living in its own ivory tower. Each industry learns its own lessons from its own failures, or attempts to, but rarely does one industry draw the lessons from another industry’s failures. I am certain some of that has to do with the fact that industries are frequently out of sync in their cycles, but also from a lack of abstraction and high level view on the nature of the industry, its processes and decisions.

A key factor is in this mess is simply the fact that the financial industry created all sorts of financial instruments working at ever higher levels of abstraction, more and more removed from the reality of the underlying relevant physical or legal entities without really understanding their systems behavior.

Derivatives, derivatives on derivatives, swaps, etc: sure, each and every one of them has a solid – and here, sorry for my friends in that space, I will qualify solid as “seemingly” solid – mathematical support. Sure, the bright mathematicians and physicist Wall Street hired by the plane load out of Europe (Russia, France, etc.) are very capable in their domain. They can understand business problems, abstract them into sophisticated models, even understand the conditions under which these models apply, provide all sorts of sensitivity analysis to take into account potential environmental changes, etc. I will not question that.
But I will say that there is no evidence whatsoever that the financial industry applied – or applies – serious systems analysis to the combination of these instruments as they are applied. The obvious aspects that systems analysts pay attention to have been totally ignored: no real systems analysis of applicability, analysis of error / exception propagation, exception management, etc.
As a consequence, each model may very well be well understood, inducing a false sense of security in that this is all under control.

When systems break, they most often break in the interface between components (the famous Mars Climate orbiter issue: http://mars.jpl.nasa.gov/msp98/news/mco990930.html), or in exception handling (the famous Ariane rocket explosion: read the report from one of my teachers, Jacques Louis Lions, http://esamultimedia.esa.int/docs/esa-x-1819eng.pdf), or in the actual decision making in particular when it involves humans (see http://architectguy.blogspot.com/2008/12/precision-speed-and-mistakes.html below, this crisis).
The financial industry – driven by its purely speculative focus, and totally blinded to the fact it no longer understands its instruments – failed and continues failing on all three counts.
For all their intellect and knowledge, the Nobel Prize winner creators of LTCM failed. The industry around it failed more than a decade ago, but little was learned from there. It went back to the same (maybe less obvious) hubris and disregard for proper risk management. With the same consequences…

Frequently enough these failures have simple root causes, overlooked only because of the overall complexity of the system and incredibly cheap to repair compared to the huge cost of these failures. Imagine if the verification systems of Société Générale had actually worked (http://en.wikipedia.org/wiki/January_2008_Soci%C3%A9t%C3%A9_G%C3%A9n%C3%A9rale_trading_loss_incident) : Kerviel would be unknown, and the company would not have lost billions of dollars (and this was limited thanks to the intervention of the company’s crack team and the help from the French government)

Other industries have learned and built into their practices both simplification of components and management of the systems to reach a level in which, when a black swan does appear, the consequences can be assessed and remedial action can be taken. The financial industry should do the same.

Decision Management can help as it increases its focus on scenario-based risk assessment. Modeling decisions and managing around them scenarios that enable the systems analysis of the business processes should allow much better understanding and control of the consequences of the decisions in changing conditions. A lot of the energy spent building predictive models based on well bounded simplifications should be shifted towards modeling the decisions that are made, the business outcomes, their dependencies with respect tot the environment, and managing vast and ever enriched portfolios of scenarios around them to keep risk to black swans in check.

This will require discipline, as it basically emphasizes analyzing highly unlikely but consequence heavy scenarios in times when things are going seemingly well.
But I believe it will be needed.

The Decision Management discipline can help, leverage it.

On a side note on all this,
- Read Nassim Taleb (http://en.wikipedia.org/wiki/Nassim_Nicholas_Taleb).
- Watch him discuss the crisis on TV (http://paul.kedrosky.com/archives/2008/10/12/nassim_taleb_ge.html)
- Watch him and Benoit Mandelbrot (http://en.wikipedia.org/wiki/Benoit_Mandelbrot) debate the crisis: http://uk.youtube.com/watch?v=DLFkQdiXPbo&NR=1

Enterprise software and usability

The long debate...

A lot has been said about this.

Of course - usability is important. It reduces learning curves, it increases effectiveness, it increases sexiness. It increases appeal and reduces cost of ownership.

But reality is more complicated than that, of course.

• Those who buy are not those who use, and the enterprise market builds for those who buy, guided by those who say what should be looked at - analysts and big vendors offering the convenience of the 'no fault' purchase ("nobody has been fired for buying ", " is in the top-right quadrant").
  The set of criteria that ends up being applied at the end of the day is functionality-based - even though users frequently do not need more functionality.Is this a good thing? Not from the perspective of the end user, but can the software vendor be blamed for increasing value for its shareholders and employees by submitting to the tyranny of the existing approach? Maybe in the long run - but debatable.
  You could as well point out that those who buy have a responsibility to their users, the value they create and ultimately their shareholders. As long as the connection between the benefits to the actual users and the value of these benefits to the shareholders do not trump the other real or perceived priorities of the buyer, I see little chance of this situation changing fundamentally.
  
• Building software is hard. Building enterprise class software is hard, and must take into account constraints and/or requirement classes that are difficult to reconcile nicely. The product managers and developers of enterprise software have to contend with difficult scalability issues, security issues, compliance issues, legacy support issues (both technical and human), long and complex release cycles, lack of acceptance by large customers of bleeding edge solutions, etc...
  None of these precludes building usable software - it just makes it harder, and when you combine this point with the previous one, it's clear that usability will frequently be partially sacrificed.
  
• Usability is relative to users, essentially cultural entities. Highly dependant on context. I have the utmost respect for well designed usability - but I represent one cultural perspective, and I know that many things that make me efficient are not efficient for my colleagues.
  This is of course why we end up with the seemingly impossible to exhaust list of configuration options in so many systems that have to cater to more than one function and more than one user type.
  Is this unavoidable? Tough to say, again. Certainly more effort could go into avoiding resorting to the "let's give them options" syndrome, but we do not live in a theoretical world - moving the responsibility to end-users themselves (or their proxies in the IT organizations) may very well represent the rational compromise between cost and benefit from the perspective of both seller and buyers.
  Fighting this is hard. But some do... You'll notice, however, that most of the "oh how usable this is" comments tend to be around software users *want* to use rather than those, even painstakingly well designed - at least from some aspects, that users *must* use.

A lot more can be said about all these subjects and much more articulate people than me have weighed in. But the fundamental reasons are above.

Is usability a lost cause in enterprise software? I don't think so. I think we are on the verge of major changes.

• The enterprise 2.0 revolution (sorry - buzzword, let's assume there is one and it fits what the pundits claim it is) will move the center of gravity to the users by reducing the role of intermediaries.That will change the distribution of power described above, and make the actual users both more relevant and more responsible in the purchasing and renewal decisions.
  
• As interoperability increases, out-of-the-box integration will cease being a major issue and stickiness will become one.Reducing the switching cost will empower the end user with the same effect.
  Of course, I am not naive enough to think that will happen fast and software vendors will make it easy on their own. It will happen under the pressure of the disruptors.
  
• As software-as-a-service turns to reality in more parts of the enterprise, the reduction in perceived, felt cost by buyers and users will result in more emphasis in the other areas neglected so far.
  
• As marketing goes "2.0" and customer acquisition becomes more viral, etc...

So I do think it will happen. But it will be tied to the empowering of the software user that will result from the "2.0" disruptive technologies and approaches. Don't expect it from those that are not threatened.

Ok, enough buzzwords for one night.