sapienti sat: Making SOA Ready for the Enterprise

Walking the line or blurring the edges?

Ab ovo

Something I have struggling with for a long time has recently come to me in a flash. Well, almost. Ever since we started showing SGF to the customers, we were inundated with various feature suggestions and enhancement requests. And one of the most persistent ones (which is incidentally the one I have been resisting most adamantly) was for dynamic routing. SGF has provided some specific forms of routing from the very beginning: between bindings, between versions, between SDLC environments, to refinement methods, etc. But apparently that was not enough, what “the people” demanded was arbitrary content- (as in: route trades with four letter security tickers to NASDAQ and with shorter ones to NYSE) and context-based (as in: route requests to the Chicago servers between UTC 14:00 and UTC 03:00 and to the Hyderabad servers from UTC 03:00 to UTC 14:00) routing capabilities. And the reason I resisted including those was my belief that such functionality represents business logic and thus clearly belongs to an ESB and not to a SOA Governance platform. However, I was usually presented with counterarguments along the lines of: but SGF already has dual binding architecture that allows to mediate between different transports and protocols, and this also is clearly an ESB feature. In fact if one looks that the high level architecture, it looks remarkably similar to those of typical ESBs.

And I would reply about the woes of maintaining business logic in two separate places and the debate would go on. To tell you the truth I was motivated not only by the purity of vision but also by the political considerations like trying to avoid competing with our flagship SOA product, that happens to be an ESB. And in the interest of full disclosure I should say that at the end I gave up and dynamic routing was included as a first-tier feature into the plans for the next release of SGF.

State of the industry

While I was fighting with myself for architectural purity and anguishing over the moral dilemma what would Turing do? the world around me was quietly moving forward. The Governance products were adding more and more ESB-like functions including: routing, arbitrary payload transformations, load balancing, retries, etc. At the same time the ESB products were starting to incorporate more and more features that are traditionally associated with governance platforms. For example Open ESB project has been working on (or at least considering to add) support for aspects for some time. Out of the 27 aspects they have on their wish list, half represent clearly Governance-related functions: Throttling, Logging, Lease, Message/Content Filter (a.k.a. censorship), Validator, Chargeback, Monitoring, Time to Recover, Regulatory Compliance, Audit Trail and Security. I believe that having realized the gap in the underlying JBI specification, the group working on the JBI 2.0 spec intends to add some “Message Exchange handler/interceptor” to codify this architecture. Ironically the first governance-related standard will most likely appear in the ESB space.

So although as both Gartner and myself agree, the value of SOA Governance capabilities embedded into an ESB or other SOA platform is to put it mildly “limited”, as these capabilities mature, it will only be a matter of time before the ESB folks realize that they can export these capabilities outside of their own realm and contend to govern the entire SOA ecosystem. Another anecdotal evidence of this trend is Mule Galaxy a self-proclaimed SOA Governance Platform which is in fact a basic Registry/Repository notable mostly for the interesting choice to expose Atom Publishing Protocol (APP) API instead of supporting any of the traditional registry standards (UDDI, ebXML or JAXR). The interesting thing there that the list of supported governable entity types includes as many SOA platform artifacts (Spring and Apache CXF descriptors, Mule’s own ESB configurations, etc) as traditional metadata types (WSDLs, policies and schemas). To me this means that ESB vendors (I have seen similar features in BEA and Iona offerings) are starting to see themselves not only as purveyors of SOA Governance capabilities but also as first-class users of those as well.

The Epiphany

happened couple weeks ago when I was talking with executive team of a very promising newcomer into the SOA Appliance marketplace. In my mind their product is the closest to what I see as a true SOA governance platform in this space. It also happens that many of their ideas, capabilities and architectural decisions are remarkably in-tune with those that we put into SGF (but this is beyond the point of this particular essay). I was really stunned when their SVP of Business Development told me that they see as their main competitors not the other appliance vendors (Layer 7, Reactivity/Cisco, Vordel or DataPower/IBM) but ESBs. And then it hit me – like the Unified field theory these similarities and mutual encroachment between ESBs and SOA Governance solutions is not coincidental but stem from the fact that they are trying to address the same fundamental problem coming from different directions.

And the name of this single underlying problem is

Mediation

It’s really simple if you look at the problem from this point of view. The gist of what SOA Governance delivers is mediation of non-functional characteristics of enterprise services, often referred to as systemic properties. So if someone wants to be even more succinct they can define SOA Governance as systemic mediation. The gist of an Enterprise Service Bus on the other hand (and I am referring here to the true original meaning of this term and not to the integration products or BPEL engines adorned with adapters and management console that are often marketed as ESBs) is all about dynamic routing of service requests and replies between consumers, producers and intermediaries, based on content and contextual considerations. If I had to use just two words to describe what ESBs do it would be semantic mediation of SOA services.

If we add the third piece to this puzzle: technical mediation which covers the mechanical functions like transport, protocol and synchronicity adaptation and service versioning, and is common to both domains, the Unified Theory of Service Mediation will look like this:

These mediation functions are essential for successful realization of any not-trivial SOA ecosystem. The cleanest way to implement them would be based on a universal service mediation platform (most likely a gateway) that should be designed to handle all flavors of mediation evenhandedly and harmoniously and would be based on cohesive common architecture. From the delivery point of view, such a platform can be offered in a number of “trims”, i.e. “full service” mediation engine, systemic-centric Governance product, semantic-oriented ESB, and minimalistic service gateway that will be limited to just technical mediation. Such solution would clearly be a “killer app” for SOA and it is unlikely to emerge from any of the existing ESB or Governance offerings which are too entrenched in their own ways of looking at the service mediation problem domain. And that, my friends, opens some really exciting possibilities…