sapienti sat: Making SOA Ready for the Enterprise

Sort of… I got e-mail from blogged congratulating me with the fact that this blog has beet evaluated by their editors and got 8.0 out of 10. It’s nice to be noticed and evaluated (especially if the marks are good). However before I let the euphoria get the better of me, it needs to be said that although sapienti sat shares the same category with Slashdot, we ended up number 743 on page 38… Only with the advent of Internet and search engines could the Top 10 lists evolve into the top 2,780:

A further humbling fact is that the Google ranking of the blogged home page is the same 5/10 as most posts in this blog and the inner pages have no ranking at all. According to the WayBackMachine the site in its current form is too young to be cached (less then 6 month old) so maybe they will soon get to the higher rankings – especially with the help of all those people whom they ranked… So thank you blogged, I appreciate the attention!

I recently came across a SearchSOA article, which talked about Burton Group’s research that found 50% failure and 30% “non success” rates of Enterprise SOA undertakings. Not much of a surprise really, unless you start comparing the expected 20% “payout” odds with some common reference points, like, for example, winning the lottery (which as everyone knows is a complete sham). One of the authors referred to this phenomenon and the resulting shift in attitudes towards the Service Oriented Architecture as “SOA fatigue” – a very appropriate term, which immediately brought associations with the yuppie flu and also reminded me of a recent conversation that I had with a CIO of a global company. When he asked me out of the blue “so what is it exactly that you specialize in?” the only thing I could come up with before the elevator doors opened was: “I help organizations which are disappointed in SOA”. That’s where that conversation ended (we had arrived) so I have no data points on how well this worked. However, the more I think about it, the more it seems that this really captures the focus of my work over the last several years. The work that started with a question: “what would it take to build an SOA platform of which no one could say that it is lacking anything true SOA should have?” and culminating in building the delta between the answer to the above and our platform of the day. So starting today, I decided to take it as my professional motto. Here it is again in bold:

I help organizations which are disappointed in SOA

Corny? – Yes. But at least not as obvious as “I sell houses” which my realtor had on his business cards, fridge magnets and “Welcome Home” slippers embroidered with his face and contact details…

Two roads diverged…

In the last week’s post, I asked a question: “is Governance worth the trouble?” and left a cliffhanger for an answer that it depends on how one looks at it. Today, I will try to untangle the issue further. But first let’s look at the diagnosis. Imagine yourself as a psychiatrist who encounters a patient that exhibits the following symptoms:

1. is perceived very differently by different observers
2. exhibit drastically different behaviors under identical circumstances
3. possesses some advanced capabilities one day and completely lacks them the next
4. radically changes the ways in which it relates to and interacts with the others

If you are an American^[1], you will undoubtedly diagnose the patient with the Multiple Personalities Disorder. Wouldn’t you agree that all of the above apply to SOA Governance? I admit that at some point the parallels end: there was no documented early abuse, hypnosis or memory loss, but in my book there are enough similarities, to warrant looking at Governance from this angle. And, as in real life, the doctor needs to identify, isolate and unravel each of the personalities before they can make progress with their patient, let’s try the same approach with ours. Now that we’ve got Governance comfortable on our couch, let us start with the history…

Earlier this year I attended a webinar, where Anne Thomas Manes, the recognized matriarch of SOA Governance, was warming up the crowd for a pitch by HP/Systinet, where she used to be the CTO. She started by saying “SOA Governance is about risk mitigation” which almost made me to do the Parisian Tailor (leave), but instead got me thinking. She is the one of the most recognized authorities in the field, so she can’t be wrong. Yet how could she be right, and talking about the same Governance to which I have dedicated the last several years of my career? Imagine yourself hearing one of the German car manufacturers to say: “Automobile is a means of ground transportation" – hard to argue with, true, yet somehow completely unfathomable. But there is a number of car companies, that live and compete (although perhaps not with the Germans) by that formula. That is when the idea for this article first hatched. Perhaps the theological disputes about the nature of Governance (how many policies can dance on the head of a PEP?) miss the point that the latter presents different personalities to different observers.

In my professional observation I identified two distinct personalities, which I named the Hauler and the Builder as a reference to one of my favourite parables attributed to Roberto Assagioli. In it a traveller walked past a construction site and saw three men doing the same work. He stopped and asked them: what are you doing? The first one replied – “Hauling stones”. The second – “Earning a living”, and the third one – “Building a Temple”. When I ran a recruitment operation, I used to tell it to some of the candidates when critiquing the resumes and discussing interviewing techniques. The diagram below illustrates how the two personalities manifest themselves to the main constituencies: technology vendors and customers.

Customer	Builder		Vendor
	Strategic > Transforming Services into Business Assets; > Building Enterprise-grade SOA Ecosystem;	Offensive > Delivering on the original SOA promise; > Creating strategic opportunities;
	Tactical > Adding Security, Logging, etc to web services; > Virtualizing endpoints; > Creating System of record;	Defensive > Closing gaps in product lines; > Checking off the boxes on RFPs > Placating the analysts;
Point of View	Hauler		Point of View

If I had to summarize each personality in a single word, Builder would be transformative and Hauler – incremental. Not that there is something wrong with the latter: both represent steps into the right direction, and any kind of improvement is way better then none.

Now, when we have identified the personalities we can analyze them in a bit more detail. Builder is a visionary and likes to relate to the same: given a choice she would go straight to the CIO/CTO. Hauler is a pragmatist, and prefers those who would benefit immediately: operations, security, etc. Builder tries to emphasize flexibility and potential, while Hauler concentrates on convenience and stability (often coming as an appliance). Builder is a generalist, trying to address all aspects of Governance in the broadest possible sense, Hauler usually tries to narrow down the problem, do one thing and do it well. Builder likes to talk about business challenges and opportunities; Hauler concentrates on technical capabilities and supported standards. Builders often see themselves as unique and irreplaceable, while Haulers tend to seek strength numbers and form or join all kinds of unions.

In retrospect it makes sense that Anne Thomas Manes chose the Hauler, when she introduced Systinet, who is the perceived leader amongst Service Registries, and thus most interested in maintaining status quo. This leads us to the final question: which of the two you (as a customer, or perhaps, a vendor) should be dealing with? The answer, as always, is highly individual but for me it feels too crowded on the Hauler’s side. And we can easy guess whose pay is better

¹ Until very recently, the phenomenon whose proper name is Dissociative Identity Disorder was almost exclusively confined to North America.

There seems to be no way to keep an easily accessible index of entries beyond the most recent ten. So I thought I would organize a table of content entry and try to keep it close to the top - although in order to achive this I will have to periodically delete and republish it again. This will also give me a chance to provide some kind of categorization that would be more relevant then chronological.

Conceptual

1. SOA Governance Defined - attempt to come up with definition that would make sense to business.
2. SOA governance – the perspectives - a summary of what others (analysts, customers, vendors) call Governance.
3. SOA Governance Terminology and Positioning - a critique of existing SOA Governance terminology and attempt to introduce some improvements.
4. SOA Governance Primer - an household example to illustrate some key concepts and differentiators of our approach to SOA Governance.
5. Service Classification - describes the approaches to service categorization and its importance for SOA
6. Adventures in the Magic Quadrant - thoughts about the latest (and as far as I know, the first) Gartner's Magic Quadrant for SOA Governance and how it reminds me of my first disaster movie.
7. Carbon Dating SOA Governance - an attempt to find out who first came up with the idea and when did it happen.
8. Demarcating between SOA Governance and ESB - a look at the similarities and the differences between SOA Governance and ESB problem domains and attempt to look at them as difference sides of the unified service mediation function.
9. Why Bother with SOA Governance? - a look at the role and importance of governance today from the point of view of an SOA vendor.
10. Multiple Personalities of SOA Governance - a humorous attempt to understand the widely different perceptions of SOA Governance through psychiatric diagnosis.

Architectural

1. SOA Governance Evaluation Guidelines - general principles which might be useful when selecting or architecting a comprehensive SOA Governance Solution.
2. Aspect-Based SOA Governance - introduction and advocacy.
3. Delegated Governance - introduction and advocacy.
4. Analysis-time Governance - introduction and advocacy.
5. Service Refinement - introduction, use cases and advocacy.
6. Brownfield-friendly Governance - introduction and advocacy.
7. Governance Information Model - overview.
8. Service Lease - introduction, advocacy and implementation details.
9. Aspect Enforcement - describes what kinds of aspects are found in an enterprise governance ecosystem, how they are used to define Governance Contracts and introduces the taxonomy of Aspect Enforcement Levels.
10. find-bind-execute or what u-boats and 20-engine planes have to do with delivering on the original promise of SOA
11. Defining an STU - defining the scope and requirements for an enterprise-grade Service Taxonomy Utility
12. SOA Governance 1.1 (1.2, 1.5, 2.0…) or Why is it so hard for Governance products to do Service Versioning right? - discussion about proper service versioning fo SOA.
13. To Common Information Model, or not to … - a discussion about the role of Governace in transitioning from JaBoWS to Enterprise SOA, the need (or lack of) for Common Information Model, and the difference between Governace Aspects and systemic -ilities.

Security

1. SOA Security Challenges - on importance or SOA Security and how it differs from the other kinds of IT security.
2. SOA Security Standards - part of the solution or the problem? Probably both…
3. A Formal SOA Security Model - introduction to a formal model and notation to talk about SOA Security.
4. Security Model Details - part II of the description of the SOA Security Model.
5. SOA Security Vision - a conclusion to [this iteration of] the SOA Security Series that defines a common set of requirements for a SOA Security Platform and provides some recommendations and best practices for implementing Secure SOA.

An insight into the obvious

I recently gave an in-house presentation with the same title, which boiled down to the following reasons:

• A recognized must-have for any SOA initiative today – when I started “selling” SOA Governance three years ago, I had to dedicate a good third of my time to proving that it is important and explaining what might happen to SOA implementations if were to be neglected. That part has been steadily shrinking with time and in the last year most audiences asked me to skip it altogether.
• Included in almost every RFI/RFP – even if the word “Governance” is not mentioned there (which is becoming less and less frequent these days) the underlying concepts, like security, monitoring, throttling, QoS, compliance, etc are found in the majority of requests that have anything to do with SOA: ESB, integration, B2B, composite applications, BPM and are creeping into even tangential subjects like Master Data Management.
• Customers keep asking for advice and position – it is harder and harder to come across at true greenfield environment or a virgin customer when it comes to SOA. So instead of delivering 101 classes and telling people to build their SOA our way, we have to deal with issues that have arisen while our customers were building their SOA Ecosystems their (or our competitors’) way. And in my experience they often tend to gravitate towards Governance (or the lack of such).
• Entry point into unassailable accounts – I once took part in a meeting with a global Systems Integrator aimed at devising a joint go-to-market strategy and observed that the SI’s marketing folks were much more interested in the Governance part then the rest of the SOA stack combined. When asked why, they explained that very often they find themselves facing a CIO who several years ago had standardized on a competitive (let’s for argument’s sake say: Tibco) platform, and just approved an additional multi-million dollar investment into that technology. If they start pushing our story (read: platform), then, regardless of the actual merits of the technologies involved, the most likely outcome would be an abrupt end of the audience and likely long-term loss of credibility. If, on the other hand, they congratulate the CIO on a wise decision and carefully steer the conversation towards Governance, they are much more likely to increase their credibility and establish an important beachhead. I personally have successfully employed this strategy on a number of otherwise hopeless accounts.
• Crown Jewel of SOA – this is another (less sales-centric but perhaps as cynical) way of looking at the above argument. An SOA implementation would never be complete without the keystone of Governance installed into it. This parallel is especially appropriate because Governance usually sits at the top (amongst the blocks on the 100,000 feet view architectural diagram – which is the only picture that really important people are likely to see) and because, unfortunately, it is often installed at the very last moment. On a more serious note, I have long maintained that Governance in the defining characteristic of SOA, because it is ultimately what will allow it to deliver on its original promise(s) and succeed where other technologically-similar approaches did not. So as far as the perceptions go, once you installed you Governance centerpiece at the top, it almost doesn’t matter whose technology is doing all the work at the bottom.
• Powerful differentiator – this one is fairly obvious as it not only separates those who have from those who have not, but highlights the completeness and cohesion of the entire SOA stack. Companies that deliberately built an SOA platform are much more likely to have a native solution then those trying to repurpose technologies from another domain.
• Reputation to uphold – through the SeeBeyond pedigree, we have a reputation of innovation and thought leadership in the SOA arena. Customers come to us not just for our products, but also seeking advice, guidance and vision. Many of our Service Offerings related to SOA aim at helping clients to transition further along the SOA maturity curve, which (as I hopefully proved above and by this entire blog) requires to have Governance in the picture.
• Because we can!

So what is the point of all those profound revelations? To me is to answer the question: is it worth the time and money for someone who is not a recognized player in the governance space to attempt becoming one? Especially if that someone is a recognized player in SOA as a whole. The anecdotal evidence seems to point to “yes”: for the past few months Tibco was actively advertising for Governance experts on various job sites, and the query for non-blog entries on microsof.com which until less then a year ago returned just a single paragraph about AmberPoint partnership, today yields respectable 376 distinct entries.

My take on this is: it depends on how one looks at SOA Governance. Which is the topic of my upcoming post.

A first post into the newly-minted Retorts category, this is in response to the comment left by Michael Poulin (I assume the one of Fidelity & МЭИС fame) on the article SOA Governance Defined.

Fortunately, I do not expect everyone to be my fan, so we are cool here. I would have been saddened by the “mess” you found in my blog (I hope people find my writings to be well-reasoned and somewhat organized), if I have not learned earlier on how to translate English-to-English through Russian). So I assumed that you just disagree with my positions and line of reasoning. Now let us get back to the substance. Yes, I am familiar with OASIS SOA RM (I personally would not call it a standard just because it emanated from a standards body) and find it of very little value or relevance to the problems faced by the SOA practitioners around the world. Which is perhaps why there has been so little movement in this area since the document was released nearly two years ago and there seems to be no plans to update/evolve it in future. And that document does not seem to mention the word “Governance” even once. One of my favorite quotes is that Camel is a horse, designed by committee… So the definition of Governance as “deals with policies and regulations” must have come from some other source – you? An anecdote comes to mind: Plato once defined a (hu)Man as "a biped, without feathers" (animal bipes implume) so as my plucked chicken I would point out that the above definition describes many things from a humble school board to the Holy Mother Church. And policy without enforcement sounds so very … Russian! (for those of you who are far from Russian culture, I am referring to a famous quote by Saltykov-Shchedrin “the severity of Russian laws is alleviated by the lack of obligation to fulfill them”.)

I coined the term “Aspect-Based Governance” (please note -based rather then -oriented) to denote the distinction with the industry’s approach to make the decision for the customers what it would be reasonable for them to govern. The term seems to have caught-on and I have seen at least a couple of vendors and a global SI using it in this sense. If you were to read the article you would see that it has nothing to do with AOP – one thing I agree with wholeheartedly is that Governance, as well as the rest of SOA, should remain implementation-agnostic. Leaving aside the idea that AOP equals to Spring (last time I checked there were languages other then Java), let me reiterate that I was far from suggesting AOP-style code injections into the service implementations which would require a naïve assumption of service implementations’ language, platform and belonging within a certain domain of control. I too am a believer in the gateway architecture for governance, or maintaining the separation between the service implementations and PEnfPs. Furthermore, as a pioneer of delegated governance, I advocate the separation between the latter and PAP (A-for Application) and PEvP (Ev-for Evaluation).

The latter part of your comment contains many unequivocal statements which make polemics somewhat difficult:

• “SOA Governance is a set of domain specific policies and regulations” – perhaps to you it is. And to me it is a way to transform services from digital artifacts into business assets. And to Miko it is something altogether different. Leaving the “minor” issue of enforcement and compliance out of Governance altogether not only threatens employment security of all those poor soles who are working on run-time governance technologies, but reminds me of another (I promise the last in this posting) historical vignette: once Descartes was asked to give the guild of Parisian tailors a lecture on solving the problem of optimal cutting of the cloth. He started it with drawing a circle and the words “for the sake of simplicity, let’s assume that human body is spherical” and was genuinely surprised when most of the audience left immediately. I guess the final test of relevance and usefulness of various definitions would be how much help they give to those struggling with SOA in the real world.
• “I do not think it is possible to regulate such things [non-functional service characteristics] at the enterprise level” – perhaps, but I was able to build a solution which does exactly that. And based on there I think you live, you might have even used it by now...  
• And I do not see a big paradox between the need to define NFSCs in the service contract and specify them through Governance solution. Ever heard of service virtualization and WSDL processing? And what about the need to expose the same service with two (five!) different sets of policies to cater to different Service Consumption Scenarios? If this paradox proves anything, it is the lack of relevance of that RM to the real world of Enterprise Architecture.

Michael, I hope I was able to respond to all substantive points in your critique, and if you are interested, would invite you to continue this discussion.

I am back after a prolonged hiatus caused by the increased workload which is not totally unrelated to the recently-announced Project Hydrazine. And I would like to start with creating a new category in my blog dedicated to the continued discussion with my readers who took time to leave lengthy and often thought-provoking comments. I feel that every comment should be answered and find regular posts to be a more appropriate media for a good retort then the constrained confines of the comment box. It’s my blog after all! So standby for more rhetoric…

Asking the eternal rhetorical questions

I recently came across a really interesting post JaBoWS is the Enemy of Enterprise SOA by Nick Malik from Microsoft. And I could not resist to comment on it. Nick was kind enough to reply with the most detailed and relevant comment I ever got, and as I started my counter-reply I realized that is deserves its own entry, so I will be posting trackbacks to both comment streams.

It seems that the gist of Nick's response was to build a case for a common (presumably across the entire enterprise) information model. Such a model would be a Really Good Thing. So good in fact, that only a mean argumentative person like myself, would ever even try to argue against it. But like other similarly Good Things (e.g. the Word Peace or the Enterprise Data Model from the 90s) it would be very hard (read - impossible) to achieve in practice for any non-trivial set of circumstances. So what shall one do - give up? No, I do not think so. A solution that I have built and would recommend to the rest of the world is Refinement - if you look at that post, I actually provided some use cases that fairly similar to the one which Nick have proposed.

So if one has the luxury of time and resources to stop the onslaught SOA in their organization while the experts analyze all the requirements and define a common information model to end all ambiguities - congratulations, you found a perfect solution to this really difficult problem. If, however, you live in the world with the rest of us and this is not an option, then perhaps the imperfect solution of Alpha building the original set of services based on their understanding of the world and just a dash of common sense and Bravo evaluating each service for a "fit" and taking the appropriate action might work just as well. Once the degree of fit and the impedance mismatch are clear, they will have a number of options:

1. to reuse the service as-is /* which I believe would be very rare */,
2. to reuse the functionality and define their own non-functional characteristics /* also too good to be common in non-trivial cases */,
3. reuse most of the functionality by refining the existing service with their own Refining Method(s) /* this is where I see the bulk of cases */,
4. evolve the service though versioning to serve the now expanded set of requirements /* this would be fairly common as well but require a higher degree of coordination between Alpha and Bravo and thus greater level of SOA Maturity across the organization */,
5.  or make an informed and responsible decision not to reuse, which, if you look at it from this prospective, is not a bad choice at all.

And a viable SOA Governance platform would allow them to make any of those choices (and to change their mind later) through its Service Virtualization, Versioning and Refinement capabilities. So going back to my argument that SOA Governance is what separates Enterprise SOA from JaBoWS and other cheap imitations, since I take the broadest possible view of the governance that includes all those things, I believe it still stands. For more on where in my view SOA Governance ends (or does not) have a look at this post.

And finally, in response to the second comment – no, I do not believe that a successful implementation of SOA governance (by the way, I was referring not just to design-time governance, but to the entire thing including analysis- and run-time as well) requires adherence to an enterprise wide information model or any other hard-to-achieve practices. And with regards to the non-functional characteristics I was referring to versus the System Quality Attributes as described here - I believe they are quite different. The latter represent the -ilities that are primarily of interest to the technical side of the house (architects, developers, operations and the rest of IT), while the former represent the things that are of interest to business today. And while there might be some overlap: e.g. security and availability are likely to end up in both sets, I doubt that SOX compliance (or, say, Patriotism or Political Correctness if our legislators ever get their way and codify them as regulatory requirements for businesses to adhere) will ever make it to the first list, and Maintainability or Portability into the second. Furthermore, I emphasized the word today to highlight the fact that governable non-functional characteristics are very fluid and are likely to change with time, geography or vertical, the set of -ilities is quite stable and I have not noticed significant changes since I started thinking about those things in early 90s.

Now that we have established what Enterprise SOA is, shall we just go and build it?

Walking the line or blurring the edges?

Ab ovo

Something I have struggling with for a long time has recently come to me in a flash. Well, almost. Ever since we started showing SGF to the customers, we were inundated with various feature suggestions and enhancement requests. And one of the most persistent ones (which is incidentally the one I have been resisting most adamantly) was for dynamic routing. SGF has provided some specific forms of routing from the very beginning: between bindings, between versions, between SDLC environments, to refinement methods, etc. But apparently that was not enough, what “the people” demanded was arbitrary content- (as in: route trades with four letter security tickers to NASDAQ and with shorter ones to NYSE) and context-based (as in: route requests to the Chicago servers between UTC 14:00 and UTC 03:00 and to the Hyderabad servers from UTC 03:00 to UTC 14:00) routing capabilities. And the reason I resisted including those was my belief that such functionality represents business logic and thus clearly belongs to an ESB and not to a SOA Governance platform. However, I was usually presented with counterarguments along the lines of: but SGF already has dual binding architecture that allows to mediate between different transports and protocols, and this also is clearly an ESB feature. In fact if one looks that the high level architecture, it looks remarkably similar to those of typical ESBs.

And I would reply about the woes of maintaining business logic in two separate places and the debate would go on. To tell you the truth I was motivated not only by the purity of vision but also by the political considerations like trying to avoid competing with our flagship SOA product, that happens to be an ESB. And in the interest of full disclosure I should say that at the end I gave up and dynamic routing was included as a first-tier feature into the plans for the next release of SGF.

State of the industry

While I was fighting with myself for architectural purity and anguishing over the moral dilemma what would Turing do? the world around me was quietly moving forward. The Governance products were adding more and more ESB-like functions including: routing, arbitrary payload transformations, load balancing, retries, etc. At the same time the ESB products were starting to incorporate more and more features that are traditionally associated with governance platforms. For example Open ESB project has been working on (or at least considering to add) support for aspects for some time. Out of the 27 aspects they have on their wish list, half represent clearly Governance-related functions: Throttling, Logging, Lease, Message/Content Filter (a.k.a. censorship), Validator, Chargeback, Monitoring, Time to Recover, Regulatory Compliance, Audit Trail and Security. I believe that having realized the gap in the underlying JBI specification, the group working on the JBI 2.0 spec intends to add some “Message Exchange handler/interceptor” to codify this architecture. Ironically the first governance-related standard will most likely appear in the ESB space.

So although as both Gartner and myself agree, the value of SOA Governance capabilities embedded into an ESB or other SOA platform is to put it mildly “limited”, as these capabilities mature, it will only be a matter of time before the ESB folks realize that they can export these capabilities outside of their own realm and contend to govern the entire SOA ecosystem. Another anecdotal evidence of this trend is Mule Galaxy a self-proclaimed SOA Governance Platform which is in fact a basic Registry/Repository notable mostly for the interesting choice to expose Atom Publishing Protocol (APP) API instead of supporting any of the traditional registry standards (UDDI, ebXML or JAXR). The interesting thing there that the list of supported governable entity types includes as many SOA platform artifacts (Spring and Apache CXF descriptors, Mule’s own ESB configurations, etc) as traditional metadata types (WSDLs, policies and schemas). To me this means that ESB vendors (I have seen similar features in BEA and Iona offerings) are starting to see themselves not only as purveyors of SOA Governance capabilities but also as first-class users of those as well.

The Epiphany

happened couple weeks ago when I was talking with executive team of a very promising newcomer into the SOA Appliance marketplace. In my mind their product is the closest to what I see as a true SOA governance platform in this space. It also happens that many of their ideas, capabilities and architectural decisions are remarkably in-tune with those that we put into SGF (but this is beyond the point of this particular essay). I was really stunned when their SVP of Business Development told me that they see as their main competitors not the other appliance vendors (Layer 7, Reactivity/Cisco, Vordel or DataPower/IBM) but ESBs. And then it hit me – like the Unified field theory these similarities and mutual encroachment between ESBs and SOA Governance solutions is not coincidental but stem from the fact that they are trying to address the same fundamental problem coming from different directions.

And the name of this single underlying problem is

Mediation

It’s really simple if you look at the problem from this point of view. The gist of what SOA Governance delivers is mediation of non-functional characteristics of enterprise services, often referred to as systemic properties. So if someone wants to be even more succinct they can define SOA Governance as systemic mediation. The gist of an Enterprise Service Bus on the other hand (and I am referring here to the true original meaning of this term and not to the integration products or BPEL engines adorned with adapters and management console that are often marketed as ESBs) is all about dynamic routing of service requests and replies between consumers, producers and intermediaries, based on content and contextual considerations. If I had to use just two words to describe what ESBs do it would be semantic mediation of SOA services.

If we add the third piece to this puzzle: technical mediation which covers the mechanical functions like transport, protocol and synchronicity adaptation and service versioning, and is common to both domains, the Unified Theory of Service Mediation will look like this:

These mediation functions are essential for successful realization of any not-trivial SOA ecosystem. The cleanest way to implement them would be based on a universal service mediation platform (most likely a gateway) that should be designed to handle all flavors of mediation evenhandedly and harmoniously and would be based on cohesive common architecture. From the delivery point of view, such a platform can be offered in a number of “trims”, i.e. “full service” mediation engine, systemic-centric Governance product, semantic-oriented ESB, and minimalistic service gateway that will be limited to just technical mediation. Such solution would clearly be a “killer app” for SOA and it is unlikely to emerge from any of the existing ESB or Governance offerings which are too entrenched in their own ways of looking at the service mediation problem domain. And that, my friends, opens some really exciting possibilities…

Why is it so hard for Governance products to do Service Versioning right?

In almost every SOA Governance-related RFP that I have come across service versioning appears near the top of the list right after security and monitoring have been dealt with. And the requirements are fairly consistent (as outlined below), why then some many Governance offerings have gaping holes in this area? I have heard vendor responses ranging from “we can capture version number as metadata” or “you can register several versions of a service separately and use ESB to route between them” to the extreme “there is no need to version services – if anything visible to the client changes, you just register a new service”. None of which seems to be useful to an organization that want to stick with SOA after the pilot project was declared a success. I think I have found the underlying cause, but let me first start with the

Basic Service Versioning Requirements

1. It should be possible for providers to publish and consumers to use multiple versions of the same service at the same time.
2. They should be clearly identifiable as versions of the same service.
3. It should be possible to designate the default version of a service, which will be invoked if no specific version has been requested. Default version doesn’t necessarily have to be the latest one.
4. It should be possible to designate compatibility relationships between service versions. “Version A is compatible to version B” means that A can process requests intended for version B and return responses that the client of B will understand. In other words it is safe to route to A requests intended for B.
5. Clients can request specific versions explicitly by passing version number with the request or implicitly by using the request that can be resolved to a specific version.
6. If explicit version number is not provided and implicit resolution fails, requests will be routed to the default version.
7. If requested version can be identified and that version is available, request will be routed to that version.
8. If requested version can be identified and that version is not available, but there is another version that is designated as compatible to it, request will be routed to the compatible version.
9. If requested version can be identified, that version is not available and no compatible version can be found, the request will be rejected.
10. Behaviors described in #1 through #9 should be available declaratively (through configuration, not coding).
11. It should be possible to implement custom versioning aspect that would be able to enhance or override behaviors described in #5 through #9. For example if requested version is unavailable and no compatible versions have been specified, such aspect instead of rejecting the request can transform it to a form that can be understood by one of the available versions. A corresponding transformation of the response would occur on the way back.
12. The system should advertise the WSDL corresponding to the default version, however clients should be able to view and retrieve WSDLs for all currently active versions.
13. The system should distinguish between non-disruptive (adding a new service version or retiring a version for which a compatible version exists) and potentially disruptive (retiring deprecated version, changing the default version, changing existing version numbers) actions. The versioning mechanism should be integrated with the change management mechanisms such as Lease and Notifications and should be able to trigger corresponding functions following a potentially disruptive action.
14. When user designates version as compatible to another, the system should be able to perform basic validation (check for missing operations, incompatible massages, etc.).

Nothing too difficult or complex, isn’t it? Why is it then so hard to come by a solution that meets most of these requirements? My theory is that

Registries are to blame!

Here is now – in the registry information models (both UDDI and ebXML Reg/Rep) Service is on the top of the food chain amongst the functional components (services can belong to an Organization, but the latter is purely a business entity encapsulating people and contact information) representing both primary and highest level entity. But in order to provide the capabilities described above one need to have a primary entity representing individual invokable versions and a higher-level construct (uber-service if you will) that will represent the logical service with all the versions and relationships between them. It does not seem possible to map such model to UDDI or ebXML RIM and remain “standards compliant”. This is why you will not find adequate support for logical service versioning in registry-centric products. They can version things: WSDLs, schemas, policies, etc but not the SOA Services, as defined here. The run-time centric solutions in theory do not have this handicap, but most of them joined various interoperability frameworks pushed by the registry vendors, to be able to check off the compatibility boxes and had to adapt their information models to those defined by the registry standards. This is why when designing SGF we decided to stay away from all the “standards” existing in or around SOA space and define our Governance Information Model based on the analysis of the problem domain and typical customer requirements, rather then various legacy compatibility considerations. The versioning-related part of the resulting model looked like this: Where:

• ServiceOffering represents the entire consumable service form the client’s perspective with all the functional and non-functional properties;
• ServiceComponent represents just the functional part with all the versions, virtualizations and refinements;
• ServiceVersion a single virtualized version, which in turn is an aggregation of individual Operations.

Needless to say that a governance solution based on such model can easily meet all of the versioning (and many other) requirements.

Like most people I like to get into things early, so for the last couple of year as the governance was moving towards the inflection point, we were brainstorming what would be the next big thing in SOA. And I am pretty sure it will have something to do with semantics. Don’t get me wrong I am not jumping on the bandwagon of semantic Web 3.0 whatever it means – it’s crowded enough there without me. I am talking about strictly about SOA. I see several things how SOA can get semantic and I touched on some of them in my blog: through meaningful service categorization, dynamic discovery and invocation and meaningful mediation. I remember after a two hour presentation about our SOA Governance solution, a customer asked me: “Ok, you have pretty much cornered the Quality of Service issues, but what about the Quality of Mediation?” I am still not sure what this means, but it sounds way cool!

In the spirit of this I have submitted a proposal to the 2008 Semantic Technology Conference which will take place in May in San Jose, California. If I am lucky, I might see you there.

We were pretty early on to catch up on the concept of SOA Governance back in 2004, however we too did not start calling Rose by that name till the fall of 2005. So who did come up with the term and when? Now, when my favorite Meme Miner stopped working I can not find an easy answer, so I had to do some research and speculation.

I started with the next best thing – the job trend, which by my pure gut feeling lags some 18 months behind the idea, shows the very first blimp in January 2006, suggesting dating the idea to mid 2004.

"soa Governance" Job Trends

"soa Governance" jobs

Another anecdotal evidence is that the first SOA Governance page appeared in Wikipedia in November 2006. Next, if you look at the pure play vendors that inhabit the magic quadrant today through the waybackmachine you will see that AmberPoint started using this term in the fall of 2005, Systinet in January 2005, SOA Software sometime in early 2005. To my knowledge the credit goes to Infravio which first used a very close term Web Services Governance referring to the “process of controlling and tracking providers, consumers and Web Services usage at every step of the lifecycle” on May 12, 2004. And finally, looking at the analyst sites, the first result on Forrester is dated January 2005 and on Gartner September 2005

How Gartner’s Magic Quadrant for Governance reminded me of my first disaster movie.

I recently come across the latest (and as far as I know the first) Gartner’s Magic Quadrant for SOA Governance. This is definitely a good trend – I see this as recognition of the importance of Governance for the overall success of SOA and a statement that these technologies are important by themselves, not as a set of capabilities of an ESB or the SOA platform of your choice. In fact they specifically excluded a prominent SOA vendor, based on the fact that their Governance technologies are only intended for their own SOA suite. I have always maintained this point of view, perhaps in a less diplomatic manner: since no SOA vendor in their right mind (however cocky) can assume that the entire SOA ecosystem will consist only of services and composite applications built on their platform, and furthermore, all of those deployments will remain within a single domain of control (i.e. if both you and your trading partner use the same platform, they will allow you to connect your management console to their ESB – yeah, right!) governance capabilities built into such suits are useless beyond PoC or an initial pilot phase. The reasoning behind it is that if someone believes in the importance of SOA Governance they will not leave ungoverned services that are developed on different platforms, exposed out of packaged applications or provided by external entities, so they will have to also invest in a standalone governance technology to cover all of these. At which point they have to either live with two completely independent and most likely non-interoperable solutions of the same problem, or most likely stop using the built-in governance capability of their SOA platform and standardize on the universal solution. Point proven (I think).

I would also agree with their overall assessment that there is no clear leader in SOA Governance yet and that in most cases even if a vendor has all the areas of SOA Governance covered, it is done by a menagerie of products that came from different pedigrees and share little or no code and underlying architecture. Which makes me quite pleased with the decision we made three years ago to build SGF as a single suite, separate from Java CAPS (then known as ICAN), based on a uniform architecture and shared code… – Sorry, could not resist here

And now, getting back to the original subject of this post. Overall this report would have provided a fairly compelling and informative reading, if only I knew a little bit less about the topic and some of the vendors and products it covered. This reminded me of the film The Crew (Экипаж) which came out when I was in high school. It was the first soviet disaster film with all the trimmings: great (for the time) special effects, engaging plot, some character development and even a hint of erotica – unheard of in the era of late soviet Puritanism. The plot is about a crew of a soviet airliner, who land their plane in a foreign mountainous city called Bidri, which was damaged by an earthquake, which is then hit by another even stronger quake. Their jumbo jet is severely damages as they take of from the spectacularly disintegrating airport, and they crew repairs it in flight by exiting outside onto the fuselage through a stopped turbine. As you would expect, everything ends well. Everyone I know (myself included) loved this film. Except my English teacher, who went to see it with her husband a veteran airline pilot, who spoiled every suspenseful moment with his comments about accuracy and plausibility of things happening on the screen. I imagine the same would be true for an American taking a systems administrator to The Net or Firewall.

So, what does it all have to do with the Magic Quadrant? Well let’s see:

• One vendor did not provide in time information about the technologies they have, but they were still included and evaluated based on a product they did not have;
• Another vendor whose known weakness is that their product is completely self-contained and standalone got kudos for federation;
• An organization whose only public message on Governance is “use our consultants to teach you best practices” got into the “visionaries” crowd. My guess is that they were evaluated on the products they are planning to build, not the ones have right now.

I have been deliberately vague in the above statements for what I hope are obvious reasons, but if you are interested in more details, write to me and we could discuss things a bit further.

What does this all mean? Perhaps, just that I am not the right audience for these type of document, that’s all.

Post-Scriptum: couple of weeks after publishing this piece I have come across the entire Magic Quadrant in question available on the web. I hope it was done with permission and would not be taken off-line. Please read for yourselves.

Let's try to do it right

In a previous post we tried to scope the problem of service categorization, identified the common traits of successful taxonomies and pointed out some limitations of existing approaches to service categorization. Today I wanted to leverage all this as a basis for defining the scope and requirements for an enterprise-grade Service Taxonomy Utility or STU. Let us start with some basic

Assumptions

about the perceived users of service taxonomies.

1. Each service provider wants to maximize the usage of their service. This does not have to be number of invocations and might mean number of consumers, uses or simply to prevent duplication. Service providers usually fulfill some need by publishing a service's availability. This could be a business need, such as usage charge, support for other revenue streams, or just good will towards partners. Service provider could also be motivated by non-business reasons, such as regulatory or policy compliance. In either case people would not be promoting their services if they do not want them to be found and utilized or at least considered.
2. Service consumers build applications that invoke services to satisfy specific needs. The searching that is done at design time or runtime by a service requester is not based upon casual curiosity, it is intended to find the best; service available to invoke for a particular purpose.
3. There can be no single taxonomy to classify all possible services from every point of view. Many taxonomies of services will coexist on multiple levels.
4. Services can appear in one or more categories in one or more taxonomies. Different consumers will use different taxonomies, that best match their view of the service domain and will follow different paths to get to a service.
5. Finally, only humans can determine what business need is satisfied by a particular service. Artificial intelligence techniques for the foreseeable future are no substitute for human judgment in determining, based on the description of the category, whether the services within that category match the required business need.

Based on this foundation we can define a set of

Functional Requirements

for a Service Taxonomy Utility (STU), which can be used in SOA implementations on enterprise, industry and global levels.

1. STU should support multiple hierarchical taxonomies.
2. STU should support non-hierarchical (“see also”) relationships between nodes.
3. STU should support typed relationships between nodes in taxonomies. The types should be user-definable but should be able to express generalization, aggregation, refinement, type-of and equivalence.
4. It should be possible to annotate nodes with plain text and XML descriptions.
5. STU should allow to import an external taxonomy.
6. STU should support unique identification of taxonomies, nodes and relationships.
7. STU should support federation by stubbing internal or external taxonomies to nodes in other taxonomies.
8. STU should support versioning of taxonomies, nodes and relationship types.
9. STU should support deprecation of taxonomies, nodes and relationship types. Users can browse and search in deprecated taxonomies and nodes, but service publishers should not be able to publish into them.
10. STU should support both direct (nodes-to-entities) and reverse (entities-to-nodes) referential models. This would ensure that category-based searches will be supported with booth cooperating service registries and repositories, which can express relationships to the categorizing taxonomy nodes as part of the metadata associated with services and with non-cooperative ones.
11. STU should provide security. There will be three primary types of actors using the STU: administrators who create and maintain taxonomies, publishers who register their services in the defined taxonomies, and users who browse the taxonomies to locate the services they need. Security requirements for each type of actor are outlined below.
    1. STU administrators should be able to create and modify taxonomies, and delegate administration rights to individual nodes and sub-trees.
    2. Service publishers should be able to register services to taxonomies, individual nodes and sub-trees.
    3. STU Users should be able to browse taxonomies and search within them. The read access should be granted on per-node basis. The users will not be able to go into or under the nodes to which they do not have access.
12. STU should provide GUI-based and programmatic access to all functionality.
13. STU should provide notification capability to inform interested parties when certain changes occur. Users should be able to register to receive notifications by event scope and type. The scopes include taxonomy, node, sub-tree; and the event types: service published, unpublished, object deprecation and change.

So what?

That is a perfectly valid question. I do not think that a COTS STU solution exists that meets all of this requirements and can be used out of the box, so I see two possible roads that lead to the desired end result: build and adapt. The first one is fairly obvious and in my estimate in about 6 man-months a production-ready solution can be put together by a small competent team. Investing a man-year will deliver lots of nice bells and whistles. The second option is more interesting – essentially what I have described so far is a narrow specialization of an XML repository. In particular, I have discussed these requirements with Farrukh Najmi one of the authors of ebXML Registry Repository standard and the lead of the freebXML Registry project, and he told me that 95% of the above requirements are met by the version 3 of the Registry. I have not had a chance to verify this myself, but if anyone out there wants to try it, I would be very grateful to hear about the outcome.

Introduction

Interoperating services in many ways resemble distributed OO systems so many of the lessons learned from more then twenty years of evolution of object technologies might be directly applicable to the SOA domain. In the beginning when OO was rapidly gaining momentum many systems and languages that did not meet the full set of criteria declared themselves to be object-based by implementing some of the required traits. None of those solutions gained significant traction and success and they quickly sank into well-deserved obscurity. Although they differed in the set of OO features their creators chose to implement, they all had left out the one which was difficult to implement and retrofit: organization of objects into hierarchies of semantic classes.

Similarly the success or failure of services as the next paradigm of enterprise and global computing depends on the ways services are specified, described, advertised, classified, discovered, selected and consumed. Thus according to Steve Burbeck, the term service-oriented should be reserved for architectures that focus on how services are described and organized in a way that supports the dynamic discovery of appropriate services at runtime. Architectural schemes that focus instead on service-to-service message protocols, or on the details of how the various servers communicate rather than what they say to each other, should be characterized as service-based. The latter, although offering a significant improvement over monolithic applications, lack the capacity to scale beyond the sub-enterprise level when service consumers and providers are under control of a single architectural authority.

The ultimate goal of service-oriented architecture is to achieve robust service interoperability through dynamic discovery and binding at run-time. For this to work service consumers and producers must have both syntactic and semantic compatibility. The former can be achieved and validated through the use of XML technologies. The latter requires human intervention. However relying on humans to ensure semantic compatibility is slow, error-prone and since it can only happen at design time can not guarantee future compatibility. Machines on the other hand are not capable of tackling this task alone. The only workable solution to combine both approaches: humans should define semantically-significant categorization schemes, assert service semantics through categorization. Service consumers then will be able to identify relevant categories at design time, allowing computers to use them at run-time as target rich environments for semantically compatible services. Since these categorization schemes will be constantly traversed, they have to be organized into hierarchical directories, or taxonomies, allowing coarse-grained browsing and fine-grained category examination.

Definitions

A cursory search on Google finds multiple definitions for taxonomy:

• “A classification of ideas in an orderly hierarchy that indicates a natural or organizational relationship.”[Wordnet]
• “Taxonomy (from Greek taxis meaning arrangement or division and nomos meaning law) is the science of classification according to a pre-determined system, with the resulting catalog used to provide a conceptual framework for discussion, analysis, or information retrieval.” [Whatis.com]
• “A taxonomy is a hierarchical, structured presentation of information by categories.” [Burbeck and Graham]

All of these definitions highlight different facets and characteristics of taxonomies but emphasize their structured nature.

There are two closely related concepts: classification and ontology which are often used interchangeably. However according to Reinout van Rees there are some important differences. The difference between a classification and a taxonomy is that a taxonomy is always hierarchical and classifies entities in a structure according to some relationships between them; while a classification may be flat and use more arbitrary (or external) grounds. A great example of taxonomy based on internal classification is the Library of Congress Classification for books. An example of external reasons would be censorship that classifies books as politically incorrect, subversive or otherwise objectionable. Taxonomy differs from ontology in that the former only organizes knowledge, while the latter also serves as a repository for it.

Common Traits of successful taxonomies

Because humans comprehend information by categorizing it, there are multitudes of taxonomies being used in almost every human activity. They serve different purposes for audiences and are presented in different formats; however all successful have some common traits.

Taxonomies are hierarchical

All of the definitions in the previous section describe taxonomies as hierarchies. This is important to reflect commonalities larger than individual species. Such hierarchy can be represented as either a tree, where every node except a single root has exactly one parent; or a forest, or collection of one or more trees; or as a polyhierarchy which has multiple roots and allows nodes to have multiple parents.

As is the case with biological taxonomies, there is usually more than one way to categorize information, so for any domain there can be more than one possible taxonomy. None of these taxonomies would be sufficient by itself yet together they provide a powerful map for organizing and navigating the information. Consequently a tree-based structure is usually too limiting for representing real-life taxonomies. On the other had polyhierarchy is often more difficult to understand and navigate: e.g. having multiple parent nodes on each level makes it impossible to use familiar breadcrumbs controls to show users where in the hierarchy they are now. As a result forest is the most practical way to represent a taxonomy, where each tree represents an alternative way of classifying the information.

The hierarchies also have to be meaningful: each category should be related to the category above it. But the semantics of these relationships may differ from each other, and can include specialization, aggregation and other domain-specific relationship types.

Taxonomies must be created and maintained by humans

Taxonomic categories must be human engineered like Yahoo Directory or Open Directory Project to reflect the semantics of the categorized entities. Using automatic categorization tools might provide a good starting point for information architects, but would never create a stable clear and useful taxonomy.

Taxonomies are meaningful

In order to be useful, taxonomies must be meaningful in the context of the problem domain, where they are applied. For example the highly developed and proven taxonomy of species from the zoological sciences would be of little use in the livestock industry where majority of categories would be left unoccupied and the few overpopulated categories would lack granularity to classify animals by breed or purpose.

Taxonomies are stable

The structure of taxonomy should change much less frequently then the information it classifies. This allows users and programs to target specific nodes of the hierarchy as target rich environments for semantically compatible entities.

Changes to taxonomies should only occur as a result of the evolution of the structure of knowledge, not the knowledge itself. For example a business taxonomy should remain the same as new companies are formed, or existing companies merge, split, diversify or go out of business, but only change when new kinds of industries emerge. This is another reason why taxonomy structure should be engineered rather then derived from the body of entities it classifies.

Taxonomies are controlled

As a key information asset, taxonomies must be owned, protected, and tightly controlled. The owners are called information brokers. Their responsibility is to define the structure, access mechanisms and policies governing access, browsing, publishing, versioning, etc.

When changes to the structure and semantics of classification categories occur, both publishers and users must be notified. It will be the broker's responsibility to notify interested parties of these changes and provide appropriate version management of categories. Techniques for change and version management will evolve. One approach is to mark categories obsolete as a sign that users of those categories should change to the new categories in a timely manner.

Taxonomies are self-describing

It should be possible to navigate a taxonomy without an external guide. To make this possible, each taxonomy should have a detailed description. If taxonomy is intended to be programmatically navigable, it should include machine-parsable information in addition to human-readable description. Such taxonomies should also use formal mechanisms for describing relationships between categories.

Taxonomies relate to other taxonomies

No single usable taxonomy can be defined for classifying all possible services. Instead most useful taxonomies emerge within various communities of interests and represent their cumulative understanding of specific knowledge domains. Sometimes these taxonomies gain broad acceptance and even become standards. When such domains intersect it creates a situation where there are several fundamentally similar yet distinct classification systems used to classify the same set of entities. When such communities of interest that use different taxonomies over the same domain have to interoperate, they need to define a mapping between their taxonomies. For example US Census Bureau maintains a mapping between North American Industry Classification System (NAICS) and Standard Industrial Classification (SIC) taxonomies.

As the number of interoperating communities of interest and taxonomies they use increases, the need for a taxonomy federation mechanism becomes apparent. A pioneering work in this area is done by the US Defense Information Systems Agency which is developing Core Taxonomy to federate all taxonomies being used within US Department of Defense.

Taxonomies have to be accepted

Regardless of how well designed and maintained, relevant and precise a taxonomy is, it will never be broadly used if it is not structured the same way its target users think: “People categorize the world not on the inherent qualities of things, but on how they interact with those things.” [George Lakoff]

Existing Approaches to Service Classification

The SOA paradigm is still new, even by the technological standards, and service classification is one of the least developed aspects of SOA implementations. Our research showed that service taxonomies in use today fall under two major categories described below.

Based on existing standards

There are many standardized taxonomies maintained by industry, national and international bodies, and some service registries use them for classifying services. For example last time I checked  HP Systinet registry had built in support for the following taxonomies:

1. North American Industry Classification System (NAICS)
2. Standard Industrial Classification (SIC)
3. Universal Standard Products and Services Codes (UNSPSC)
4. IS0 3166 Geographic Taxonomy
5. UDDI Type Taxonomy
6. UDDI Keyword Taxonomy

These taxonomies are carefully engineered, widely used and well understood. In other words with regards to classifying services they meet all the criteria outlined above but with one important exception — in most cases they are not applicable to services. For example the 2002 NAICS classification has 15 subcategories under code 1111 Oilseed and Grain Farming, and just 7 subcategories (only 3 of which are meaningfully distinct) under code 518 Internet Service Providers, Web Search Portals, and Data Processing Services, which for the foreseeable future would be much more relevant to the SOA players then Oilseed Farming.

Created by SOA Pundits

The need for service classification has long been accepted by the technology experts. Many of them came with their own service taxonomies. Below is a small selection of proposed taxonomies for SOA services that appeared in recent publications:

Despite coming from people of very diverse backgrounds (Architect, Executive and Analyst), these taxonomies are remarkably similar: they are layered, well structured, based on deep understanding of SOA principles. They are also absolutely useless to service provider who wants to offer a Customer Invoicing Service, or service consumer in need of a Credit Card Processing Service. These taxonomies (According to the above definitions, these are actually not taxonomies but classifications) are too small to sufficiently narrow services down in order to create a target rich environment, but most importantly they will never be accepted by business community because business users do not relate to services in these terms.

In my next post I will introduce the concept Service Taxonomy Utility which is intended to facilitate development of the relevant and useful service taxonomis.