sapienti sat: Making SOA Ready for the Enterprise

Two roads diverged…

In the last week’s post, I asked a question: “is Governance worth the trouble?” and left a cliffhanger for an answer that it depends on how one looks at it. Today, I will try to untangle the issue further. But first let’s look at the diagnosis. Imagine yourself as a psychiatrist who encounters a patient that exhibits the following symptoms:

1. is perceived very differently by different observers
2. exhibit drastically different behaviors under identical circumstances
3. possesses some advanced capabilities one day and completely lacks them the next
4. radically changes the ways in which it relates to and interacts with the others

If you are an American^[1], you will undoubtedly diagnose the patient with the Multiple Personalities Disorder. Wouldn’t you agree that all of the above apply to SOA Governance? I admit that at some point the parallels end: there was no documented early abuse, hypnosis or memory loss, but in my book there are enough similarities, to warrant looking at Governance from this angle. And, as in real life, the doctor needs to identify, isolate and unravel each of the personalities before they can make progress with their patient, let’s try the same approach with ours. Now that we’ve got Governance comfortable on our couch, let us start with the history…

Earlier this year I attended a webinar, where Anne Thomas Manes, the recognized matriarch of SOA Governance, was warming up the crowd for a pitch by HP/Systinet, where she used to be the CTO. She started by saying “SOA Governance is about risk mitigation” which almost made me to do the Parisian Tailor (leave), but instead got me thinking. She is the one of the most recognized authorities in the field, so she can’t be wrong. Yet how could she be right, and talking about the same Governance to which I have dedicated the last several years of my career? Imagine yourself hearing one of the German car manufacturers to say: “Automobile is a means of ground transportation" – hard to argue with, true, yet somehow completely unfathomable. But there is a number of car companies, that live and compete (although perhaps not with the Germans) by that formula. That is when the idea for this article first hatched. Perhaps the theological disputes about the nature of Governance (how many policies can dance on the head of a PEP?) miss the point that the latter presents different personalities to different observers.

In my professional observation I identified two distinct personalities, which I named the Hauler and the Builder as a reference to one of my favourite parables attributed to Roberto Assagioli. In it a traveller walked past a construction site and saw three men doing the same work. He stopped and asked them: what are you doing? The first one replied – “Hauling stones”. The second – “Earning a living”, and the third one – “Building a Temple”. When I ran a recruitment operation, I used to tell it to some of the candidates when critiquing the resumes and discussing interviewing techniques. The diagram below illustrates how the two personalities manifest themselves to the main constituencies: technology vendors and customers.

Customer	Builder		Vendor
	Strategic > Transforming Services into Business Assets; > Building Enterprise-grade SOA Ecosystem;	Offensive > Delivering on the original SOA promise; > Creating strategic opportunities;
	Tactical > Adding Security, Logging, etc to web services; > Virtualizing endpoints; > Creating System of record;	Defensive > Closing gaps in product lines; > Checking off the boxes on RFPs > Placating the analysts;
Point of View	Hauler		Point of View

If I had to summarize each personality in a single word, Builder would be transformative and Hauler – incremental. Not that there is something wrong with the latter: both represent steps into the right direction, and any kind of improvement is way better then none.

Now, when we have identified the personalities we can analyze them in a bit more detail. Builder is a visionary and likes to relate to the same: given a choice she would go straight to the CIO/CTO. Hauler is a pragmatist, and prefers those who would benefit immediately: operations, security, etc. Builder tries to emphasize flexibility and potential, while Hauler concentrates on convenience and stability (often coming as an appliance). Builder is a generalist, trying to address all aspects of Governance in the broadest possible sense, Hauler usually tries to narrow down the problem, do one thing and do it well. Builder likes to talk about business challenges and opportunities; Hauler concentrates on technical capabilities and supported standards. Builders often see themselves as unique and irreplaceable, while Haulers tend to seek strength numbers and form or join all kinds of unions.

In retrospect it makes sense that Anne Thomas Manes chose the Hauler, when she introduced Systinet, who is the perceived leader amongst Service Registries, and thus most interested in maintaining status quo. This leads us to the final question: which of the two you (as a customer, or perhaps, a vendor) should be dealing with? The answer, as always, is highly individual but for me it feels too crowded on the Hauler’s side. And we can easy guess whose pay is better

¹ Until very recently, the phenomenon whose proper name is Dissociative Identity Disorder was almost exclusively confined to North America.

An insight into the obvious

I recently gave an in-house presentation with the same title, which boiled down to the following reasons:

• A recognized must-have for any SOA initiative today – when I started “selling” SOA Governance three years ago, I had to dedicate a good third of my time to proving that it is important and explaining what might happen to SOA implementations if were to be neglected. That part has been steadily shrinking with time and in the last year most audiences asked me to skip it altogether.
• Included in almost every RFI/RFP – even if the word “Governance” is not mentioned there (which is becoming less and less frequent these days) the underlying concepts, like security, monitoring, throttling, QoS, compliance, etc are found in the majority of requests that have anything to do with SOA: ESB, integration, B2B, composite applications, BPM and are creeping into even tangential subjects like Master Data Management.
• Customers keep asking for advice and position – it is harder and harder to come across at true greenfield environment or a virgin customer when it comes to SOA. So instead of delivering 101 classes and telling people to build their SOA our way, we have to deal with issues that have arisen while our customers were building their SOA Ecosystems their (or our competitors’) way. And in my experience they often tend to gravitate towards Governance (or the lack of such).
• Entry point into unassailable accounts – I once took part in a meeting with a global Systems Integrator aimed at devising a joint go-to-market strategy and observed that the SI’s marketing folks were much more interested in the Governance part then the rest of the SOA stack combined. When asked why, they explained that very often they find themselves facing a CIO who several years ago had standardized on a competitive (let’s for argument’s sake say: Tibco) platform, and just approved an additional multi-million dollar investment into that technology. If they start pushing our story (read: platform), then, regardless of the actual merits of the technologies involved, the most likely outcome would be an abrupt end of the audience and likely long-term loss of credibility. If, on the other hand, they congratulate the CIO on a wise decision and carefully steer the conversation towards Governance, they are much more likely to increase their credibility and establish an important beachhead. I personally have successfully employed this strategy on a number of otherwise hopeless accounts.
• Crown Jewel of SOA – this is another (less sales-centric but perhaps as cynical) way of looking at the above argument. An SOA implementation would never be complete without the keystone of Governance installed into it. This parallel is especially appropriate because Governance usually sits at the top (amongst the blocks on the 100,000 feet view architectural diagram – which is the only picture that really important people are likely to see) and because, unfortunately, it is often installed at the very last moment. On a more serious note, I have long maintained that Governance in the defining characteristic of SOA, because it is ultimately what will allow it to deliver on its original promise(s) and succeed where other technologically-similar approaches did not. So as far as the perceptions go, once you installed you Governance centerpiece at the top, it almost doesn’t matter whose technology is doing all the work at the bottom.
• Powerful differentiator – this one is fairly obvious as it not only separates those who have from those who have not, but highlights the completeness and cohesion of the entire SOA stack. Companies that deliberately built an SOA platform are much more likely to have a native solution then those trying to repurpose technologies from another domain.
• Reputation to uphold – through the SeeBeyond pedigree, we have a reputation of innovation and thought leadership in the SOA arena. Customers come to us not just for our products, but also seeking advice, guidance and vision. Many of our Service Offerings related to SOA aim at helping clients to transition further along the SOA maturity curve, which (as I hopefully proved above and by this entire blog) requires to have Governance in the picture.
• Because we can!

So what is the point of all those profound revelations? To me is to answer the question: is it worth the time and money for someone who is not a recognized player in the governance space to attempt becoming one? Especially if that someone is a recognized player in SOA as a whole. The anecdotal evidence seems to point to “yes”: for the past few months Tibco was actively advertising for Governance experts on various job sites, and the query for non-blog entries on microsof.com which until less then a year ago returned just a single paragraph about AmberPoint partnership, today yields respectable 376 distinct entries.

My take on this is: it depends on how one looks at SOA Governance. Which is the topic of my upcoming post.

Asking the eternal rhetorical questions

I recently came across a really interesting post JaBoWS is the Enemy of Enterprise SOA by Nick Malik from Microsoft. And I could not resist to comment on it. Nick was kind enough to reply with the most detailed and relevant comment I ever got, and as I started my counter-reply I realized that is deserves its own entry, so I will be posting trackbacks to both comment streams.

It seems that the gist of Nick's response was to build a case for a common (presumably across the entire enterprise) information model. Such a model would be a Really Good Thing. So good in fact, that only a mean argumentative person like myself, would ever even try to argue against it. But like other similarly Good Things (e.g. the Word Peace or the Enterprise Data Model from the 90s) it would be very hard (read - impossible) to achieve in practice for any non-trivial set of circumstances. So what shall one do - give up? No, I do not think so. A solution that I have built and would recommend to the rest of the world is Refinement - if you look at that post, I actually provided some use cases that fairly similar to the one which Nick have proposed.

So if one has the luxury of time and resources to stop the onslaught SOA in their organization while the experts analyze all the requirements and define a common information model to end all ambiguities - congratulations, you found a perfect solution to this really difficult problem. If, however, you live in the world with the rest of us and this is not an option, then perhaps the imperfect solution of Alpha building the original set of services based on their understanding of the world and just a dash of common sense and Bravo evaluating each service for a "fit" and taking the appropriate action might work just as well. Once the degree of fit and the impedance mismatch are clear, they will have a number of options:

1. to reuse the service as-is /* which I believe would be very rare */,
2. to reuse the functionality and define their own non-functional characteristics /* also too good to be common in non-trivial cases */,
3. reuse most of the functionality by refining the existing service with their own Refining Method(s) /* this is where I see the bulk of cases */,
4. evolve the service though versioning to serve the now expanded set of requirements /* this would be fairly common as well but require a higher degree of coordination between Alpha and Bravo and thus greater level of SOA Maturity across the organization */,
5.  or make an informed and responsible decision not to reuse, which, if you look at it from this prospective, is not a bad choice at all.

And a viable SOA Governance platform would allow them to make any of those choices (and to change their mind later) through its Service Virtualization, Versioning and Refinement capabilities. So going back to my argument that SOA Governance is what separates Enterprise SOA from JaBoWS and other cheap imitations, since I take the broadest possible view of the governance that includes all those things, I believe it still stands. For more on where in my view SOA Governance ends (or does not) have a look at this post.

And finally, in response to the second comment – no, I do not believe that a successful implementation of SOA governance (by the way, I was referring not just to design-time governance, but to the entire thing including analysis- and run-time as well) requires adherence to an enterprise wide information model or any other hard-to-achieve practices. And with regards to the non-functional characteristics I was referring to versus the System Quality Attributes as described here - I believe they are quite different. The latter represent the -ilities that are primarily of interest to the technical side of the house (architects, developers, operations and the rest of IT), while the former represent the things that are of interest to business today. And while there might be some overlap: e.g. security and availability are likely to end up in both sets, I doubt that SOX compliance (or, say, Patriotism or Political Correctness if our legislators ever get their way and codify them as regulatory requirements for businesses to adhere) will ever make it to the first list, and Maintainability or Portability into the second. Furthermore, I emphasized the word today to highlight the fact that governable non-functional characteristics are very fluid and are likely to change with time, geography or vertical, the set of -ilities is quite stable and I have not noticed significant changes since I started thinking about those things in early 90s.

Now that we have established what Enterprise SOA is, shall we just go and build it?

Walking the line or blurring the edges?

Ab ovo

Something I have struggling with for a long time has recently come to me in a flash. Well, almost. Ever since we started showing SGF to the customers, we were inundated with various feature suggestions and enhancement requests. And one of the most persistent ones (which is incidentally the one I have been resisting most adamantly) was for dynamic routing. SGF has provided some specific forms of routing from the very beginning: between bindings, between versions, between SDLC environments, to refinement methods, etc. But apparently that was not enough, what “the people” demanded was arbitrary content- (as in: route trades with four letter security tickers to NASDAQ and with shorter ones to NYSE) and context-based (as in: route requests to the Chicago servers between UTC 14:00 and UTC 03:00 and to the Hyderabad servers from UTC 03:00 to UTC 14:00) routing capabilities. And the reason I resisted including those was my belief that such functionality represents business logic and thus clearly belongs to an ESB and not to a SOA Governance platform. However, I was usually presented with counterarguments along the lines of: but SGF already has dual binding architecture that allows to mediate between different transports and protocols, and this also is clearly an ESB feature. In fact if one looks that the high level architecture, it looks remarkably similar to those of typical ESBs.

And I would reply about the woes of maintaining business logic in two separate places and the debate would go on. To tell you the truth I was motivated not only by the purity of vision but also by the political considerations like trying to avoid competing with our flagship SOA product, that happens to be an ESB. And in the interest of full disclosure I should say that at the end I gave up and dynamic routing was included as a first-tier feature into the plans for the next release of SGF.

State of the industry

While I was fighting with myself for architectural purity and anguishing over the moral dilemma what would Turing do? the world around me was quietly moving forward. The Governance products were adding more and more ESB-like functions including: routing, arbitrary payload transformations, load balancing, retries, etc. At the same time the ESB products were starting to incorporate more and more features that are traditionally associated with governance platforms. For example Open ESB project has been working on (or at least considering to add) support for aspects for some time. Out of the 27 aspects they have on their wish list, half represent clearly Governance-related functions: Throttling, Logging, Lease, Message/Content Filter (a.k.a. censorship), Validator, Chargeback, Monitoring, Time to Recover, Regulatory Compliance, Audit Trail and Security. I believe that having realized the gap in the underlying JBI specification, the group working on the JBI 2.0 spec intends to add some “Message Exchange handler/interceptor” to codify this architecture. Ironically the first governance-related standard will most likely appear in the ESB space.

So although as both Gartner and myself agree, the value of SOA Governance capabilities embedded into an ESB or other SOA platform is to put it mildly “limited”, as these capabilities mature, it will only be a matter of time before the ESB folks realize that they can export these capabilities outside of their own realm and contend to govern the entire SOA ecosystem. Another anecdotal evidence of this trend is Mule Galaxy a self-proclaimed SOA Governance Platform which is in fact a basic Registry/Repository notable mostly for the interesting choice to expose Atom Publishing Protocol (APP) API instead of supporting any of the traditional registry standards (UDDI, ebXML or JAXR). The interesting thing there that the list of supported governable entity types includes as many SOA platform artifacts (Spring and Apache CXF descriptors, Mule’s own ESB configurations, etc) as traditional metadata types (WSDLs, policies and schemas). To me this means that ESB vendors (I have seen similar features in BEA and Iona offerings) are starting to see themselves not only as purveyors of SOA Governance capabilities but also as first-class users of those as well.

The Epiphany

happened couple weeks ago when I was talking with executive team of a very promising newcomer into the SOA Appliance marketplace. In my mind their product is the closest to what I see as a true SOA governance platform in this space. It also happens that many of their ideas, capabilities and architectural decisions are remarkably in-tune with those that we put into SGF (but this is beyond the point of this particular essay). I was really stunned when their SVP of Business Development told me that they see as their main competitors not the other appliance vendors (Layer 7, Reactivity/Cisco, Vordel or DataPower/IBM) but ESBs. And then it hit me – like the Unified field theory these similarities and mutual encroachment between ESBs and SOA Governance solutions is not coincidental but stem from the fact that they are trying to address the same fundamental problem coming from different directions.

And the name of this single underlying problem is

Mediation

It’s really simple if you look at the problem from this point of view. The gist of what SOA Governance delivers is mediation of non-functional characteristics of enterprise services, often referred to as systemic properties. So if someone wants to be even more succinct they can define SOA Governance as systemic mediation. The gist of an Enterprise Service Bus on the other hand (and I am referring here to the true original meaning of this term and not to the integration products or BPEL engines adorned with adapters and management console that are often marketed as ESBs) is all about dynamic routing of service requests and replies between consumers, producers and intermediaries, based on content and contextual considerations. If I had to use just two words to describe what ESBs do it would be semantic mediation of SOA services.

If we add the third piece to this puzzle: technical mediation which covers the mechanical functions like transport, protocol and synchronicity adaptation and service versioning, and is common to both domains, the Unified Theory of Service Mediation will look like this:

These mediation functions are essential for successful realization of any not-trivial SOA ecosystem. The cleanest way to implement them would be based on a universal service mediation platform (most likely a gateway) that should be designed to handle all flavors of mediation evenhandedly and harmoniously and would be based on cohesive common architecture. From the delivery point of view, such a platform can be offered in a number of “trims”, i.e. “full service” mediation engine, systemic-centric Governance product, semantic-oriented ESB, and minimalistic service gateway that will be limited to just technical mediation. Such solution would clearly be a “killer app” for SOA and it is unlikely to emerge from any of the existing ESB or Governance offerings which are too entrenched in their own ways of looking at the service mediation problem domain. And that, my friends, opens some really exciting possibilities…

Why is it so hard for Governance products to do Service Versioning right?

In almost every SOA Governance-related RFP that I have come across service versioning appears near the top of the list right after security and monitoring have been dealt with. And the requirements are fairly consistent (as outlined below), why then some many Governance offerings have gaping holes in this area? I have heard vendor responses ranging from “we can capture version number as metadata” or “you can register several versions of a service separately and use ESB to route between them” to the extreme “there is no need to version services – if anything visible to the client changes, you just register a new service”. None of which seems to be useful to an organization that want to stick with SOA after the pilot project was declared a success. I think I have found the underlying cause, but let me first start with the

Basic Service Versioning Requirements

1. It should be possible for providers to publish and consumers to use multiple versions of the same service at the same time.
2. They should be clearly identifiable as versions of the same service.
3. It should be possible to designate the default version of a service, which will be invoked if no specific version has been requested. Default version doesn’t necessarily have to be the latest one.
4. It should be possible to designate compatibility relationships between service versions. “Version A is compatible to version B” means that A can process requests intended for version B and return responses that the client of B will understand. In other words it is safe to route to A requests intended for B.
5. Clients can request specific versions explicitly by passing version number with the request or implicitly by using the request that can be resolved to a specific version.
6. If explicit version number is not provided and implicit resolution fails, requests will be routed to the default version.
7. If requested version can be identified and that version is available, request will be routed to that version.
8. If requested version can be identified and that version is not available, but there is another version that is designated as compatible to it, request will be routed to the compatible version.
9. If requested version can be identified, that version is not available and no compatible version can be found, the request will be rejected.
10. Behaviors described in #1 through #9 should be available declaratively (through configuration, not coding).
11. It should be possible to implement custom versioning aspect that would be able to enhance or override behaviors described in #5 through #9. For example if requested version is unavailable and no compatible versions have been specified, such aspect instead of rejecting the request can transform it to a form that can be understood by one of the available versions. A corresponding transformation of the response would occur on the way back.
12. The system should advertise the WSDL corresponding to the default version, however clients should be able to view and retrieve WSDLs for all currently active versions.
13. The system should distinguish between non-disruptive (adding a new service version or retiring a version for which a compatible version exists) and potentially disruptive (retiring deprecated version, changing the default version, changing existing version numbers) actions. The versioning mechanism should be integrated with the change management mechanisms such as Lease and Notifications and should be able to trigger corresponding functions following a potentially disruptive action.
14. When user designates version as compatible to another, the system should be able to perform basic validation (check for missing operations, incompatible massages, etc.).

Nothing too difficult or complex, isn’t it? Why is it then so hard to come by a solution that meets most of these requirements? My theory is that

Registries are to blame!

Here is now – in the registry information models (both UDDI and ebXML Reg/Rep) Service is on the top of the food chain amongst the functional components (services can belong to an Organization, but the latter is purely a business entity encapsulating people and contact information) representing both primary and highest level entity. But in order to provide the capabilities described above one need to have a primary entity representing individual invokable versions and a higher-level construct (uber-service if you will) that will represent the logical service with all the versions and relationships between them. It does not seem possible to map such model to UDDI or ebXML RIM and remain “standards compliant”. This is why you will not find adequate support for logical service versioning in registry-centric products. They can version things: WSDLs, schemas, policies, etc but not the SOA Services, as defined here. The run-time centric solutions in theory do not have this handicap, but most of them joined various interoperability frameworks pushed by the registry vendors, to be able to check off the compatibility boxes and had to adapt their information models to those defined by the registry standards. This is why when designing SGF we decided to stay away from all the “standards” existing in or around SOA space and define our Governance Information Model based on the analysis of the problem domain and typical customer requirements, rather then various legacy compatibility considerations. The versioning-related part of the resulting model looked like this: Where:

• ServiceOffering represents the entire consumable service form the client’s perspective with all the functional and non-functional properties;
• ServiceComponent represents just the functional part with all the versions, virtualizations and refinements;
• ServiceVersion a single virtualized version, which in turn is an aggregation of individual Operations.

Needless to say that a governance solution based on such model can easily meet all of the versioning (and many other) requirements.

We were pretty early on to catch up on the concept of SOA Governance back in 2004, however we too did not start calling Rose by that name till the fall of 2005. So who did come up with the term and when? Now, when my favorite Meme Miner stopped working I can not find an easy answer, so I had to do some research and speculation.

I started with the next best thing – the job trend, which by my pure gut feeling lags some 18 months behind the idea, shows the very first blimp in January 2006, suggesting dating the idea to mid 2004.

"soa Governance" Job Trends

"soa Governance" jobs

Another anecdotal evidence is that the first SOA Governance page appeared in Wikipedia in November 2006. Next, if you look at the pure play vendors that inhabit the magic quadrant today through the waybackmachine you will see that AmberPoint started using this term in the fall of 2005, Systinet in January 2005, SOA Software sometime in early 2005. To my knowledge the credit goes to Infravio which first used a very close term Web Services Governance referring to the “process of controlling and tracking providers, consumers and Web Services usage at every step of the lifecycle” on May 12, 2004. And finally, looking at the analyst sites, the first result on Forrester is dated January 2005 and on Gartner September 2005

How Gartner’s Magic Quadrant for Governance reminded me of my first disaster movie.

I recently come across the latest (and as far as I know the first) Gartner’s Magic Quadrant for SOA Governance. This is definitely a good trend – I see this as recognition of the importance of Governance for the overall success of SOA and a statement that these technologies are important by themselves, not as a set of capabilities of an ESB or the SOA platform of your choice. In fact they specifically excluded a prominent SOA vendor, based on the fact that their Governance technologies are only intended for their own SOA suite. I have always maintained this point of view, perhaps in a less diplomatic manner: since no SOA vendor in their right mind (however cocky) can assume that the entire SOA ecosystem will consist only of services and composite applications built on their platform, and furthermore, all of those deployments will remain within a single domain of control (i.e. if both you and your trading partner use the same platform, they will allow you to connect your management console to their ESB – yeah, right!) governance capabilities built into such suits are useless beyond PoC or an initial pilot phase. The reasoning behind it is that if someone believes in the importance of SOA Governance they will not leave ungoverned services that are developed on different platforms, exposed out of packaged applications or provided by external entities, so they will have to also invest in a standalone governance technology to cover all of these. At which point they have to either live with two completely independent and most likely non-interoperable solutions of the same problem, or most likely stop using the built-in governance capability of their SOA platform and standardize on the universal solution. Point proven (I think).

I would also agree with their overall assessment that there is no clear leader in SOA Governance yet and that in most cases even if a vendor has all the areas of SOA Governance covered, it is done by a menagerie of products that came from different pedigrees and share little or no code and underlying architecture. Which makes me quite pleased with the decision we made three years ago to build SGF as a single suite, separate from Java CAPS (then known as ICAN), based on a uniform architecture and shared code… – Sorry, could not resist here

And now, getting back to the original subject of this post. Overall this report would have provided a fairly compelling and informative reading, if only I knew a little bit less about the topic and some of the vendors and products it covered. This reminded me of the film The Crew (Экипаж) which came out when I was in high school. It was the first soviet disaster film with all the trimmings: great (for the time) special effects, engaging plot, some character development and even a hint of erotica – unheard of in the era of late soviet Puritanism. The plot is about a crew of a soviet airliner, who land their plane in a foreign mountainous city called Bidri, which was damaged by an earthquake, which is then hit by another even stronger quake. Their jumbo jet is severely damages as they take of from the spectacularly disintegrating airport, and they crew repairs it in flight by exiting outside onto the fuselage through a stopped turbine. As you would expect, everything ends well. Everyone I know (myself included) loved this film. Except my English teacher, who went to see it with her husband a veteran airline pilot, who spoiled every suspenseful moment with his comments about accuracy and plausibility of things happening on the screen. I imagine the same would be true for an American taking a systems administrator to The Net or Firewall.

So, what does it all have to do with the Magic Quadrant? Well let’s see:

• One vendor did not provide in time information about the technologies they have, but they were still included and evaluated based on a product they did not have;
• Another vendor whose known weakness is that their product is completely self-contained and standalone got kudos for federation;
• An organization whose only public message on Governance is “use our consultants to teach you best practices” got into the “visionaries” crowd. My guess is that they were evaluated on the products they are planning to build, not the ones have right now.

I have been deliberately vague in the above statements for what I hope are obvious reasons, but if you are interested in more details, write to me and we could discuss things a bit further.

What does this all mean? Perhaps, just that I am not the right audience for these type of document, that’s all.

Post-Scriptum: couple of weeks after publishing this piece I have come across the entire Magic Quadrant in question available on the web. I hope it was done with permission and would not be taken off-line. Please read for yourselves.

Let's try to do it right

In a previous post we tried to scope the problem of service categorization, identified the common traits of successful taxonomies and pointed out some limitations of existing approaches to service categorization. Today I wanted to leverage all this as a basis for defining the scope and requirements for an enterprise-grade Service Taxonomy Utility or STU. Let us start with some basic

Assumptions

about the perceived users of service taxonomies.

1. Each service provider wants to maximize the usage of their service. This does not have to be number of invocations and might mean number of consumers, uses or simply to prevent duplication. Service providers usually fulfill some need by publishing a service's availability. This could be a business need, such as usage charge, support for other revenue streams, or just good will towards partners. Service provider could also be motivated by non-business reasons, such as regulatory or policy compliance. In either case people would not be promoting their services if they do not want them to be found and utilized or at least considered.
2. Service consumers build applications that invoke services to satisfy specific needs. The searching that is done at design time or runtime by a service requester is not based upon casual curiosity, it is intended to find the best; service available to invoke for a particular purpose.
3. There can be no single taxonomy to classify all possible services from every point of view. Many taxonomies of services will coexist on multiple levels.
4. Services can appear in one or more categories in one or more taxonomies. Different consumers will use different taxonomies, that best match their view of the service domain and will follow different paths to get to a service.
5. Finally, only humans can determine what business need is satisfied by a particular service. Artificial intelligence techniques for the foreseeable future are no substitute for human judgment in determining, based on the description of the category, whether the services within that category match the required business need.

Based on this foundation we can define a set of

Functional Requirements

for a Service Taxonomy Utility (STU), which can be used in SOA implementations on enterprise, industry and global levels.

1. STU should support multiple hierarchical taxonomies.
2. STU should support non-hierarchical (“see also”) relationships between nodes.
3. STU should support typed relationships between nodes in taxonomies. The types should be user-definable but should be able to express generalization, aggregation, refinement, type-of and equivalence.
4. It should be possible to annotate nodes with plain text and XML descriptions.
5. STU should allow to import an external taxonomy.
6. STU should support unique identification of taxonomies, nodes and relationships.
7. STU should support federation by stubbing internal or external taxonomies to nodes in other taxonomies.
8. STU should support versioning of taxonomies, nodes and relationship types.
9. STU should support deprecation of taxonomies, nodes and relationship types. Users can browse and search in deprecated taxonomies and nodes, but service publishers should not be able to publish into them.
10. STU should support both direct (nodes-to-entities) and reverse (entities-to-nodes) referential models. This would ensure that category-based searches will be supported with booth cooperating service registries and repositories, which can express relationships to the categorizing taxonomy nodes as part of the metadata associated with services and with non-cooperative ones.
11. STU should provide security. There will be three primary types of actors using the STU: administrators who create and maintain taxonomies, publishers who register their services in the defined taxonomies, and users who browse the taxonomies to locate the services they need. Security requirements for each type of actor are outlined below.
    1. STU administrators should be able to create and modify taxonomies, and delegate administration rights to individual nodes and sub-trees.
    2. Service publishers should be able to register services to taxonomies, individual nodes and sub-trees.
    3. STU Users should be able to browse taxonomies and search within them. The read access should be granted on per-node basis. The users will not be able to go into or under the nodes to which they do not have access.
12. STU should provide GUI-based and programmatic access to all functionality.
13. STU should provide notification capability to inform interested parties when certain changes occur. Users should be able to register to receive notifications by event scope and type. The scopes include taxonomy, node, sub-tree; and the event types: service published, unpublished, object deprecation and change.

So what?

That is a perfectly valid question. I do not think that a COTS STU solution exists that meets all of this requirements and can be used out of the box, so I see two possible roads that lead to the desired end result: build and adapt. The first one is fairly obvious and in my estimate in about 6 man-months a production-ready solution can be put together by a small competent team. Investing a man-year will deliver lots of nice bells and whistles. The second option is more interesting – essentially what I have described so far is a narrow specialization of an XML repository. In particular, I have discussed these requirements with Farrukh Najmi one of the authors of ebXML Registry Repository standard and the lead of the freebXML Registry project, and he told me that 95% of the above requirements are met by the version 3 of the Registry. I have not had a chance to verify this myself, but if anyone out there wants to try it, I would be very grateful to hear about the outcome.

Introduction

Interoperating services in many ways resemble distributed OO systems so many of the lessons learned from more then twenty years of evolution of object technologies might be directly applicable to the SOA domain. In the beginning when OO was rapidly gaining momentum many systems and languages that did not meet the full set of criteria declared themselves to be object-based by implementing some of the required traits. None of those solutions gained significant traction and success and they quickly sank into well-deserved obscurity. Although they differed in the set of OO features their creators chose to implement, they all had left out the one which was difficult to implement and retrofit: organization of objects into hierarchies of semantic classes.

Similarly the success or failure of services as the next paradigm of enterprise and global computing depends on the ways services are specified, described, advertised, classified, discovered, selected and consumed. Thus according to Steve Burbeck, the term service-oriented should be reserved for architectures that focus on how services are described and organized in a way that supports the dynamic discovery of appropriate services at runtime. Architectural schemes that focus instead on service-to-service message protocols, or on the details of how the various servers communicate rather than what they say to each other, should be characterized as service-based. The latter, although offering a significant improvement over monolithic applications, lack the capacity to scale beyond the sub-enterprise level when service consumers and providers are under control of a single architectural authority.

The ultimate goal of service-oriented architecture is to achieve robust service interoperability through dynamic discovery and binding at run-time. For this to work service consumers and producers must have both syntactic and semantic compatibility. The former can be achieved and validated through the use of XML technologies. The latter requires human intervention. However relying on humans to ensure semantic compatibility is slow, error-prone and since it can only happen at design time can not guarantee future compatibility. Machines on the other hand are not capable of tackling this task alone. The only workable solution to combine both approaches: humans should define semantically-significant categorization schemes, assert service semantics through categorization. Service consumers then will be able to identify relevant categories at design time, allowing computers to use them at run-time as target rich environments for semantically compatible services. Since these categorization schemes will be constantly traversed, they have to be organized into hierarchical directories, or taxonomies, allowing coarse-grained browsing and fine-grained category examination.

Definitions

A cursory search on Google finds multiple definitions for taxonomy:

• “A classification of ideas in an orderly hierarchy that indicates a natural or organizational relationship.”[Wordnet]
• “Taxonomy (from Greek taxis meaning arrangement or division and nomos meaning law) is the science of classification according to a pre-determined system, with the resulting catalog used to provide a conceptual framework for discussion, analysis, or information retrieval.” [Whatis.com]
• “A taxonomy is a hierarchical, structured presentation of information by categories.” [Burbeck and Graham]

All of these definitions highlight different facets and characteristics of taxonomies but emphasize their structured nature.

There are two closely related concepts: classification and ontology which are often used interchangeably. However according to Reinout van Rees there are some important differences. The difference between a classification and a taxonomy is that a taxonomy is always hierarchical and classifies entities in a structure according to some relationships between them; while a classification may be flat and use more arbitrary (or external) grounds. A great example of taxonomy based on internal classification is the Library of Congress Classification for books. An example of external reasons would be censorship that classifies books as politically incorrect, subversive or otherwise objectionable. Taxonomy differs from ontology in that the former only organizes knowledge, while the latter also serves as a repository for it.

Common Traits of successful taxonomies

Because humans comprehend information by categorizing it, there are multitudes of taxonomies being used in almost every human activity. They serve different purposes for audiences and are presented in different formats; however all successful have some common traits.

Taxonomies are hierarchical

All of the definitions in the previous section describe taxonomies as hierarchies. This is important to reflect commonalities larger than individual species. Such hierarchy can be represented as either a tree, where every node except a single root has exactly one parent; or a forest, or collection of one or more trees; or as a polyhierarchy which has multiple roots and allows nodes to have multiple parents.

As is the case with biological taxonomies, there is usually more than one way to categorize information, so for any domain there can be more than one possible taxonomy. None of these taxonomies would be sufficient by itself yet together they provide a powerful map for organizing and navigating the information. Consequently a tree-based structure is usually too limiting for representing real-life taxonomies. On the other had polyhierarchy is often more difficult to understand and navigate: e.g. having multiple parent nodes on each level makes it impossible to use familiar breadcrumbs controls to show users where in the hierarchy they are now. As a result forest is the most practical way to represent a taxonomy, where each tree represents an alternative way of classifying the information.

The hierarchies also have to be meaningful: each category should be related to the category above it. But the semantics of these relationships may differ from each other, and can include specialization, aggregation and other domain-specific relationship types.

Taxonomies must be created and maintained by humans

Taxonomic categories must be human engineered like Yahoo Directory or Open Directory Project to reflect the semantics of the categorized entities. Using automatic categorization tools might provide a good starting point for information architects, but would never create a stable clear and useful taxonomy.

Taxonomies are meaningful

In order to be useful, taxonomies must be meaningful in the context of the problem domain, where they are applied. For example the highly developed and proven taxonomy of species from the zoological sciences would be of little use in the livestock industry where majority of categories would be left unoccupied and the few overpopulated categories would lack granularity to classify animals by breed or purpose.

Taxonomies are stable

The structure of taxonomy should change much less frequently then the information it classifies. This allows users and programs to target specific nodes of the hierarchy as target rich environments for semantically compatible entities.

Changes to taxonomies should only occur as a result of the evolution of the structure of knowledge, not the knowledge itself. For example a business taxonomy should remain the same as new companies are formed, or existing companies merge, split, diversify or go out of business, but only change when new kinds of industries emerge. This is another reason why taxonomy structure should be engineered rather then derived from the body of entities it classifies.

Taxonomies are controlled

As a key information asset, taxonomies must be owned, protected, and tightly controlled. The owners are called information brokers. Their responsibility is to define the structure, access mechanisms and policies governing access, browsing, publishing, versioning, etc.

When changes to the structure and semantics of classification categories occur, both publishers and users must be notified. It will be the broker's responsibility to notify interested parties of these changes and provide appropriate version management of categories. Techniques for change and version management will evolve. One approach is to mark categories obsolete as a sign that users of those categories should change to the new categories in a timely manner.

Taxonomies are self-describing

It should be possible to navigate a taxonomy without an external guide. To make this possible, each taxonomy should have a detailed description. If taxonomy is intended to be programmatically navigable, it should include machine-parsable information in addition to human-readable description. Such taxonomies should also use formal mechanisms for describing relationships between categories.

Taxonomies relate to other taxonomies

No single usable taxonomy can be defined for classifying all possible services. Instead most useful taxonomies emerge within various communities of interests and represent their cumulative understanding of specific knowledge domains. Sometimes these taxonomies gain broad acceptance and even become standards. When such domains intersect it creates a situation where there are several fundamentally similar yet distinct classification systems used to classify the same set of entities. When such communities of interest that use different taxonomies over the same domain have to interoperate, they need to define a mapping between their taxonomies. For example US Census Bureau maintains a mapping between North American Industry Classification System (NAICS) and Standard Industrial Classification (SIC) taxonomies.

As the number of interoperating communities of interest and taxonomies they use increases, the need for a taxonomy federation mechanism becomes apparent. A pioneering work in this area is done by the US Defense Information Systems Agency which is developing Core Taxonomy to federate all taxonomies being used within US Department of Defense.

Taxonomies have to be accepted

Regardless of how well designed and maintained, relevant and precise a taxonomy is, it will never be broadly used if it is not structured the same way its target users think: “People categorize the world not on the inherent qualities of things, but on how they interact with those things.” [George Lakoff]

Existing Approaches to Service Classification

The SOA paradigm is still new, even by the technological standards, and service classification is one of the least developed aspects of SOA implementations. Our research showed that service taxonomies in use today fall under two major categories described below.

Based on existing standards

There are many standardized taxonomies maintained by industry, national and international bodies, and some service registries use them for classifying services. For example last time I checked  HP Systinet registry had built in support for the following taxonomies:

1. North American Industry Classification System (NAICS)
2. Standard Industrial Classification (SIC)
3. Universal Standard Products and Services Codes (UNSPSC)
4. IS0 3166 Geographic Taxonomy
5. UDDI Type Taxonomy
6. UDDI Keyword Taxonomy

These taxonomies are carefully engineered, widely used and well understood. In other words with regards to classifying services they meet all the criteria outlined above but with one important exception — in most cases they are not applicable to services. For example the 2002 NAICS classification has 15 subcategories under code 1111 Oilseed and Grain Farming, and just 7 subcategories (only 3 of which are meaningfully distinct) under code 518 Internet Service Providers, Web Search Portals, and Data Processing Services, which for the foreseeable future would be much more relevant to the SOA players then Oilseed Farming.

Created by SOA Pundits

The need for service classification has long been accepted by the technology experts. Many of them came with their own service taxonomies. Below is a small selection of proposed taxonomies for SOA services that appeared in recent publications:

Despite coming from people of very diverse backgrounds (Architect, Executive and Analyst), these taxonomies are remarkably similar: they are layered, well structured, based on deep understanding of SOA principles. They are also absolutely useless to service provider who wants to offer a Customer Invoicing Service, or service consumer in need of a Credit Card Processing Service. These taxonomies (According to the above definitions, these are actually not taxonomies but classifications) are too small to sufficiently narrow services down in order to create a target rich environment, but most importantly they will never be accepted by business community because business users do not relate to services in these terms.

In my next post I will introduce the concept Service Taxonomy Utility which is intended to facilitate development of the relevant and useful service taxonomis.

Or what u-boats and 20-engine planes have to do with delivering on the original promise of SOA

When I was younger (a lot younger) I lived by an airfield and after a year or so driving past the flight school billboards I decided that I would like to learn to fly. So one day I stopped by and looked at the classes they were offering. One thing that struck me as odd was that in order to fly a twin engine plane one had to take much longer classes and pay significantly more money. Intuitively that did not make sense – twin engine aircraft seemed conceptually the same as a singe engine one only twice as safe. Well actually it is the other way around – it is enough for one of the engines to fail for the entire thing to become really unstable and virtually uncontrollable in the hands of an unskilled pilot, making it actually twice less safe… What does it have to do with SOA? Well the old “monolithic” [this is supposed to be a derogatory term in the SOA-speak] applications behaved like U-boats – at the end of the day either everyone was fine and victorious or everyone was dead. Now we transition into a wonderful world of SOA and break it into a Composite Application consuming couple dozen enterprise services (some of which might be outside of the domain of control where the new application belongs) we effectively turned our U-boat into a twenty-engine plane: it is enough for any one of those services to go down or become otherwise unavailable for the entire application to get into serious trouble. For those less statistically inclined, this decreases expected reliability by a factor of twenty – not good.

When SOA was sold to the IT and business communities back at the turn of the century, it came with a beautiful vision called find-bind-execute. It promised composite applications and services dynamically finding other services they need in registries and other target-rich environments and if the one they use goes down a replacement can be located before the operator can say “trouble ticket”. Sometimes I close my eyes and can still see it – makes me feel warm all over. However this vision did not eventuate, primarily because computers can not do semantics and figure out on their own how to invoke a replacement service which supposedly does the same thing but uses slightly different interface and protocol. A contributing factor was also that registries, which were suppose to enable all this splendor, once they implemented all the security, workflow, federation and extensibility features, have become too heavy-weight to support real-life find-bind-execute scenarios. So the pendulum have swung the other way and now everyone doing SOA is supposed to look services up in a registry at design time, import the WSDL into their tool of choice at development time and write their consumers to a specific service implementation. It seems that the entire SOA world is busy building twenty-engine planes… Makes you wonder why there are no high profile SOA disaster stories like: company transforms core business applications to SOA, suffers meltdown due to service outages in the news yet. My guess is that we will se some in the coming year. And with computer’s ability to “understand” WSDLs and replace unavailable services with “similar” ones at least a decade out, is there a hope for SOA, or is it one of those technologies that came before it’s time?

We think that there is. In our SOA Governance solution we could not get all the way back to the original vision, but we built a set of technologies that got us at least half-way there. Here is how SGF solves the above problem and enables building self-healing composite applications:

1. Our Governance platform clearly separates the functional and non-functional aspects of enterprise services and enables SOA participants to express them in the form of reusable Service Components and Governance Contracts that can be combined together to form consumable Service Offerings.
2. Developers can code their consumers to the functional components, and implement the find-(re)bind-execute logic using queries like: find me the offering for the credit card processing service that implements the ChargeCreditCard component and provides a supported security mechanism, guarantees 99.99% availability and has the lowest per-transaction costs.
3. Consumer-side convenience API can hide some of the complexities from the developers and where possible provide automatic re-binding to alternate services based on configurable business policies.
4. Combined with a service lease feature it will result in significantly higher dependability of Composite Applications.
5. Having all of this build on top of a formal Governance Information Model (GIM), guarantees reliable and unambiguous interoperability between all participants in a SOA ecosystem.

This is why we introduced a separate SPECIFY Aspect Enforcement Level in GIM, which allows us to designate Governance Aspects that support dynamic (re)binding.

While this approach does not completely resolve the problem of excessive engines on our big SOA plane, it allows to bring aboard enough spares and other safety features to feel confident about the journey. Roger that, wing leader!

Please bookmark with social media, your votes are noticed and appreciated:

When we introduced aspect-based SOA Governance, we defined aspects as key non-functional characteristics of enterprise services that are of importance to business today. This is [intentionally] a very broad definition that covers a lot of things ranging from security and monitoring to QoS metrics, censorship and regulatory compliance. It is fairly obvious that the governance platform should be able to treat such diverse characteristics quite differently and we capture this diversity by categorizing the aspects by enforcement level. When we designed our solution we have identified four such enforcement levels and everything that I have encountered so far fit neatly into this taxonomy.

Before we dive into these levels and their meaning, I would like to remind how Sun SOA Governance Solution uses the aspects. Our fundamental approach is based on governance-by-contract. We create end user consumable Service Offerings by combining service implementations with one or more reusable Governance Contracts. The contracts are constructed from clauses and each clause contains a statement about a specific governance aspect. These statements are made by associating an aspect with a specific value, for example:

• Throughput = 10 transaction per second
• Availability = 99.999%
• Time to Recover = 5 hours
• Usage Metering is true
• Security specified by <ws-security Policy document>
• Clearance level is “Top Secret”
• The service complies with the European Privacy laws

Aspect Enforcement Levels are directly related with the kinds of values that can be used to make statements about the aspects:

• ADVERTISED – the lowest level when the associated value is in human-readable form only with no formal semantics and enforcement strategy. For example, in order to reason about the Sarbanes Oxley Compliance of a composite application, one needs to know that the services it depends upon have been audited for compliance as well. The corresponding governance clause can simply state: the service has been audited and found to be in compliance with Section 404 and 803 of Sarbanes-Oxley Act.
• SPECIFIED – similar to ADVERTISED, but the aspect value is specified in a formal machine-readable form with exact semantics. Tariff (cost of service invocation to the consumer) is a good example of aspect that can use this level.
• REPORTED – similar to SPECIFIED, but implies that the system can and will actively monitor compliance and report violations. Report here means simply make known to someone and can include things like raising an alert, sending a message, including a warning with the reply or simply writing to a log. This level is typically used with various QoS aspects, such as throughput, latency, availability, reliability, time to recover, etc.
• ENFORCED – similar to REPORTED, but means that in addition to reporting non-compliance the system will attempt to take a corrective action when it occurs. The nature of the corrective action depends on the type of the aspect. For restrictive aspects, such as security and throttling, it usually means simply rejecting the request. For affirmative aspects, such as performance or availability, governance engine might attempt to use service management capabilities to remedy the situation, for example issue the Increase Capacity request to the container hosting the service implementation or failover to an alternative implementation. There are many more possibilities, for example compliance aspects might transform the requests and/or responses to censor classified information or replace personally identifiable attributes with surrogates, etc.

The two higher levels have to be supported by the run-time infrastructure and require that corresponding aspect enforcement methods to be implemented and deployed into the governance engine or, in case of delegated enforcement, registered with it.

It is also worth noting that an aspect doesn’t have to be always associated with a single enforcement level. Sometimes, depending on business requirements and context, it could make sense to support two or more levels for the same aspect. For example throttling can be enforced for external consumers, over which an organization has no direct control, and just reported for internal ones where alerting operations should suffice. The following table illustrates how a Throughput aspect can be defined at each level:

In particular, Sun SOA Governance Solution supports development of context-aware aspect enforcement methods that adaptively change behavior depending on the enforcement level of the aspect with which they are associated. For example lease and throttling aspects included with the framework support both enforced and reported semantics.

Conclusion

I hope that I made a good case for the need to recognize and support all four aspect enforcement levels in a comprehensive SOA Governance platform. In a future post I also plan to show how the SPECIFIED level helps to deliver on the original vision of SOA that has been largely forgotten in the recent rush of “practical” implementations.

The forgotten key to service dependability

If I have to name the feature of SOA infrastructure and governance that is at the same time most overlooked and critical to the overall success of post-pilot implementations, the honor would undoubtedly go to the Service Lease.

Let us start with a

Hypothetical Example

Once upon a time there was a talented developer Alice who implemented a ReallyUsefulService. She tested, documented and deployed it as was prescribed by the Company’s Process and as the last step published it to the SOA System of Record (some people like to call it a registry, but that’s a topic for another discussion) along with the corresponding SLA and all other relevant metadata. Some time later Boris, the technical lead in the Business Systems Group, was given a task to develop a BusinessCriticalApplication. Being wise in the ways of SOA he started by searching the registry for viable reuse opportunities, and found the ReallyUsefulService which was available and satisfied both his functional and non-functional requirements and constraints. He told his team to use the service and they lived happily ever after… Let’s pause here for a while, what exactly does it mean to have service published in a registry? For Boris it most likely means that the service will be available at the specified end point with the advertised contract, features and policies forever. How can he be so naïve? The Sun will burn out in a few billion years, but in an absence of any specific expiration date this assumption is most convenient and as good as any… Alice, on the other hand, when she was publishing the service, meant all of this to be guaranteed for the next instant, after which all bets are off. This is a very dangerous disconnect which will very likely lead many adopters of SOA to painful and costly surprises when things that were once in a registry start to change. But do not despair! Luckily there is a very simple answer – Service Lease.

The Concept

Service lease means that a binding (either explicit or implicit) exists between the service consumer, governance platform and (ultimately) provider which guarantees that from the time it has been granted until it expires, nothing of importance to either party will change. This doesn’t mean that implementing service lease requires to perform the full find-bind-execute cycle at runtime – in its simplest form the lease is just the expiration date placed on the service contract by the Governance Infrastructure that can guide prospective consumers at design time to make informed decisions about dependability of services they want to use.

The Benefits

The lease in the true spirit of SOA further reduces the coupling between the service consumer and the service provider, by limiting the amount of time consumers and providers may be bound. Without the notion of a lease, a consumer could bind to a service forever and never rebind to its contract again. This would have the effect of a much tighter coupling between the service consumer and the service provider.

The lease provides some degree of protection to both service consumers and producers. For consumers it is a guarantee that service should remain available and SLA would remain in force for the duration of the lease. For producers it is a guarantee that they will be free form any SLA obligations after the last lease expire; and if a producer needs to somehow change its implementation, security, QoS, or simply temporarily take the service down for maintenance, it may do so at this point. The implementation can change without affecting the execution of the service consumers, because those consumers must request a new contract and lease. When the new contract and lease are obtained, they are not guaranteed to be identical to the previous ones. They might have changed, and it is the service consumer’s responsibility to understand and handle this change.

The use(funless) of lease is not limited to registries and design-time in general. At run-time the governance engine should be aware of the lease of the services that are being invoked. Depending on the policy, it can notify the participants (consumers, providers, operations) that service is being used outside of its lease window or lease is about to expire, or it can reject the requests for out-of-lease services.

The Implementation

A practical implementation of Service Lease that we have in our SOA Governance Solution is based on the use of Lease Tokens. The basic token includes the expiration date; effective date in order to support early announcement of services and perhaps even at-your-own-risk availability, and issue date to support and track lease extensions.

When a service is “published” (i.e. approved for public consumption) the Governance Officer can define the lease terms and create a token that describes them. The token is be stored in the registry and made available via both the GUI and API. It is also visible to the Service Delivery Agent (run-time governance engine) to implement a variety from lease policies including embedding warnings with the replies, raising alerts and rejecting out-of-lease invocations. The mechanism is designed to be extendable and can support the following additional scenarios:

• Token inclusion with each request, to force the clients to do service lookup and present token as a proof.
• Issuing tamper-proof tokens using XML signature to prevent token forging by rogue consumers.
• Issuing traceable tokens containing unique IDs to allow the infrastructure to keep track of all consumers presently bound to a service.

The Alternatives

To my knowledge no other offering on the market today supports the notion of lease, so how do people manage? To tell the truth – I really do not know… The most reasonable workaround seems to be the subscription/notification mechanisms supported by most high-end registries. They allow consumers and other interested parties to register interest in various resources; including services and the system will automatically notify them when anything about those resources changes. Although a generally useful feature, it has a number of draw-backs when used as a substitute to service lease, including:

• It does not advertise the lease terms as a part of the service contract, so the prospective users can not made informed decisions about service dependability.
• It relies on inherently unreliable human behavior (users have to register for notifications of services they use) in order to be notified about the changes. And keep the subscriptions up to date as their contact details, responsibilities and employment status change.
• It is inherently reactive – the notifications are sent after the service information has changed in the registry, which typically happens as the last step of the update process after the changes has been deployed into the ecosystem. In all likelihood by this time something that depended on the affected service will be broken by the time the developer reads the notification e-mail and implements the necessary changes.
• It does not give the service providers a clear liability-free way to change service contract, SLAs or conduct maintenance.

Conclusion

SOA is new. SOA Governance is even newer. But most concepts it is based upon are not. Isn’t publishing service to the registry just a form of advertising? And advertisers have learned (many of them the hard way) that it is very unsafe to put an add without the expiration date… So, the SOA users of the world, – unite and demand from your Governance vendors to support service lease!

Last year my boss Jesse Lambeth came up with a fun little example to illustrate some key concepts and differentiators of our approach to SOA Governance: separation of functional and non-functional, governance by contract not by policy, distinction between functional service and consumable service offering, reusability of contracts and functional services, and interaction across domains of control.

Once upon a time in a far-far-away economy there were two lines of business, one is a home owner (consumer) and the other is a lawn care service (provider). The consumer needed someone to take care of her lawn and came up with the following requirements:

Armed with those requirements he took the phone book (service registry), navigated the index (service taxonomy) and started browsing through the landscaping section (target-rich environment). She reviewed the ads placed by different providers until she found the one she liked:

Having reviewed the characteristics of each service offering for applicability to his requirements, and realized that (as in happens in real life) none of them is a perfect match, the consumer then selected the Lawn Care Comprehensive Plan with the Gold Contract as the closest match to his requirements.

Note that from the consumer’s point of view, the service is not just the Lawn Care Basic Plan but also incorporates the appropriate contract as well which governs the relationship between consumer and provider. The actual person that comes out to deliver the service now understands not only what is to be functionally done for the home owner, but also when, where, and how. Also please take another look at the last paragraph of the above example which reminds us how dangerous it is to advertise something without clearly specifying a period during which the offer and the terms are valid and brings un to the notion of service lease, which will be the subject of a future post.

Please bookmark with social media, your votes are noticed and appreciated:

In a recent post I proclaimed a need for a formal Governance Information Model and a foundation for a proper SOA Governance platform. Today I would like to get a bit deeper and introduce the model that we have developed over the last three years. For those who prefer pictures to words or do not have the patience you can go straight to the main and auxiliary models.

For the rest of you, especially for those who are uttering the dreaded word proprietary, I will start with:

Why?

We started with looking at the existing standards, including UDDI, ebXML, WS-Policy to start with. None of them seemed appropriate because although all of them have been designed to be infinitely extendible and expressive all of them lacked the key first-class abstractions that exist in the SOA problem domain. Those of us with the data-modeling background know that everything can be expressed as name-value pairs (which I think are the highest normal form) but such representation does not lead to intuitive, maintainable or robust designs. As a result every one who have tried to use these standards for Governance, ended up using them in highly proprietary ways. Let me illustrate with a simple WS-Policy document:

<wsp:Policy xmlns:L7p="http://www.layer7tech.com/ws/policy" xmlns:wsp="http://schemas.xmlsoap.org/ws/2002/12/policy">
  <wsp:All wsp:Usage="Required">
   <L7p:HttpBasic/>
   <L7p:SpecificUser>
     <L7p:UserLogin stringValue="admin"/>
     <L7p:UserUid stringValue="3"/>
     <L7p:IdentityProviderOid longValue="-2"/>
     <L7p:UserName stringValue="admin"/>
   </L7p:SpecificUser>
   <L7p:AuditAssertion/>
   <L7p:HttpRoutingAssertion>
     <L7p:ProtectedServiceUrl stringValue="http://www.jasongaylord.com/webservices/zipcodes.asmx"/>
   </L7p:HttpRoutingAssertion>
  </wsp:All>
</wsp:Policy>

I highlighted in blue all the lines that come from the standard specification, and the rest is – yes you guessed it right, proprietary! And thus requires the same degree of coupling (i.e. shared knowledge) as a completely proprietary model, yet they have to fit into Procrustean bed of syntax designed to meet someone else’s purpose. Actually we came fairly close to mapping our GIM to ebXML RIM 3.0 which was the closest fit between the three candidates, but about that time the future of Sun Service Registry became much less certain.

How?

Having reached that conclusion we set out to define the model. We wanted it to strike the right balance between strictness (strongly typed, verifiable, validatable and ultimately executable) and extendibility (because we can never think of everyone’s requirements). We wanted it to reflect the key view points (which later evolved into roles). As a result at the heart of the model are 3 objects: Service Component, Governance Contract and Service Offering representing the points of view of SOA Developer, Business Analyst and Governance Officer respectively.

What?

This is what we ended up with. Was it a success? We think so. It served a basis for six different versions of SGF, survived several implementations, dozens real-life use cases. We used that same model to derive the data model that serves as our registry/repository, the XML schemas that are used on the wire, and even the Java language classes that run inside our solution. For the last 2 years it had no significant changes. Now let us see if it survives the public scrutiny.

The time is now!

The real life usage of SOA governance solutions is evolving even compared to the last year. Until recently most companies that introduced SOA Governance did so in some form of pilot project, which usually represented a "greenfield" environment: service consumers, composite applications and often the services themselves were being developed at the same time. So the key factors on which SOA Governance solutions were evaluated were centered around their design- and run-time capabilities and operational characteristics but did not include the "brownfield" environment friendliness factor. Consequently SOA Governance vendors had little incentive to invest in those capabilities of their products. But all of this is about to change, and the ability to support effective and painless introduction into existing IT environments will soon become one of the key differentiators in the SOA Governance marketplace. We have experienced this first hand during a recent implementation of Sun Service Governance Framework (SGF) at a large European media company, during which it turned out that the majority of issues with development, implementation and rollout of the governance solution were directly related to the "brownfield" category. Specifically they included:

1. The need to reconcile and integrate Service Governance with the SDLC used by the client’s IT organization.
2. Ability to support service governance across multiple development, testing, staging and production environments.
3. The need to provide support for “uncooperative” clients – the ones which are impossible or not feasible to change to accommodate governance-related service changes.
4. The need to quickly and efficiently bring large numbers of existing services under the control of the Governance Solution.
5. The need to support safe and effective sharing and migration of governance data between multiple environments.

It was estimated that without the above capabilities, the total effort required to introduce the Governance solution into the IT SOA landscape would exceed half of a man-year.

State of the union

I have not seen any full-spectrum SOA Governance products or technologies that provide noteworthy "brownfield" environment friendly capabilities described above. There are number of design-time only governance products (Registries) that provide federation capabilities which could be utilized to support some form of migration and reconciliation of governance data across multiple environments. There are also some run-time only governance products which allow import and export of governance data and can be used to ease some of the pains of reconciling Governance with the SDLC. But that’s about it!

Existing governance methods and solutions are focused on governing services in the context of established SOA environment complete with underlying governance infrastructure. Let me bring an example: WebMethods in their definitive whitepaper on the subject write: "Ensure that governance capability-related milestones are synchronized with SOA adoption milestones so that you do not end up trying to retro-fit governance after the fact" and "the right time for governance is before you put any services into place" - great advice if you only deal with clients that have never played with SOA before! This situation is most representative for "greenfield" environment and is highly atypical for real-life enterprise IT. This static nature of service governance can become a significant barrier for its introduction (and consequently the success of SOA overall) in "brownfield" environments with its existing sets of services, legacy consumers, third-party composite applications and established software development processes and practices.

The Answer

When we first recognized this problem (and the shortcomings of our own governance solution) we set out to define the list of capabilities and enhancements to SGF that would solve the challenge of [near] painless introduction of SOA Governance into existing SOA-based IT environments. This is what we end up with:

1. Staging-aware SOA Governance which aims to resolve the disconnect between the fact that governance is essentially an oversight activity and thus should happen in (or at least as close as possible to) production with the need to put governance artifacts through the same QA processes as the rest of the IT assets.
2. SDLC support in Governance which addresses the fact that transition from so called monolithic or siloed applications to SOA has in fact, from the SDLC point of view, made the entire IT environment even more monolithic than it was before that transition. In the past at least these applications were independent form one another and could have been taken through SDLC phases one-at-a-time. As companies embrace SOA they are facing potentially infinitely connected mesh of services, consumers and composite applications and the only guaranteed safe option becomes to take through SDLC the entire snapshot of enterprise IT, making it more difficult and costly then ever to introduce new changes required by the business. Extending Governance solution with SDLC capabilities makes it possible to take individual services, consumers and entire composite applications through the lifecycle stages as required by the IT practices and procedures.
3. Transparent Governance Mode which resolves the tension between the need for a Governance platform to transform services and the need to support legacy clients that can not (easily) change to accommodate governance-related changes to service interfaces. For example declaring that a certain service has to be authenticated with WS-Security requires changing the WSDL to reflect the fact that it now needs wsse-compliant header.
4. Bulk operations which would allow to quickly and consistently bring under the umbrella of governance groups of existing services, spread throughout the Enterprise.

I believe that brownfield-friendliness will be a decisive differentiator amongst the SOA Governance products in 2008 so I am planning to talk bout each of these capabilities in more detail in future posts.

And how long did it take us to introduce our solution into the infrastructure of the client I mentioned in the beginning of this post? Less then a man-week.!

Note: I am aware that WebMethods is now part of Software AG, but I could not get to that paper from their website.

Please bookmark with social media, your votes are noticed and appreciated: