Whatever... [Deepak Alur's Blog]: Deepak Alur's Blog @ Sun

Hmmm...If Web 2.0 is nothing but read/write web as Hal Stern says it is, then I might as well disconnect my broadband and go back to reading good old fashioned printed material. At least, I know who the frick wrote and published that damn thing. Nicholas Carr points out some intersting muck about the amorality of Web 2.0 and you can see some examples of some of the mess created by good old human nature in the new Web 2.0. And then there is the news of tightening of the posting rules for Wikipedia, which is an often used example as a Web 2.0 site. If humans are going to be humans (a.k.a monkeys) and do their business good old fashioned way on the read/write Web 2.0, then this Web 2.0 will end up being a conglomeration of incoherrent, inconsistent, unreliable, uncorroborated bunch of crap. Or, the Web 2.0 will keep tightening up the rules until it is no longer the read/write web, which I think will be called Web 3.0. Or wait a minute, is that Web 1.0? Bah, Humbug!

Web2.0

Today, I presented on Sun SOA at our Software Summit attended by Sun and our partner folks. There were lots of comments and feedback on my presentation and there were some pretty good discussions.

A discussion touched upon the topic of SOA and Standards which I blogged about a while ago. On one of the slides, I was trying to address the confusion about SOA and Web Services by saying that SOA != WebServices, but that Web Services was the current popular implementation that seems to be gaining traction with our customers and the industry. And a few slides later I showed a picture (see image map below) to show how we were trying to grasp all the standards (approved, emerging, conflicting and so forth) mapped to different SOA aspects.

I might have confused my dear audience a little (can't blame them!), because on one hand I was saying that SOA != WebServices, and on the other hand most of the standards I was mapping to different SOA aspects were Web Services related. But the reality is that SOA today is being mostly viewed from a WebServices implementation perspective. What do you think?

PS: On the plus side, it was great to meet the John Clingan and other Sun bloggers in person.

SOA Software Standards WebServices

You must not miss what Captain Bray posted, here and here, to relate (in another way) to what I was trying to say in my previous post. Very funny, Captain Bray!
Humor WebServices Software Standards Specifications

One of the things I often hear about Service Oriented Architecture (SOA), when compared to past architecture styles, is how it is based on industry standards and sets you free from all kinds of burdens related to integration and interoperability. But, just the mere number of standards and specifications that one has to consider when attempting to SOA is mind numbing. I happened to chance upon one of Thomas Erl's (I realized later he has many similar) websites called specifications.ws which shows a (stained-glass-like) mosaic of standards that can put you in a kind of a trance or even a coma if you are not careful. I do like his attempt at distinguishing first and second generation web services. Not only are there many many standards/specifications to consider, you also have to consider:

• Which ones are specifications (not yet standards) by one or more vendors?
• Which ones are specifications (not yet standards) submitted to standard bodies?
• Which ones are specifications ratified by the standard bodies as industry standards?
• Which ones (standards or specifications) overlap (some do) and which to choose / use in such cases?
• And worst of all, there is not a single place to go to for these standards!

On that last note, some of the several standard bodies you run into when investigating SOA standards are W3C, OASIS, DMTF, BPMI, WFMC, IETF, Liberty Alliance, to name a whew! And you will definitely not miss the WS-I effort underway to develop and publish interoperability profiles (not standards) based on accepted standards in the industry. (Note:WS-I is not a standards body.)

In reality there is not any holistic standard for SOA. And I seriously doubt there ever can be, will be such a holistic standard. Although there seems to be an effort underway at OASIS to create a SOA Reference Model, you won't learn much about how to do SOA and how all the standards relate in typical SOA environments.

Anyway, after reviewing several specifications and standards, I am exhausted. I am working on putting together a list of all relevant standards and specifications and will publish my take on all this sometime soon.

Meanwhile, if you have any words of wisdom, I am listening.

It is encouraging to see James Gosling blog about Service-Oriented Architecture (SOA). It is encouraging because:

1. We are not alone in this confusion about the overlaps and differences between all things in SOA.
2. Hey! It is James Gosling talking about SOA!

I agree with James that there is a camp that things SOA and OO are distinct camps. To me SOA is a goal and Object Orientation (Design/Programming/Modeling) is one of the fundamental ways to get there. SOA is the destination and the journey, and OO is the vehicle to get there in good shape. I say in good shape because you can still get there without using OO, but you might not be in as good of a shape as you would have if you had used OO.

For example, say you want to go from point A to point B and you have a choice between using different modes of transportation (car, bicycle, skateboard, Segway, on foot, etc.). Which one would you use? What? You want to know how far you are going? Ok. Let's say you are going to the next block to meet a neighbor. What do you use? What if you are going a few blocks to get a carton of milk from a neighborhood store? What if you are going the same few blocks but to get your groceries for the whole week? What if you are going a few hundred miles to visit your family? Would you use a skateboard or a Segway to travel a few hundred miles to visit your family? (Not unless you want to make a new world record). Now, would you choose to use the same mode of transportation for all of these? Of course not. [If you do, you need to go back to the beginning and read again until you get it right. ]

Why not think about SOA the same way? If SOA is the goal, the destination, and the journey, then which means (modeling, design, programming, language) would you choose to get there?

The following is an informal comparison of SO and OO and how things fall in or out of place:

To reiterate what James said, I quote him: "Proper OO structuring is always a good idea."
Great advice from the wise. I for one am going to follow it.

SOA OO Software Architecture Programming

Looks like analysts can't help but keep coining terms to one up each other. Now Burton Group analysts are talking about Superplatform, which I quote from this article :

"The Burton Group defines the superplatform, an outgrowth of the middleware market, as a tightly integrated suite of products that provides a platform for enterprise computing."

I find this illuminating statement in that article:

"While scalability is still an issue for some organizations when choosing a superplatform, "less than 5%" will find Windows is not scalable enough."

Less than 5% ? What 5%? Don't you want to know how analysts come up with these numbers? Well, if you do, you might have to pay $ to get that information. Anyway, is it 5% of all enterprises? Or 5% of Fortune 500?

Want some more words of wisdom? Here goes:

"If you're Visa, and you process 8,000 transactions per second, you might have trouble. But if you're an insurance company, no problem."

Hmmm....

And finally the comment about Java portability not being important is somewhat twisted as one reader points out here.

"For several organizations, vendor lock-in is an important issue, but the portability benefits of choosing a Java platform become less as enterprise environments get more sophisticated and complex."

So, are you ready for your superplatform?

Architecture and design quality is of utmost importance in any mission critical system. Yet how many times have you seen quality take a back seat to time to market and other factors that are deemed more important to the business. In current development scenarios, how many times have implemented solutions quickly and deployed them production without adequately ensuring scalability, performance, security and other qualities of service (QoS). And all this because it is more important to release on time than to release with quality. The business forces justify going live sooner than later at the cost of overall systemic quality.

Even though over the years, many approaches (Patterns, Idioms, Processes, Methodologies, etc.) have evolved to focus and improve system quality, we seldom find enough time or effort to adopt and implement these best practices. We find little or no time to research, educate and adopt them within our teams. As a result, for most development teams, quality becomes an afterthought in the development process. In some cases we rely on a few key individuals in the team to ensure the quality of implementations. This works well in small and highly skilled teams, but when the teams get large with members of varying skill levels, the technical leadership is burdened beyond their ability to ensure adherence and conformance to best practices. The quality of the system under development takes a spiral dive. Sometimes, we comfort ourselves thinking that we can refactor our system as and when we discover the deficiencies. I think we even kid ourselves that we can do so even after going live. And so, in our quest to meet fast evolving business needs, we wrongly rely on future refactoring to justify such quick and non-ideal implementations. Even if we know we will not have time to refactor, we go ahead any way and implement whatever solution we come up with that works at the moment. Because, that is what the project demands, right?

So what's the big deal? The big deal is that we pay a price! Sometimes a hefty one. When we find a defect in an implementation, we can find and fix it relatively easily if the defect is truly related to implementation and not architecture/design. But fixing a design problem, like you chose the wrong design or something central to the system, we can't really fix it that easily without ripping out the guts of the system.

So in theory, identifying and fixing defects early on is the least expensive approach. It might even be possible to contain the impact of such fixes if detected in this early stage. But, despite having known this fact over the years, little has been done in the industry to provide tools and solutions to help identify design and architectural defects early on in the development lifecycle. Thus, in practice, it becomes a difficult task. In any case, even if we successfully identify a design or architecture defect, but do it late in the development cycle, it becomes next to impossible to fix it. So most systems in this stage will go live despite the looming problems ahead, hoping to try to patch it up as they go on. And most likely, the patches are going to be cosmetic and external such as allocating more resources (hardware/personnel) instead of eliminating the core defect in the system.

So, don't phunk with architecture and design when you can pay a little more attention from the beginning and get it right.

[With sincere apologies to Black Eyed Peas for borrowing the term Phunk, which they make it sound so cool.]

Some references:

• Software Defect Reduction Top 10 List by Barry Boehm and Victor R Basili
• Return On Investment from Software Process Improvement by Thomas McGibbon

Software Architecture Design

What does architecture mean to you? Do you care about architecture? If not, should you care?

Depending on what you do, the term architecture signifies different things. And the fact that it is used as a verb (i.e. to architect) and a noun (i.e. this architecture) complicates things.

For instance, my civil engineering friends talk about building architecture. They might also talk similarly about building design. Now, consider my friends who work in automobile engineering. I hear them talk a about automobile design. But I don't hear any talk about automobile architecture. Why? What makes these two engineering kind different? And finally, consider software engineering in which I can claim some expertise, at least more than I can in any other engineering domain. In software, we talk a lot about both architecture and design. We also talk about software construction. Most often there is no clear delineation between these. We seem to transcend between architecture and design and sometimes even construction.

So when you build a building, you architect it, design it and construct it. When you build a car, do you just design it and construct it. Does a car have architecture?

I have some views on this, but before taking on them here, I truly want to understand your take on this. What do you think? Do you architect? Do you design? Do you do both?

Software Architecture Design

While interacting with my colleagues and customers, I have seen the terms Offshoring and Outsourcing used interchangably. Sometimes they say Outsourcing whereas they really are talking about Offshroing and vice versa. Sometimes they are talking about both. Here is my attempt to compare and contrast the two terms and to also show the relationship between the two.

Basically, Outsourcing and Offshoring while often used together can also be used in a mutually exclusive manner. Let me explain what I am thinking using the following 4 basic combinational models. I will use the same definitions of the terms Owner and Outsourcer from my earlier post on Outsourcing Models.

1. No outsourcing, No offshoring - Onshore, Internal:
   Nickname: In-On : In this model, the Owner does not really Outsource, but might be delegating the activity to an internal group within the Owner's company. This activity is performed by the internal group onshore. No outsourcing, No offshoring
2. No Outsourcing, Internal Offshoring - Offshore, Internal:
   Nickname: In-Off : This is same as the above model, except that the activity even though is within the same company, is shifted to be performed offshore at the company's location in another country than the Owner's home country. This is offshoring without outsourcing.
3. Outsourcing, No Offshoring - Onshore, External:
   Nickname: Out-On : This is the first model of outsourcing where the Outsourcer is basically in the same location/country as the Owner.
4. Outsourcing, Offshoring - Offshore, External:
   Nickname: Out-Off : This is the second model of outsourcing where the Outsource is offshore. This model is the one that gets most attention in the industry thus leading to the two terms (Outsourcing and Offshoring) being used interchangeably in my opinion.

In defining the models above, I am tempted to use the term Insourcing to describe In-On and In-Off models, because that is what it is. But there are other usages of that term that led me to avoid using it for the time being. In-Off and Out-On models demostrate the mutually exclusive nature of outsourcing and offshoring.

Outsourcing is an increasingly popular trend in the IT industry. However, from an architectural prespective, there are many nuances about outsourcing that I have been focussing on lately to understand.

Let me discuss different models of outsourcing I have seen from a software development perspective. I suspect these models are applicable for any systems development. But, software is my game and I am sticking to it. Let us categorize the project activities broadly into 5 areas: Requirements/Analysis, Design, Development, Deployment and Management/Maintenance. The Owner is an entity (team, department, division, company) that owns the project and the Outsourcer is an entity (mostly in another company) that executes various tasks assigned by the Owner. The Owner is always in charge of the requirements so outsourcing that activity may not make much sense in most cases. Anyway, I have seen several different models of outsourcing when it comes to IT organizations working with an Outsourcer. I have categorized them as follows:

Model 1: All In (AI)
This is the traditional model of development where everything is done inhouse, i.e. no outsourcing.

Model 2: Development Out (DO)
Outsource only the development activity. The Owner is still responsible for design and is trying to leverage lower development costs by outsourcing the development activity. Once the Outsourcer completes development, the code is brought inhouse for deployment, management and maintenance.

Model 3: Design Out, Develop Out (DODO)
[No, not that Dodo, although there might be some similarities] This model extends Model 2, but in this case the design also becomes a responsibility of the Outsourcer. This is more suitable if the Owner has limited IT development capabilities (e.g. skills & resources).

Model 4: Design In, Rest Out (DIRO)
In this model, the Owner designs the system and delegates the development, deployment and management of the system to the Outsourcer. I don't see a value in this model for most scenarios. If you are going to hand off the management of the system to the Outsourcer, let them design as they see fit.

Model 5: All Out (AO)
This is the inverse of Model 1 where the Owner delegates all activities (except for Requirements) to the Outsourcer. This is similar to the ASP model. This is the model that most Outsourcers might prefer. If executed properly, this can be a win-win scenario for both parties.

Some of the customers I have talked to have been experimenting with Model 2 and Model 3. And this model exhibits the pain points of architecture and design. When you are designing a system and having someone else implement it, it becomes a tedious and sometimes impossible task to govern the architecture and design of the system on a continuous basis. A lot of architects in this model spend their cycles on the phone talking to the developers in a different world trying to communicate the design intentions and the confirm and validate the developer implementations to ensure those intentions are addressed and implemented.

What do you think? Do these models make sense? Have you seen other models? What are the (architectural/design/development related) pain points for architects when dealing with outsourcing?

May 26 2005: Also see this followup.

Ever find yourself wondering the difference between standards and specifications? A while ago I was working with one of our customers - T N Subramanian (TN) Chief Architect at RouteOne. TN first described to me about how there is this tension between Standards and Specifications in the industry. Although I do not recall the exact words he said, I was reminded of it again today after I presented on Pragmatic SOA to an audience in San Francisco. Randy Heffner, VP at Forrester Research, spoke before me and briefly mentioned and contrasted Standards and Specifications. I have been meaning to write on this topic for a while, so this brief entry ought to do it.

The problem is you can find any number of specifications proposed by any number of people. In the SOA and Web Services space, there are numerous standards and specifications, many of them competing with each other and this confuses the heck out of anyone. So as an architect, how do you deal with such an ocean of standards and specifications?

I think, the key is to remember that a specification in itself is useless without a community backing and a roadmap for standardization. I will only incorporate a standard into my implementation roadmap and not just a specification.

So the next time you look at a specification relevant to you, consider where it is coming from and where it is going. Unless the specification is submitted to standardization, it remains just that - a specification. So, as an architect, you probably do not want to adopt that specification until it becomes an official standard. Until then though, we can watch and learn.

One of the things I keep thinking about is how the concept of architecture^[1] has evolved in software development over the years. Gone are the days when we (sometimes meticulously) planned, designed and implemented each and every module and sub-system to build a system. And most of the time, using such a long (and waterfall?) life-cycle for development, the thing that we built ended up being close to obsolete when finished. But businesses were not as agile that they were willing to live with such a system. Many businesses bought systems and modified their processes to adapt to the capabilities, pitfalls, drawbacks of a product they bought or built. When businesses adapt to IT, you lose agility and competitive advantage. You want IT to adapt to business requirements so business keeps going at its pace without hindrance from IT. So back then architecture was a rigid concept and a goal, but, today architecture is very fluid adapting to changing business and technical environments. Due the the rapidly changing business and technology scenario, there is no longer a constant architecture to build towards. The architecture has to evolve as we are building the system taking into considerations technology obsolescence, evolving and emerging standards, changing business requirements and everything else that goes with it. Managing architectural quality^[2] becomes extremely challenging in this constantly changing^[3] environment.

Leaving architectural quality aside for a while, a number of changes in the industry can help us in implementing fuild architectures. On the enterprise software side, that is where Service Oriented Architecture (SOA) comes into play with loosely coupled services helping us build dynamic composite applications leveraging existing and new services. On the infrastructure side, grid computing hopefully fulfils the promise of seamless expansion of unlimited compute power and demand.

These times are challenging, but have never been more interesting. Are you up for it?

_{[1] The definition of the term Architecture can raise a decent debate among architects and developers, but that is to discuss another day. [2] Defining Architecture Quality is tricky and sometimes subjective. [3] Isn't that an oxymoron?}

What does the term "micro-architecture" mean to you? You probably have heard it somewhere. But, its meaning can vary depending on the domain/context most relevant to you: hardware or software (in broad terms). Micro-Architecture is not a new term. In the area of microprocessor design, the term micro-architecture has been used for a long time to describe processor design. A web search for micro-architecture yields almost all references to microprocessor design. There is even an IEEE committee on micro-architecture.

So, here is one definition of the term from Wikipedia:

Microarchitecture consists of a set of microprocessor design techniques used to implement the instruction set (including microcode, pipelining, cache systems, etc.).

And, here is another from The Anatomy of a High Performance Microprocessor:A Systems Perspective by Bruce Shriver and Bennett Smith:

Microarchitecture is the term used to describe the resources and methods used to achieve architecture specification. The term typically includes the way in which these resources are organized as well as the design techniques used in the processor to reach the target cost and performance goals. The microarchitecture essentially forms a specification for the logical implementation.

What I want to talk about is the usage of the term micro-architecture in software architecture. Here is one such perspective titled Micro-Architecture of Software Components (circa 1995), which tries to connect micro (object/component) level concerns to macro (application) level concerns. But that paper while filled with good intentions, I get lost quickly because I find it too confusing to link the two levels of abstraction. I feel the difference in abstraction levels make it hard to link those two levels. But, most references to micro-architecture take this approach to connect the high-level software architecture with low-level coding/implementation. Here is another paper with an interesting title One Architecture Does Not Fit All: Micro-Architecture Is As Important As Macro-Architecture. In this paper, micro-architecture is treated as software component engineering focussed on composition, compatibility and interoperability of componenets. And this too quickly gets into a discussion of low-level component design issues.

In the micro-architecture definition by Shriver & Smith above, if we replace "processor" with "system", the definition becomes pretty generic and can apply for hardware or software or anything in between. That leads to our definition of the same term as applied in software architecture.

Now I want to discuss how we started using the term in the area of enterprise software architecture. In the second edition of Core J2EE Patterns, we said this about micro-architecture:

We define micro-architecture as a set of patterns used together to realize parts of a system or subsystem. We view a micro-architecture as a building block for piecing together well-known, cohesive portions of an overall architecture. A micro-architecture is used to solve a coarser-grained, higher-level problem that cannot be solved by a single pattern. Thus, a micro-architecture represents a higher level of abstraction than the individual patterns.

Our goal was, and still is, to document reusable combinations of patterns, where each combination (micro-Architecture) solves a problem that is too large for any one pattern to solve.

In the diagram I have attempted to show the conceptual decomposition of a overall Architecture into micro-Architectures, patterns, components and their relationships. However, there are some missing parts that I should mention that are not shown in the diagram. It is not always possible to describe an architecture entirely as a set of micro-Architectures. Partly because much work needs to be done in the area of documenting reusable micro-Architectures. The missing parts are contextual in nature and are gaps in the architecture that are not addressed by micro-Architectures alone. These gaps (which I tentatively want to call bridges & connectors) are filled by individual patterns and/or custom implementations to make the architecture complete. Thus you can view an overall architecture as a union of these 2 sets:

Architecture = {µArch1, µArch2, ... , µArchN} + {bridges & connectors}

Next, I will try to describe more about the relationship between µ-Architectures, Patterns, and Frameworks.

What do you think?

Service granularity is a recurring hot discussion in design teams. So, I am glad that my amigo John Crupi started this discussion on SOA Granularity. The issue of granularity is a never ending discussion, and as John mentions is an easy way to start a debate. The issue of granularity is not just an issue in SOA. We have always had to deal with object/procedure/service <insert-your-favorite-here> granularity in designing a system. I think SOA magnifies the granularity issue by bringing it to the forefront of the design discussion.

IMO, granularity is a subjective matter. What is coarse-grained for one is not coarse-grained for another. I think this subjectivity is one of the reasons that granularity becomes a hot topic debate amongst designers. While granularity is one aspect that must be paid close attention to, I think it is better not to issue edicts around it. Because the moment you define a law of granularity, someone is going to break it! Trust me.

Pragmatically speaking, a (composite) SOA application will contain services of varying levels of granularity as shown in the diagram. I think what John is pointing out is that you want to ensure that the services that are exposed externally are the ones that are on the top (i.e. relatively coarse-grained). Anyway, I do agree with John that when talking about SOA services, we should be talking about business services. But, that does not really imply that we should always make a service coarse-grained. So, can a business service be fine-grained? It depends, on the business it serves.

For instance, let's say you are a famous online auction company and many successful small business partners use/depend on your auction platform. Let's say that these users require a service to be exposed that returns the latest bid price for a given item number. Is this too fine-grained? It sure feels like it when compared to other services like Create Auction,  End Auction, etc. But what if this is indeed business speak. That is, your business managers want to provide a Latest Bid Price service. And what if your partners using this service are willing to pay for it. Will you not expose this service because it is fine-grained? Hell, no! You are going to expose that service. Why? Because it is going to make money! Whether that be a Dime, a Rupee, a Yen or a Yuan. (OK, it is not always about money, but surely about generating some business value.)

So while we lean towards making SOA services as coarse-grained as necessary, remember that whether a service is coarse-grained or fine-grained or in between, if it makes business sense, it must be a good service to expose to the world.

I think this picture is useful to show the difference between hacking or not.

I wonder what the famous bard would do if he had to write software instead. We might have had classics today such as The Coding of the Shrew, or A Midsummer's Night Screen to name a couple. But, i think The Comedy of Errors still is a fitting title for a software book, I guess he was thinking ahead. And then perhaps we would have then had the saying To Hack or Not to Hack. Who knows.

Anyway, here is a picture that I use in my presentations to represent the value of using patterns in design and architecture. Overall, I have gotten good response to this picture from my audiences. Thought it might be useful to share in case others feel like using this.

What do you think?