Friday Sep 26, 2008

I would like to share some points from a recent conversation about requirements in a distributed ESB platform.This post is not going to talk about WS-Security or LDAP or encryption or any specific technology. Instead it is going to cover some of the different and maybe unexpected ways that security concerns are applied in the ESB/SOA space.

The conversation was centered on  ESBs, security, and SOA. The architects I was talking to have been looking at the ESB space for around 18 months and have been using Service Mix as a prototype platform thus far. They are considering moving to Open ESB which is what sparked the conversation in the first place.

As with a lot of large organizations I have been talking to lately, they have a preference for either open source or commercial open source as well as standards based software. Their preference is driven by a desire to avoid vendor lock in and to mitigate risk if a vendor or product line ceases to be available. The standards requirement is motivated by a desire to achieve the greatest level of interoperability and portability (and all of the other -ilities that "come" with standards).

Here is a description of a couple of challenges they've run across:

They look at the ESB as a facilitator/processor for two types of messages: data at rest and data in flight. Data at rest is any data that is persisted in a permanent storage facility (i.e. database) while data in flight includes real time data feeds and data from semi persistent sources like JMS queues. The data comes in all types of formats. Some are XML based. Some are binary, proprietary, text, etc. They also have a large interest in data streaming and processing real-time data feeds (similar to stock ticker) where there is a continuous open connection to the bus and data continuously streams through the pipe. There is no ack protocol and they want to insure that no data is ever lost.

Censorship

Because the data being processed has different levels of sensitivity, there is a need to be able to perform censorship on any message leaving the bus. The bus itself is considered a trusted zone (within it's own security domain). But any external communication must be filtered based on the authorization of the requester - or sometimes the destination. For example, it may be necessary to filter Social Security numbers out of a message that leaves the corporate boundaries. Fields or entire sections of a message may need to be removed or masked in order to insure the recipient all of the data they are entitled to but none that is restricted from them.

Identity Propagation

Service requests made from the local ESB node to any remote nodes (like for an invoke in a BP) must sometimes preserve the identity of the original requester and sometimes make the request under the authority of the ESB itself (using a certificate-based identity assigned to the ESB or the particular service itself). There is no requirement to propagate the actual user credentials, just a validated distinct name. Once a user has been authenticated, there is no requirement for re-authentication. Authorization, however is dynamic. Each access to a secure resource requires the resource to check the authorization of the authenticated user.

Secure Deployment

There is also a requirement for secure distribution of binaries. When there is a new service or a new version of a service that has been approved for use within the distributed ESB, it is distributed and deployed by a central authority. Not only does the binary need to be transported and deployed on remote ESB nodes, but there must be a guaranty that the bits that arrive at the remote ESB node are the same bits that were sent.

In other words, they need a secure deployment service that would allow complete remote deployment with no configuration to be done on the target ESB node. The intention is to minimize staff at the remote sites and relieve the local admins from the need to maintain or administer the ESB.

Friday Sep 05, 2008

This entry comes from a discussion I was involved in with a group looking to build a widely distributed ESB. The question started out around how to have a distributed NMR (Normalized Message Router) in Open ESB. But the real meat of the conversation was around how to communicate between geographically discrete nodes in an enterprise spanning ESB. If you don't know about Open ESB, you can get all the information you need at http://open-esb.org

A lot of ESB implementations solve the distribution problem by using a JMS solution as the message store. By using JMS, you get message persistence and the ability to have any node that can connect to the JMS server pick up the message and process it. On the surface this looks like a solid solution. And in some cases it is. Especially in cases where the ESB is mostly local and owned by a single organization (maybe even a single business unit).

The problem arises when you need to have an ESB solution that spans organizational or geographic boundaries. One of the biggest problems in JMS (and MOM solutions in general) is the fact that there is no defined wire protocol. All JMS clients require the use of the appropriate client libraries in order to interact with the JMS server. They also require that specific well-known ports are open outside the firewall. The particular ports vary depending upon which JMS vendors are used.  When you have nodes distributed all over the globe and owned by different business units and organizations, this can be quite problematic. Each node must have all of the client libraries required by all of the JMS providers used in all of the other nodes that may potentially be communicated with.It is also necessary to know all of the ports that all of these JMS servers are listening on.

Ouch.

So when implementing a widely distributed ESB, what is really needed is a generic wire protocol to use to exchange messages between nodes in different silos (business units, organizations, networks, or maybe even different continents). One potential candidate would be HTTP. HTTP is a simple, well defined  protocol used on standard TCP/IP networks. It doesn't require any specific client libraries in order to publish or receive a message.And there is a single well known port to connect to (mostly).

This is the point in the conversation where the group we were talking to brought up the fact that HTTP is a request/response protocol whereas JMS is not. Much of the communication across a widely distributed ESB is asynchronous (actually from a practical standpoint ALL communication across a widely distributed ESB should be asynchronous). So it would seem that HTTP is not a very good choice in this case.

I would respectfully have to disagree.

The communication we are talking about is between ESB nodes; not an external service consumer and the ESB. When a message is sent from one ESB node to another, the only response the target node needs to make is an acknowledgment that the message has been received (or a no-op if you don't care). When the remote node is finished processing the message, it will initiate a new communication with the original ESB node and send the response. There needs to be a correlation id associated with the message in order for the original node to complete the process the remote request was part of.

This is not fundamentally different than the case with JMS. There are two ways to get a response with a JMS message. You can set up a temporary destination as the reply queue or you can have a well known destination from which a response is read. In the former case, the temporary destination is used as a synchronous response. In the later, the message needs a correlation id in order to be associated with the asynchronous request.  For all intents and purposes, this is the same as the HTTP case described above.

In many cases these calls to external ESB nodes (service calls) are orchestrated by something like BPEL. Using BPEL makes the use of correlation Ids easy and supports long running processes.It isn't required that BPEL is used. But it is one of the options and has built in support for message correlation.

I am not sure that the group we talked to walked away from the conversation 100% convinced. But I think that after they have time to digest things and internalize them. They will agree that it is a viable option.

Relating all of this back to Open ESB, the nice thing is that there is a choice. You can configure your external calls to use any type of communication you want. I will talk more about that in an upcoming entry.




Thursday Sep 04, 2008

I recently had a visit with an interesting company that provides services for healthcare payers, providers, and vendors. Their business encompasses the seemingly contradictory slow IT adoption of healthcare businesses while being an innovator in the web and web 2.0 space.Since this particular entry is very specific to the Java CAPS product, I am going to use some terminology that is also very specific to Java CAPS.


The company been a Java CAPS customer for the past several years. They are currently running projects on CAPS 5.1.3. Current projects use eGate, eInsight, a few eVision pages and several of the eWays. They are deciding whether to keep CAPS and migrate to Release 6 or change vendors and rewrite their existing projects over time. Obviously the sales team doesn't want the CAPS presence to go into "maintenance mode" and be phased out of the company. Some of the directors were not familiar with CAPS except by word of mouth but had potential integration projects coming up and wanted a briefing.The hope was that if we could show them that the direction and the future of the CAPS line was promising they may not only renew but expand the presence of Java CAPS in the enterprise.

We spent some time discussing how the adoption of GlassFish and NetBeans was accelerating the pace of innovation at Sun and it was agreed by the directors in the group that transparent development brought about by Sun's open source model was a very positive thing to have as long as the commitment to the commercial product was not compromised. 

The current release of Java CAPS (Release 6) added a JBI container to the platform. The idea behind adding the JBI container was to provide the product with an extensibility platform. The future potential of the pluggable micro kernel architecture was appreciated. We took the opportunity to point out that Sun was the only major player in the SWI (Software Infrastructure) domain to offer a comprehensive and open platform that could be leveraged with third party vendors and collaborators as well as in house product offerings.(yes, I know it is a cheap plug for Sun. It is also true.)

The JBI container and some of the new components in Java CAPS come straight from the Open ESB community. Open ESB is the project in which new ESB components and capabilities are developed and then they are adopted into the commercial product once they reach a level of maturity that is appropriate to enterprise deployment.

One their largest concerns was the migration path to Release 6 and beyond. They have made a significant investment in developing OTDs and services (JCDs and BPs) that utilize them and want to be assured that their investments will not need to be thrown away or require major re-working.I was pleased to be able to tell them that virtually all of the projects they had created in CAPS 5.1 would migrate unchanged into CAPS Release 6.

Since they have a mature Java EE developer skill set in the company, they also want to make sure that these skills can be leveraged in the future. They were pleased with the prospect of creating the equivalent of JCDs (Java Collaboration Definitions) using standard MDBs and stateless session beans. They want to be able to use the existing OTDs (Object Type Definitions) in standard EE projects along with the JCA adapters.

Another item that came out of the discussion was that they have a need for a large EMPI (Enterprise Master Patient Index) solution. It is interesting that they were relatively unaware of the eView / eIndex product line in the CAPS 5.x releases. The Sun MDM (Master Data Management) Suite is a direct decendent of the former SeeBeyond Master Patient Index application eIndex. The difference is that now the whole thing has been open sourced under the Mural community. The MDM suite is a bundling of the mural components along with a lot of the other Java CAPS and Sun software products.

All in all it was a positive meeting and I think everyone was happy with the direction in which the Java CAPS product line is heading.

I know this entry is a little preachy and somewhat of an advertisement for Java CAPS. But I did want to share it because there are a lot of Sun's ESB customers using various versions of Java CAPS and it's predecessors and I wanted to show that other companies have the same questions as they do in whether or not to pursue Sun's ESB stack. I think they should.


Tuesday Aug 19, 2008

This entry is about an experience I had where a large insurance company was looking at our Java CAPS product as just a BPM solution. The adventure here to me was that the best way to help this company was NOT to do the POC they were requesting.

[Read More]
I created this blog to capture and share some of the experiences I encounter in real world situations where companies are struggling to solve their IT problems by leveraging ESB and SOA technologies.[Read More]

This blog copyright 2008 by Michael Jenkins