4 Generations of Services

While working at MCI in 1995, I was exposed for the first time to the benefits of service orientation. I was responsible for the order entry application for 1-800-MUSIC-NOW (which turned out to be a marketing flop), in which call center agents needed to process credit card payments for music CDs. Fortunately for me, MCI was already accepting credit card payment for long distance service. Even more fortunate was that a forward-thinking architect working in the credit-card processing group in Nashville had anticipated future uses of the software his group had developed. What we had feared would require several months of effort turned into about a week of coding to properly access the TCP/IP-based credit card authorization service.

I am sure my experience at MCI is not unique. Across industries, smart architects have been exposing functionality as reusable services. This got me thinking about the how service oriented architectures have been evolving over the years. Those of us working to expand capabilities in SOA tooling have recognized that services and SOA have been around far longer than the "web service standards" that have made SOA so popular over the last few years. It is apparent that in the software industry, we have gone through phases of maturity in our service-oriented implementations.

The long term vision of SOA is to enable rapid assembly of applications by orchestrating services into new business processes. Unlike the modest cost savings I experienced at MCI, the cost reductions achieved in a mature SOA could be immense. The traditional 12 or 18 month development cycles involving armies of analysts and developers are replaced by a business analyst drawing a flow graph of the new process, then clicking a button for deployment to enterprise-ready infrastructure. The "service oriented" approach is measured in days or weeks using a fraction of the engineering resources.

The industry is a long way from realizing the SOA vision, of course, but service orientation has certainly evolved a good deal in the past decade. To better understand this evolution, it is helpful to categorize technologies into generations. For example, programming languages have been categorized as they matured, from machine code to assembly, then FORTRAN, C/C++ and Java, and finally 4GLs like PowerBuilder, SQL, ColdFusion. Each generation achieved higher levels of abstraction and more statement power, making it ever-easier to translate process descriptions and algorithms into machine-executable form.

As service orientation has matured, a number of generations of services have emerged. These generations are defined loosely by how well they support or implement the major characteristics of SOA: standards-based, loosely coupled, coarse-grained, and business-orientation enabling an analyst to understand and manipulate it. Early generations reflect less of these core SOA values, and later generations are more comprehensively based on these values. Looking at the landscape of services over the last 5 years, I see 4 generations of services:

1st Generation Services - Simple services coded in a 3GL language like C, C++, C#, or Java, which don't use modern service standards like WS-* or REST. These services tend to tightly couple the consumer with underlying resources. Older distributed computing technologies like CORBA and DCOM fall into this category as well.

2nd Generation Services - Services that are standards-based and fairly simple, like implementing an operation to retrieve, modify, create, or delete a data set on a database. These services can often be auto-generated from other sources, such as from a Java or C# class, an EJB, or a database query. These services tend to reflect a method on an object, or expose an underlying implementation strategy like a relational table. They are easy to create, but because they are technology-oriented rather than business-oriented, they are unwieldy to use directly in a business process. Instead, they require combining with other services and logic to provide the proper level of granularity for orchestration.

3rd Generation Services - Truly "service-oriented", these services are aligned with a step in the business process. Loose coupling is achieved by explicitly defining data formats that are the payload of the service request and response, and these formats are driven by analysts that know the business process at hand, not by technologists attempting to optimize execution times and storage requirements. These services are often created by stitching together and transforming 1st and 2nd generation services to achieve a coarse-grained service that is suitable for orchestration and at the same time achieves loose coupling.

4th Generation Services - These are 3rd generation services that are institutionalized as managed, secure, governed, and reusable services. 4th generation services involves an ecosystem of SOA-aware technologies and procedures, which allow construction and management of business processes and higher level services. Given time to achieve 4th generation services, a company will maximize the benefits of SOA, enabling them create and modify business processes quickly to meet the changing demands of the business.

The precise boundaries of the generations and the metrics used to categorize a service can certainly be debated. But the concept is valuable in that it helps clarify the goal, to ultimately maximize the benefits of an SOA. These concepts also help justify implementing services in projects that have more modest goals. Maybe you know a function will be re-used across departments. Or, you anticipate that your basic service will be used by a 3rd generation service, thereby growing the foundation that will become an SOA based on 4th generation services. We don't need it all today, but we can derive concrete benefits in the short-term while maneuvering towards an even more attractive set of benefits in the future.

SOA Sweet Spot: Integration

A survey released this week from AmberPoint, titled "State of SOA Adoption Survey", shows steady growth in SOA adoption among respondents. While I always take a vendor-sponsored survey with a grain of salt, AmberPoint seems to have found their way to real SOA implementers in the corporate environment. Systems Integrators, vendors, and consultants were culled out of the distribution list, leaving only end-users: "a database of IT professionals who have an understanding for SOA concepts and methodologies.... a large population of architects, operations staff and developers." Only a fraction of respondents were AmberPoint customers (15%). The promising news from the survey is that the vast majority of respondents (98%) viewed SOA projects as having a high degree of success. That figure breaks down with 38% stating their projects were a success, achieving all desired goals, and 60% cateogorizing projects as "partially successful", meeting most of the goals.

I happen to agree with the results, as they are consistent with a steady growth in SOA projects that I have seen. One of the more compelling survey questions deals with what problems companies are solving with SOA - Integration tops the list by far, with 75% of respondents saying SOA addresses integration-related issues. I saw one article this week about flailing SOA projects (Time for a ’stimulus package’ for SOA? ), but there seems to be a developing consensus that SOA doesn't solve every problem. No surprise there. Like any technology, SOA is good at solving many problems, but some problems are better left to other solutions. What this survey clearly indicates is that integration is SOA's sweet spot.

Junk Yard Parts

Dana Blankenhorn at ZDNet talked today about how the open source ecosystem is a lot like the Pull-a-Part yards in the South and Midwest (see http://blogs.zdnet.com/open-source/). A Pull-a-Part is an auto junk yard that inventories its parts for easy identification and retrieval. I remember my first experience at a junk yard. My dad dragged me along in the hunt for a replacement radio for our Datsun B-210 Hatchback. It was cheap, too: $15 if we used our tools to pull the radio, or $20 if they pulled it out for us. To my surprise, Dad opted for the more expensive package, not wanting to waste his efforts if the radio didn't work.

Dana makes the point that Sourceforge is full of software "parts", and any system you build is made of many parts:

You have to locate them, put them together yourself, and use your own tools. With over 166,000 projects in stock, you're bound to find the one you need.

You can't build today's complex sites and systems from scratch, any more than you can build your own car. But with parts, the right tools, skill, and patience, you can build something very good, very quickly.

And this is why open source is the development platform of choice. Everything you need is visible, right in the yard.

Of course, despite the title of this post, what you get from Sourceforge is not junk. True, many projects are immature, but the leading projects rival the quality and depth of features found in leading commercial products. And if you use the right tools and skills in building your system using quality parts, the results can be excellent.

I would extend Dana's analogy by noting that many leading open source projects go beyond just making parts available, as they provide value-added services and support. Just as Dad didn't want to waste time pulling a part that might not work, as a user, you can pay a small premium for the added assurance that your open source "part" will fit nicely into your project. In addition to free support from the project's community, you can buy support to guarantee answers in a reasonable timeframe, or buy consulting time to help you install the part into your system. Fortunately, there is ample room for both types of users in the open source community: those like Dad willing to pay for a little extra assurance, and those like me willing to pull their own radio (I'm certain now that I would have broken it!). As it turned out, we installed the radio and it worked great. I did my part by handing Dad the tools.

The Un-ESB: Spring Integration

SpringSource recently announced a new project, Spring Integration. See The Server Side at http://www.theserverside.com/news/thread.tss?thread_id=47868 for an active discussion and links to related resources.

Much of the discussion revolves around how the new project works with, or competes against other integration software. I am accustomed to seeing an integration package touted by its marketing department as the new category killer - the best ESB, ETL, EII package available. So it struck me as conspicuous that Mark Fisher, Spring Integration's project lead, failed to categorize the software into one of these well-known (but over-hyped) integration strategies.

From the information I've been able to glean from Mark's blog and the press announcements, the Spring Integration project appears to be a set of tools that let you add ESB-like capabilities to your application. Some of Mark's quotes mimic the value propositions you hear from ESB vendors, for example:

"What this does is it provides a layer of insulation that manages those different endpoints and routes them to the same service implementation..."

It sounds like a service developed using Spring Integration is made available through any number of transports. This is a core ESB feature. So, why wouldn't SpringSource just come out and say that Spring Integration is Spring's implementation of an ESB? After pondering this for a while, I've concluded that its my own biases causing my desire to slap a label on this technology. As project leader for XAware.org, I get questions from analysts and corporate users all the time, asking what category XAware fits into. The analysts like Gartner, Forrester, and IDC educate the masses of corporate management on what technology to buy. A CIO needs to know he's spending 7 figures on the "Best XYZ" on the market. As a software company, you better have a great XYZ to sell to big corporations.

But SpringSource doesn't have to play that game. They're not talking to your CIO. Instead, they are providing technology features directly to implementers. In this case, Mark Fisher and team member Alef Arendsen have both stated that Spring Integration implements integration patterns described by Gregor Hohpe's book, "Enterprise Integration Patterns". You can see many of these patterns at Gregor's site, here:

http://www.enterpriseintegrationpatterns.com/eaipatterns.html

I applaud the SpringSource team for not branding their project an ESB, even though it shares a similar feature set. Though SpringSource has not issued a specific list of patterns supported by the project, I like the direction they are heading. The integration market is fragmented and full of marketing hype. By pointing to well-defined integration patterns, SpringSource strips the marketing glaze and displays a transparency that is very refreshing.

Extreme Governance?

There was a lot of talk about governance this week at Gartner's Application Architecture, Development & Integration Summit in Las Vegas. What is SOA governance? Simply put, SOA governance defines the policies, procedures and rules of how an organization implements and manages its SOA. Just like we have a development process guiding how we develop core software (XAware uses Agile/Scrum), SOA governance defines the process for SOA development initiatives across the enterprise. Issues to be managed include who can define services, who implements services, permissions for who can access them, on what platform they run, and who pays for development and maintenance.

At the show, I talked with many enterprise users to get reactions to the governance talks. The message from Gartner is clear - Paulo Malinverno stated in one presentation that an SOA with no governance is doomed to failure. But most of the companies I talked to, in trying to implement SOA, are either not implementing governance, or are still struggling with how to do it. I am convinced many others have governance, but are calling it by a different name. The common strategy seems to have a single point of authority, such as an enterprise architecture group, which defines the policies and ensures compliance.

In a conversation with Gartner analyst Roy Shulte, I asked his thoughts on SOA governance. In his view, governance is merely "the workflow in IT". Again, this sounds like nothing more than a process to manage services and other components in the SOA.

Just Enough Governance

Paolo and others caution against too much governance, which might smother and cripple a fledgling SOA initiative. In this respect, I look at agile development processes as the model for appropriate levels of controls. Scrum and XP both value working software over documentation and heavy-handed processes. In fact, I like the term "extreme governance" as a moniker for SOA governance in an agile environment. For developers, the connotation is light-weight, as-needed controls. For the executive who skims over details, the company's "extreme governance" process should win bragging rights in the fight to maintain firm control over the IT rebels. We'll fill him in on the play on words later.

For my part, I would simply recommend to users to stay pragmatic. Services are becoming easy to build, so the number within the enterprise will naturally grow rapidly. The key is understanding that some of the services should become "investment engines" that deserve more oversight to extract meaningful, enterprise-wide value. Services on the edge of an enterprise require more control than services deployed for departmental use. Services with broad use potential within the enterprise should be institutionalized and deployed on enterprise-scale, operationally sound infrastructure. To me, staying pragmatic and applying controls in the proper contexts is the definition of "extreme governance".