SOAs & ESBs

Dr. Dobb's Journal February, 2005

Ensuring that your ESB delivers an SOA

By James Pasley

James is chief architect at Cape Clear Software. He can be contacted at james.pasley@capeclear.com.

The design principals of Service-Oriented Architectures (SOAs) are widely considered to be best practice when solving integration problems or as part of business process modeling. An Enterprise Service Bus (ESB) gives you the tools and infrastructure necessary to build an SOA. But with all the components and tools provided, you can lose sight of the SOA while working on the ESB. Consequently, care must be taken to ensure that the end result is a new architecture, and not just a better way to perform point-to-point integration. In this article, I present an overview of SOA, describe the ESB's role, and present tasks you can expect to tackle when deploying an SOA.

Service-Oriented Architecture

An SOA is a way of approaching the task of creating software. You can think of it as either a collection of architectural concepts or a programming model. These concepts should be understood and followed carefully to make effective use of an ESB. Within an SOA, all functionality is provided by services. This provides a level of flexibility in deploying and reusing services not previously attained—services that:

Are remotely accessible over asynchronous or synchronous transports.
Have a well-defined interface described in an implementation-independent manner.
Are self contained, and can be used independently of any other services.
Perform a specific task.
Are loosely coupled. In other words, loosely coupled APIs should be: coarse-grained, representing an overall business function or service; based on the exchange of documents rather than the datatypes used in RPC-style APIs (this typically means that operations will have a single input or output parameter that is defined using XML Schema); accept a range of different documents that contain the requisite information; not include explicit or implicit state assumptions (that is, be connectionless); and are dynamically discoverable at runtime rather than statically bound.

However, an SOA is more than just a collection of services—it adds a number of principals, of which interoperability is the most important and the core to the idea of SOA. Interoperability is achieved through the use of Standards. The web-services Standards stand out above all others in satisfying this requirement.

Another core feature of an SOA is the reuse of existing assets. Deployment of an SOA within an organization is an exercise in creating services from the existing IT systems and infrastructure.

Once an SOA is in place, the existing IT systems within an organization can be viewed as services that provide business functions. These are easily integrated because they provide well-defined interfaces and can be accessed using standard protocols and transports. This provides the basis on which to orchestrate services into new services that reflect business processes.

The Role of the ESB

The deployment of an SOA requires the conversion of existing systems into services. The tasks involved in achieving this may be repeated for each system and a common set of components may be needed to provide additional functionality. An ESB is a collection of servers and components that provide these functions and a set of tools to assist in the conversion of existing systems to services. The ESB:

Solves integration problems at the transport layer by supporting Standards-based transports such as HTTP. It also supports a variety of commonly used transports such as MQSeries, or even shared folders for batch-processing applications, so that existing systems can be exposed as services supporting new transports.
Provides for the routing of messages across various transports.
Provides tools to help expose the existing systems as web services; for example, a wizard to convert CORBA IDL to WSDL. In addition, it provides the runtime piece to convert SOAP requests to the web service into IIOP requests to the CORBA Object. You can expect to find such tools for multiple classes of systems such as Java, J2EE, databases, and mainframes.
Provides adapters to a variety of packaged applications such as SAP or PeopleSoft. However, with the proliferation of web-services Standards, many of these applications are providing web-service interfaces directly. Therefore, the need for the ESB to provide such adapters reduces over time.
May also host services. While the emphasis remains on the reuse of existing systems, an ESB can be used as a powerful application-development environment. Features such as database integration and adapters to applications (SAP, for example) ensure rapid application development.
Addresses quality-of-service issues such as security and data validation, which arise as a result of exposing services to a variety of clients.
Provides for the transformation of both XML and other document formats.

The Lifecycle of an Integration Project

The lifecycle of a software project that uses an ESB can be thought of as constructing an SOA in layers, starting at the existing infrastructure. As such, there is a natural progression to the development of services within an SOA; see Figure 1.

The steps involved can be grouped together into three categories:

Exposing the existing systems as core services.
Building business services from these core services.
Using the business services to achieve business process modeling.

An SOA emphasizes the reuse of existing assets, so exposing the existing systems is a natural starting point. This must be balanced with the emphasis on creating services that reflect business needs and not technology. This means that the top-down approach should also be considered. That said, in this article, I follow a bottom-up approach because it facilitates a better explanation of the tasks involved.

Exposing Core Services

Exposing existing IT systems as services is typically the first task you perform when using an ESB. ESBs provide a number of tools to help with this. For programming APIs (such as Java, EJB, or CORBA), wizards automatically generate WSDL from the APIs. For messaging systems, transports are provided to expose particular queues as web services.

For batch-processing systems, the documents processed are often text-based and typically consist of fixed-width fields. The payload of the exposed web service should be XML to achieve maximum interoperability. This requires transformation of the payload document. ESBs can provide tools (such as Cape Clear's Data Interchange) to facilitate the creation of transforms between varieties of document formats. Because no formal description of the data formats may exist, this task cannot reach the same level of automation as is achieved for programming APIs. Depending on the nature of the existing systems, the creation of these transforms can be a significant part of the development effort. As a result, the tools to support this should be reviewed carefully when selecting an ESB, to ensure that this task does not become a programming one.

With the tools provided by the ESB, you have solved a number of integration issues and have added web-service front-ends to your existing systems. The web services have the following features that, depending on the nature of the existing systems, may not have been present before:

Can be invoked remotely, over a standard transport by a variety of web-service clients.
Has an XML-based payload format.
Uses a standardized protocol envelope (SOAP Envelope).
Has a formal description of the interface (WSDL and XML Schema).
The payloads in messages are validated against the WSDL/XML Schema to ensure that invalid data is not passed on to the application.

Building Business Services

The first stage in the lifecycle of a project solves many integration issues by creating web services from the existing systems. This fulfills a number of the characteristics of a service as defined by the SOA. The web services can be invoked remotely, have well-defined interfaces, and (depending on the systems they were created from) are self contained. However, other aspects may not be there. Web services that conform to the SOA definition can be thought of as "business services," based on the requirement that they perform a specific task or business function.

This is the point at which you need to take a step back and consider what you've done. To turn these web services into services as defined by SOA, you may have to reconsider the API to each service and evaluate whether they can be described as loosely coupled.

The web services created from mainframe or batch-processing systems most likely already meet these remaining requirements because they process documents in an asynchronous manner. The web services created from the other systems are more likely to be fine-grained, RPC-based, synchronous systems. If the web services automatically generated from the existing systems cannot be described as loosely coupled, then new descriptions (WSDL) should be created.

To illustrate loosely coupled services, consider how a customer might purchase car insurance. In the first scenario, the customer could use the telephone to carry out the transaction. This approach is synchronous—it requires an operator to be available to handle the customer's request. The telephone conversation forms a connection in the context of which the interactions take place. The exchanges are fine-grained—the operator asks a series of individual questions to obtain the necessary information.

Now consider a second scenario, in which the customer fills out a form and mails it to the insurance company. This approach is course grained and handles documents. It is asynchronous—it does not require an operator to be present at the exact time the customer fills out the form. This second scenario is consistent with the principals of an SOA, while the first one is not.

Look at this example again, but this time in code. Consider the Java API in Listing One that is to be exposed as a web service. It has a method to initiate the interaction—one that provides some additional data. Another method is used to return the quoted price, and finally, one that processes the application and bills the customer.

Running a JAX-RPC-compliant, WSDL-generation tool over this kind of a Java interface results in the WSDL in Listing Two. This solves a number of integration issues; for example, a .NET client that invokes on this web service can now be written. However, to create a loosely coupled service from this, a new WSDL interface must be designed. For this, you need a WSDL editor.

The granularity of the API should be increased to provide fewer operations, but ones that accept documents as parameters. At the same time, the API should be made connectionless by eliminating the use of the reference parameter. This leaves you with two operations—calculateQuote and processApplication. Both of these operations accept the application document as a parameter. This may seem inefficient, but it is done to ensure that the API is connectionless. Applying these principles to the WSDL results in Listing Three. In this new WSDL:

The data types from the original WSDL have been combined under a single element (InsuranceApplication) to form a document.
There are just two operations: calculateQuote and processApplication.
Both of the operations take the entire InsuranceApplication as input.
The responses are returned to the caller asynchronously, via a callback API (InsuranceServiceCallBack).

In light of the aforementioned example, the following guidelines should be applied to WSDL files as part of evaluating whether they can be described as loosely coupled. New WSDL files should be created where necessary.

portTypes should have a small number of operations that reflect the business tasks.
Each operation should have a single parameter defined as a document.
The API should not include the concept of a session, or operations that depend on data passed to previous operations.
The API should support asynchronous interactions. A separate interface (portType) should be provided for responses that the server needs to send back to the client.

Implementing the New Service

An ESB provides tools to generate Java skeleton implementations for web services. This new WSDL can be used to generate such a Java skeleton, providing the basis for the implementation of the new service. This service can act as a client of the auto-generated service.

Of course, the equivalent changes could be made to the original Java API. However, there are benefits to working at the XML level. An important part of an SOA is the definition of data types within the XML Schema that can be reused across different services. The use of common data types defined using XML Schema should be considered as early as possible when working with an SOA. This greatly reduces the amount of data transformation that must be performed later when reusing these services.

Implementing Service Policies

By now, you have created business services that can be easily integrated and reused. However, exposure to business partners or deployment within an enterprise requires additional functionality. This is functionality that can be provided by the ESB itself and is typically common to all services.

These features can also be described by the WSDL or by policy statements included within the WSDL; see Figure 2. This step in the project lifecycle can be considered the point at which the WSDL changes from being a description of the API to becoming a contract between the service and its clients. The additional features provided by the ESB, such as security, management capabilities, auditing, logging, or even custom features can be added. These can be added across the different services implemented using an interceptor or message-handler framework. This is an area where you can expect an ESB to provide components that can be easily combined with the services you have built.

Building Business Processes

With services that reflect business functions in place on the ESB, new services that model complete business processes can be created. Business Process Execution Language (BPEL) is a language designed for just this purpose. A BPEL editor should be provided to enable the existing services to be combined into processes. The ease with which new business processes can be developed depends in part on how successful the design of the business services has been. BPEL is a language designed to model business processes—it is not a place to solve integration issues. These should be solved in the lower layers.

Conclusion

An ESB provides effective tools and services that can be used to construct an SOA. However, careful thought and planning is required in order to ensure that the resulting system is truly an SOA. This requires a good understanding of the principals of SOA to evaluate the output of the tools provided by the ESB. The effort put into the design of individual services exposed on the ESB results in benefits when the task of business-process modeling is performed.

DDJ

Listing One

public class CustomerDetails {
    public String name;
    public String street;
    public String city;
    public String state;
}
public class CarDetails {
    public String make;
    public String model;
}
public class BillingInfo {
    public String creditCardNumber;
    public String exprityDate;
}
public interface InsurancePortType {
    public String createApplication( CustomerDetails customer );
    public void setCarDetails( String reference, CarDetails details );
    public Decimal calculateQuote( String reference );
    public void processApplication( String reference, BillingInfo info );
}

Back to article

Listing Two

<?xml version="1.0" encoding="UTF-8"?>
<definitions name="Insurance"
    targetNamespace="http://capeclear.com/CarInsurance.wsdl"
    xmlns="http://schemas.xmlsoap.org/wsdl/"
    xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/"
    xmlns:tns="http://capeclear.com/CarInsurance.wsdl"
    xmlns:xsd="http://www.w3.org/2001/XMLSchema"
    xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/"
    xmlns:xsd1="http://capeclear.com/CarInsurance.xsd">
    <types>
        <xsd:schema targetNamespace="http://capeclear.com/CarInsurance.xsd">
            <xsd:complexType name="BillingInformation">
                <xsd:sequence>
                    <xsd:element name="creditCardNumber" type="xsd:string"/>
                    <xsd:element name="expiryDetails" type="xsd:string"/>
                </xsd:sequence>
            </xsd:complexType>
            <xsd:complexType name="CustomerDetails">
                <xsd:sequence>
                    <xsd:element name="name" type="xsd:string"/>
                    <xsd:element name="street" type="xsd:string"/>
                    <xsd:element name="city" type="xsd:string"/>
                    <xsd:element name="state" type="xsd:string"/>
                    <xsd:element name="age" type="xsd:string"/>
                </xsd:sequence>
            </xsd:complexType>
            <xsd:complexType name="CarDetails">
                <xsd:sequence>
                    <xsd:element name="make" type="xsd:string"/>
                    <xsd:element name="model" type="xsd:string"/>
                    <xsd:element name="serialnumber" type="xsd:string"/>
                </xsd:sequence>
            </xsd:complexType>
        </xsd:schema>
    </types>
    <message name="setCarDetailsResponse" />
    <message name="createApplicationResponse">
        <part name="return" type="xsd:string"/>
    </message>
    <message name="setCarDetails">
        <part name="reference" type="xsd:string"/>
        <part name="carDetails" type="xsd1:CarDetails"/>
    </message>
    <message name="calculateQuoteResponse">
        <part name="return" type="xsd:decimal"/>
    </message>
    <message name="processApplicationResponse" />
    <message name="processApplication">
        <part name="reference" type="xsd:string"/>
        <part name="billingInfo" type="xsd1:BillingInformation"/>
    </message>
    <message name="createApplication">
        <part name="customer" type="xsd1:CustomerDetails"/>
    </message>
    <message name="calculateQuote">
        <part name="reference" type="xsd:string"/>
    </message>
    <portType name="InsurancePortTypeServer">
        <operation name="calculateQuote">
            <input message="tns:calculateQuote"/>
            <output message="tns:calculateQuoteResponse"/>
        </operation>
        <operation name="createApplication">
            <input message="tns:createApplication"/>
            <output message="tns:createApplicationResponse"/>
        </operation>
        <operation name="processApplication">
            <input message="tns:processApplication"/>
            <output message="tns:processApplicationResponse"/>
        </operation>
        <operation name="setCarDetails">
            <input message="tns:setCarDetails"/>
            <output message="tns:setCarDetailsResponse"/>
        </operation>
    </portType>
</definitions>

Back to article

Listing Three

<?xml version="1.0" encoding="UTF-8"?>
<wsdl:definitions name="InsuranceService"
    targetNamespace="http://capeclear.com/CarInsurance.wsdl"
    xmlns="http://schemas.xmlsoap.org/wsdl/"
    xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/"
    xmlns:tns="http://capeclear.com/CarInsurance.wsdl"
    xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/"
    xmlns:xsd="http://www.w3.org/2001/XMLSchema"
    xmlns:xsd1="http://capeclear.com/CarInsurance.xsd">
    <wsdl:types>
        <xsd:schema targetNamespace="http://capeclear.com/CarInsurance.xsd">
            <xsd:element name="InsuranceApplication">
                <xsd:complexType>
                    <xsd:sequence>
                       <xsd:element name=
                         "CustomerDetails" type="xsd1:CustomerDetails"/>
                       <xsd:element name=
                         "CarDetails" type="xsd1:CarDetails"/>
                       <xsd:element name=
                         "BillingInformation" type="xsd1:BillingInformation"/>
                    </xsd:sequence>
                </xsd:complexType>
            </xsd:element>
            <xsd:complexType name="BillingInformation">
                <xsd:sequence>
                    <xsd:element name="creditCardNumber" type="xsd:string"/>
                    <xsd:element name="expiryDetails" type="xsd:string"/>
                </xsd:sequence>
            </xsd:complexType>
            <xsd:complexType name="CustomerDetails">
                <xsd:sequence>
                    <xsd:element name="name" type="xsd:string"/>
                    <xsd:element name="street" type="xsd:string"/>
                    <xsd:element name="city" type="xsd:string"/>
                    <xsd:element name="state" type="xsd:string"/>
                    <xsd:element name="age" type="xsd:string"/>
                </xsd:sequence>
            </xsd:complexType>
            <xsd:complexType name="CarDetails">
                <xsd:sequence>
                    <xsd:element name="make" type="xsd:string"/>
                    <xsd:element name="model" type="xsd:string"/>
                    <xsd:element name="serialnumber" type="xsd:string"/>
                </xsd:sequence>
            </xsd:complexType>
        </xsd:schema>    </wsdl:types>
    <wsdl:message name="calculateQuote">
        <wsdl:part element="xsd1:InsuranceApplication" name="reference"/>
    </wsdl:message>
    <wsdl:message name="calculateQuoteResponse">
        <wsdl:part name="return" type="xsd:decimal"/>
    </wsdl:message>
    <wsdl:message name="processApplication">
        <wsdl:part element="xsd1:InsuranceApplication" name="application"/>
    </wsdl:message>
    <wsdl:message name="processApplicationResponse">
        <wsdl:part name="policyNumber" type="xsd:string"/>
    </wsdl:message>
    <wsdl:portType name="InsuranceServiceCallBack">
        <wsdl:operation name="QuoteDetails">
            <wsdl:input message="tns:calculateQuoteResponse"/>
        </wsdl:operation>
        <wsdl:operation name="ApplicationDetails">
            <wsdl:input message="tns:processApplicationResponse"/>
        </wsdl:operation>
    </wsdl:portType>
    <wsdl:portType name="InsuranceService">
        <wsdl:operation name="calculateQuote">
            <wsdl:input message="tns:calculateQuote"/>
        </wsdl:operation>
        <wsdl:operation name="processApplication">
            <wsdl:input message="tns:processApplication"/>
        </wsdl:operation>
    </wsdl:portType>
</wsdl:definitions>

Back to article