What is Enterprise Integration?

The author provides key guidelines that endeavor to simplify this tedious, yet imperative, task.

Introduction

The past decade has seen the world change more than ever before. Technology innovations are no more a fanciful pastime that attracted only the enthusiasts, geeks, engineers and scientists. They fuel the economies of today.

Heraclitus, the Greek philosopher professed 2500 years back that — “the only constant is change.” In the new millennium, this age-old observation seems to have a bigger significance than ever before. This is perhaps the reason why the role of the CIO in an organization is of paramount importance. Technology is driving the way companies evolve, organize, dissolve, merge and succeed.

According to recent studies, conducted by several independent organizations and media entities, the biggest predicament that CIOs of large enterprises face is the issue of Enterprise data integration. In order to stay ahead of competitors, there is always persistent pressure to reduce costs from the finance department. Since the tech meltdown, the budget squeeze is a reality that every CIO faces. It is common knowledge that many projects are nipped at the buds as CIOs succumb to budgeting pressures.

While it is extremely important for CIOs to reduce Total Cost of Ownership, the focus must be on using strategies that work best for the Enterprise. Most CIOs agree that the biggest challenge is evolving the right strategy for their Enterprise Integration (EI).

The past decade or so has seen the enterprise computing market grow at exponential rates. This has meant that a number of large and small players have entered the market with an eye on garnering further market share. Every vendor tries to resolve the problems and challenges an enterprise faces in his/her own unique approach. This has resulted in different interpretations and explanations for most terms used in the enterprise computing parlors.

If you analyze the definitions of this term provided by most vendors, you will observe that some of the facts are common in all of them. I will attempt to arrive at a definition that does justice to these multiple interpretations. EI is all about integration of all the IT activities of an organization, so that data is made available seamlessly irrespective of the applications and technologies being used. EI enables the sharing of data and business functions across applications.

For a growing organization it is vital that information is shared across departments, locations, users and partners. Most organizations start off with custom-built applications that are designed for specific needs. As the organization grows, the IT section adds more applications depending again on the growing needs of the organization. Most decisions are almost always taken based on current demands; rarely does a growing organization take into consideration the demands of tomorrow.

Once an organization reaches the size of an enterprise company, the challenges are even more humongous. That is when even a multi-million dollar IT budget sans the right strategies fail.

EI is the right strategy to ensure that every department gets the benefits of years of IT investment. EI does not mean a complete revamp of the IT applications and infrastructure. It is all about ensuring that investments made in the past are protected, and the future is bright and safe.

To get the best out of the past, the CIO needs to ensure that an EI project is kicked off, which will extract information and functionality of a proprietary application used by a department or a team within the enterprise and make that available to every organization within the enterprise. EI essentially evolves a framework that can be used by the enterprise to generate a “single view” of all their enterprise data and infrastructure.

By doing so the company can adapt rapidly to business changes, take decisions faster and implement those decisions without delays. Every organization, big or small, starts off with a cluster of small stand-alone applications that would have served the organization during the early days of IT investment. A change to traditional Enterprise Resource Planning (ERP) typically means that these investments are scrapped and new applications employed. However, in the case of an EI project, existing users can still use stand-alone software applications while data from these applications can be made available to other users in different departments within the Enterprise.

This ensures that existing users need not learn new applications, familiarize with new user interfaces and adapt to new applications. At the same time, other users can benefit from these independent applications at the end of the EI project.

When new applications are being implemented, the EI framework makes it congenial for all departments and users within the enterprise to make best use of a new application as the business processes are integrated as part of the EI integration.

Return on Investments (ROI)

It is apparent that an EI project deployed correctly results in major cost advantages. Because of the complexity of integration and the mixture of tangible and intangible benefits, it is rarely possible to build an integration value case that is purely financially based. It is safe to estimate that in an EI project you can avoid a rewrite of at least 30% of legacy applications. There are unaudited reports on avoiding complete rewrites of up to 60% of legacy applications by some enterprises. This, for an enterprise, means savings of millions of dollars. Apart from the direct cost savings, an EI project usually takes much lesser time than a fresh implementation of an ERP system.

While planning an EI project, it is important to make an estimation of the tangible ROI benefits at least. These include:
1. Reduced development costs for new projects;
2. Reduced maintenance costs;
3. Reduced resource needs; and
4. Reduced training costs.

A CIO planning an EI project should project the tangible benefits as a part of the budgeting exercise so that decision makers on the finance outlay understand the benefits.

Concurrently, the CIO needs to take into confidence the real beneficiaries of an EI project, different departments and business units within the enterprise, before he embarks on the project. It is important to assign values depending on the functions of the business units. A classic example often quoted by management gurus is a just-in-time process for a manufacturing unit obtained out of well-thought EI implementation. This could result in reduced inventory and hence reduced investments on stocks, which improves cash flow of the company. It is difficult to calculate such benefits upfront. However, a CIO can convince the CFO on such benefits.

EI projects can also help an enterprise improve customer relations since, usually, in an EI project these days a CRM system is also planned. However, it is difficult to calculate the exact benefits even after an EI project has been implemented. Still, a smart CIO can convince the management on the intangible benefits associated with an improved CRM business. Happy customers always mean a healthy business!

What drives an EI Project?
There are several technology and business drivers for an EI project. Apart from cost benefits and ROI calculations, there are business needs that force an enterprise to think on the lines of EI.

While the newer software are feature rich, they rarely do a good job of replacing all legacy applications, owing to many reasons. For example, users are often reluctant to move from a user interface and business process that they are comfortable with and have familiarized over many years. New ERP systems may not have features that certain departments require and customizing applications may be very difficult. In such cases, CIOs prefer to retain the legacy applications while using the new applications to complement or extend the latter.

Globalization is driving the present day economies. To achieve growth targets, most enterprises prefer an inorganic growth model, where they acquire other companies. This means that the CIO needs to integrate the IT systems of two different organizations, which in many cases use different technologies and are addressed by different vendors. It will be a foolish decision to replace IT systems and the right strategy is to integrate the systems of both companies in the new enterprise.

Yet another trend is the reduced cost of communication and the advent of Internet. It is cheaper today to integrate applications across continents, owing to cheaper infrastructure costs. Internet has no boundaries and applications are moving on to the web.

Finally, availability of newer technologies has opened up new avenues. XML and web services are redefining the way businesses are done. Open Standards and technologies are in vogue and every vendor is offering APIs, connectors to work with their legacy products. It is easier to integrate an enterprise today, though the sizes of projects and resources required are indeed large.

Different types of EI

Data Synchronization 

What is data synchronization? Consider an example of two retail chains merging. The CIO’s biggest headache is to ensure compatibility of data between both business processes. Data synchronization is the process through which you synchronize data from one system to another while ensuring consistency. This core technology set, often referred to as a message broker or integration server, is used as the foundation for every category of integration project. Data synchronization is enabled by technology that connects applications and data stores to a common middleware platform (or bus), routes the message-based information between these different systems based on content, rules or subscription and dynamically transforms the data so that it is in the native format of the target system(s).

Data synchronization transforms application-specific information into a shared enterprise resource, reducing costs and clerical errors associated with entering and maintaining consistent information across disparate systems. The result is improved data quality and increased productivity by ensuring consistent data access across all applications.

Business Process Instrumentation and its benefits

Business Process Instrumentation, also called Business Process Management (BPM), is the combination of the process re-engineering concepts of the 80’s and the workflow technologies of the 90’s. BPM is focused on modeling and instrument (replace with instrumentating) the workflow between individual functions in different applications in order to automate and streamline previously independent business processes. It can be thought of as the “integration logic” of a cross application workflow.

In many cases, BPM is primarily focused on “long running” business processes that may take hours, days, months or longer to complete, as well as business processes that span multiple applications and include interaction with users (often called workflow). For example, the process of shipping a computer to the customer of a mail-order computer company is a long running business process that starts with the request-to-ship at the computer company’s warehouse and ends when the return period has ended, which might be 30 or 60 days. The process itself might span both the back office warehouse application as well as the customer support and order entry applications. Managing long running processes requires that state of the processes be maintained through the entire duration — be that hours, days, weeks or months. BPM tools typically include modeling, automation and management components as well. The modeling components allow business analyst to define, view or manage individual processes without having to understand the underlying technical details of how the process is actually implemented.

BPM enables organizations to automate and streamline their manual processes and fuse previously autonomous operational and management systems. The result is a more agile business and efficient business better able to respond to changes in the marketplace.

Business Activity Monitoring and its benefits

The goal of Business Activity Monitoring (BAM) is to provide management with immediate awareness of changing business events across the enterprise, so that appropriate and timely decisions can be made. Real-time, event-driven business activity monitoring solutions provide real-time alerts via graphical dashboards and other notification mechanisms, enabling management to immediately react and intervene when key performance indicators change. BAM solutions also complement BPM solutions by providing real-time process monitoring capabilities that enable BPM systems to dynamically and immediately react to changes in the business environment.

By using real-time information to remove delays in managing and executing enterprises critical business processes, BAM reduces costs and enables faster execution of business processes.

Compound Application Development and its benefits

A compound application is an application or a new application functionality that combines the data and functionality of an enterprise’s existing applications and data stores, adding new business process logic, custom code and user-facing front ends. Amalgamated applications leverage individual functions or business processes within back-end operational systems and use them as components to build new applications or application functionality. Amalgamated Application Development uses a service-oriented development approach, whereby existing functions and business processes, available as services, are combined in support of new application functionality.

Amalgamated applications enable organizations to gain greater reuse from existing applications and IT investments, facilitating quicker time to market for new solutions that rapidly address changing business dynamics and challenges.

Combining all four categories

Even if you don’t think you need all four integration capabilities right now, the chances are you will in the future. New business opportunities breed new integration projects, or what looks like a small project normally expands dramatically in scope once the initial business benefits are realized. This technology assembly approach dramatically increases the complexity and costs when integration projects evolve and new technologies are required to implement integration strategies. It is, therefore, usually better to choose a complete integration suite with services than to try and assemble and manage all the necessary capabilities piecemeal.

How difficult it is to obtain and use a complete integration suite
Most integration platforms are really nothing more than “suites” that represent assemblies of formerly independent products. A comprehensive integration suite with a consistent architecture backed with a vendor provided service should be a prime consideration while deciding on an integration strategy. The better the strategy, faster will your integration projects be completed. 

What are the basic functions of an integration suite?
There are seven capabilities that a comprehensive integration suite should provide. These include:
· Connection;
· Abstraction;
· Coordination;
· Storage;
· Amalgamation;
· Development; and
· Management.

Let us investigate them one at a time.

Connection
Connection provides a channel of communication between the integration suite and each of the applications that are being integrated. Most integration suites achieve connection by using adapters. Adapters are pre-written pieces of code that work by speaking the native protocols of the target resources and connecting them to the integration suite or message bus. The use of adapters greatly speeds integration and can significantly reduce costs associated with writing code for each connection.

When choosing an integration suite, look at the number and types of adapters it includes. Look out for:
· Can the suite connect to your existing applications? 
· Is there a mechanism for building custom adapters, should you need to? 
· How easy is it to build and use adapters?

Abstraction
Abstraction is the process of representing the data and functionality of your existing applications in some consistent form. This alleviates the need to have to understand and work with diverse and different protocols, data and programming models used by each of the applications you are integrating. Three of the common approaches to abstraction include:

· Using J2EE: This approach can work well for applications and business logic, but it is not as good at dealing with data;
· Using XML and Web services: This approach works well for loosely coupling applications that are already Web service enabled. Legacy and custom applications may need to be retrofitted with Web service “wrappers”; and
· Using “canonical form”: Canonical form basically means a standardized representation of the data or functions. Putting something into canonical form would mean that each set of data and application functionality would be translated into a standard format or representation. This approach can be very flexible and powerful, if the canonical form can be exposed or projected in various standard-based object and data formats concurrently, e.g. Java, .NET, Web Services, Relational, C++, etc.

When choosing an integration platform, consider the flexibility of its abstraction. Is it easy to represent both data and functionality? Can it represent your legacy applications without requiring that they be rewritten? Do you have both J2EE and .NET-centric applications within your organization? What happens if your Java-based organization acquires a company where all the applications or development are based on .NET?

Coordination
In an integrated system, information is passed between disparate applications in the form of messages. Coordination is the capability to manage the flow of messages in your integrated system. Most integration platforms include message brokers that are responsible for guaranteed delivery, data transformation and intelligent routing of messages.

When choosing an integration platform, the performance and reliability of the messaging engine is of prime importance, as these will directly affect the scalability and robustness of your integrated system. However, also consider the messaging engine’s capabilities: 
· Does it support synchronous, asynchronous and publish-and-subscribe routing? 
· Does it support content-based routing? 
· What data transformation tools are available and how easy are they to use? 
· Is it integrated with a message warehouse for persisting all messages?

Storage

Integration generates data (metadata, messages, state information of long-running processes, indexes on federated databases, etc.) and every integration platform needs a mechanism for storing, retrieving and analyzing data.

Unfortunately, many integration platforms include only rudimentary capabilities for storage. In such cases, to gain robust and scalable storage, you must purchase and integrate an external database, thus adding cost and complexity to your integration project.

When choosing an integration platform, look for one that has a built-in shared metadata repository for all the integration touch points that will be generated and used by the integration platform. Also, scour for one that has a built-in message warehouse and pay particular attention to its capabilities for analyzing the message data in support of reporting and activity monitoring. Other parameters to be taken into account are:
· How well does the platform’s storage mechanism support service-oriented development of amalgamated applications? 
· Does it support distributed environments? 
· Will it impede the overall performance and scalability of the integration platform under heavy load?

Instrumentation
Instrumentation is another name for modeling and automating business processes. It can be as detailed as writing the code that tells the messaging engines what to do and when to do it.

However, instrumentation normally refers to business process modeling, whereby business analysts define and model graphical process flows without thinking about the underlying coding complexities involved in automating them. The instrumentation engine is able to “read” these process models and execute them. Most integration platforms come with built-in graphical tools for process modeling. A few of them enable standard-based XML-based code to be generated from the process diagrams, thus allowing interoperability with third party modeling tools.

When choosing an integration suite, consider how much flexibility it gives you with regard to instrumentation, e.g.:
· Does it provide a powerful graphical modeling tool? 
· Does it support interoperability with third party BPM products? 
· Does the integration platform support direct coding of processes and what scripting language does it use? 
· How easy is it to orchestrate complex business processes? 
· Does it support human workflow interaction? 
· Does it have a rules based engine that provides analysts with quick and easy ways to define rules that change the process flow?

Why is development important in integration projects?
In the context of integration, development normally refers to building amalgamated applications and portals, often the ultimate deliverable in an integration project. Amalgamated applications make use of all the capabilities of the integration platform we have described so far, adding new business logic and user-facing front ends to the abstracted services and business processes previously discussed.

When choosing an integration platform, look for the same capabilities that you would want in any rapid development environment, such as:
· Is the suite (tools, platform, service and environment) easy to use? 
· Does the architecture support business process modeling? 
· Can you build all the components of your integration solution using the same suite (E.g. adapters, message maps, transformations, business activity monitoring solutions, etc.)?
· Does it support the development technologies you already use? 
· Does it work with your favorite third party development tools?

Management
Integrated systems are among the most difficult to manage because of their loosely coupled nature. It is therefore vital that your integration platform provides good end-to-end management capabilities, both during development and operation of your integrated system. Ideally, your integration platform will keep a record of every single message that passes through the system in a message warehouse. Analysis of the message warehouse allows for quick debugging and identification of problems. Other important management features include: configuration management, queue and process monitoring, detailed event logs and usage histograms. When choosing an integration platform, consider the completeness and ease-of-use of the management tools. Does the platform save enough diagnostic data? Can that data be analyzed in real time? Are the management tools extensible? Does the platform support the use of standard-based (e.g. SNMP) third party management tools? Can the messages that flow through all business processes be traced?

Conclusion
By deploying an Enterprise Integration strategy, IT Managers and CIOs can ensure that the cost of development of a project is reduced considerably and at the same time ensure both tangible and intangible results. It is imperative that you use standards, technologies and methods that have been proven, while designing the solution.