eWEEK content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
Integrating a far-flung organizations technology products and strategies is one of the toughest, messiest and most failure-prone jobs in IT. Making sure that integration projects result in success takes business knowledge, clearly defined goals, realistic expectations and the right technology.
In this Special Report, well outline the major challenges and directions in the integration space, as well as examine the technologies that can make integration efforts easier to carry out and cheaper to finish.
Achieving economies of scale, ensuring that all parts of the business are working with correct as well as current information (two separate challenges) and keeping IT system maintenance costs down all depend on a consistent, universal and integrated information management strategy.
Unlike typical IT budget line items, integration efforts need to be made on an ongoing basis and supported by all parts of an IT organization. This is an area where leaving well enough alone fails miserably as a management strategy, especially in large enterprises with many thousands of identifiable applications.
“If you have one ERP [enterprise resource planning] system, you have two, and then you start getting into integration between those two,” said Jim Ivers, director of product marketing at EAI (enterprise application integration) software vendor WebMethods Inc., in Fairfax, Va. “A lot of the applications that have been running a very long time are there because they do a good job.”
Ivers gave as an example global semiconductor and electronics manufacturer Motorola Inc. (a WebMethods customer), which shows the typical scope of the integration challenge at a global company.
“Motorola is tying 9,000 applications together, and theyve scratched the surface of 10 percent of those,” Ivers said. “Theyve saved $38 million a year. The age of people building new apps is coming to an end, and more people are working at building these composite applications that cut across those stovepipes. There is more process development than application development.”
Tools for Integration
Tools for Integration
Whatever the integration demands, it staffers need to balance two approaches to integration: integration through consolidation and integration through distributed systems. Consolidation generally offers the best results but takes longer to do, while heterogeneous systems can be brittle and difficult to administer but are quick to deploy.
Integration projects also will generally fall into either database- or application-centric efforts, depending on the complexity of the business logic that manipulates this data. Bypassing applications to directly manipulate base tables is fine if the user applications that manipulate this data are simple and data integrity rules well-understood.
When applications grow complex (and ERP systems are the classic example), application-level APIs need to be used to ensure the application always has a consistent view of the data.
Although application-level integration is often the best approach, its also very expensive, because every interface is proprietary.
A core group of EAI vendors offers help here: Mercator Software Inc., SeeBeyond Technology Corp., TIBCO Software Inc., Vitria Technology Inc., WebMethods and, to an extent, IBM, as it integrates its CrossWorlds Software Inc. acquisition into its product line. All provide large sets of custom adapters, data transformation and routing tools, and reliable messaging.
These are big-budget enterprise plays, offering long-term returns for enterprise-level commitments. Building more ad hoc solutions has a place in smaller projects or shorter-term development efforts. Data Junction Corp.s Data Junction, Microsoft Corp.s BizTalk Server, Sonic Software Corp.s SonicXQ and Sterling Commerce Inc.s Gentran product line are all lower in cost and require less corporate investment than all-encompassing EAI products, although they dont reach as far.
Integration Scenarios
Integration Scenarios
There are some scenarios that demand integration strategies.
The first is the most familiar to organizations: the basic need to connect diverse internal systems to share information, reuse system resources, preserve data integrity and ensure consistent system configuration. The most powerful tool an organization has in this scenario is its ability to minimize diversity and maximize interoperability through centralized standards-setting, purchasing and budget approval.
Tightly written request-for-proposal documents that require outside vendors to prove compatibility with existing systems make the purchasing process itself a major partner in future integration successes. Of course, sticking to published, nonproprietary standards as a purchasing policy wherever possible allows for a competitive bidding process without any sacrifices in interoperability.
The second major integration scenario is change in business needs. The most common driver is when organizations develop the need to establish tight relationships with business partners, an arrangement clearly dependent on the ability of IT staff to make intraorganizational interoperability work.
EDI (electronic data interchange) is the long-standing business-to-business stalwart here, although organizations using EDI are increasingly switching to Internet-based EDI links instead of EDI VANs (value-added networks). The emerging best-practices approach is to use HTTP over Secure Sockets Layer, with SSL authenticated using client-side digital certificates.
“What Im seeing is that customers want to get away from VANs,” said Rick Clements, director of product marketing at SeeBeyond, in Redwood Shores, Calif. “Its not that their connections are not successful, but the cost is enormous. Many of the customers that I talk to want to leverage EAI to avoid these costly network connections. The common approach is to employ a specification like AS2 [Applicability Statement 2] to allow companies to share purchase orders without having to go through a VAN.”
AS2 (whose full name is HTTP Transport for Secure Peer-to-Peer Business Data Interchange over the Internet) is one way companies using EDI can lower costs. Its a proposed standards draft thats been submitted to the Internet Engineering Task Force. It provides for reliable HTTP-based transport of EDI or XML messages while AS1 (Applicability Statement 1) does the same using SMTP. (The IETFs EDI page at www.ietf.org/ids.by.wg/ediint.html has more information.)
Both AS1 and AS2 are already supported by many leading integration and EDI server vendors. The Web site of Drummond Group Inc. (www.drummondgroup.com), a consultancy doing AS1 and AS2 interoperability testing, provides compatibility test results. (Particularly interesting are the issues the group found with public-key infrastructure interoperability.)
Another business change driving integration efforts is the implementation of an internal information portal. Portals provide a single window into enterprise information for employees, and they depend heavily on consistent document taxonomies, document markup efforts and consolidated data repositories.
XML databases are emerging as strong players here because they provide a way to organize documents using markup tags without requiring major document changes or data re-entry.
Finally, adoption of new industry standards or regulatory changes will sometimes mandate integration.
For example, the financial sector is making a concerted attempt to embrace straight-through processing, an effort to reduce the average settlement time of financial transactions from the current three days to 24 hours or less. This is something that will be possible only through reliable, event-driven application integration servers connecting all the parts of the settlement process.
The medical industry is facing similar challenges as a result of Health Insurance Portability and Accountability Act mandates to integrate data from the doctors office back through to the insurer.
All these intraorganizational challenges are best handled through a dual strategy of industry-specific consortia establishing standards and deployment of leading application integration tools to provide reliable, standards-based integration. XML-based document schemas are delivering great value here and are rapidly being adopted in B2B efforts.
Emergency Integration
Emergency Integration
In addition to B2B efforts, there is another integration scenario IT managers need to keep in mind: the emergency-style variety that happens all at once when mergers or acquisitions occur. This is a worst-case scenario and one thats the most difficult to handle from a technical standpoint.
“The main technical challenge [with integration] is that the infrastructure is chaotic and antiquated,” said David Linthicum, chief technology officer and executive vice president of Mercator, in Reston, Va. “Mergers and acquisitions sometimes fail because they cant get the systems integrated. Thats a failure of the IT group. Supply chain integration setups fail because they cant get the system integrated. … Those things leave you scratching your head in terms of the work that needs to be done to get more competitive.”
Direct CEO and CIO involvement is a must when mandating core standards among sometimes vastly different IT architectures.
Web services interfaces and heterogeneous database gateways are two key technologies here, as both can be deployed in a few weeks, allow ad hoc deployments and require little or no changes to underlying infrastructure to work. ´
West Coast Technical Director Timothy Dyck can be reached at [email protected].
Related stories:
- Playing Integration Game
- XML: New Options, New Worries