Too often, promising integration projects wind up as expensive flops. The FBI's failed four-year, $170 million Virtual Case File project is only the latest example to make the news. On May 24, FBI Director Robert Mueller reported to Congress that the first phase of the replacement system won't be ready until 2006, at a cost yet to be announced.
While few integration projects are that costly in pure dollar terms, they can still make or break an organization. With the increasing commoditization of both hardware and software, technology alone is no longer the key. Rather, business success hinges on how well those products are integrated to advance business goals.
"The applications themselves are not the differentiator," says John Schmidt, president of the Integration Consortium, an industry group working to establish standards, guidelines and best practices for integration projects. "It's how well you can glue them all together and connect with customers and suppliers."
Schmidt has worked in the IT industry for 27 years and has spent the past 15 helping major North American and European retail, financial and telecommunications corporations integrate their systems. Computerworld asked Schmidt and other experienced systems integrators for advice that could help IT managers with their integration projects.
Getting Agreement
Systems integration is often thought to refer to the effort to make systems work together harmoniously. Perhaps less understood, however, is that those systems must also be closely aligned with the overall business strategy.
"The most critical phase of the project includes really understanding its purpose before it starts and interviewing all stakeholders to find out their definition of what will make the project successful," says Bob Woodruff, special assistant to the CEO of project management consulting firm Robbins-Gioia LLC in Alexandria, Va.
Unfortunately, few companies appear to heed this advice. Woodruff has worked in project management for 20 years and says that people with the most clout in an organization tend to get their projects funded, whether or not they are the most important projects for the company as a whole. This can wreak havoc for the IT staff.
"These sponsors are typically unaware of the impact caused by inserting new technology into an already existing environment," he says. "This leaves the IT manager in the unenviable position of trying to integrate systems that just don't work well together."
For example, Woodruff is overseeing the update of his own firm's enterprise architecture. Systems that are based on Oracle or SQL databases are simple to integrate -- just write some SQL queries, pull the data out, reformat it and dump it into the other system. Systems that export the data into an Excel spreadsheet require a bit more work. Even worse are the proprietary, nondatabase systems, some of which require manual re-entry of data into the new systems.
"Some pieces we may not convert because of the cost involved, the timing, the real value to the company," Woodruff says. "If we are going to replace it in a year or two, we will just leave it."
He advises avoiding projects without sufficient executive sponsorship or funding, pet projects that provide only short-term gain and projects that have ill-defined requirements.
To be fair, though, IT can also be guilty of failing to coordinate integration projects. Michael Kuhbock, CEO of K-Bear Corp., a business strategy consulting firm in Calgary, Alberta, has seen cases where IT departments bought middleware licenses, assembled project teams and created budgets before consulting with the business units.
"On one case, it took another three months to bring the business on board," Kuhbock says. "That happens day-to-day in the market."
Assembling the Team
After hashing out the project scope and purpose, it's time to assemble the integration team. That can involve a mix of internal and external resources.
"IT managers can find the expertise they need in several places," says Liz Mann, managing director of Mycroft Inc., an identity management and security firm in New York. "They can find it within their own organization, they can utilize product-specific expertise from the vendors, and they can use expertise from integration consulting companies."
Major projects require expertise in a broad array of technologies.
"There have been many waves of application technology over the years that seem to move in regular seven-year cycles -- for example, mainframes to mini- to microcomputers, or monolithic to client/server to Web service applications," Schmidt says. "The shift from one wave to the next is neither instantaneous nor necessarily economically justified, so an integration methodology must deal with three to four generations of technology at once."
While the right mix of technical skills is critical, integration projects can be scuttled by the "religious" wars surrounding some technologies.
"You should stay agnostic as applied to technology and stay away from those who are devout," says Gerard McGowan, vice president of technology and services at Innovativ Inc., an integrator in Edison, N.J.
Schmidt says this also applies when selecting a consultant or vendor to join the project. When someone tries to present a single solution, such as implementing a service-oriented architecture, he shows that person the door.
"It is not that a service-oriented architecture is bad, but it is presented as if it will solve all your integration problems," he explains. "There are no silver-bullet solutions, and you should avoid that pitfall."
Limits of Standards
An issue raised by several of the integrators is that while you should definitely look to incorporate industry standards when designing an enterprise architecture, they don't necessarily guarantee interoperability.
"Even successful standards, such as TCP/IP, are not universal," says Schmidt. "When it comes to software standards such as Cobol or Java, interoperability and transportability come at the expense of vendor-specific extensions, forcing developers to use a less-than-ideal core set of 'pure' language features."
This is particularly true when implementing new technologies. McGowan points out that while voice over IP is becoming more popular, it's still better to rely on a single vendor than to hope that pieces coming from different companies will work together properly.
"There is talk about generic [Session Initiation Protocol] stacks and open-source SIP, but when you try to integrate that stuff, it tends to get relatively ugly, relatively quickly," McGowan says.
The same applies for authentication, according to Mann. The Security Assertion Markup Language (SAML) is designed to allow the exchange of authentication materials between dissimilar authentication systems or multiple versions of a single system that have been installed across a corporate or divisional boundary. But developers have options in how they implement the standard, so although different products can be made to talk to one another, they won't necessarily do so out of the box.
"You may find that two vendors are SAML-compliant, but when you try to make that exchange, it doesn't work," Mann says. "It is not that either one failed to implement the standard, but they both did it slightly differently."
But these problems tend to disappear, or at least get easier to deal with, as time goes on. "The more mature technologies tend to have good, well-proven cross-vendor support," says McGowan.
No Five-Year Plans
Because technology and business needs are constantly changing, you can't operate with Soviet-style five-year plans. Mergers and acquisitions, software and hardware updates, changing economic conditions and numerous other factors all mandate achieving shorter-term results. So break larger projects down into small pieces.
"I think people recognize today that new technology deployments are an ongoing process," says Mann. "They don't do these 'pie in the sky' projects anymore."
She recommends breaking longer-term projects into 90-day chunks, each with a discrete deliverable.
"If you take too long to deliver infrastructure and discrete successes are not noticeable, you will probably lose your funding or lose momentum on your project," Mann says.
In working on smaller pieces, however, don't lose sight of the bigger picture. It requires a balance between the strategic and the tactical, and every bit must advance the overall long-term business and IT plans.
"Integration is often seen as a project-based technological event rather than a mission-critical, enterprisewide business strategy," Kuhbock cautions. "We need to change that."
Integrating Expertise
Successful integration requires that you continually gain and update information on the best ways to link systems. One way to achieve this is to become active in trade groups, such as the Integration Consortium, that develop and share information on how to integrate systems. You probably won't be the first one to try a particular type of project, and you'd be better off learning the lessons from people who have been through it before than finding out on your own 12 months into the project.
"It is important for end users and suppliers to engage in collaborative efforts in the industry," says Schmidt. "When you do, play an active role in these organizations; don't just read the papers others write."
Kuhbock emphasizes that integration failures often boil down to weaknesses in the personnel, not just the technology, and training on integration is essential for success.
"Then we can achieve a 100% success rate instead of the failure rates we hear about now," he says. "An automobile manufacturer wouldn't make it if three out of 10 cars it built couldn't make it off the lot."
Robb is a Computerworld contributing writer in Los Angeles. Contact him at drewrobb@attbi.com.