Front-end integration with lightweight architecture

Often-daunting integration projects don't have to be difficult

Integration is the third rail of corporate IT. The mere mention of the word raises terrifying thoughts of huge budgets, endless meetings and extremely complicated software. Integration projects can be daunting, often requiring the skills of highly specialized IT staff.

It doesn't have to be that hard. On the open Web, countless sites have circumvented the headaches associated with expensive, drawn-out integration to quickly and easily integrate various data sources. As these sites suggest, a lot of corporate integration projects -- things like consolidating employee phone books after a merger -- can actually be done quite easily with Web 2.0 technologies that have been updated for corporate applications. This new approach to integration emphasizes a lightweight architecture that shifts IT's focus from the back end to the front end. As a result, an organization can reduce its backlog of integration projects and dedicate its top people to genuinely difficult challenges such as supply chain integration.

Given the challenges of previous computing eras, FOI (Fear of Integration) is understandable. In the 1970s, getting two computers to talk to each other was pretty straightforward. IT laid a physical wire between the two machines, developed a proprietary protocol, messed around on either end for a while, and presto, the two machines were talking to each other. Needless to say, this integration was very expensive and cumbersome. Computers generally were hooked up to each other only if the need was very pressing and the machines were in close geographic proximity to one another.

Later, when a machine had to talk to more than a handful of other machines, packet switching networks based on protocols like X.25 arose. Two companies that participated in the same network could send data to each other once they negotiated a common data format. This type of integration fostered the evolution of closed supply chain systems like electronic data interchange.

When the Internet opened for commercial use, anyone could connect to anyone else with a variety of open protocols, yet there was still no standard for data format. Getting divisions within a company, let alone multiple companies, to interact with one another was still very complicated.

For example, consider the average e-commerce site a couple of years ago. To accept credit cards over the Web, it needed to sign a deal with a company such as CyberCash, and that entailed choosing a client API -- available in a variety of flavors, including, Solaris C++, Windows C++, Java 1.1, and Java 1.2 -- and sending engineers to a training class.

To show physical store locations on a map required the same type of API deal with MapQuest. To start a frequent flyer miles promotion took an intermediary like Netcentives, which required the online retailer to lease a Netcentives computer, add that computer to its cluster and send it data via a proprietary protocol. The leased computer would then forward the data on to Netcentives.

Today, all of these contortions sound ridiculous. Yet we forget that this is what the industry was doing only a few years ago.

Web service simplicity?

Enter the world of Web services. Conceptually, Web services are simple -- self-describing text files are sent around. In the most basic implementations, there are no binary formats. The elements are named and therefore can be in any order. The data is hierarchical and therefore relatively well organized. Everyone can generate or parse a text file and communicate over HTTP, so data exchange becomes really simple.

Web services are definitely not for creating an efficient, technically elegant protocol for machine-to-machine communications, as CORBA and DCOM do. But in the real world, where high-power processors sit idle most of the time, agreement among endpoints is more important than technical efficiency. If self-describing text files over HTTP is what everyone can agree on, so be it.

Unfortunately, the simplicity of Web services often fails to translate to simple integration. At an executive forum in 2006, the audience was asked for a show of hands on the following question: "Who thinks it's easier to build an app that communicates to a Web service than it is to write an app that communicates to a database?" Of this very sophisticated audience of senior IT architects and managers, only two people raised their hands.

After having spent countless amounts of time and money implementing service-oriented architectures, companies find it still incredibly difficult to build new user-facing applications that tie services together. Why? Because enterprises are trying to build these new applications with traditional technologies such as Java 2 Platform Enterprise Edition (J2EE).

It is hard enough using this approach to build a basic HTML application that talks to a single database. It is nearly impossible when using the same tools to create applications with rich interfaces that integrate with multiple services including RSS, Simple Object Access Protocol, REST and database stored procedures.

However, that's the direction IT is taking. The days when each enterprise application is an island are coming to an end. Even things as simple as an employee directory now need to integrate the HR systems of multiple divisions, accommodate cross-reporting and virtual teams, and integrate outsourced third parties. Like it or not, essentially every enterprise application today requires integration. But not every enterprise application needs to integrate at the back end.

Moving to mash-ups

Over the last few years, "mash-up" applications have exploded onto the Internet. This new breed of Web application integrates data from a variety of sources and services under a rich user interface to deliver a superior user experience.

For example, shows listings of real estate rentals and sales pulled from the popular classified ads site, These listings are displayed on interactive maps pulled from a Web map server, Google Maps. Users can then drag the maps in any compass direction to see what is available for sale or rent in a given region. And this popular application was created by just one developer.

Naturally, mash-ups are having an impact on user experience expectations. Customers, employees and other users are no longer willing to wait for new pages to load when they click on links, or wait for call center reps to click through multiple systems and re-enter account information just to answer simple questions.

Everybody wants their data immediately, and they want it presented in an understandable, at-a-glance format. Bottom line, enterprises must upgrade their infrastructures to match customer expectations or risk losing customers.

Lightweight liftoff

Rising to meet the challenge, front-end integration has become the enterprise equivalent to the mash-up movement. Front-end integration works by integrating Web services and databases at the Web tier, using lightweight (as opposed to heavyweight) architecture.

Heavyweight architecture uses complicated infrastructure software such as J2EE with complicated APIs on a small cluster of expensive symmetric multiprocessing machines. Lightweight architecture, on the other hand, employs straightforward, open-source software stacks with service-oriented APIs on large clusters of commodity machines.

Here are four common ways of achieving a lightweight architecture. Three of them are open-source solutions, and the fourth is Microsoft's attempt:

LAMP. This stack includes the Linux operating system, the Apache Web server, the MySQL database and the PHP/Perl/Python scripting languages. LAMP runs a vast majority of the massively scalable Web sites, including Google, Yahoo and Amazon. LAMP is also the favorite deployment stack for most of the Web 2.0 crowd, including Friendster, Facebook and Flickr. Both IBM and Oracle support PHP, and there has been an upswing of enterprise adoption of the LAMP stack.

Open-Source Java. A lot of Java developers are moving away from full-blown J2EE to lightweight Java, which includes Tomcat, Spring and Hibernate, among other open-source Java projects. There are some big Web sites running this architecture, including ETrade and eBay. Lightweight Java is built on open-source projects, which are a very different set of Java APIs than J2EE and are not officially sanctioned by Sun or the JCP. And Sun's recent decision to release Java under the GPL license goes a long way toward bringing the Java and LAMP worlds together with a single virtual machine.

Ruby on Rails. RoR has gotten quite a bit of traction with the intelligentsia but very little yet from corporate customers or massively scalable Web sites. Some people complain about it because it is a code generator. In any case, it is a cool scripting language and framework and a nice addition to the lightweight trend.

Skinny .Net. Companies like JetBlue and Charles Schwab are starting to make very selective use of .Net so that it is more lightweight. Windows was always meant to run on commodity machines, but getting it to run on large clusters or grids of commodity machines is a challenge. On the other hand, .Net offers the best tools in the business and great XML support.

Each of these four approaches differs from the others, but they are remarkably consistent when contrasted with the old way of doing things. With lightweight architecture, it is very simple to integrate data from a variety of internal and external sources and present the data with a rich Asynchronous JavaScript in XML (AJAX) user interface.

AJAX enables a Web browser to retrieve data without having to refresh the page. For example, a call center application can use AJAX to integrate data from internal CRM and ERP systems as well as external systems at, say, FedEx, to give a call center rep a single screen display of information pulled from the three different back-end systems.

Optimized integration options

Of course, back-end integration is still necessary to create non-user-facing services that require multiphase commits or long-lived processes. Multiphase commits involve coordinating across multiple data stores to ensure that data is updated on all of them or none of them, most commonly for financial types of transactions where money is being moved.

Long-lived processes involve coordinating multiple steps across many systems, such as handling exceptions in supply chain management where a shipment needs to be diverted automatically to a new plant. These types of projects are very demanding, and the most popular technology for back-end integration is J2EE servers and their integration server offshoots from vendors such as IBM and BEA Systems.

Today, IT has options when it comes to integrating applications. As the growing use of mash-ups demonstrates, lightweight architecture on the front-end can deliver very useful integration projects quickly and economically.

Yared is the founder and CEO of ActiveGrid. 

This story, "Front-end integration with lightweight architecture" was originally published by LinuxWorld-(US).

Copyright © 2007 IDG Communications, Inc.

7 inconvenient truths about the hybrid work trend
Shop Tech Products at Amazon