Closely Held Applications

Maintaining some 300 homegrown applications isn't easy, but consumer electronics retailer Crutchfield Corp. says that doing so makes it more responsive to customers' needs.

Imagine if you wrote and maintained every major application in your enterprise. The good news is that you wouldn't have to wait for vendors to fix integration problems before making application upgrades. The bad news is that when application conflicts arose, fixing them would be entirely your problem. And as your infrastructure changed from, say, client/server to three-tier to service-oriented architecture (SOA), you would need to migrate everything yourself.

That's the situation at Crutchfield Corp., a Charlottesville, Va.-based catalog and online merchant of consumer electronics. The company has a staff of 500 people, with some 550 desktops and 100 servers to manage. And the IT organization wrote 85% of the code that runs inside the company, according to CIO Steve Weiskircher. His staff maintains 300 applications that run on Windows servers. Those applications include the Web site, warehouse management, point-of-sale, general ledger and order entry systems. The order entry system alone has more than 1 million lines of code.

With that many applications to manage, it's not surprising that Crutchfield has a relatively large IT staff for its size, including 18 full-time developers. But why take on the burden? "It's due to our entrepreneurial nature," Weiskircher says. "We know our business and data better than anyone else." By making the investment, he says, Crutchfield is able to achieve a level of business responsiveness that enterprise application software vendors routinely promise but never seem to deliver.

The advantage of owning the code is that the IT staff can make changes quickly instead of waiting on vendors' revision schedules and priorities. "That does come at a cost when you have things like Windows XP SP2 or any other change to the infrastructure," Weiskircher says. Those types of changes cause the same problems for Crutchfield's homegrown applications that they cause for commercial applications.

Steve Weiskircher, CIO of Crutchfield Corp.
Steve Weiskircher, CIO of Crutchfield Corp.

Propagating Change

There are times when Crutchfield does look for outside help. A decision to move to Windows XP Service Pack 2, which was precipitated by the spyware battle that call center staffers faced while working on the Web, was delayed when Crutchfield discovered that its own applications were accessing areas of the registry and file system that SP2 was trying to close. With only four people available to manage the desktops, Weiskircher estimates that it would have taken six months to identify all of the application conflicts and track down the problems. "We didn't know everything that had changed since SP2, so we would have had to look for lots of variables," he says.

Weiskircher eventually ran across Identify Software Ltd.'s AppSight Black Box application performance monitoring software, which automated testing and helped identify and trace problems. Using Black Box, the project went forward. It required a month for testing, followed by a week to upgrade 250 PCs in the contact center, where the spyware problem was most acute.

"The bulk of our problems—between 80% to 90%—were the result of permissions changes on specific portions of the file system or registry," says Bill Hamilton, client support team leader. Finding a specific key that has had its permissions changed was like looking for a needle in a haystack. "AppSight was like having a big red arrow pointing to the needle," he says.

While fixing all of the company's Visual Basic 6 programs to accommodate SP2 required a bit of work, Weiskircher says being able to work around problems with third-party software beats waiting for patches. "We can just adjust our code around that product," he says.

EyEing An ERP System

Crutchfield has weighed the pros and cons of going with a complete ERP system from a vendor such as SAP AG, but it rejected the idea. The problem with that scenario, Weiskircher says, is that you "buy a product off the shelf and then adjust your business processes to it." That's fine for some well-defined applications such as human resources or accounting. "But for what is unique in your organization, you can probably do better than someone who doesn't know your space or environment or business rules," he says.

Stuart Selip, an analyst at Burton Group in Midvale, Utah, isn't sure so much of the application infrastructure needs to be homegrown. "Essentially, they're in the software business," he says. Such investments are worthwhile only if an organization has a unique need that can't be satisfied by enterprise software offerings and if the company derives significant competitive advantage from custom applications, he says. But Weiskircher says Crutchfield's systems enable a close relationship with customers that its competitors can't match.

Crutchfield is looking into revamping its application infrastructure and is comparing each application against off-the-shelf alternatives. Weiskircher will OK a move to an off-the-shelf product only if it fits cleanly into the organization and can do 80% of what the homegrown application does without a lot of customization. The applications must lower staff overhead for maintenance and management, freeing up resources to focus on more business-critical systems.

Crutchfield will be doing a lot of development over the next few years: Its application code is currently a mix of Visual Basic 6, VB.Net, C, C++, C#, ASP and ASPX. Its system architectures are a mix of client/server in the call center, three-tier in the online retail store and an n-tier Web architecture. The next step will be toward an SOA, Weiskircher says.

The question is how to get there with those 300 applications over the next three years. The company wants to migrate to the .Net platform, which means moving some code from the procedural VB6 to the more object-oriented VB.Net. That's a lot of work.

"The goal is not to replace the VB6 code. That is a byproduct of the project. It is really a move toward an SOA and providing more flexibility to the business," Weiskircher says.

Crutchfield has also benefited from the flexibility of Web services and the .Net platform. Weiskircher discovered that he could consolidate business processes from the company's three retail channels into one Web service. "We can make incremental changes to reach the desired architecture over time without wholesale rewriting of applications," he says.


Crutchfield Corp.


Revenue: $220 million

Employees: 500, including 250 in the call center

IT infrastructure: 300 custom-developed applications residing on 100 Windows servers run every major operation from order entry to the general ledger.

Percentage of business applications built and maintained in-house: 85%

Programming staffers required: 18

Projects: Recently completed upgrade to Windows XP SP2; reviewing applications against off- the-shelf alternatives; starting three-year migration to an SOA.

Copyright © 2005 IDG Communications, Inc.

7 inconvenient truths about the hybrid work trend
Shop Tech Products at Amazon