IT and Its Reputation at a Crossroads

Computerworld - The IT profession is at a turning point. One sizable group of IT practitioners already knows what needs to be done. Another continues to apply the same old ways of doing things that result in the same old horrendously expensive systems that often don't work. What differentiates one group from the other? The manner in which they perceive and respond to complexity.
People not skilled in the use of effective techniques for dealing with complexity usually fall back on the use of clumsy, slow-moving, bureaucratic ways of doing things. In most situations, these approaches aren't up to the challenge. They fail, and the reputation of the IT profession is tarnished each time that happens.
A good example of this is the recent collapse of a multiyear, multimillion-dollar project to upgrade application systems used by the Internal Revenue Service. People on this project looked at the complexity involved and were totally overwhelmed. They responded by adopting cumbersome procedures that tried to handle everything. Analysts analyzed and programmers programmed as documents piled up and the years went by. Nothing useful got done.
Another approach would be to respond to complexity by making rigorous use of the six core techniques I outlined in a previous column . You can use these techniques to reduce complexity into manageable, self-contained pieces. You can make progress right away on the simpler pieces and build solutions to the more complex pieces over time.
One reader who belongs in that group of IT practitioners who know what needs to be done sent me a wonderful way to solve the problem. It's elegant in how it uses just a handful of techniques to address complexity. "Process each account in its own thread - not whole files; it greatly reduces complexity and latency and increases scalability," he wrote. "Use pipe-and-filter with no intervening files instead of file-process, file-process, etc. This architecture was described by Mary Shaw at Carnegie Mellon years ago.
"Design and write a separate thread for each business case, beginning with the most common or simplest, e.g., domestic exempt salaried employee," he continued. "The IRS could have been processing all simple 1040 returns in a matter of months on the new system instead of taking years trying to develop monolithic, all-case, highly complex systems. In my scenario, there might be 30 or more separately designed and written processes handling business case threads."
Go to Google, type in "pipe-and-filter," and read some of the references that come up. It's a great approach. It uses combinations of four core techniques: