Premier 100 Q&A: AT&T's Eslambolchi on software code, SOAs, security

He foresees software security problems of 'biblical proportions'

PHOENIX -- AT&T's top IT leader, Hossein Eslambolchi, this week warned of security problems of "biblical proportions" unless more is done to improve the quality of software code. To help accomplish that, he is working to improve the education of software engineers under a new program at the University of California.
Eslambolchi, who spoke at this week's Computerworld Premier 100 IT Leaders Conference, is president of AT&T's Global Networking Technology Services and serves as the company's chief technology officer and CIO. In an interview with Computerworld's Patrick Thibodeau, he also talked about how AT&T is turning to a service-oriented architecture (SOA) environment to consolidate systems and grid technologies, and touted network improvements at the company that can immediately detect security issues for corporate customers.
AT&T has been retiring many legacy systems. What technologies are you betting on? Let's say you have voice services, ATM services, frame services, IP services -- they all have different ordering systems, so we collapsed them into the "Concept of One," which is now called the Global Integrated Order Management system. It uses a service-oriented architecture, so it's really an object-oriented model. In the old model, if you had 60 services, you write the software 60 times. That's the way it used to be at AT&T. Now we write the software only once, and if I need to get access to address validation I go to my service-oriented architecture, extract intelligence, come back and rout the traffic.
What has an SOA meant for application development and the custom applications? We have 400 systems left. I would say about 80% of what is left is already legacy system, which we are retiring over the next couple of years. We will end up, when all is said and done, by the end of [the] 2006, 2007 time frame, with no more than 50 systems managing the entire global AT&T network, from [what was] originally 800 systems.

Are you standardizing on any particular hardware platform? We are hardware-agnostic. We use HP, Sun, IBM, and for storage, Hitachi and EMC. The differentiation factor comes in on the software layer that we are building right on top of the hardware. We use multiple companies and I don't differentiate them, to be honest with you, unless there is a price differentiation. If you design a service-oriented architecture that could run on any hardware, whether Sun has it, HP has it or IBM has it, whoever gives you the better-cost structure, that's how you use it.
Does grid have a place at AT&T? Grid computing does have a place. There is a technology that we have been working on called AAN, which is called Application Aware Networking. It is creating this infrastructure to better utilize your computing power. For example, 5% of Intel servers and 10% of Unix servers are utilized today, globally, that's all. So we have this huge computing power out there that we're not taking advantage of. We have built a software layer, based on the dynamic resource allocation software algorithm that we developed, that does a load balancing of the application based on the different servers that you have in a server farm. So you have much different SLAs that you can create with a much higher level of computational efficiency.
So this improves your ability to reduce the number of systems? Absolutely. That's been one of the key attributes: lowering the number of servers that we have in our infrastructure significantly -- and lowering our cost.
What has this done for your IT cost? We used to have 55,000 people in AT&T five years ago; at the end of last year, we were probably around 21,000 people -- it's all through the "Concept of One" and the "Concept of Zero" [hands-free, self-operating network] automation. So our cost has gone down dramatically. If you have better MIPS processing, your cost will come down, you end up with lower power, lower utility, lower space and all of those are costs that we have taken out as result of what we have done.

What were the biggest challenges in accomplishing this? Honestly, it's cultural. Just getting the people and the mind-set to think about competitiveness in a different environment, in which the prices are coming down, and you have to take your cost with it. Technology has been important. Getting the people to innovate in the right area is very important. In AT&T, the way it [software] was designed ... you buy off-the-shelf as a last resort. I inverted the order. I said, "Whatever software we have that we can reuse, great, but if you can't reuse the software because of a new application, we are going to buy off-the-shelf as a first priority and develop it internally as a last resort." And that has changed the environment dramatically around.
Doesn't the move to commercial software increase your security risks? You can't develop software all internally so you have to buy from somewhere. [But] you are absolutely correct. ... That's not the fault of the network, that's how the software industry drives code, and 90% of the software problem is attributable to the way people write software in the industry. ... The only way to solve the problem is [getting] people to write better software code. Well, how do you write better software code? You have to go back to education; you have to teach them how to write better software code. Sometimes when you outsource to cheap labor in other countries, you may not get good software code. ... We cannot develop all software ourselves. We don't have the luxury anymore to develop everything -- you have to rely on the innovation in the industry itself.
That's why we have innovated this technology, like Internet Protect. Now, we have a software database infrastructure where we do sampling data of the entire network so if somebody is trying to do a reconnaissance on your network we will detect that immediately. If somebody is trying to attack you, we will know that immediately. We will also understand what patching is required. I do not believe that the security problem is going to be solved in the short term from software. If we don't get back to the root cause, which is writing better software code, we're going to end up, in my opinion, [with] a problem of biblical proportion in this industry. You are going to have 8,000 bugs, viruses and worms every week hitting corporate networks, globally. That's a rate of one every five minutes. That's how bad this could become.

What are you doing to get the vendors to write better code? I started a new center at the University of California at San Diego. [AT&T] created the center for network systems, and this center is a combination of computer engineers, expertise and electrical engineering, [combined] into a new field called network science and network engineering. So that will help people understand the quality and reliability and security [issues]. We kicked off the center last year.
That sounds like a good long-term solution, but what are you doing today to get vendors to write better code? What you can do in the short term? You need to give them better requirements. The second thing, you do a software code design review -- you review every piece of the module, every piece of the code, line by line. We do that working with the vendor. We do that for the ones that we believe may have problems. These are the type of things you can do. The long-term solution has to be solved industrywide, and this requires a different mind-set about the education in this country and globally.
Do you think the use of offshore outsourcing is hurting software quality? I wouldn't say it's hurting, but I would say the cycle time to really get the bugs out and integrate the software -- it's a little bit longer than normal, very candidly. Just the language barrier, the requirements barrier, it all plays into that. In the short term, you are going to get a productivity hit. The question is how well can you come down off that productivity hit to a steady state. And that's really a function of what application, how many lines of code and what software engineers and what company you offshore to. And those are the factors you have to be very cognizant of. We don't directly offshore to India, China. We have outsourced some of our development to IBM. They probably offshore some there, but we actually work in a very collaborative fashion with IBM.

Copyright © 2005 IDG Communications, Inc.

7 inconvenient truths about the hybrid work trend
Shop Tech Products at Amazon