The A-Z of Programming Languages: Bourne shell, or sh

Computerworld is undertaking a series of investigations into the most widely-used programming languages. Previously we have spoken to Alfred v. Aho of AWK fame, S. Tucker Taft on the Ada 1995 and 2005 revisions, Microsoft about its server-side script engine ASP, Chet Ramey about his experiences maintaining Bash, Bjarne Stroustrup of C++ fame and to Charles H. Moore about the design and development of Forth. We've also had a chat with the irreverent Don Woods about the development and uses of INTERCAL, as well as Stephen C. Johnson on YACC, Luca Cardelli on Modula-3, Walter Bright on D, Simon Peyton-Jones on Haskell and more recently, Larry Wall, creator of the Perl programming language.

On this occasion we speak to Steve Bourne, creator of the Bourne shell, or sh. In the early 1970s Bourne was at the Computer Laboratory in Cambridge, England working on a compiler for ALGOL68 as part of his PhD work in dynamical astronomy. This work paved the way for him to travel to IBM’s T.J. Watson Research Center in New York in 1973, in part to undertake research into compilers. Through this work, and a series of connections and circumstance, Bourne got to know people at Bell Labs who then offered him a job in the Unix group in 1975. It was during this time Bourne developed sh.

What prompted the creation of the Bourne shell?

The original shell wasn’t really a language; it was a recording -- a way of executing a linear sequence of commands from a file, the only control flow primitive being goto a label. These limitations to the original shell that Ken Thompson wrote were significant. You couldn’t, for example, easily use a command script as a filter because the command file itself was the standard input. And in a filter the standard input is what you inherit from your parent process, not the command file.

The original shell was simple but as people started to use Unix for application development and scripting, it was too limited. It didn’t have variables, it didn’t have control flow, and it had very inadequate quoting capabilities.

My own interest, before I went to Bell Labs, was in programming language design and compilers. At Cambridge I had worked on the language ALGOL68 with Mike Guy. A small group of us wrote a compiler for ALGOL68 that we called ALGOL68C. We also made some additions to the language to make it more usable. As an aside we boot strapped the compiler so that it was also written in ALGOL68C.

When I arrived at Bell Labs a number of people were looking at ways to add programming capabilities such as variables and control flow primitives to the original shell. One day [mid 1975?] Dennis [Ritchie] and I came out of a meeting where somebody was proposing yet another variation by patching over some of the existing design decisions that were made in the original shell that Ken wrote. And so I looked at Dennis and he looked at me and I said “you know we have to re-do this and re-think some of the original design decisions that were made because you can’t go from here to there without changing some fundamental things”. So that is how I got started on the new shell.

Was there a particular problem that the language aimed to solve?

The primary problem was to design the shell be a fully programmable scripting language that could also serve as the interface to users typing commands interactively at a terminal.

First of all, it needed to be compatible with the existing usage that people were familiar with. There were two usage modes. One was scripting and even though it was very limited there were already many scripts people had written. Also, the shell or command interpreter reads and executes the commands you type at the terminal. And so it is constrained to be both a command line interpreter and a scripting language. As the Unix command line interpreter, for example, you wouldn’t want to be typing commands and have all the strings quoted like you would in C, because most things you type are simply uninterpreted strings. You don’t want to type ls directory and have the directory name in string quotes because that would be such a royal pain. Also, spaces are used to separate arguments to commands. The basic design is driven from there and that determines how you represent strings in the language, which is as un-interpreted text. Everything that isn’t a string has to have something in front of it so you know it is not a string. For example, there is $ sign in front of variables. This is in contrast to a typical programming language, where variables are names and strings are in some kind of quote marks. There are also reserved words for built-in commands like for loops but this is common with many programming languages.

So that is one way of saying what the problem was that the Bourne Shell was designed to solve. I would also say that the shell is the interface to the Unix system environment and so that’s its primary function: to provide a fully functional interface to the Unix system environment so that you could do anything that the Unix command set and the Unix system call set will provide you. This is the primary purpose of the shell.

One of the other things we did, in talking about the problems we were trying to solve, was to add environment variables to Unix system. When you execute a command script you want to have a context for that script to operate in. So in the old days, positional parameters for commands were the primary way of passing information into a command. If you wanted context that was not explicit then the command could resort to reading a file. This is very cumbersome and in practice was only rarely used. We added environment variables to Unix. These were named variables that you didn’t have to explicitly pass down from the parent to the child process. They were inherited by the child process. As an example you could have a search path set up that specifies the list of directories to used when executing commands. This search path would then be available to all processes spawned by the parent where the search path was set. It made a big difference to the way that shell programming was done because you could now see and use information that is in the environment and the guy in the middle didn’t have to pass it to you. That was one of the major additions we made to the operating system to support scripting.

How did it improve on the Thompson shell?

I did change the shell so that command scripts could be used as filters. In the original shell this was not really feasible because the standard input for the executing script was the script itself. This change caused quite a disruption to the way people were used to working. I added variables, control flow and command substitution. The case statement allowed strings to be easily matched so that commands could decode their arguments and make decisions based on that. The for loop allowed iteration over a set of strings that were either explicit or by default the arguments that the command was given.

I also added an additional quoting mechanism so that you could do variable substitutions within quotes. It was a significant redesign with some of the original flavor of the Thompson shell still there. Also I eliminated goto in favour of flow control primitives like if and for. This was also considered rather radical departure from the existing practice.

Command substitution was something else I added because that gives you very general mechanism to do string processing; it allows you to get strings back from commands and use them as the text of the script as if you had typed it directly. I think this was a new idea that I, at least, had not seen in scripting languages, except perhaps LISP.

Page Break

How long did this process take?

It didn’t take very long; it’s surprising. The direct answer to the question is about maybe 3-6 months at the most to make the basic design choices and to get it working. After that I iterated the design and fixed bugs based on user feedback and requests.

I honestly don’t remember exactly but there were a number of design things I added at the time. One thing that I thought was important was to have no limits imposed by the shell on the sizes of strings or the sizes of anything else for that matter. So the memory allocation in the implementation that I wrote was quite sophisticated. It allowed you to have strings that were any length while also maintaining a very efficient string processing capability because in those days you couldn’t use up lots of instructions copying strings around. It was the implementation of the memory management that took the most time. Bugs in that part of any program are usually the hardest to find. This part of the code was worked on after I got the initial design up and running.

The memory management is an interesting part of the story. To avoid having to check at run time for running out of memory for string construction I used a less well known property of the sbrk system call. If you get a memory fault you can, in Unix, allocate more memory and then resume the program from where it left off. This was an infrequent event but made a significant difference to the performance of the shell. I was assured at the time by Dennis if this was part of the sbrk interface definition. However, everyone who ported Unix to another computer found this out when trying to port the shell itself.

Also at that time at Bell Labs, there were other scripting languages that had come into existence in different parts of the lab. These were efforts to solve the same set of problems I already described. The most widely used “new” shell was in the programmer’s workbench -- John Mashey wrote that. And so there was quite an investment in these shell scripts in other parts of the lab that would require significant cost to convert to a the new shell.

The hard part was convincing people who had these scripts to convert them. While the shell I wrote had significant features that made scripting easier, the way I convinced the other groups was with a performance bake off. I spent time improving the performance, so that probably took another, I don’t know, 6 months or a year to convince other groups at the lab to adopt it. Also, some changes were made to the language to make the conversion of these scripts less painful.

How come it fell on you to do this?

The way it worked in the Unix group [at Bell Labs] was that if you were interested in something and nobody else owned the code then you could work on it. At the time Ken Thompson owned the original shell but he was visiting Berkeley for the year and he wasn’t considering working on a new shell so I took it on. As I said I was interested in language design and had some ideas about making a programmable command language.

Page Break

Have you faced any hard decisions in maintaining the language?

The simple answer to that is I stopped adding things to the language in 1983. The last thing I added to the language was functions. And I don’t know why I didn’t put functions in the first place. At an abstract level, a command script is a function but it also happens to be a file that needs to be kept track of. But the problem with command files is one of performance; otherwise, there’s not a lot of semantic difference between functions and command scripts. The performance issue arises because executing a command script requires a new process to be created via the Unix fork and exec system call; and that’s expensive in the Unix environment. And so most of the performance issues with scripting come from this cost. Functions also provide abstraction without having a fork and exec required to do the implementation. So that was the last thing I added to the language.

Any one language cannot solve all the problems in the programming world and so it gets to the point where you either keep it simple and reasonably elegant, or you keep adding stuff. If you look at some of the modern desktop applications they have feature creep. They include every bell, knob and whistle you can imagine and finding your way around is impossible. So I decided that the shell had reached its limits within the design constraints that it originally had. I said ‘you know there’s not a whole lot to more I can do and still maintain some consistency and simplicity’. The things that people did to it after that were make it POSIX compliant and no doubt there were other things that have been added over time. But as a scripting language I thought it had reached the limit.

Looking back, is there anything you would change in the language's development?

In the language design I would certainly have added functions earlier. I am rather surprised that I didn’t do that as part of the original design. And the other thing I would like to have done is written a compiler for it. I got halfway through writing a shell script compiler but shelved it because nobody was complaining about performance at the time.

I can’t think of things that we would have done particularly differently looking back on it. As one of the first programmable scripting languages it was making a significant impact on productivity.

If the language was written with the intention of being a scripting language, how did it become more popular as an interactive command interpreter?

1 2 Page 1
Page 1 of 2
Shop Tech Products at Amazon