It's easy to become so dazzled by the inexorable march of computing technology that we lose sight of the profound moral and ethical questions that it raises. Every year brings with it a new generation of CPUs, smaller and faster disk drives, cheaper and higher-density RAM chips, more capable development tools -- gosh, this must be the best of all possible worlds! But these advances carry with them as well the potential for more tragic, more far-reaching abuses of a computer's totally amoral abilities to crunch numbers, manage data, or control devices. "Foo!" you may respond (or perhaps "Foobar!"), "my programming [or engineering] job has no social implications and my heart is pure." The reality, unfortunately, is that virtually all of us work with computers in such a way that we encounter important ethical and moral issues on a daily basis -- whether we recognize them or not.
We often pride ourselves on the cleanness and elegance of programming, the tidy little digital universe that we can control so absolutely with our properly structured routines, and the way we can distance ourselves from the more mundane concerns of "ordinary people" who have to wear suits and go to work at fixed hours. But no system is completely closed; the effects of everything we do ripple out into the society we live in, and can sometimes have an impact beyond our wildest imaginings. Think about the power Bill Gates has over the direction of an industry worth hundreds of billions of dollars, and the lives of all the people that work within that industry! Bill Gates didn't acquire his power by inheritance, military action, or demagoguery -- he literally just thought it into existence. Your actions and mine as programmers or engineers aren't likely to have quite the same cosmic, long-term effects as those of Bill Gates -- but the potential is there.
As a sensitization exercise, let's look aside at the use and misuse of a rather different high technology. During the recent civil unrest in Los Angeles (which I experienced at a somewhat closer perspective than I would have preferred), roving journalists employed minicams to bring arson, assaults, rioting, and looting to the television screens in real time. Among the specific images that stick in my mind are an "action shot" of hoodlums throwing trash cans through windows of one of the city government's office buildings, a close-up of an innocent Hispanic driver being dragged out of his car and beaten by a mob, a protracted scene of streams of people emptying a furniture store while two policemen sat in their car watching, and a panoramic view of the home of one of the Rodney King trial jurors that included the juror's street address and a glimpse of his four-year-old daughter peeking through the door.
I am sure that the minicam journalists would tell us that they were just doing their job and that we shouldn't blame the messengers for bad tidings. But how does the possession of a minicam and a press badge release a citizen from his obligation to aid his fellow citizens in distress and uphold public order? At what point do real-time images of looters operating with impunity under the very noses of a paralyzed police force cease to be news and become an invitation to other opportunists to join in the looting? Imagine that the angry mob who thrashed the Hispanic passerby had instead decided to go burn down the home of the juror I mentioned or (worse yet) injured or kidnapped his daughter? What entitles some journalist with a thousand dollars worth of high-tech hardware on his shoulder to put a four-year-old's life at risk in the pursuit of "news?"
Before we get too self-righteous about the minicam wielders, though, let's direct our attention back to our own field. Violent injuries or deaths that can be directly attributed to computer-logic errors are mercifully rare so far, but they definitely have occurred: In Europe, an incorrectly programmed microcomputer-controlled fuel-injection system led to a fatal truck accident; in Texas, a misprogrammed X-ray machine delivered radiation overdoses that brought about the demise of at least one cancer patient. No doubt we can look forward to more of these "computer homicides" with the introduction of fly-by-wire airliners, driverless rapid transit trams, "closed-loop" therapy in intensive care units, robotized factories, and the like. But wait! Computers don't kill people, people kill people. So who should shoulder the responsibility for these unfortunate events? The authors of the compilers that were used to build the faulty applications, the application programmers themselves, the programming managers or equipment manufacturers that did not insist on sufficiently thorough testing, the managers who selected and purchased the faulty equipment, or the equipment operators who put human lives at the mercy of computerized machinery without adequate safeguards? Not an easy question.
Perhaps the cases I just mentioned are too dramatic for your tastes. Let's reach back, therefore, to the Jurassic age of computers for an utterly different sort of ethical problem: one of the first recorded accounts of a major breakdown in privacy protection for computer users. Here's Fernando J. Corbato discussing Project MAC and the Compatible Time Sharing System (CTSS) circa 1964 (IEEE Annals of the History of Computing, vol. 14, 1992):
In order to build the system simply, we put in a lot of constraints, and one of the constraints was that a single user couldn't change the directory, and conversely only one user at a time could be logged into a single directory... Now for the system there was a single system programming directory where the commands were [stored], and a couple of system programmers were twisting my arm saying, "Gee, it's really awkward, but we have to get two installations queued up in line. Just let us take off the interlock on that directory only, and we'll let at least two people be logged in at once on the same directory." That sounded innocent enough sort of, but what happened was that the editor that was commonly used created a temporary file in that directory called something like .temp. Unfortunately, it had no way of distinguishing who had invoked it.
So it turned out that the system administrator of the day who was trying to change the message of the day and another system programmer who was worrying about entering some new passwords both used the editor. They both ended up working at the same time, they both created .temp files and/or wrote over the .temp file. And, lo and behold, before anyone quite realized what had happened, the message of the day turned into being the password file. And the way it was discovered was that somebody started to log in soon thereafter (they were on the ninth floor) and they stared in wonderment as the passwords unfolded in front of them...
This incident sounds pretty benign, until you remember that it occurred at MIT on one of the very first systems to be regarded as an "information utility," in daily use by graduate students, researchers, and many of the foremost computer scientists of that era. In a cut-throat academic environment like MIT, careers can be made or broken by an indiscreet memo falling into the wrong hands, the premature disclosure of research data that ultimately proves to be wrong, or the preemptive publication of some new technique that should have been credited to someone else. In other words, this little anecdote shows all too graphically how lives can be changed (and not for the better) as a result of someone failing to think through all the implications of a minor change to the system software. For another incident that lies ethically somewhere between the two extremes of death and passwords, consider the notorious Internet Worm.
On November 2, 1988, Robert T. Morris, a 23-year-old computer science student at Cornell University, unleashed a "worm" program that took advantage of a trapdoor in the UNIX sendmail program and a bug in the finger program to propagate itself from one BSD 4.3 system to another. (The term "worm," incidentally, was coined by John Brunner in his 1975 novel The Shockwave Rider, which foreshadowed the cyber-punk novels of Gibson and Sterling.) Within a matter of hours, the worm had spread across the continent and brought thousands of UNIX systems to their knees. Whatever Morris's intentions, it's clear that the worm was not simply an idle prank; the worm used a sophisticated, multipronged attack strategy and was based on a deep knowledge of UNIX system internals. Also, inspection of the Cornell system logs revealed that Morris was testing his worm as early as October 19, and tried several implementations before arriving at the "successful" version. Some people have defended Morris with the observation that the worm didn't damage any files and (due to the efforts of teams at MIT, Berkeley, and other sites) was fairly rapidly dissected and neutralized. But what about the hundreds of thousands of man-hours that were lost forever by those programmers, engineers, professors, students, and researchers who depended on the worm-infested UNIX machines? Some machines were not reconnected to the Internet until as late as November 10, and the mail backlog was not cleared until November 12.
Like a gun or a minicam, the immense power and the even more immense stupidity of the computer make it a dangerous device in the hands of the malicious, the ignorant, the morally blind, or the merely oblivious. Every time we create a programming tool, automate some task that previously required a real-live person, change someone's password for our own convenience as the network manager, "loan" a copy of Lotus 1-2-3 or Microsoft Excel to a friend, consult for the government, or even put a customer's name and credit rating into a database, we should be thinking carefully about the broader ethical and moral implications of our actions. Whose time might we be stealing, whose expectations might we be confounding, whose trust might we be betraying, whose position might we be compromising, whose children might we be endangering? It's required, but hardly sufficient, to act honorably according to our own lights; we must also seek to broaden our understanding of the moral and ethical issues and learn how others have dealt with challenging situations or ambiguous predicaments. While I'm certainly no expert in these areas, there are three books that I would commend to you as a starting point.
The seminal work on the social issues of computing is Joseph Weizenbaum's Computer Power and Human Reason: From Judgment to Calculation. This book, which was published in 1976, had a deep influence on me at the time and I give it a lot of credit for steering me away from an isolated, antisocial, hacker existence. Weizenbaum's description of the "compulsive programmer" is classic:
The computer programmer is a creator of universes for which he alone is the lawgiver. So, of course, is the designer of any game. But universes of virtually unlimited complexity can be created in the form of computer programs. Moreover, and this is a crucial point, systems so formulated and elaborated act out their programmed scripts. They compliantly obey their laws and vividly exhibit their obedient behavior. No playwright, no stage director, no emperor, however powerful, has ever exercised such absolute authority to arrange a stage or a field of battle and to command such unswervingly dutiful actors or troops.
One would have to be astonished if Lord Acton's observation that power corrupts were not to apply in an environment in which omnipotence is so easily achievable. It does apply. And the corruption evoked by the computer programmer's omnipotence manifests itself in a form that is instructive in a domain far larger than the immediate environment of the computer. To understand it, we will have to look at a mental disorder that, while actually very old, appears to have been transformed by the computer into a new genus: the compulsion to program.
Wherever computer centers have become established, that is to say, in countless places in the United States, as well as in virtually all other industrial regions of the world, bright young men of disheveled appearance, often with sunken glowing eyes, can be seen sitting at computer consoles, their arms tensed and waiting to fire their fingers, already poised to strike, at the buttons and keys on which their attention seems to be as riveted as a gambler's on the rolling dice. When not so transfixed, they often sit at tables strewn with computer printouts over which they pore like possessed students of a cabalistic text. They work until they nearly drop, twenty, thirty hours at a time. Their food, if they arrange it, is brought to them: coffee, Cokes, sandwiches. If possible, they sleep on cots near the computer. But only for a few hours, then back to the console or the printouts. Their rumpled clothes, their unwashed and unshaven faces, and their uncombed hair all testify that they are oblivious to their bodies and to the world in which they move. They exist, at least when so engaged, only through and for the computers.
Weizenbaum's book was written to be accessible to laymen, so the first part of the book is tutorial in nature: "Where the Power of the Computer Comes From," and "How Computers Work." Once the introductory material is out of the way, however, the book is equally valuable to the experienced computer practitioner. Weizenbaum discusses problem solving and modeling, the psychology of computer interactions, natural-language processing, self-organizing complex programs (he refers to these as "incomprehensible programs"), and artificial intelligence. (Weizenbaum was the author of the ELIZA program.) In retrospect, nearly 20 years later, much of his discussion of AI was prophetic even if it has become moot, because the field has lost most of its credibility and is frankly floundering. Weizenbaum's book ends with a chapter called "Against the Imperialism of Instrumental Reason" that should be required reading for every computer science student.
The second book I would like to bring to your attention is Computer Ethics: Cautionary Tales and Ethical Dilemmas in Computing, by Tom Forester and Perry Morrison. This short but superb text surveys the entire gamut of computer-related ethical, moral, and social issues, including computer-assisted crimes, software theft, hacking, viruses, unreliable computers, invasion of privacy, artificial intelligence, the Strategic Defense Initiative ("Star Wars"), and computerization of the workplace. Many detailed case histories are included, and these are supported by an extensive bibliography. The authors have a direct, vivid, precise writing style that I admire enormously!
The third book that I think you would enjoy is Computers Under Attack: Intruders, Worms, and Viruses, edited by Peter J. Denning, past editor-in-chief of Communications of the ACM. The book begins with a history and description of the Internet by Denning, then continues with a collection of articles on various security topics by authors such as Ken Thompson, Clifford Stoll, Maurice Wilkes, Richard Stallman, and Paul Saffo. The extremely detailed analysis of the Internet Worm, some of which was previously published in Communications of the ACM, is particularly interesting. The common PC viruses are also listed and their implementations are explained. The last part of the book surveys the reactions to the Internet Worm by all sectors of society, ranging from legislators to hackers.
Copyright © 1992, Dr. Dobb's Journal