Dr. Dobb's Journal February, 2005
Perhaps the title this month should be "Programmer Paradigms." Several things I've read recently have got me thinking about different kinds of programmers: the lone coder versus the connector, the troubleshooter versus the architect, the guerrilla versus the soldier. Few of us, perhaps, would fit unambiguously into one category or another, but we can probably all see what label fits us best. The challenging question, though, is: How well does the kind of programmer you are fit into the time and situation in which you currently find yourself? And if the fit is not ideal, can you change yourself? Can you change your environment? (I'm just asking; I don't have the answers.)
A Slashdotter asked a disturbing question last November. "Is the lone coder dead?" CyNRG asked. "The little guy. The one-person software company. Can it still exist today?" CyNRG's corn wasn't economics or code complexity or even Microsoft's dominance of the software market, but software patents. Will the need to ensure that no idea in your code is covered by some obscene obvious-tweak patent soon mean that the Lone Coder can no longer afford to sell software? "The amount of money required to perform the due diligence research seems like it would be greater than the amount of money needed to develop the software, or even the total revenues that the software could ever generate," CyNRG moaned. "Please someone tell me I'm wrong!" CyNRG can't have been satisfied with the discussion that this plea generated on Slashdot. Mostly it was variations on bogus news reports and autopsies of the Lone Coder's death (death by overdose of Jolt, and so on); arguments that he didn't die, he just moved overseas; and rejection of the whole Lone Coder theory in favor of a Second Coder on the Grassy Knoll. To a comedian, everything looks like a straight line. I still think that the fate of the Lone Coder is an interesting question, but if you want to send me your thoughts on it, I just wanted you to know that I've read all the obvious jokes. We all are Lone Coders, though, when it comes to programming for that very special client—ourselves. And we all keep looking for better tools for our homemade-software toolkits.
I really want Mathematica to be a tool in my toolkit. Whenever a new version comes out, I hack away for a weekend trying to get a handle on it, but the learning curve rises ahead of me and I only advance a small step, while I've fallen back one and a half since my last effort. The learning curve has so far kept me from exploiting the serious power that's in this product.
The new string-handling capabilities in the latest release could change that. Wolfram Research suggests using the new features for web site manipulations, data mining, or DNA processing. While I'm not heavily into DNA processing, manipulating text is something I do a lot of—whether for something as mundane as preprocessing text for web publishing or as frivolous as trying to write a program to generate crossword puzzles for my own amusement. Mathematica's analytical power combined with the new string-handling capabilities immediately suggested some uses. Such as, "how many unique words are there in a sample of my DDJ columns?" "What are the most common words?" "How about a bar graph showing the frequencies of words in my columns?" This time I found myself getting farther up that curve.
Mathematica supports different programming paradigms and modes of operating, but it is above all designed to make it easy to play around with variables and calculations in an interactive way. You work in a workbook, which remembers values of previous calculations so that you can build up very elaborate but flexible what-if structures.
I wrote the following code to read all my 2004 "Programming Paradigms" columns and combine their texts into one long string. Nothing startling, but note the specification of multiple files with Grep-like notation in the second line. This makes myColumns be a list with entries like "PP0401.txt," "PP0402 revised.rtf," and so on:
SetDirectory["../Documents/Paradigms/"]
myColumns = FileNames["PP04*"]
For[rawText = "";
i = 1,
i 2 Length[myColumns],
i++,
rawText = StringJoin[rawText,
Import[myColumns[[i]], "Text"]]]
The 32K-word string rawtext needed some massaging before I could do statistics on it. I appreciated the interactive nature of Mathematica when it came to identifying the punctuation and other characters that I needed to strip out to have just words left. It took several passes with different word lists to come up with a list of 25 characters to ignore.
Mathematica is big on treating actions as data; the first line below (which I've abbreviated) defines a list of replacement operations (replace "." by " "), which is then used by the built-in function StringReplace. Other built-in functions turn letters to lowercase and return a list of the individual words. (Lists in Mathematica are denoted by braces: {}.)
ignoreChars = {"." -> " ", "?" -> " "...}
wordList = StringSplit[ToLowerCase
[StringReplace[rawText, ignoreChars]]]
Mathematica's programming paradigm of choice is functional programming, and like that classic functional language Lisp, Mathematica loves lists. And because Mathematica was created to do symbolic math, it knows about operations like Union, which treats lists as sets. When applied to a single list of words, Union eliminates all duplicate words.
uniqueWords = Union[wordList]
Now I had my vocabulary list for my 2004 columns and could start doing statistics on it.
Length[uniqueWords]
That one-liner told me that I used 4766 unique words in the 12 columns of 2004. Most of the other statistical manipulations that I'd like to do, like computing the most common words or looking at word sequences, are also one-liners. But graphing the results wasn't much more difficult, after I loaded the Graphics module.
For[favoriteWords = {};
i = 1,
i 2 Length[uniqueWords],
i++,
AppendTo[favoriteWords, Length
[Cases[wordList, uniqueWords[[i]]]]]
]
<< Graphics'Graphics'
BarChart[favoriteWords, BarOrientation ->
Horizontal, BarLabels -> uniqueWords]
A glance at that graph shouted "Too Much Information," so I went back a step and trimmed down the favoriteWords list to just the most common ones, testing different parameters to get a readable graph. And so on. Before the weekend was over, I had built a statistical "Programming Paradigms" column generator. I was relieved to see that my Mathematica skills aren't good enough that I can replace myself as a columnist.
There's nothing here that couldn't be done in any other language, and in some languages it could doubtless be done just as succinctly. But Mathematica is powerfully interactive and operates at a symbolic level that can be exploited to do surprising things. And its own vocabulary is very deep: If the quick-and-dirty code I wrote proved inefficient, I could have spent a little more time plumbing the language's depths for other ways to do the calculations. That wasn't necessary in this case, because Mathematica is well tuned under the hood. Operations that look wildly inefficient generally run faster than you'd guess. On my 1.8-GHz iMac G5, these wasteful string manipulations ran too fast to bother optimizing.
And I didn't even exploit all the new string manipulation features.
This one won't go in my own toolkit, but it might find a place in yours.
Spheres, a spin-off from Caltech, has come out with a new programming language that is event based. That doesn't sound too revolutionary, but the kinds of events they have in mind are financial transactions and attempts to break into a computer system. A programming language designed from the ground up to deal with such events is certainly worth a look. An early version, called EPL (Event Programming Language) Server, is available and can be used royalty free.
EPL Server isn't intended to replace other programming languages, but to interface with them and to simplify a small set of tasks. The model is designed to:
What's new about EPL is not the technology, which is familiar. Other companies and industry groups are working on standards and technologies for business event processing. As reported on Cnet's Builder.com.com, KnowNow has developed similar software to send data to many destinations based on an event. Efforts are afoot to create an industry-wide language for automating business processes, called Business Process Execution Language (BPEL), for which languages or rule engines to handle events will presumably be required. The fact that we will probably soon have billions of RFID tags broadcasting their meagre bits of information everywhere we go suggests that we need a model of programming that deals opportunistically with the events that impinge on it. And so on. EPL Server is an interesting tool that may become an important tool for swimming in that sea of digital events.
But what I find interesting about EPL Server, in the context of programmer paradigms, is the philosophical model. The EPL programmer is less an architect, more a troubleshooter. The language encourages a worldview that accepts that things happen on their own schedule and need to be dealt with. It is a philosophy of dealing with what is thrown at you rather than trying to engineer top-down control. EPL Server is certainly not a Lone Coder's programming language. But there is an aspect of it that at least feels incompatible with the top-down control philosophy.
Here is a different psychological profile of a programmer: One who possesses social skills, enjoys working in groups, and is a born networker. Such a programmer would hardly mourn the death of the Lone Coder. This is the kind of programmer that I like to call "Them."
Occasionally, I have to pretend—to myself, especially—to be one of Them. I found a book that helps. Fearless Change, by design pattern experts Mary Lynn Manns and Linda Rising (Addison-Wesley 2005; ISBN 0201741571), is not a book that loners would naturally gravitate toward. The authors unabashedly admit to being connectors, and the whole thrust of their book (and of the patterns they present) is connecting. It's all about design patterns for effecting change in groups.
To a comedian, everything looks like a straight line, and if all you have is a hammer, every problem looks like a nail. So naturally, these born networkers argue for solving the problem of introducing change into groups by their preferred method of working to recruit others to your cause, rather than by the time-honored technique of core-dumping every detail of your brainstorm onto everyone within three feet of you and stomping out of the room in a huff when the clueless losers don't immediately get it.
To each his or her own? Yes, but sometimes their own works better than our own; and those least like us have the most to teach us, if we are open to learning from them. As a natural loner, I find that this book speaks only to that underdeveloped social lobe of my brain, but the question is: How well does it do that? Pretty well, I think, on balance, but for those of us whose need is great, there is a design pattern to help. The authors reference it on page 253:
Teach yourself to play a role so that observers believe you are extroverted, bold, and outgoing. Teach yourself to recognize the situations in which this role is appropriate and to gather your resources and play the role. http://csis.pace.edu/~bergin/patterns/introvertExtrovert.html.
It's just one of the places in the book where a subversive, Machiavellian tone comes through. I like that.
I don't know why no one has yet come up with a set of design patterns for reading books. It's such a common activity, and so often poorly done. Think how much time you would have saved by now if you'd had a few good book-reading patterns like the following:
Design patterns are a good example of the kind of information that books actually do convey well. Design patterns benefit from having some whiskers. It takes time to know that a candidate really merits being elected to patternhood.
Information in books is also information that you may have come across before. But familiarity does not necessarily make information worthless. As the authors point out, if we never needed to be reminded of things we have already learned, we would never make the same mistake twice. Familiar information, presented in a systematic way, can be very useful if that system works for you.
Is that the case with this book? You'll have to decide. The book consists of a collection of patterns, each described in about three pages, plus 12 chapters discussing a philosophy of organizational change and describing how to use the patterns, plus four case studies, including Sun's experience in using patterns in introducing J2EE applications. Here are some of the themes that the authors emphasize in the chapters:
And here are some of the patterns:
In my humble opinion, the great insight of the early 21st Century will be that individuals are more powerful than institutions; that a dedicated indigenous guerrilla movement can always wear down an invading army; that you can't defeat terrorism with bombs; that ad hoc, self-organizing task groups work and management-composed committees don't.
The authors of this book, unlike this columnist, avoid such political pronouncements, but they clearly understand the power of guerrilla movements. Their book shows little respect for the structures of power, except as obstacles to get around or weaknesses to exploit. As I said, Machiavellian.
"How do I sell my executive team on doing this stuff?" they ask, and, quoting consultant Jim Highsmith, they answer, "Don't. Just do it. They don't know what you're doing anyway." That's also a recipe for resisting change: The authors describe the powerful passive resistance that Carly Fiorina met when she tried to impose top-down changes at HP.
The book freely acknowledges that people are not rational, that we use facts to justify emotionally-arrived-at decisions. It then presents techniques for getting things done anyway, despite this fundamental irrationality, by the use of proven patterns.
Whether the things you get done are rational or not is another question.
Okay, that's it for this month. Thanks for reading.
DDJ