Technical writing is a demanding, yet largely unappreciated vocation. When you further restrict the focus to technical writing about operating systems, programming interfaces, and development tools for trade magazines and book publishers, you've got a vocation that is not only demanding and unappreciated, but decidedly peculiar as well. There's probably no other field where an author can reach so massive and sophisticated an audience and, at the same time, be obligated to contend with such frequent technological advances, short product life cycles, shoddy vendor documentation, hair-raising nondisclosure agreements, fuzzy facts (and even more fuzzy release dates), poorly understood market forces, and high socioeconomic stakes. It's a dirty job, scorned by academia and "real" programmers, and the author who tries to stay on the cutting edge often finds himself on the bleeding edge instead.
To be perfectly fair, one of the reasons that technical writing carries so little prestige in the programming field is undoubtedly because it so faithfully follows the 90/10 rule. At least 90 percent of the articles and books on programming are (to put it in terms suitable for a family magazine) mere dreck: mindless verbiage generated by hacks who simply rehash company backgrounders and product manuals without adding any value whatsoever, or by journeymen who grind out boilerplates according to editorial outlines. The 10 percent of the programming articles and books which can be said to have some redeeming value come from two types of authors: "shooting stars" who make one or two significant contributions and are never heard from again, and a remarkably small core group of superstars who steadily produce excellent articles and books year-in and year-out.
What characterizes the superstar technical writers? They all have an obvious affection for programming coupled with years of experience in the front lines of software development. They exhibit an attention to detail and respect for accuracy that borders on fanatical. They write about things they've done, not things they've heard. They are quick to acknowledge the ideas and accomplishments of others, wasting little time on turf wars or battles for precedence. They are open-minded, eclectic, widely read, and historically savvy. They are especially facile at deducing a logical structure from scattered fragments of information (or imposing a structure, if necessary) and explaining this structure to others. They feel a deep compulsion to write and are highly efficient at it. And finally, they each have a unique style and a genuine gift for the beauty of the language (many have a wicked sense of humor as well). In short, we're talking about people like Jeff Duntemann, Charles Petzold, Michael Abrash, Jeff Prosise, and Andrew Schulman.
Andrew Schulman has a distinctive approach that is reminiscent of the Greek classics: He doesn't just give you the facts, he grabs you by the collar and shows you (in sometimes painful detail) where, why, and how he got the facts, embedding them in a philosophical framework that explains why they are important and how they may safely be applied. Over the last few years, Andrew has contributed a series of brilliant essays to BYTE, DDJ, and PC Magazine, coauthored and edited the fascinating book Undocumented DOS, and saved my personal bacon on the book Extending DOS--all this in his "spare time" while gainfully employed at Lotus and then at Phar Lap Software. More recently, Andrew has gone into the writing racket full time, and his latest book (with David Maxey and Matt Pietrek) is Undocumented Windows.
Before you API purists out there decide to turn the page, let me say that I, too, have little patience with books obsessed with forbidden lore for its own sake. (The childish and tiresome New Hackers Dictionary is a perfect example.) However, Undocumented Windows does not fall into this category. Dealing with the Windows API as a pristine entity is not a practical strategy; many of the functions overlap, interact, or have nonintuitive side effects. Additionally, functions or behaviors that are undocumented today may well be documented tomorrow, as Microsoft's agenda vacillates or goes through one of its periodic startling metamorphoses. The first-generation Windows books--Petzold, Yao, Richter, and Heller--supplemented the Microsoft Windows SDK, but made little attempt to go beyond it. I believe that Undocumented Windows is the first true example of a second-generation Windows programming book, because it takes you behind the scenes to show how the various Windows modules are put together, how they depend on each other, and how they have evolved from the real-mode-only Windows version 1.03 to the protected-mode-only Windows version 3.1.
Of course, Andrew being a fellow cursed with insatiable curiosity, a liberal helping of forbidden lore is bundled in as well--orphaned or senseless code fragments from the Windows kernel, details of the data structures behind the various types of Windows handles, and documentation for hundreds of previously undocumented functions with evocative names like Death, Resurrection, PrestoChangoSelector, TabTheTextOutForWimps, WinOldAppHackOMatic, UserSeeUserDo, Bunny_351, Brute, and FixUpBogusPublisherMetaFile. The number of hours the authors must have spent disassembling object code, tracing program execution, poring over memory dumps, combing through Windows SDK and DDK source code and header files, and attempting to correlate vaguely related interfaces such as OS/2's DosPTrace to create this book simply boggles the mind. Chapter 1 of Undocumented Windows, entitled "This Was Not Supposed to Happen," includes an apologia that clarifies the authors' motivation:
A key goal of Microsoft Windows is to be more orderly than MS-DOS. DOS is a "house of cards," with memory-resident (TSR) programs, device drivers, disk caches, memory managers, DOS extenders, networks, and multitasking environments (such as Windows itself) all competing for control of your machine. From the software developer's perspective, Windows often looks a lot saner. It provides a wide-ranging and seemingly all-inclusive collection of services--such as protected mode, multitasking, dynamic linking, window management, and graphics--that plain-vanilla DOS doesn't offer. Often, Windows lets developers concentrate on making a program do what it is supposed to do rather than on the underhanded shenanigans--including the use of undocumented system functions--that are necessary to create a great DOS application.
The idea of "undocumented Windows," then, is really somewhat alarming. Using undocumented functions is exactly the sort of problem Windows was supposed to solve! Making use of functions that Microsoft has implemented but not documented fits in perfectly with the free-wheeling style of DOS, but it seems to contradict the entire spirit and purpose of Windows. By providing an API much more extensive and capable than DOS's, Windows is supposed to make such tricks unnecessary.
But how likely a scenario is this? How many commercial Windows applications can really "play by the rules" and still be marketable, with decent performance and with the features users expect? The idea that the Windows API can totally replace low-level coding seems, unfortunately, no more reasonable than the idea that C++ can totally replace assembly language--in other words, not very reasonable at all.
What we will see in this chapter is that key commercial Windows applications, including Microsoft's own, use undocumented API calls. In some cases, these calls have since been documented by Microsoft, though only after developers went ahead and used them anyway, without Microsoft's blessing. In other words, real-world use of the Windows API has driven the documentation, rather than the other way around. Writing only to the documented Windows API sounds great, but has failed in the real world.
What went wrong with the lovely notion of Windows programming without tricks, without low-level, nonportable code, without undocumented shenanigans? What went wrong, mostly, is that Windows succeeded. By winning the operating system wars, Windows is now paying the price of success: large numbers of programmers are banging on the system, and they need to make it do all sorts of things for which it was probably never intended. The use of undocumented features, in other words, is the inevitable price of success. MS-DOS paid this price, and now Windows will. Interestingly, Windows too is now being called a "house of cards."
The remainder of Chapter 1 rambles through a variety of topics: dynamic linking, protected mode, the Microsoft "Open Tools" strategy, portrayals of the use of undocumented API functions by Norton Desktop and certain Microsoft products, the FTC investigation, Microsoft's celebrated (and in my judgment totally mythical) "Chinese Wall," and the first-ever in-print account of Microsoft's shameful bullying of famous Windows guru Michael Geary. (How a gentleman like Mike Geary, who has devoted untold thousands of hours to helping other Windows programmers on Genie and CompuServe, could ever find himself in the gunsights of Microsoft's corporate lawyers is an unfathomable mystery and a chilling manifestation of the Dark Side of the Force at Microsoft.)
Chapters 2, 3, and 4 of Undocumented Windows are methodological; they explain the static analysis of Windows executables and how to use various debuggers, "spy" programs, and other utilities to poke around in Windows' innards. Chapter 3 is particularly interesting because it constitutes, in essence, a crash course in how to disassemble Windows DLLs and applications back to their source code. Each subtask--from identifying callback procedures to dumping out the file's binary resources--is carefully explained, using TASKMAN.EXE (Microsoft Windows Task Manager) as a practical example. In fact, you could pretty quickly write your own customized Task Manager from the annotated TASKMAN source code included in this book.
Chapters 5 through 8 are devoted to in-depth examinations of Windows KERNEL.EXE, USER.EXE, undocumented messages, and GDI.EXE, respectively. All of the crucial hidden data structures are dissected, and each exported API function not found in the official Microsoft manuals is documented here with its parameters, flags, return values, and bugs. Most function entries are accompanied by source code for a short program that exercises the function. Much "legal" information that is inexplicably missing, incompletely documented, or incorrectly documented in the Microsoft manuals is also found here. For example, the enigmatic "free system resources," the structure of local heaps and atom tables, transformations between task, module, instance, and window handles, and a method for determining whether your application is running in an OS/2 2.0 Windows-compatibility session. In the course of all this, Schulman and Maxey occasionally take Microsoft to task for its "do as we say, not as we do" attitude:
SetSelectorLimit() should be equivalent in functionality to the DPMI Set Segment Limit (Int 31H AX=08H) function. However, KERNEL seems almost never to rely on one documented DPMI function, when multiple DPMI functions, or avoiding DPMI altogether, will do. Go figure.
The last two chapters of Undocumented Windows cover SYSTEM.DRV and TOOLHELP.DLL, and they are followed by two appendices. The first appendix is a WINIO library reference; I'll say more about WINIO in a moment. The second appendix is an annotated bibliography of articles, books, samizdat documents, software, and other primary sources invaluable to any Windows reverse-engineer wanna-be. The bibliography is by turns amusing, informative, and ironic, and near the end tips his hat to Hofstadter by reviewing the same book he is writing.
A jumble of material on undocumented functions and internal data structures in KERNEL, USER, and GDI. Apparently a second book is planned to cover Windows DLLs such as SHELL, 16-bit device drivers, 32-bit VxDs, DPMI, interrupts, and other lower-level aspects of Windows. Contains an extensive bibliography, with only one recursive self-reference, which one of the coauthors uses just to talk about different books and articles he likes: some of them don't even have anything to do with Windows!
I should mention that the example programs in Undocumented Windows resemble no other Windows application source code you've ever seen or are likely to ever see again. Not only do they exploit a bewildering variety of undocumented API functions that you've never heard of outside this book, but they don't use the documented functions that you've always heard of. This is because the programs are written with the aid of an interface library-cum-application framework, called WINIO, that was first described by Schulman and Maxey in Microsoft Systems Journal (July 1991). WINIO conceals the Windows API and the message-oriented, event-driven nature of Windows applications beneath replacements for the more familiar C runtime-library functions such as gets() and printf(). The authors defend their approach thusly:
That a Windows application does not have to directly use the Windows API, that you can put a layer on top of this API, surprises so many Windows programmers that we could almost claim that this fact is "undocumented." Certainly Microsoft's SDK manuals never suggest that you could write a Windows application in any way other than peppering your code with direct calls to TextOut(), BeginPaint(), and so on. The idea that a Windows program must contain direct, explicit Windows API calls--that it's not a "true" Windows application if it isn't descended from the original GENERIC.C--is part of the same reverence for the Windows API that we seek to undermine by disassembling this API and looking at the code. It may seem odd to introduce a way of hiding the Windows API, in a book otherwise devoted to exposing even lower level portions of it. However, revealing undocumented Windows API calls and then covering up the existing documented ones are really just two sides of the same coin: questioning the Windows API, instead of taking it on face value. The API is just code; we can do with it what we will.
Although I share Andrew's antipathy for the Hungarian gibberish and the multipage switch() statements found in Microsoft SDK sample programs, I sometimes feel that WINIO verges on throwing the baby out with the bath water. It's nice to make things easier for traditional C programmers, but they're going to lose out by not learning and using the new concepts inherent in the Windows programming model. Still, I must admit that the authors' use of WINIO substantially simplifies their example source code, and therefore, on balance, must be considered an asset to the book. A companion disk bound into the book contains source code for the example programs, the WINIO library, and miscellaneous Windows spelunking aids. Some of the more interesting utilities on the disk include:
Copyright © 1992, Dr. Dobb's Journal
Undocumented Windows is not completely flawless. The presentation is resolutely centered on the C programming language and relies extensively on C pointer idioms even within the narrative text, which makes the book difficult for Visual Basic, Turbo Pascal/Windows, or Smalltalk programmers to digest. The commenting and formatting of the C source code is uneven. There are few diagrams, and the production values are disappointing; the book would have benefited enormously from the attention of a skilled book designer, a strong-willed manuscript editor, and a few diligent copy editors. (It will be interesting to see whether Addison-Wesley awakens to Schulman's importance before he decides to take his projects to a publisher that will give them more professional treatment.) But these quibbles do not change the fact that the release of Undocumented Windows is one of most important events of 1992 for our little corner of the universe, and every serious Windows programmer should own a copy of this book. The days when Microsoft could get away with saying, "Pay no attention to that man behind the curtain" are over.