After 30 years, Ted Nelson's dream of universal hypertext is close to realization. Last fall, Autodesk bought Ted's Xanadu project, or an important piece of it. Later this year, if all goes well, Autodesk will release Xanadu/Server, a radically different approach to data storage and perhaps the most ambitious multimedia and hypertext system ever commercially available.
The release of Xanadu/Server should be enormously satisfying to Ted, and to the faithful who have followed the glacial movement of Xanadu toward producthood over the decades. And it should be big news: There is likely to be a flurry of press coverage around the release, for several reasons. The product is genuinely interesting, and Autodesk will put some effort into promoting the release. But the event will also get press because Ted Nelson is always quotable.
Unfortunately, some of that press coverage is likely to be wrong, because the product and its connection with the 30-year-long Project Xanadu are complicated. And what Autodesk is building is not the whole of Project Xanadu, but a key component. DDJ will probably have more to say about Ted, Xanadu/Server, Project Xanadu, and the dream of universal hypertext when Autodesk actually releases the product. But before the noise begins, here is an advance look at a radical piece of software, and at the radical guy who dreamed it up.
"People referring to Xanadu have a quite natural confusion between Xanadu the server and Xanadu the projected world-wide publishing repository," Ted explains over dinner in a restaurant down the street from the Palo Alto offices of Xanadu Operating Company (XOC). XOC is the Autodesk subsidiary that is developing Xanadu the server. We've just come from the XOC offices, where Ted has reassembled virtually his entire original Xanadu team: Roger Gregory, Mark Miller, Eric Hill, Roland King, and others. What they are working on at XOC is merely Xanadu the server; that's all Autodesk bought. Xanadu the projected world-wide publishing repository remains the exclusive province of Theodor Holm Nelson.
I wonder if I have just heard one of the reasons for this "quite natural confusion" when Ted immediately goes on to tell me that the server and the publishing repository are really the same idea. This idea has obsessed him for 30 years, spinning off several books, entertaining lectures, and provocative views on software interface design as a branch of cinema. Perhaps only someone with Ted's views on the interconnectedness of information would think of this web of themes as a single idea.
If it is one idea, it has many faces. "There are a thousand ways to describe it," he says. The Evil of Files is one approach. The phrase comes from a seminar Ted gave this spring in one of Terry Winograd's classes at Stanford.
"Most people act as though God invented files," he grumbles over a salmon steak. "Because it is simple to implement a hierarchical directory structure, and it is simple to implement a model in which all of memory is cleared and you bring in something new as a great lump, this comes to seem the way computers ought to work."
Ted's rejection of this model is total. "From the very beginning, before I really understood that model -- well, I never deeply accepted it. I understood it but I said, 'Wait a minute. What we need is --' and I've been on that same path ever since."
He doesn't have a single name for that path, either. "You can call it multithreading or zippered lists or whichways, the term I used in the 1970s. The basic idea is, you have a pool of material and the different objects you want to deal with are threads through the pool of material. So you can pull out the threads. The same object can be on many of these different threads."
This is fundamentally different from the file model. "In the file model, if you want to use the same material twice you copy it into the different files. So it loses its identity."
In Ted's model, the Xanadu model, "each document will be able to use the same material in different ways, and if you do it right You'll be able to find out which documents use the same material and see them side-by-side."
This approach grew out of what Ted wanted to do with computers. "I have so many uses for this that anything else seems a totally wrong use of computers. In my personal life, my plan since 1960 has been to have all my writings be in a solid block of hypertext, interconnected among all the uses of the same material."
That's not how it has worked out. Ted has published several books "the old way," and it bothers him. "Every book I write in the old method is written wrong, and therefore does not participate in the plan I originally formulated. So each of these mock achievements is a heartbreak to me."
Hence the Xanadu model, in which documents are represented the way Ted Nelson wants them to be represented.
Xanadu documents, not files, are the focus of the Xanadu/Server system.
Understanding the document model of Xanadu/Server is a prerequisite to understanding anything else about it. And understanding the document model requires learning a lot of new terminology. Some of the terminology may be absolutely necessary, because new ideas are being discussed (or at least ideas not seen in personal computer operating systems). Other terms may be of value chiefly in emphasizing the fact that there are new concepts here. But there is among Ted's proteges an enjoyment of terminology for its own sake. Earlier, in the XOC offices, Ted referred to me as "Michael," and there was a momentary confusion, Michael McClary thinking that Ted was referring to him. "Do you mean we've overloaded Michael?" he asked.
Ted, a self-described neologian, enjoys this.
So what is a Xanadu/Server document? Here's what it is not: A Xanadu/Server document is not a discrete stream of sequential bytes with a static content and clear boundaries.
If this sounds more muddled than the usual model of a document, it is in fact partly an attempt to resolve some of the muddle in the usual model. Conventionally, we often refer to two different things by the same name. "The DDJ Editorial Calendar" can refer to a persistent entity that passes through various revisions and can appear on various disks and in printed form. At the same time, it can refer to a specific ordering of bytes in a particular representation at a particular time.
This is a confusion of the identity of the document with its state, and it's a confusion that is more or less built into the conventional model. The Xanadu/Server document model deals with the confusion by being explicit about the identity and the state of the document. The model also distinguishes between editions of a document, which embody temporal changes in the document, and variants of a document, such as what-if scenarios.
The identity and the state of a document in the Xanadu/Server system are distinguished by having distinct objects assigned to each. A document's identity, that is, the user's perception of the document as a cohesive whole, is represented by a globally unique, system-generated, persistent object called a "Bert." A document's state is represented by a globally unique, system-generated, persistent object called a "Stamp." Each Stamp is associated with one Bert, and is a snapshot of the document's state at a particular time. Stamps cannot change.
Managing versions of documents, allowing multiple users to access a single document, and tracing the development of a document through time are all accomplished through manipulating the associations among Berts and Stamps. The system is subtle and elegant.
The Xanadu/Server model differs most clearly from the conventional model in its use of virtual documents. A simple document may consist of a sequence of bytes in a particular location; then it is similar in its state representation to a conventional document. But a document can also include parts of other documents in the form of virtual copies. The virtual copy is a kind of hot link; not a static copy, but a use of the original. It is possible to create Xanadu documents entirely by pasting together virtual copies of other works. Such a virtual document is effectively a frame placed around existing chunks of data to create a new whole from the parts.
Because no new copy is created, this virtual copy approach is not expensive in terms of space. And since the Xanadu/Server system is designed to support this mechanism efficiently, it is also not expensive in terms of time.
The key to all this interconnectedness is the connections themselves, the links. Now that hypertext is ubiquitous, it might seem that some part of Xanadu is already realized. But the links of the Xanadu system are so richly realized that they are fundamentally different from the connections in products like Guide and HyperCard. The richness is there, again, because it was something that Ted needed for his own purposes.
"I'm an amateur philosopher," he tells me over coffee. "I was a philosophy major in college and I have probably about 200,000 notes accumulated in the past 30 yers. You see, I decided in the fall of 1960 that I was not going to write any books until I had a decent editing system."
I observe that he is still working on it.
"Still working on it. Unfortunately I had to write three books the wrong way. But most of my stuff is still hanging in notes. I have literally millions of notes squirreled away on file cards in storage."
Digging through the Xanadu/Server documentation later, trying to understand the complexities and the depth of the linking system, I find it helpful to keep those file cards in mind.
The Xanadu/Server documentation describes links as objects created by end users to assert relationships between the various components of their data. Exactly how the end users do this is not the business of Xanadu/Server, however. Applications running on Xanadu/Server will provide various interfaces to the documents.
As I kept hearing Xanadu links described as omnidirectional, I wanted to challenge this claim. Aren't backlinks, I wondered, fundamentally different from forward links? Thinking about hypertext links in products like Guide or HyperCard, I envisioned clicking on a word and jumping to a new document. The backlink in this case would seem to be from the document as a whole to the word, but is it? The user interface is necessarily different, since you can't really click on an entire document as you can click on a word. But more than that, the backlink's job is not really to follow a path through the space of links so much as to backtrack through the temporal dimension of link traversal. Backing out of a link traversal is like an Undo operation more than it is like a navigation operation.
I finally realized that I was missing the point. Xanadu/Server is not a user tool, although it will probably ship with sample user tools. It is a back end, and so far as the API goes, links are entirely omnidirectional. Specific applications may apply a handedness to certain kinds of links. But only to certain kinds, and I also realized that I was missing another point in thinking of links purely as associations between chunks of hypertext. Links in Xanadu/Server can be used to map font and other attribute information to parts of documents, to implement electronic mail services, and to implement all the navigational operations of the system, both within and between documents. Everything is linked.
Because links are so fundamental to Xanadu, they are richly realized. A link is not just a connection between two chunks of data. A Xanadu link has two kinds of what are called endsets: a type space endset and a linkend endset.
The type space endset specifies what type of link this is; the set of possible types is open, and new types can be defined by applications. Some possible link types are criticism, version change, definition, footnote, cross-reference, and effectively any other relationship imaginable.
The linkend endset, and there is always at least one, describes the object pointed to by the link. Any number of endsets is possible. Each endset has a structure that consists of:
There's another point about the omnidirectionality of links. There is apparently a philosophical principle here that application developers are expected to adhere to. Links are not supposed to be hidden. The user is supposed to be able to find and follow all links from a particular document. Even if the user does not have permission to read the chunk linked to, he or she should be able to see that the link is there. On the other hand, the user is supposed to have the same ability to see all links to the document being read. So if somebody creates a link to something you have written, you can determine that this is the case.
The projected world-wide publishing repository that Ted Nelson plans to build on top of this product depends on this facility, because it provides the mechanism for automatic payment of royalties to authors.
"On bad days I feel I could have saved the world if I had only been more efficient," Ted says.
Xanadu has been a long time coming, but it is a big chunk of newness to swallow all at once. I haven't touched here on the efforts the XOC team are putting into the more conventional aspects of the system, but looking at Xanadu/Server as a conventional product, the goal is to provide high-end DBMS integrity and performance for freeform, multimedia, real hypertext, seamless-across-file, platform, and network boundaries. Not a small ambition.
Xanadu is "a generalized facility of great power," as Ted puts it, and the characterization hardly seems an exaggeration. But the practical question ultimately arises, will it actually work?
"Naturally, I'm holding my breath. I haven't been involved in the implementation for 11 years. Whenever I worry about this question I remember the immortal words of General Buck Turgidson in Dr. Strangelove: 'Hell, my boys'll get through!'"
Copyright © 1991, Dr. Dobb's JournalBeyond Files
Xanadu Links
The Bert context provides a document identity to use in viewing the endset. The original context is a Stamp ID representing the state of the context in which the link was created. The path and end contexts together specify fully the material being linked. This entire structure, this link object, resides in the same amorphous, non-file-oriented space in which documents reside; and links are editable and readable. All of this flexibility in specifying types and power in specifying context seems to me to make Xanadu links a genuinely new idea of as-yet unrealized power.Saving the World