Getting to Xanadu

Ted Nelson’s Xanadu is one of the most important and most maligned ideas in computer history. Why has it held such sway after more than 50 years? Why isn’t there any working software for download or a GitHub community of devs? Is Xanadu so difficult to achieve that no one call pull it off, or are there other factors? These are questions I’ve attempted to answer for myself.

Truthfulness relies on an unchanging record of the past.

Gary Wolf’s 1995 investigation for Wired into what he called The Curse of Xanadu is the de facto online history, not only for its scathing criticisms of Nelson (which he later disputed in an open letter) but for the rich detail and fascinating revelations about the people and companies behind the project. Where else could you find out about Autodesk’s involvement and the $5M they spent on trying to achieve Nelson’s dream?

Certainly personal factors and generally being viewed as a crazy person are problems shared by all inventors and geniuses. If it were just Ted’s fault, others would have come forward and made Xanadu without him. A deeper analysis shows that there are technical and market reasons that keep Xanadu from being real. They fall roughly into four buckets:

1. Things that already exist
2. Things we don’t need
3. Things Xanadu proposed but didn’t deliver
4. Things Xanadu is missing

 

Where It All Started

Nelson’s books Computer Lib and Dream Machines, published as a single volume and bound together, are the geek’s version of Be Here Now by Ram Dass. The books are amazing and indescribable, remarkable for their breadth and freshness, completely unorganized, and impossible to navigate. These are the first hand accounts of a brilliant traveler to a new place, witnessed and narrated for us in stream-of-consciousness style. Ram Dass was also a former academic turned on by the new and drug-induced culture of the sixties, and like him, Nelson conveyed enough “key driver” ideas to have inspired generations since. By 1980 in Literary Machines, he had formalized the main goals of Xanadu into a list of 17 rules. Let’s see which of these buckets each rule falls in.

17 Rules of Xanadu

1. Every Xanadu server is uniquely and securely identified.
We could say that every device today can be uniquely identified by an IPv6 address, a MAC address, or a domain name. “Securely identified” is an interesting phrase, but blockchains like Bitcoin meet the requirement. This Xanadu rule goes in the “exists” category.

2. Every Xanadu server can be operated independently or in a network.
This is true of nearly all computers, so it exists.

3. Every user is uniquely and securely identified. This is the same as rule #1. With peer networking, there is no difference between servers and users. Already exists.

4. Every user can search, retrieve, create and store documents.
This one definitely exists.

5. Every document can consist of any number of parts each of which may be of any data type.
HMTL covers this requirement in an ugly way. I would not vote for HTML as the way to organize composite documents going forward, but it does exist.

6. Every document can contain links of any type including virtual copies (“transclusions”) to any other document in the system accessible to its owner.
This definitely does not exist yet and it’s the thing that seems to drive Ted crazy. We’ll get to why it doesn’t exist in a minute. It goes in the “proposed” category.

7. Links are visible and can be followed from all endpoints.
This is another aspect of Xanadu that’s only proposed and doesn’t exist. Really, it’s a feature of rule #6 in that once a document is transcluded in another, you can see and follow links both ways.

8. Permission to link to a document is explicitly granted by the act of publication.
This is the first example of a Xanadu feature we don’t need. A tool for readers and writers should not get involved in the concerns of platform or content providers.

9. Every document can contain a royalty mechanism at any desired degree of granularity to ensure payment on any portion accessed, including virtual copies (“transclusions”) of all or part of the document.
Again, we are messing where we shouldn’t be messing. This “feature” is only here to enable rule #15.

10. Every document is uniquely and securely identified.
This is a terrific idea and sorely needed. Enormous problems are caused by making humans take on the naming and management of documents. Is a blockchain answer the way to address this proposed idea?

11. Every document can have secure access controls.
It can be argued that we’ve claimed to have this forever but the current systems are all pretty bad. Today’s best solution to secure access is public key encryption like we see in blockchains, but losing a key has catastrophic results and will require better custodial and backup solutions in the future. I’m putting this in the “exists” bucket.

12. Every document can be rapidly searched, stored and retrieved without user knowledge of where it is physically stored.
This absolutely exists in many forms, the most obvious of which is Google. Of course, Ted is talking about a more personalized system of searching and saving documents that interest you, but the tech exists.

13. Every document is automatically moved to physical storage appropriate to its frequency of access from any given location.
This isn’t even a user feature, it’s housekeeping. Not needed. If sharding and redundant access to remote documents is required, solutions like IPFS are coming online to provide that.

14. Every document is automatically stored redundantly to maintain availability even in case of a disaster.
This is more housekeeping but definitely useful. Systems already exist to do this. Bitcoin is a great example of redundant storage of common data which is needed by many people.

15. Every Xanadu service provider can charge their users at any rate they choose for the storage, retrieval and publishing of documents.
Could this be the straw that broke the camel’s back? The requirement to turn Xanadu into a copyright management business certainly isn’t a user feature nor necessary technologically.

16. Every transaction is secure and auditable only by the parties to that transaction.
This is a terrific and important idea which is only now coming to fruition, especially with privacy-focused cryptocurrencies like Zcash. Although recent, it exists.

17. The Xanadu client–server communication protocol is an openly published standard. Third-party software development and integration is encouraged.
And strangely, here at the end, is a rule that was violated even before work began on the code. Perhaps Nelson didn’t believe enough in his publishing business model to allow anyone to see the software as it was being developed. This aspect of Xanadu was only proposed and not delivered.

The Missing Rules

I mentioned earlier that, in addition to what was proposed, Xanadu is missing some features that would help it attain commercial success today. Let’s examine them.

18. The system provides multiple views of the same data and makes it easy to switch between them.
This idea is from Douglas Engelbart and his NLS system. Xanadu was never shown with multiple views, such as slideshows, timelines, or outlines.

19. Text is stored without formatting.
This idea is discussed at length in Nelson’s papers and even mentioned in Computer Dreams, but it’s not on the Xanadu rule list. It should be. Today’s systems mix text, formatting, and data structure which makes it difficult to extract any of these later.

20. Metadata can be canonical.
This idea, which I expound upon in the Mimix whitepaper, addresses Nelson’s desire to have certain data or documents accepted as absolutes, such as the signed copy of the Declaration of Independence and its previous versions.

21. Data is atomic.
Another idea intrinsic to Xanadu but not revealed until later was that it was to hold all data in unalterable “atoms.” What appeared to be documents on screen, including your own writing, were merely atoms strung together. Ted described a complex system of altering data later and changing its address, which never took off.

22. Streams can be shared, disassembled, and re-used.
Although Ted’s papers refer to the idea of a set of Xanadu documents as a “stream,” he doesn’t develop the idea of how streams would be shared or utilized by others. These must be built-in to the system and offer abilities beyond today’s document or app sharing.

23. Your data is kept private.
End-to-end encryption is necessary today for many reasons. Only you should have access to the materials you read and write inside Xanadu or its successor.

Sorting It Out

If we rearrange the rules into the four buckets: Exists, Unnecessary, Missing, and Proposed, we get something like this:

A quick glance at this diagram shows that much of what’s needed to create Xanadu already exists or can be cobbled together from existing pieces by using API’s. An interesting side-effect of diagramming things this way is that it makes clear what was proposed but not delivered. The first two (#6 and #7) are the things that Ted talks about today in his YouTube videos.

Delivering on the Promise

If half of the problem has already been solved (or doesn’t need to be), we are left asking the question, “How difficult are the remaining pieces?” Judging by what’s left of the original 17 rules, I would say, “Achievable.” The last rule, open source, we can dispatch by simply starting from that point. Mimix or any worthy successor to Xanadu must be open source.

The Holy Grail of Xanadu was transclusion, the idea of including part of one document inside another and being able to refer back-and-forth between them. We’ve actually had transclusion for some time. Microsoft delivered it in their OLE object linking and embedding technology and HP went so far as to ship an entire version of Windows, called HP NewWave, based around transclusion of atomic objects. Today you can create a logo in Adobe Illustrator, transclude it in a Photoshop document, then double-click it from Photoshop to edit the original in Illustrator. But what Ted is referring to is transclusion while reading others’ works and writing your own. This, we do not have.

It turns out, that kind of transclusion is actually easy. It requires two things which were missing from Nelson’s proposal. The first is massive storage. It simply wasn’t possible to keep thousands of books or papers on the personal computers of the day, or even on your personal account on a university or corporate computer. Instead, Ted thought we should link to copies of the same data. That made sense in the days when schools and institutions had all the computing power and storage, but that’s not the case today. You can buy an external terabyte drive on Amazon for $50. Cloud storage is cheap, too.

Nelson liked to use the example of the Declaration of Independence, and so do I. While recognizing that documents would need to be copied and virtualized, he held to the idea that links would be to the original document or its alias. He also suggested that all links would be bidirectional, meaning that any elementary school child who quoted the Declaration of Independence could expect to have his or her paper linked back from the original document. That’s simply preposterous. Imagine the huge numbers of scientific papers which all refer to facts in the periodic table. Should visiting that table produce links to all of them? Balderdash!

If we think of transcluding a document from our own local storage rather than from some remote, shared location, we can greatly simplify the problem and produce a system that actually works for the user. Instead of linking to a remote copy of the Declaration controlled by someone else (or a copy of a copy), why not just keep a copy in your own local storage? If it comes from a trustworthy source, you can rely on it going forward. Taking the Declaration from the Library of Congress is a 100% reliable source. In Mimix, we call this canonical metadata. It’s not necessary for you to refer back to the original Declaration, just pull up your own (canonical) copy which is known to be correct. As I’m writing now, I’m referring to my own saved versions of all these documents I’ve mentioned.

With multiple terabytes of personal storage, even the most prolific author or researcher could have his or her own copy of every document he or she ever examined, along with all the user’s annotations. The periodic table (or a book about the history of France) isn’t going to change after you download it, and if it does, that’s easy for software to find and fix. You would still have your old copy, of course.

In working on my nuclear book, I’ve amassed a collection of over 250,000 pages of PDFs along with my highlighting and notes inside each. There’s no software I’ve found that’s really up to the job, but the terrific and free Docear (rhymes with “dog ear” as one would do to a book) is a good place to start. Its interface is tedious and not well integrated with a writing environment. It also doesn’t make any connections for me between my data and my writing, nor offer any alternative views. But it does let me keep my notes together with the original sources they came from and let me find either of these when I need to.

Keeping a copy of every document you read in your own personal storage and accessing them with the same tool that you use for your own writing opens the gateway to a true Xanadu-style system. Now, if we draw links between our writing and the documents we read, it’s easy for the system to retrieve the referenced files and show them on the screen at the same time — even with the graphical links that Nelson preferred. After all, we have both documents on our own disk! In this environment, it makes to sense to show links from every document as well as to it. In other words, every paper that you read or write about the Declaration of Independence should be linked from your view of that original document, but not from every else’s in the entire world.

The storage issues notwithstanding, I think Nelson skipped over the idea of local copies because he was trying to build a business around publishing and copyright management. The idea of every user having his or own copy of a document would be anathema to such a system. All three unnecessary features of Xanadu hinged on the publishing business model. In a last ditch effort to commercialize Xanadu after Autodesk pulled out, the owners approached Kinko’s (now known as FedEx Office) with a copyright and document management system. Even for them, it was unclear who would get paid and how. The entire idea of your writing and research tool being bound-up with a publisher and his royalties is simply awful. This, at the end of the day, is probably why Xanadu failed.

Building the Missing Pieces

If the missing pieces I’ve described are truly missing in that no deployed system exists which offers them, then we have something to aim for. If we can build a program that incorporates all of the existing aspects, skips the unnecessary ones, delivers on the unfulfilled promises, and fills in the holes in the original idea — then we can have a salable product! LOL.

We’ve already seen a brute-force way to include the existing technologies. A variety of frameworks, application programming interfaces, and services can cover everything in green on our chart. It would be preferable to build a new system which incorporates open source code for these features rather than relying on outside providers to continue to offer them. The biggest unfulfilled promise of Xanadu was transclusion and we’ve already seen how that can be done by simply reading and writing with a tool that keeps copies of what you’re doing. All that’s left are the orange items on our chart.

Multiple Views

We work with multiple views of data every day, but the process is messy and mostly manual. We often need to move data from something that looks like a list to something that looks like an email, a chart, or a slideshow. Today’s tools offer a million ways to present data but it’s still surprisingly difficult to move the same information around into different views.

Douglas Engelbart’s revolutionary NLS system had this idea built-in from the start, in 1968. Tragically, NLS was almost impossible to use. If anything, it offered the antithesis of a user interface and died commercially after a series of acquisitions and mergers. Along the way, we lost the idea of a consolidated system for multiple views.

Mimix proposes to re-frame the current document paradigm into a system of data called atoms and ancillary data about your data called metadata. If you are writing about Gaudí, the factual sources you consult would be metadata. So would any formatting you applied. Together, these are Mimix views.

Format-Free Text

I’ve mentioned formatting as an example of metadata because it needs to be separated from the writing itself. Nelson has complained that today’s WordStar and SGML-inspired systems are messing up the real data we’re after, and he’s absolutely right. Mimix does not allow the co-mingling of data and formatting and these aspects are formalized into the design. Formatting is just another kind of Mimix metadata which can be applied or removed, both of which are recorded in the stream.

Canonical Metadata

Nelson put a great emphasis on the idea of referencing the “one true version” of a document (or its copy or alias), but he didn’t address how that truthfulness would be determined. This is a real problem which hasn’t been addressed in any software that I know of. Taking a simple example, a college student might write a paper for music class without being exposed to such divergent concepts as diatonic set theory or serial music. “Google it,” we might reply today. But this is an artifact of the library card catalog, itself an artifact of the printing press. Instead of telling children “Let’s go to the library,” we tell them to go to Google. This is something of an improvement but we’re not home yet.

Much of the data that children would find in a library could be called canonical in computer terms, or simply “correct.” Everyone who inquires as to the distance to the Sun should receive the same answer. In this world of fake news and unreliable sources, it is more important than ever to declare and classify data according to its authenticity and sources. A more complex example is, “Who shot JFK?” What sources would you choose as canon? These are decisions best left to each individual author, but his or her choices should be clear not only to the writer but also to readers. In a Mimix stream, the author’s canonical sources are available to inspect and alter.

Atomicity

Truthfulness relies on an unchanging record of the past. Although facts and opinions change, our interactions with them in the past were based on our knowledge then. Our actions in the future must be based on the future values. The values that some data had yesterday were the basis of all of yesterday’s decisions and should not be erased. When changes are needed, we can simply record a new atom of data with the new values.

Nelson referred to this idea in his paper on the Xanalogical Structure of Documents, but the proposed method of accessing and “updating” these atoms was unnecessarily complex. Using the mass storage now available to us, we can simply snapshot every piece of data as it comes in and also snapshot every change to it. By replaying our snapshots, we could see where the data started and how it had changed.

Whenever there are long streams of documents and you need to only access a small part, some kind of addressing system is necessary. Two programmers working for Nelson came up with a system called tumblers based on the transfinite numbers they had studied in college. The method relied on literal rows, columns, and character counts to identify chunks of text, making it fragile and unreliable after text was changed.

Since the 1950’s, we’ve had regular expressions. I’ll be the first to admit that I hate them, but they do work. Rather than trying to specify the literal character positions of the data we want, regular expressions let us find the text we need syntactically. To grab the Preamble from the Constitution, I only need to write:

/we the people(.*)america./gi

Once I’ve referred to that piece of the original text, I could format it any way I like, display it on a slide, or quote it in the Schoolhouse Rock video that still rings in my mind. Selecting just the first letter and applying special formatting like a drop-cap is easy with regular expressions, too.

The Constitution isn’t changing anytime soon, but how does the concept of atomicity apply to your own writing which might change frequently? Let’s take two very different use cases. A screenwriter needs to create characters and then draw from many resources to develop a narrative around that character. There are sure to be numerous references, notes, illustrations, etc. and these will grow and evolve over time. But keeping them atomic lets the author go back and see the original data from which the notes were derived. If some material had been deleted, it would be impossible to refer to it going forward but easy to access it going backward.

A doctor needs to keep copious notes on his or her patients as well as a variety of canonical documents such as lab tests or radiology studies. Every prescription written must be maintained. There is some list of what drugs the patient is on right now versus all the drugs he or she has ever been prescribed. Today, this is mostly in paper files. In Mimix, the patient is an atom and all of these ancillary data are applied to that atom over time. The metadata itself is atoms, too. In other words, everything known about Vioxx at the time it was prescribed to patients is quite different from what is known now.

Streamsharing

Sharing today is second nature and we have all kind of systems for sharing screens, apps, and they content they produce. Unfortunately, none of these systems makes it easy to do anything useful with the data after it’s shared. These are siloed applications in computer terms with data only moving up and down in the same system but not outside it. A corporate webinar or conference keynote makes a great example. Some useful content might be displayed by the presenters but it’s trapped in the video recording and can’t be extracted. At best, the audience gets a list of links they can click later. Most of the valuable information in the content stream is lost unless the viewer keeps separate notes.

Mimix introduces the idea of streamsharing, giving others a functional copy of your own research from which they can extract data, metadata, and annotations or add their own. More properly, Mimix recreates this idea from the original work of Vannevar Bush who in 1945 proposed a stream recording system called Memex based on microfilm. A Memex user who received a reel of film would be able to extract any frames, rearrange them, print them, delete some, and compile the whole thing onto a new spool. The medium carrying the information was the information itself, giving the recipient the same powers as the author.

My Mimix stream recording idea differs in several ways from app or document sharing. First of all, once shared, the stream belongs to the recipient who retains all its contents. Secondly, the stream is live and can not only be played linearly like a video but also accessed for its atoms and metadata. The new user can extract or discard any part of the stream he wishes, a concept not possible with today’s software. Because Mimix data is not bound up in documents or presentation formats, the system can show the stream recipient a summary of everything inside it and allow that person to choose what he or she wants to extract.

In addition to selectively importing the stream, a recipient is free to rewind and make changes to a “past” view of the stream’s contents. The audience member or potential new inventor could discard some part of the stream’s metadata halfway through and substitute new data of his own, changing the views thereafter. Scientific researchers could use this feature to look through old papers with the lens of new information, for example.

A Mimix stream naturally incorporates all the documents referenced in its metadata, so a Shakespeare commentary stream would include the full text of all the plays mentioned as well as any other reference sources the author consulted. The massive storage of today’s systems makes it possible and desirable for each reader and writer to have a full digital library which he or she can annotate and reference going forward.

Encryption

I leave for last the topic that came first in the history of computing and that is encryption. Your favorite news source is full of stories about why we need encryption, from corporate data breaches to government hacks and rampant surveillance. The earliest computers didn’t worry much about encryption for several reasons. Many were released before strong encryption techniques were popularized. Home computers didn’t seem to need encryption (back then) and they didn’t have the processors or storage to provide it anyway. Corporate and university computers were managed by the institutions that owned them and users didn’t worry about how to protect the data.

Things have changed drastically since then. The prevailing idea around security is, “Be afraid. Be very afraid.” Any reasonable successor to Xanadu (or any non-trivial software, really) should provide end-to-end encryption as a built-in and non-optional feature. In a system like this, the Mimix software would only store “garbage data,” of no use to anyone without your private key to decrypt it. When you share your streams with others, a different set of garbage data is sent to them which only they can decrypt. When your data is backed up in the cloud or elsewhere, only the garbage data is recorded, requiring your key to decrypt it later.

As mentioned earlier, this kind of public/private key encryption system comes with a risk. If you can’t decrypt your data, neither can anyone else. This reinforces the need for key backup systems and custodial oversight, such as making sure a trusted friend or associate has your key.

Can We Get to Xanadu?

I think we can. If we incorporate the technologies that already exist and build the missing pieces, I see no technical barriers to its completion. As for the market, it’s huge. Like word processors and spreadsheets which were once expensive and limited to big business owners, a tool like Mimix which is a hit with early adopters (education, scientific, medical, creative) can evolve into a version for everyone.

Are you excited about the Mimix ideas? I am, and I’d love to hear your thoughts. Check out the whitepaper and write me anytime at david@mimix.io. And thanks for reading!

— D
Key West, Florida
August 26, 2018