Vitanuova for 2002 November 16 (entry 7)
< Song parody joke
Sick >

I've been asked to collect in one place a set of pointers to everything substantial I've written in public about trusted computing. So I've tried to do that below; I hope I'm not missing anything. In most cases, you'll have to skip over a substantial amount of unrelated material in each diary entry.

I have also written one piece about trusted computing which is probably not well-known to most readers here; this is a message to Nick Moffitt's "crackmonkey" mailing list, responding to an anti-trusted computing piece in Linux and Main. (Topics: running a non-Microsoft operating system on TCPA or Pd hardware; what is an "end-to-end security" measure; coupling policy enforcement to access to decryption keys; why this could put users at a disadvantage even though it gives them new functionality; DRM support in hard drives; document revocation; claims of scary features in Windows Media Player; network effects, interoperability, reverse engineering; CBDTPA; Palladium does not give Hollywood what it wants; incentives for users and companies.)

I'm soon going to start writing about technology on an EFF-sponsored web log, as I've already mentioned. That will probably mean that I'll write very little about technology here in the future.

By the way, I think my original post about epistemology was the most interesting of everything I've written about trusted computing. I know at least one other person who agrees with me.

That particular discussion also made its way into a science fiction story:

"It is like Descartes," Murray said, accelerating up a new hill.

"Yeah?" Liam said. "Who's God, then?"

"Crypto," Murray said.

More seriously, though, I think computers have given us a new kind of understanding of the problems of epistemology, much as they've given us new ways of thinking about psychology. This is not to say that the metaphors we take from computing are more accurate or ultimately more faithful to reality. But they're undeniably present.

When we think about thinking and about experience, we draw on what we know about computers and what we've encountered as programmers and as computer users. Turing, who had more knowledge of computing than any of his contemporaries, made this parallel explicit and started decades of academic debate about just how literally it ought to be taken. In the past, metaphors for simulation were always dreams (Chuang-tzu, Descartes), the theater and the arts (Plato), or games. But now we have actual simulations (physics experiments run entirely in software!) and emulation (hardware environments, tantamount to sensory experience, conjured up by emulator code). Just as nobody who has had a vivid dream can be quite sure of reality, nobody who has run a program under an emulator can have quite the same certainty about reality.

Only in our time could someone like Ed Fredkin seriously suggest that our universe was a computer simulation. The idea that it is a simulation is old; people used to say that the world was the dream of God, or, in Carroll, of the Red King:

"It's only the Red King snoring," said Tweedledee.

"Come and look at him!" the brothers cried, and they each took one of Alice's hands, and led her up to where the King was sleeping.

"Isn't he a lovely sight?" said Tweedledum. Alice couldn't say honestly that he was. He had a tall red night-cap on, with a tassel, and he was lying crumpled up into a sort of untidy heap, and snoring loud -- "fit to snore his head off!" as Tweedledum remarked.

"I'm afraid he'll catch cold with lying on the damp grass," said Alice, who was a very thoughtful little girl.

"He's dreaming now," said Tweedledee: "and what do you think he's dreaming about?"

Alice said, "Nobody can guess that."

"Why, about you!" Tweedledee exclaimed, clapping his hands triumphantly. "And if he left off dreaming about you, where do you suppose you'd be?"

"Where I am now, of course," said Alice.

"Not you!" Tweedledee retorted contemptuously. "You'd be nowhere. Why, you're only a sort of thing in his dream!"

"If that there King was to wake," added Tweedledum, "you'd go out -- bang! -- just llke a candle!"

"I shouldn't!" Alice exclaimed indignantly. "Besides, if I'm only a sort of thing in his dream, what are you, I should like to know?"

"Ditto," said Tweedledum.

"Ditto, ditto!" cried Tweedledee.

He shouted this so loud that Alice couldn't help saying, "Hush! You'll be waking him, I'm afraid, if you make so much noise."

"Well, it's no use your talking about waking him," said Tweedledum, "when you're only one of the things in his dream. You know very well you're not real."

"I am real!" said Alice, and began to cry. "You won't make yourself a bit realer by crying," Tweedledee remarked: "there's nothing to cry about."

"If I wasn't real," Alice said -- half-laughing through her tears, it all seemed so ridiculous -- "I shouldn't be able to cry."

"I hope you don't suppose those are real tears?" Tweedledum interrupted in a tone of great contempt.

Today, Alice would see a computer running an enormous simulation in place of the snoring King.

It's common to hear that software is different from mental processes because we can see exactly how software works, but can't do the same for mental processes. This may be true, but software can't see inside of itself any more than a person can really see the operation of his or her own mind.

Some programmers don't believe that. "I can read my program's code or disk, or even in memory while my program is running! That's how viruses work; they have to be able to read their own code in order to copy themselves!"

But anybody who's played A. K. Dewdney's Core War knows about the unreliability of programmatic introspection. Core War programs are fighting other programs in RAM, trying to corrupt and destroy and crash the enemy code, while protecting their own code. In the multithreaded version of Core War, it was common for programmers to implement a copying routine which tried to relocate code in memory, an integrity-checking routine which looked for errors introduced by writes from hostile code, or perhaps both (a routine to make multiple copies of a program and then constantly compare them against one another to try to detect the presence of errors as soon as they were introduced).

The trouble is that all such self-monitoring code runs in the same memory space as the rest of a Core Wars program. As such, it is itself vulnerable to attack. A checksumming routine could be disabled or altered in mid-checksum. A copying routine could be altered and induced to write a corrupted copy. An entire subroutine could be overwritten with a block of code performing an entirely different function. Any individual test can be reversed, turning "if x" into "if not x".

As I've discussed elsewhere, programs which examine themselves for reasons such as enforcing restrictions against a user are also vulnerable to modification. Perhaps a program checksums itself before running (as many programs do); the result of the checksum has to be examined somewhere, and that examination has to be performed by code, and that code is modifiable. Virus scanners also want to protect themselves against alteration, so they frequently run all kinds of self-tests. A clever virus which was specialized to attack a particular scanner would find the proper entry point to bypass the self-tests, but also initialize any relevant memory locations to indicate that the self-tests had completed successfully. If self-tests occur later on, during the virus scanner's actual execution, the virus would have to determine where these tests occur and overwrite their code, too. But there are at most a finite number of tests which can occur, and an attacker can ultimately locate and alter all of them.

We can actually get much sneakier than this. A program generally assumes that it can obtain and dereference pointers to (1) various system objects or data structures, and (2) its own code. This assumption is generally valid on most systems, subject to some kind of security policy. But there is no technical reason that the result of dereferencing a pointer has to be correct. There is no reason that the machine (if designed by or under the control of an attacker) need ever perform according to its specifications. It's entirely possible to create a virtual machine in which reads from memory occur successfully and describe a consistent but completely inaccurate memory state. Techniques like this are known to the authors of rootkits, which are now able to alter operating system kernels. You can imagine a rootkit which alters an operating system kernel so that software on the system no longer has access to read the memory where the rootkit resides, but believes it does; accesses to that memory could result in plausible but completely misleading results.

You can proliferate examples of what attacks like this might be capable of. The really interesting one for me is a program which is deceived about its own state in memory. It might go through and analyze itself and conclude "OK, my code has just done this" or "my code is about to do that" or "I can guarantee that my code will never execute the following instruction" or, as above, "my code has a valid checksum or signature, showing that it's unmodified from its original version". And the program can be justified in concluding any of these things, but it can still be totally wrong; it can't look at itself directly, but always and only through the mediation of the underlying system, which might be a completely deceptive virtual machine.

Of course, this kind of deception can occur for reasons other than an attack by a human or a rival Core Wars automaton. Imagine that you've written an implementation of an error-correcting code which looks at blocks of memory and finds and corrects errors resulting from hardware faults or cosmic radiation. Now, your error-correcting code also has to run in memory, and it can use the same error-correcting system and can try to correct errors within the ECC routine itself. But since the error-correcting code is running in memory, if a hardware fault occurs, the routine may give an inaccurate answer, and falsely claim that there is no error at all.

Our own thought processes are famously opaque to us, as are our emotions; even when we practice introspection, it seems that we don't really have reliable knowledge of how we think. (You can make an empirical claim about this, as George Lakoff seems to do regularly, and say that people's claims about their own thought processes are inconsistent with their behavior or with other things they say about themselves. You can also make a more abstract claim and delve into thought experiments in which consciousness is produced by a mechanism to which we have no access but includes the illusion of access to part of that mechanism. Feynman once had a dream that he had figured out some surprise aspect of how his mind worked, but when he woke up, his insight turned out to be totally baseless.)

In that sense, we are in the position of software: we can try to look at ourselves, but we can't have independent confirmation that what we see is really there. There is always a philosophical possibility that we are implemented somewhere or in some way we don't know about, that we have a particular feature we can't anticipate, and in general that any part of our experience, not just our senses, could be unreliable. A computer program implemented on reliable hardware which performs according to spec is guaranteed never to add 2 and 2 and get 5 (or to divide 4195835 by 3145727 and get 1.3337), but the software itself can't tell whether the results of instructions it requests are correct, having no truly independent way to verify them. "To them, reality would be the results of the execution of their opcodes." If a hardware flaw does occur, a program's results will be wrong, and the program will not necessarily have any way to tell. The same is true of humans; as sane people, we can see why 2 plus 2 equals 4, and see why it's impossible for 2 plus 2 to equal 5; but it's by no means impossible for a person to have the experience of believing that 2 plus 2 equals 5. There is no guarantee that we will remain sane and there is no ultimately certain way that we can test our own sanity at any chosen moment.

One strange result of insanity is that it is only sometimes apparent to the person afflicted with it. Many people with serious mental illness are aware of it, and understand its significance, but even this is not guaranteed to make the experience seem any less real. (Understanding that a phobia is unreasonable, for example, may not lessen its power at all.) Some people who become insane are adamant that they are sane and that everyone else is insane, and many people have been shaken by the plausibility of such claims. Your brain could change in such a way that almost any given aspect of your experience was altered; in fact, it may already have changed that way, and you may not know it, and you might never know it.

My observation is that computers have made such possibilities more present and more plausible to us, and given us a new way to think about them. They have made us speak of what we are "programmed" to do or to think, how little we know about our own "programming", and how little we can hope to control it.

The classic question in the face of this has been "Are you insane at this moment? How do you know?". This question seems to infect any epistemology which aims at absolute certainty; even if we assume that almost every conclusion of Descartes followed necessarily from what came before it, the possibility would still remain that he was insane and had simply imagined some particular detail, or systematically added or dropped a "not" at some point in the deductive process. (Imagine if you were selectively blind to a particular color; you wouldn't have any direct experience to indicate that you were missing anything. Only other people's disagreement with you would suggest a problem.) The existence of insanity is corrosive. People have struggled for a long time to try to redeem Cartesian arguments. I think they have never done so. If someone asked me how I know whether I'm insane, I would say that there's a lot of evidence for my sanity but that it's always possible that I'm totally delusional. I can't look inside my mind to see whether it's working properly; even if I could, I would still have to use my mind to make that evaluation, so that insanity could always alter my judgment to make itself undetectable -- just like a rootkit patching a kernel to conceal itself.

I should talk more about the possibilities of breaking the abstraction barrier which makes the "implementation" of our subjective experience opaque. It seems that we often think of this as disturbing or improper; we seem to feel that we ought to deal with our own consciousness only through its high-level interfaces (such as language). If you are actually analogous to software running on an interpreter, you have many different abstraction layers, and (like a complex software system) you can be affected on almost any of these, and (also like a complex software system) one layer may not have any real sense of what is happening at another layer. I think there is a real moral sense to this: when B. F. Skinner argued that we could work more directly on lower levels by taking advantage of insights from biology -- in particular, by practicing conditioning on people -- people were horrified, and, as he observed, complained that it was contrary to human dignity to do so. Maybe one meaning of human dignity is interacting with someone's mind on a very high level (say, at the level of dialogue and debate); violating human dignity would them mean interacting with someone at a lower level ("close to the machine", in some sense).

[Main]
Support Bloggers' Rights!
Support Bloggers' Rights!


Contact: Seth David Schoen