different from this cellular-automaton system. The difference is a detailed difference

different from this cellular-automaton system. The difference is a detailed difference. This brainlike neural network was produced by the long history of civilization, whereas the cellular automaton was created by my computer in the last microsecond. The problem of abstract AI is similar to the problem of recognizing extraterrestrial intelligence: How do you determine whether or not it has a purpose? This is a question I don’t consider answered. We’ll say things like, “Well, AI will be intelligent when it can do blah-blah-blah. ” When it can find primes. When it can produce this and that and the other. But there are many other ways to get to those results. Again, there is no bright line between intelligence and mere computation. It’s another part of the Copernican story: We used to think Earth was the center of the universe. Now we think we’re special because we have intelligence and nothing else does. I’m afraid the bad news is that that isn’t a distinction. Here’s one of my scenarios. Let’s say there comes a time when human consciousness 1s readily uploadable into digital form, virtualized and so on, and pretty soon we have a box of a trillion souls. There are a trillion souls in the box, all virtualized. In the box, there will be molecular computing going on—maybe derived from biology, maybe not. But the box will be doing all kinds of elaborate stuff. And there’s a rock sitting next to the box. Inside a rock, there are always all kinds of elaborate stuff going on, all kinds of subatomic particles doing all kinds of things. What’s the difference between the rock and the box of a trillion souls? The answer is that the details of what’s happening in the box were derived from the long history of human civilization, including whatever people watched on YouTube the day before. Whereas the rock has its long geological history but not the particular history of our civilization. Realizing that there isn’t a genuine distinction between intelligence and mere computation leads you to imagine that future—the endpoint of our civilization as a box of trillion souls, each of them essentially playing a video game, forever. What is the “purpose” of that? 192 HOUSE_OVERSIGHT_016995