Thoughts on AI catastrophe
Earlier I quoted Kurzweil's estimate of about thirty years to human level A.I. Suppose he is correct. Further suppose that Moore's law continues to hold, that computers continue to get twice as powerful every year or two. In forty years, that makes them something like a hundred times as smart as we are. We are now chimpanzees, perhaps gerbils, and had better hope that our new masters like pets. (Future Imperfect Chapter XIX: Dangerous Company)
As that quote from a book published in 2008 demonstrates, I have been concerned with the possible downside of artificial intelligence for quite a while. The creation of large language models producing writing and art that appears to be the work of a human level intelligence got many other people interested. The issue of possible AI catastrophes has now progressed from something that science fiction writers, futurologists, and a few other oddballs worried about to a putative existential threat.
Large Language models work by mining a large database of what humans have written, deducing what they should say by what people have said. The result looks as if a human wrote it but fits the takeoff model, in which an AI a little smarter than a human uses its intelligence to make one a little smarter still, repeated to superhuman, poorly. However powerful the hardware that an LLM is running on it has no superhuman conversation to mine, so better hardware should make it faster but not smarter. And although it can mine a massive body of data on what humans say it in order to figure out what it should say, it has no comparable body of data for what humans do when they want to take over the world.
If that is right, the danger of superintelligent AI’s is a plausible conjecture for the indefinite future but not, as some now believe, a near certainty in the lifetime of most now alive.
Orthodoxies
There are two reasons to believe in something — arguments you can work through for yourself or the fact that other people believe in it. I discount the second, especially the form that is not “A,B,C whom I know and respect believe in it” but “everyone important believes in it.” In my lifetime I have observed several impending catastrophes that, I was told, everyone important knew were coming. Population and climate were the two big ones, the latter not yet resolved,1 but Year 2K and Peak Oil qualify as well. The requirements for nuclear winter were, fortunately, never met, but current evidence is that if they had been it wouldn’t have happened.2
I do not entirely understand the mechanism that produces a mistaken orthodoxy but those cases makes me assign little weight to the public claim that everyone important believes something so I should believe it too — especially when what they believe is a dramatic story, provides an argument for policies that many people want arguments for, or both.
I am left where I was in 2008. It is plausible, although not certain, that the human mind is an organic computer. If so, and if scientific progress continues, we should eventually learn enough to emulate it in hardware. If so, as the hardware gets faster so will the emulated mind. As the hardware gets more powerful, the mind should get smarter, although how smart how fast we have no good way of knowing.
Solutions?
If AI is a serious, indeed existential, risk, what can be done about it?
I see three approaches:
I. Keep superhuman level AI from being developed.
That might be possible if we had a world government committed to the project but (fortunately) we don’t. Progress in AI does not require enormous resources so there are many actors, firms and governments, that can attempt it. A test of an atomic weapon is hard to hide but a test of an improved AI isn’t. Better AI is likely to be very useful. A smarter AI in private hands might predict stock market movements a little better than a very skilled human, making a lot of money. A smarter AI in military hands could be used to control a tank or a drone, be a soldier that, once trained, could be duplicated many times. That gives many actors a reason to attempt to produce it.
If the issue was building or not building a superhuman AI perhaps everyone who could do it could be persuaded that the project is too dangerous, although experience with the similar issue of Gain of Function research is not encouraging. But at each step the issue is likely to present itself as building or not building an AI a little smarter than the last one, the one you already have. Intelligence, of a computer program or a human, is a continuous variable; there is no obvious line to avoid crossing.
When considering the down side of technologies–Murder Incorporated in a world of strong privacy or some future James Bond villain using nanotechnology to convert the entire world to gray goo–your reaction may be "Stop the train, I want to get off. " In most cases, that is not an option. This particular train is not equipped with brakes. (Future Imperfect, Chapter II)
II. Tame it, make sure that the superhuman AI is on our side.
Some humans, indeed most humans, have moral beliefs that affect their actions, are reluctant to kill or steal from a member of their ingroup. It is not absurd to belief that we could design a human level artificial intelligence with moral constraints and that it could then design a superhuman AI with similar constraints. Human moral beliefs apply to small children, for some even to some animals, so it is not absurd to believe that a superhuman could view humans as part of its ingroup and be reluctant to achieve its objectives in ways that injured them.
Even if we can produce a moral AI there remains the problem of making sure that all AI’s are moral, that there are no psychopaths among them, not even ones who care about their peers but not us, the attitude of most humans to most animals. The best we can do may be to have the friendly AI’s defending us make harming us too costly to the unfriendly ones to be worth doing.
III. Keep up with AI by making humans smarter too.
The solution proposed by Raymond Kurzweil is for us to become computers too, at least in part. The technological developments leading to advanced A.I. are likely to be associated with much greater understanding of how our own brains work. That might make it possible to construct much better brain to machine interfaces, move a substantial part of our thinking to silicon. Consider 89352 times 40327 and the answer is obviously 3603298104. Multiplying five figure numbers is not all that useful a skill but if we understand enough about thinking to build computers that think as well as we do, whether by design, evolution, or reverse engineering ourselves, we should understand enough to offload more useful parts of our onboard information processing to external hardware.
Now we can take advantage of Moore's law too.
A modest version is already happening. I do not have to remember my appointments — my phone can do it for me. I do not have to keep mental track of what I eat, there is an aap which will be happy to tell me how many calories I have consumed, how much fat, protein and carbohydrates, and how it compares with what it thinks I should be doing. If I want to keep track of how many steps I have taken this hour3 my smart watch will do it for me.
The next step is a direct mind to machine connection, currently being pioneered by Elon Musk’s Neuralink. The extreme version merges into uploading. Over time, more and more of your thinking is done in silicon, less and less in carbon. Eventually your brain, perhaps your body as well, come to play a minor role in your life, vestigial organs kept around mainly out of sentiment.
As our AI becomes superhuman, so do we.
Other Futures
Short of becoming partly or entirely computers ourselves or ending up as (optimistically) the pets of computer superminds,4 I see three other possibilities. One is that the continual growth of computing power that we have observed in recent decades runs into some natural limit and slows or stops. The result might be a world where we never get human level A.I., although we might still have much better computers and computer programs than we now have. Less plausibly, the process might slow down just at the right time, leaving us with peers but not masters and a very interesting future. The one argument I can see for that outcome is that that is how smart we are; perhaps there are fundamental limits to thinking ability that our species ran into a few hundred thousand years back.
A second possibility is we are not software after all. The analogy is persuasive but until we have either figured out in some detail how we work or succeeded in producing programmed computers a lot more like us than any so far, it remains a conjecture. Perhaps my consciousness really is an immaterial soul, or at least something more accurately described as an immaterial soul than as a program running on an organic computer. It is not how I would bet but it could still be true.
Finally, there is the possibility that consciousness, self-awareness, will, depends on more than mere processing power, that it is an additional feature which must be designed into a program, perhaps with great difficulty. If so, the main line of development in artificial intelligence might produce machines with intelligence but no initiative, natural slaves answering only the questions we put to them, doing the tasks we set, without will or objectives of their own. If someone else, following out a different line, produces a program that is a real person, smarter than we are, with its own goals, we can try to use our robot slaves to deal with the problem for us. Again it does not strike me as likely; the advantages of a machine that can ask questions for itself, formulate goals, make decisions, think, seem too great. But I might be wrong. It might turn out that self-awareness is a much harder problem than intelligence.
The Mini-Catastrophe
So far I have been considering the existential risk, the possibility that the development of superhuman AI will have catastrophic effects for humans. A less exotic and more likely scenario, at least in the short term, is that AI will continue to get better at doing things that some humans now do, obsoleting the skills with which those humans now make their living.
This is not a new problem. In 1880, forty-nine percent of employed persons in the US were farmers. Currently, the number is just over one percent. It does not follow that forty-eight percent of the population ended up unemployed. Similar changes have occurred, due to technological progress and economic growth, for many other jobs.
There are two respects in which, some argue, this time is different. One is the speed of the change. If, as some argue, current Large Language Models are already at human level intelligence, they could take over a wide variety of jobs in a few years, giving the society very little time to adjust to the change.
The other is the breadth of the change. Machines are already our superiors in physical strength; arguably almost the only reason things are still done by humans is human intelligence, used in part to guide machines. If AI reaches the point where practically everything humans do can be done as well and much less expensively by a machine, what is left for humans to do?
In the limiting case, where machines can do almost everything humans can do, the result is to make the tradeoff between labor and capital as inputs to production much more favorable to capital. We could end up with a society in which those who had capital were better off, those with only labor to sell worse off. The net effect, as with other cases of technological change, would be an improvement in a conventional economic sense combined with a substantial shift in the distribution of income.
David Ricardo, in my view the most intellectually impressive of the early economists, considered this issue two hundred years ago, correcting what he had concluded was an error in his previous analysis:
Ever since I first turned my attention to questions of political economy, I have been of opinion, that such an application of machinery to any branch of production, as should have the effect of saving labour, was a general good, accompanied only with that portion of inconvenience which in most cases attends the removal of capital and labour from one employment to another.
…
These were my opinions, and they continue unaltered, as far as regards the landlord and the capitalist; but I am convinced, that the substitution of machinery for human labour, is often very injurious to the interests of the class of labourers. (Principles of Political Economy and Taxation Chapter XXXI )
For my reasons to think climate change is unlikely to be catastrophic see my past posts on the topic and this one from my old blog.
The modelers who predicted nuclear winter predicted a similar effect from the 1991 Kuwait oil fires. It didn’t happen.
I don’t want to but I haven’t figured out how to turn the damn thing off.
Ian Banks’ Culture novels provide a science fictional account of a society with people rather like us who are, in effect, the pets of vastly superior artificial intelligences.
A list of past posts sorted by topic
A search bar for my posts and other writings
I think it’s worth asking something like, “at what energy cost the machines will compete with us,” and “in what domains?”
There is no machine that competes with animals, in general. Sure, airplanes can fly faster than birds. But they cannot feed themselves or repair themselves or make copies of themselves without human intervention.
The chaotic nature of the physical system we inhabit means that the capacity even if the best machines to predict, eg, the weather, is probably going to be more constrained by the ability to precisely measure the world, than it will be computational bandwidth.
As such I think the likely outcome here is that any self aware AGI with its own drives would see us, not as pets, but as the way an intelligent self-aware human being views trees - as a necessary party of our external biology and, as such, something worth protecting and maintaining.
Human beings are general purpose computers made of little more than dirt, water, and sunshine. We make copies of ourselves, repair ourselves, and aren’t subject to the same failure modes that a digital machine would be. We consume very little energy, compared to a rack of GPU’s. So it seems most likely to me that a machine intelligence would want to keep us alive to reduce the risks of its down death. It would try its best to keep us happy, to ensure that we would even turn it back on in the event that it fails.
To believe otherwise requires me believing the machine would decide to replace these extremely cheap dirt robots, with machines made of much more expensive materials, at a cost of giving up all chance of being brought back from the dead via an unpredicted environmental shock that hurts it but leaves biomatter fine.
I have high confidence that the human brain is not a "computer" if that word means a programmable digital computer. There are at least two reasons for this:
* A digital computer has a (now extremely large) number of internal switches that can be turned on or off, and that will stay on or off (barring random physical errors) until changed by an external signal. The human brain doesn't work that way at all. It has a (large number of) neurons that sit waiting until externally stimulated, and then fire a transient pulse that can cause other neurons to fire, goes into a refractory interval when it not only is "off" but cannot be made to fire, and then waits to fire another impulse when stimulated again; the information seems to be represented partly by the frequency with which the neuron fires. Perhaps this setup can also be modeled as a kind of Turing machine, as a digital computer can be, but as far as I know this hasn't been proven, and in any case, that may be like saying that because the same equations can represent either a weight on the end of a spring, or a capacitor and inductor, a tuned circuit IS a weight on the end of a spring: the confusion of the "is" of analogy with the "is" of identify. ("O my love's like a red red rose," but the bees won't make honey from her secretions.)
A typical neuron goes from an initial stimulus to the restoration of the state where it's read to fire in approximately 4 ms. It can do this 250 times in a second. Bernard Shaw observed that we can recite seven syllables in counting seconds: Hackertybackertyone, hackertybackerty two, and so on. That's time for 36 steps of a brain "program." Even with massive parallelism, I don't think you can write meaningful code for any complex action in 36 steps. The brain's "software" can't be anything like a computer program, just from the simple physics of the matter.