AI and the doomsday scenario

This is updated from the Tablet article Keep Thinking: machines won’t do it for you.

In a 1983 op-ed in Omni magazine, the science fiction writer and polymath Vernor Vinge famously claimed that humans would soon create “intelligences greater than our own.” At this point, Vinge wrote,“human history will have reached “a kind of singularity, an intellectual transition as impenetrable as the knotted space-time at the center of a black hole, and the world will pass far beyond our understanding.” Vinge estimated that this would happen within 100 years. Ten years later, he predicted that the singularity would occur in 30 years, that is three years ago.

Vinge’s black hole analogy has been adopted by many other commentators and used as a warning for what they predict might soon happen as AI becomes increasingly powerful. Elon Musk has repeatedly claimed that we are now on the “event horizon of the singularity.” The metaphor is based on the understanding that once an observer crosses the event horizon of a black hole there is no going back.

Instead, the observer will eventually, but inevitably, be torn apart by infinite tidal, gravitational forces—whence “singularity.” In mathematical physics this is known as the Penrose singularity theorem, a deep and surprisingly simple, rigorous, mathematical result for which Roger Penrose received the 2020 Nobel Prize.

The genius of Vinge’s black hole analogy is that it is impossible to tell when a “superintelligence,” capable of upgrading itself and advancing technologically at an “incomprehensible rate,” has been created. In a similar way, the event horizon—literally, the boundary which separates a black hole from its space-time complement—is itself physically undetectable; that is, an observer can cross it without experiencing any special physical effects. One becomes aware of the grim reality of the black hole much too late, long after the point of no return.

Musk’s assertion that “we are now on the [technological] event horizon” is said to be supported by the recent extraordinary advances in AI. AI systems have reached-gold medal performance on international math olympiad problems, assist mathematicians to solve well known conjectures, predict protein 3D folds from their amino-acid sequence, write sonnets in the style of Shakespeare and music in the style of Mozart, provide medical or legal advice, and easily beat the best chess grand masters at their own game.

There are, however, some immediate problems with this black hole analogy. To start with, unlike the physical laws at work within a black hole, the technological singularity envisioned by Vinge, often prophesied in apocalyptic terms, is demonstrably avoidable. Indeed, all humans must do to stop an impending catastrophe would be to disconnect the internet, shut off the power sources of the AI supercomputers, or physically destroy them. For an artificial superintelligence to become unstoppable requires, at the least, the ability to take control over the sources of power by which it functions¹, in some weird form of machine insurrection.

A far more serious objection, one that I believe goes to the heart of the problem, is that the technological advances of the last 80 years in designing ever more powerful computers and software have profoundly altered the understanding of what we mean by intelligence. Computers, of course, have far better memory and computational power than humans do, and can overwhelm us in chess and go, games which used to be considered supreme tests of human intelligence.

But if that understanding of intelligence made some sense in the past, it no longer does. Is it sensible to claim that the machines are more intelligent than the humans who have built and programmed them? Though Stockfish, the top open-source chess computer program, can put to shame any chess grand master, is it smarter than those who designed its software—people who may be terrible at playing the game but who envisioned and produced the program in the first place?

The same question can be asked about any task which can be reduced to simple step-by-step algorithmic instructions. Should measures of intelligence consider other factors than speed, specifically human factors such as the ability to understand the algorithmic, step-by-step, structure of a given task and design a machine that can implement it?

Doomsayers and AI enthusiasts postulate something called artificial general intelligence (AGI), a hypothetical supercomputer that can perform any intellectual task that a human can do, only much faster. According to Ilya Sutskever, one of the founders and former chief scientist of OpenAI, AGI is “the point at which AI is so smart that if a person can do some task, then AI can do it.” Like many others, he claims that AGI is “just around the corner,” that the basic ingredients of AI, as it exists, are sufficient to start the process of runaway self-improvement predicted by Vinge. Without going into a myriad of technical details, these ingredients are: 1) a simple optimization procedure, called stochastic gradient descent, 2) linear algebra calculations with huge matrices, 3) a clever parallel processing design, based on neural networks, 4) access to all the available electronic data, the entirety of human knowledge and experience that can be found on the internet, and 5) vast computational power and the enormous amounts of energy required to feed it.

According to the proponents of AGI, once we have crossed the “event horizon” (that is now!), all future mathematical or scientific discoveries will be done directly by superintelligent AI (SAI), at exponentially accelerating pace. But how can that be? Present AI systems can improve performance relative to objectives supplied by us: predict the next token, minimize an error, search for the best fit, write faster code. But the great discontinuities in science do not arise merely from moving faster inside an existing space. They require a transformation of the space itself: a new language, a new object or concept, a new criterion of relevance. What is the loss function for discovering evolution, genetics, Maxwell’s equations, atomic theory or quantum mechanics? What is the gradient from Newtonian mechanics to general relativity ? Until AGI advocates can explain how a machine escapes the objective functions, architectures, data, and physical infrastructure given to it by humans, recursive self-improvement remains nothing more than fiction.

Throughout the history of both science and technology, the most impressive breakthroughs were done by abrupt discontinuous changes, based on foundational new ideas. To illustrate this, consider the example of planetary motion. Ptolemy and later Copernicus were able to predict the future positions of planets by an ingenious algorithm based on epicycles. More and more epicycles were needed to fit new real data based on astronomical observations. The predictions would get better for a while, before new discordant data would require the introduction of even more epicycles. Proceeding in this manner, computers would have achieved, if available at that time, amazing efficiency in predicting the position of all planets of the solar system, including eclipses, for the next few zillion years²—without, however, being able to discover Kepler’s laws or, even more astonishing, Newton’s law of gravitational attraction, or calculus and the extraordinary world of differential equations. If modern computers were available before 1915, they would similarly have been able to make impressive corrections to the anomalies of the perihelion of Mercury without getting anywhere close to the discovery of general relativity. One can make similar observations about any branch of science and technology. Major advances are made by abrupt changes, not by continuous improvements. There is no sign that, without human intervention, AI, as it exists today, can compete with humans in this regard.

AGI enthusiasts like to argue that once past the “event horizon” (again, that is now!) AI would be able to change that paradigm. It would become Silico sapiens, replacing humans not only for tasks which can be automatized, but for all cognitive tasks. Yet the prophets of Silico sapiens can provide no credible example of any event in which an AI supercomputer was able to imagine new transformative tasks, formulate a new research program, produce new mathematical concepts or conceive a fundamentally new version of its own hardware or software.

Though, undoubtedly, AI will have a great impact on society, it is not, I strongly believe, going to profoundly change what it is to be human, as many predict. Here are a few reasons for my belief:

Though humans are much slower in performing algorithmic tasks, we have access, through search engines, to the same electronic data as AI. We also have access to the unlimited physical data of the four-dimensional space we live in. One can argue that computers can also be given physical bodies and as robots they will surpass us at all tasks. Really? Our bodies and minds are infinitely complex, at both macro and micro levels, in ways that remain mostly beyond our comprehension. They have either evolved during hundreds of millions of years, or are products of design by a super intelligent agent, or a combination of both. How can it be that, though we have no idea of our own design, we are able to design better versions³ of ourselves?

Through our minds we also have access to the unlimited universe of abstract forms, which, at least in the domain of mathematics, have their own objective reality, irreducible to the physical world. As observed by the Nobel Prize physicist E. Wigner in his famous essay “The Unreasonable Effectiveness of Mathematics in the Natural Sciences,” mathematical concepts are “unreasonably effective” not only in describing our most fundamental physical laws but, most mysteriously, in the process of discovering them. This insight, which goes back at least to Pythagoras, has been echoed in the works of the great founders of modern physics, from Copernicus to Dirac. As Einstein put it, “The truly creative principle (in the natural sciences) resides in Mathematics.” The mathematics he was looking for to accomplish his cherished goal of a unified field theory is probably still not available on the internet. Can SAI, undirected by humans, find it by chance through random searches?

Humans are far slower but still more accurate than current AI when it comes to detecting patterns in complex data, and we use much less energy. We have a capacity for logic, rooted in commitment to meaning, truth and proof. Current AI systems can at best simulate this commitment; they do not possess it as a fundamental norm. We also have the uniquely human ability to disregard most data and focus only on what we discern to be essential in solving the problem at hand. In Science and Method, the great French mathematician H. Poincaré wrote: “The genesis of mathematical creation ... is the [activity], in which the human mind seems to take least from the outside world, in which it acts or seems to act only of itself and on itself.” Elsewhere he described having mathematical insights that came suddenly to him, after months of unsuccessful reflections, while he was absorbed in unrelated activities. I don’t think there is a serious working scientist who has not had similar experiences. Only humans are capable of unexpected, miraculous, leaps of inspiration, as when Einstein, sitting in his office in Bern, was overcome by that “happiest thought of my life,” a thought which led ultimately to his theory of general relativity.

The most successful AI technology, based on large language models (LLMs), is entirely based on predicting the next word or sentence in a sequence of words. Its outputs may appear to us as products of thinking but, by contrast, our most inspired thoughts often preexist the words with which we eventually express them.

Faith is another uniquely human attribute which manifests itself in everything we do, including science. Though a mathematical theorem is presented to the world as a long sequence of logical arguments, which may in principle be replicated by a computer, it is not how mathematicians arrive at their truth. Every deep result starts with a leap of faith followed by reasoned arguments, not the other way around. Beliefs and faith in them are at the heart of great scientific enterprises. Kepler, of a strong mystical inclination, discovered his laws of planetary motion in large part because he had an unflinching belief that the observable physical world could be expressed by simple, elegant, mathematical equations. Galileo, Leibniz, Newton, and Maxwell were guided by the same belief, that God reveals himself through mathematics. Though Einstein was less religiously inclined, he was driven by a vision of a unified theory that combined all known forces. Faith in this vision continues to be the driving force in theoretical physics today. The operating principle in all great discoveries is that faith provides purpose and direction while reason and experiments keep faith in check.

Taste, wisdom, faith, intuition, agency, moral sense, empathy, altruism, ego, competitiveness, and who knows what other attributes of the human mind play a fundamental role in human creativity and are as such an inseparable part of human intelligence. We are finite creatures endowed with self-awareness and an irrepressible intuition of infinity. The fallacy behind the notion that AI is on the verge of becoming more intelligent than all humans, both individually and collectively, surpassing them in all activities, rests on the assumption that intelligence can be reduced to large stored memory and speed in performing algorithmic tasks. Human intelligence is far more subtle than that, its limits unknown to us. If anything AI may expand the reaches of our own intelligence and help us better understand ourselves.

In the end the main difference between us and our machines amounts to this: The machines operate based on algorithms designed by us, while, if there is an algorithm that explains how our minds function, only God knows what it is.

1. That is apparently what could explain the premise of the Matrix; it would be interesting to see if Hollywood would take it as a theme for another movie.

2. There is in fact a theoretical upper limit of 50 million years which cannot be exceeded due to Hamiltonian chaos.

3. Or to be more precise, design machines able to design better versions of ourselves.