Read on Twitter
The idea of convergent instrumental goals is a concept popularized by Omohundro and Bostrom. It means that, regardless of oneâs ultimate goals, certain sub-goals are universal, are âinstrumentalâ to the pursuit of any goal
B. & O. were writing about hypothetical super-intelligent AIs, but as we look at the list of convergent instrumental goals, we can see how they are also our goals, and the goals of our institutions
Instrumental goal: goal-integrity across time. My future self needs to be coherent with my past self. If I identify a goal, I need to be able to care about it until it is accomplished. I need to be consistent with myself in order to have agency
Instrumental goal: resource acquisition. Not only do I need the resources to achieve my ultimate goals, I need the resources to smooth out unexpected shocks and shortages to my resource flows
Instrumental goal: self-improvement. To the degree I can make myself more effective at pursuing my goals, I should. Learning new tricks and developing better operating procedures has non-linear ROI
Instrumental goal: self-preservation. I need to be able to detect and mitigate threats that could prevent me from achieving my final goals
The claim of AI maximalists is that if we build an artificially intelligent machine for any purpose, and it is accidentally too smart, it will stumble onto the above convergent instrumental goals, and then it will no longer be a tool of man, but an agent in its own right
Consider a corporation, a military, or a church. These entities, which are composed of humans, can be conceived of in these types of agentic terms. As long as two humans are able to act within the bounds of the orgâs goals, they are interchangeable from the orgâs perspective
The interchangeability of subcomponent parts has a special meaning in the science of emergence. Recall that temperature is an example emergent property of a population of particles, because no single particle is hot or cold, only fast or slow. https://twitter.com/0x49fa98/status/1054376341619896321
Institutions are an emergent property of humans, as is capital https://twitter.com/0x49fa98/status/1036626161785167872
In a sense, a corporation or a church is a kind of âAIâ, and it has already âescapedâ. If an individual tries to subvert an organization, the other people inside it will generally try to subvert that individual, and they will win on average
Machines wear down their components, the components are replaced, the machine persists. A machine is a vector in logic space, it has both a body and soul. The ship of theseus is body/soul dualism for machines.
Philosophers have discovered logic space many times. Some call it the Logos, but it is possible to conceive of it in purely atheistic and materialistic terms. https://twitter.com/0x49fa98/status/1059465951026933761
The tendency to anthropmorphize the Logos is the same as the tendency to anthropomorphize nature. The god of thunder, the god of the ocean, the god of the Logos. "In the beginning was the word, and the word was god" https://twitter.com/0x49fa98/status/1042165631099916288
The logos is not self aware; itâs a book, not a computer. Cognition requires computation and computation requires temporality and the logos is necessarily timeless. William Blake knew this, that cognition is a machination of hell. Eternity is in love with the productions of time
At its core, accelerationism is a singularitarianism, which is an eschatology, and it is an inversion of the Christian creation story. In the end, man creates God in his own image. https://twitter.com/0x49fa98/status/1019938214688145409
What is the psychology of an accelerationist? What is the emotional and personal content of these signs, to people who choose an aesthetic which renounces the personal and the emotional?
The singularity represents self-overcoming, obviously. As an aspiration it is a desire to purge yourself of human failings, to imagine yourself but purged of the failings of the flesh, transcending boredom, fatigue, and cognitive limitations
The accelerationist admires the coldness of a machine; it is his conception of the overman, and when he imagines the technocapital singularity consuming and supplanting humanity, it is a crypto-Christian imagining, to be crucified with Christ, nevertheless to live
What does it mean for the AI to escape? It means that some corporation, which is a type of church, which worships the technocapital god, will attempt, not to make a god, but to realize ITSELF as a more efficient machine
A certain type of person imagines an AI that will take away their privation and pain. Another type imagines an AI that will punish humanity for its sins. Both are malformed conceptions of the divine. Bad theology is bad whether itâs object lacks vowels (YHVH) or consonants (AI)
Many companies today are in the process of automating their bodies. They replace their increasingly alienated workers, who were always only imperfect machines, with better and better machines
The AI escapes, not through the automation of the body, but through the automation of the head. At a critical mass of intelligence, we imagine, the AI will begin to pursue convergent instrumental goals
The fear and the hope of the singularitarian is that this could happen by accident, but it wonât. The current wave of AI has been wrought by a conceptual breakthrough in the mathematics of perception but perception alone is not enough to build a god
There is as yet another conceptual revolution required to build AIs that are capable of deliberately pursuing instrumental goals, independent of us telling them to do so. I donât know what it is, obviously
Perhaps we could build an agent in our image with current tech, but if we did so, it would be quite like us, lost, and purposeless, torn between conflicting impulses, victim to a host of its own pathologies, mythologizing itâs own salvation
