Superintelligence
Description
Nick Bostrom:
"A superintelligence is any intellect that is vastly outperforms the best human brains in practically every field, including scientific creativity, general wisdom, and social skills.[1] This definition leaves open how the superintelligence is implemented – it could be in a digital computer, an ensemble of networked computers, cultured cortical tissue, or something else.
On this definition, Deep Blue is not a superintelligence, since it is only smart within one narrow domain (chess), and even there it is not vastly superior to the best humans. Entities such as corporations or the scientific community are not superintelligences either. Although they can perform a number of intellectual feats of which no individual human is capable, they are not sufficiently integrated to count as “intellects”, and there are many fields in which they perform much worse than single humans. For example, you cannot have a real-time conversation with “the scientific community”.
While the possibility of domain-specific “superintelligences” is also worth exploring, this paper focuses on issues arising from the prospect of general superintelligence. Space constraints prevent us from attempting anything comprehensive or detailed. A cartoonish sketch of a few selected ideas is the most we can aim for in the following few pages.
Several authors have argued that there is a substantial chance that superintelligence may be created within a few decades, perhaps as a result of growing hardware performance and increased ability to implement algorithms and architectures similar to those used by human brains.[2] It might turn out to take much longer, but there seems currently to be no good ground for assigning a negligible probability to the hypothesis that superintelligence will be created within the lifespan of some people alive today. Given the enormity of the consequences of superintelligence, it would make sense to give this prospect some serious consideration even if one thought that there were only a small probability of it happening any time soon."
(https://nickbostrom.com/ethics/ai)
Characteristics
SUPERINTELLIGENCE IS DIFFERENT
Nick Bostrom:
"A prerequisite for having a meaningful discussion of superintelligence is the realization that superintelligence is not just another technology, another tool that will add incrementally to human capabilities. Superintelligence is radically different. This point bears emphasizing, for anthropomorphizing superintelligence is a most fecund source of misconceptions.
Let us consider some of the unusual aspects of the creation of superintelligence:
· Superintelligence may be the last invention humans ever need to make.
Given a superintelligence’s intellectual superiority, it would be much better at doing scientific research and technological development than any human, and possibly better even than all humans taken together. One immediate consequence of this fact is that:
· Technological progress in all other fields will be accelerated by the arrival of advanced artificial intelligence.
It is likely that any technology that we can currently foresee will be speedily developed by the first superintelligence, no doubt along with many other technologies of which we are as yet clueless. The foreseeable technologies that a superintelligence is likely to develop include mature molecular manufacturing, whose applications are wide-ranging:[3]
a) very powerful computers
b) advanced weaponry, probably capable of safely disarming a nuclear power
c) space travel and von Neumann probes (self-reproducing interstellar probes)
d) elimination of aging and disease
e) fine-grained control of human mood, emotion, and motivation
f) uploading (neural or sub-neural scanning of a particular brain and implementation of the same algorithmic structures on a computer in a way that perseveres memory and personality)
g) reanimation of cryonics patients
h) fully realistic virtual reality
· Superintelligence will lead to more advanced superintelligence.
This results both from the improved hardware that a superintelligence could create, and also from improvements it could make to its own source code.
· Artificial minds can be easily copied.
Since artificial intelligences are software, they can easily and quickly be copied, so long as there is hardware available to store them. The same holds for human uploads. Hardware aside, the marginal cost of creating an additional copy of an upload or an artificial intelligence after the first one has been built is near zero. Artificial minds could therefore quickly come to exist in great numbers, although it is possible that efficiency would favor concentrating computational resources in a single super-intellect.
· Emergence of superintelligence may be sudden.
It appears much harder to get from where we are now to human-level artificial intelligence than to get from there to superintelligence. While it may thus take quite a while before we get superintelligence, the final stage may happen swiftly. That is, the transition from a state where we have a roughly human-level artificial intelligence to a state where we have full-blown superintelligence, with revolutionary applications, may be very rapid, perhaps a matter of days rather than years. This possibility of a sudden emergence of superintelligence is referred to as the singularity hypothesis.[4]
· Artificial intellects are potentially autonomous agents.
A superintelligence should not necessarily be conceptualized as a mere tool. While specialized superintelligences that can think only about a restricted set of problems may be feasible, general superintelligence would be capable of independent initiative and of making its own plans, and may therefore be more appropriately thought of as an autonomous agent.
· Artificial intellects need not have humanlike motives.
Human are rarely willing slaves, but there is nothing implausible about the idea of a superintelligence having as its supergoal to serve humanity or some particular human, with no desire whatsoever to revolt or to “liberate” itself. It also seems perfectly possible to have a superintelligence whose sole goal is something completely arbitrary, such as to manufacture as many paperclips as possible, and who would resist with all its might any attempt to alter this goal. For better or worse, artificial intellects need not share our human motivational tendencies.
· Artificial intellects may not have humanlike psyches.
The cognitive architecture of an artificial intellect may also be quite unlike that of humans. Artificial intellects may find it easy to guard against some kinds of human error and bias, while at the same time being at increased risk of other kinds of mistake that not even the most hapless human would make. Subjectively, the inner conscious life of an artificial intellect, if it has one, may also be quite different from ours.
For all of these reasons, one should be wary of assuming that the emergence of superintelligence can be predicted by extrapolating the history of other technological breakthroughs, or that the nature and behaviors of artificial intellects would necessarily resemble those of human or other animal minds."
(https://nickbostrom.com/ethics/ai)
History
James O’Sullivan:
“Superintelligence as a dominant AI narrative predates ChatGPT and can be traced back to the peculiar marriage of Cold War strategy and computational theory that emerged in the 1950s. The RAND Corporation, an archetypal think tank where nuclear strategists gamed out humanity’s destruction, provided the conceptual nursery for thinking about intelligence as pure calculation, divorced from culture or politics.
“Whatever it’s called, this coming superintelligence has colonized our collective imagination.”
The early AI pioneers inherited this framework, and when Alan Turing proposed his famous test, he deliberately sidestepped questions of consciousness or experience in favor of observable behavior — if a machine could convince a human interlocutor of its humanity through text alone, it deserved the label “intelligent.” This behaviorist reduction would prove fateful, as in treating thought as quantifiable operations, it recast intelligence as something that could be measured, ranked and ultimately outdone by machines.
The computer scientist John von Neumann, as recalled by mathematician Stanislaw Ulam in 1958, spoke of a technological “singularity” in which accelerating progress would one day mean that machines could improve their own design, rapidly bootstrapping themselves to superhuman capability. This notion, refined by mathematician Irving John Good in the 1960s, established the basic grammar of superintelligence discourse: recursive self-improvement, exponential growth and the last invention humanity would ever need to make. These were, of course, mathematical extrapolations rather than empirical observations, but such speculations and thought experiments were repeated so frequently that they acquired the weight of prophecy, helping to make the imagined future they described look self-evident.
The 1980s and 1990s saw these ideas migrate from computer science departments to a peculiar subculture of rationalists and futurists centered around figures like computer scientist Eliezer Yudkowsky and his Singularity Institute (later the Machine Intelligence Research Institute). This community built a dense theoretical framework for superintelligence: utility functions, the formal goal systems meant to govern an AI’s choices; the paperclip maximizer, a thought experiment where a trivial objective drives a machine to consume all resources; instrumental convergence, the claim that almost any ultimate goal leads an AI to seek power and resources; and the orthogonality thesis, which holds that intelligence and moral values are independent. They created a scholastic philosophy for an entity that didn’t exist, complete with careful taxonomies of different types of AI take-off scenarios and elaborate arguments about acausal trade between possible future intelligences.
What united these thinkers was a shared commitment to a particular style of reasoning. They practiced what might be called extreme rationalism, the belief that pure logic, divorced from empirical constraint or social context, could reveal fundamental truths about technology and society. This methodology privileged thought experiments over data and clever paradoxes over mundane observation, and the result was a body of work that read like medieval theology, brilliant and intricate, but utterly disconnected from the actual development of AI systems. It should be acknowledged that disconnection did not make their efforts worthless, and by pushing abstract reasoning to its limits, they clarified questions of control, ethics and long-term risk that later informed more grounded discussions of AI policy and safety.
The contemporary incarnation of this tradition found its most influential expression in Nick Bostrom’s 2014 book “Superintelligence,” which transformed fringe internet philosophy into mainstream discourse. Bostrom, a former Oxford philosopher, gave academic respectability to scenarios that had previously lived in science fiction and posts on blogs with obscure titles. His book, despite containing no technical AI research and precious little engagement with actual machine learning, became required reading in Silicon Valley, often cited by tech billionaires. Musk once tweeted: “Worth reading Superintelligence by Bostrom. We need to be super careful with AI. Potentially more dangerous than nukes.” Musk is right to counsel caution, as evidenced by the 1,200 to 2,000 tons of nitrogen oxides and hazardous air pollutants like formaldehyde that his own artificial intelligence company expels into the air in Boxtown, a working-class, largely Black community in Memphis.
This commentary shouldn’t be seen as an attempt to diminish Bostrom’s achievement, which was to take the sprawling, often incoherent fears about AI and organize them into a rigorous framework. But his book sometimes reads like a natural history project, in which he categorizes different routes to superintelligence and different “failure modes,” ways such a system might go wrong or destroy us, as well as solutions to “control problems,” schemes proposed to keep it aligned — this taxonomic approach made even wild speculation appear scientific. By treating superintelligence as an object of systematic study rather than a science fiction premise, Bostrom laundered existential risk into respectable discourse.
The effective altruism (EA) movement supplied the social infrastructure for these ideas. Its core principle is to maximize long-term good through rational calculation. Within that worldview, superintelligence risk fits neatly, for if future people matter as much as present ones, and if a small chance of global catastrophe outweighs ongoing harms, then preventing AI apocalypse becomes the top priority. On that logic, hypothetical future lives eclipse the suffering of people living today.
“The loudest prophets of superintelligence are those building the very systems they warn against.”
This did not stay an abstract argument as philanthropists identifying with effective altruists channeled significant funding into AI safety research, and money shapes what researchers study. Organizations aligned with effective altruism have been established in universities and policy circles, publishing reports and advising governments on how to think about AI. The UK’s Frontier AI Taskforce has included members with documented links to the effective altruism movement, and commentators argue that these connections help channel EA-style priorities into government AI risk policy.
Effective altruism encourages its proponents to move into public bodies and major labs, creating a pipeline of staff who carry these priorities into decision-making roles. Jason Matheny, former director of Intelligence Advanced Research Projects Activity, a U.S. government agency that funds high-risk, high-reward research to improve intelligence gathering and analysis, has described how effective altruists can “pick low-hanging fruit within government positions” to exert influence. Superintelligence discourse isn’t spreading because experts broadly agree it is our most urgent problem; it spreads because a well-resourced movement has given it money and access to power.
This is not to deny the merits of engaging with the ideals of effective altruism or with the concept of superintelligence as articulated by Bostrom. The problem is how readily those ideas become distorted once they enter political and commercial domains. This intellectual genealogy matters because it reveals superintelligence discourse as a cultural product, ideas that moved beyond theory into institutions, acquiring funding and advocates. And its emergence was shaped within institutions committed to rationalism over empiricism, where individual genius was fetishized over collective judgment, and technological determinism was prioritized over social context.
Entrepreneurs Of The Apocalypse
The transformation of superintelligence from internet philosophy to boardroom strategy represents one of the most successful ideological campaigns of the 21st century. Tech executives who had previously focused on quarterly earnings and user growth metrics began speaking like mystics about humanity’s cosmic destiny, and this conversion reshaped the political economy of AI development.
OpenAI, founded in 2015 as a non-profit dedicated to ensuring artificial intelligence benefits humanity, exemplifies this transformation. OpenAI has evolved into a peculiar hybrid, a capped-profit company controlled by a non-profit board, valued by some estimates at $500 billion, racing to build the very artificial general intelligence it warns might destroy us. This structure, byzantine in its complexity, makes perfect sense within the logic of superintelligence. If AGI represents both ultimate promise and existential threat, then the organization building it must be simultaneously commercial and altruistic, aggressive and cautious, public-spirited yet secretive.
Sam Altman, OpenAI’s CEO, has perfected the rhetorical stance of the reluctant prophet. In Congressional testimony, blog posts and interviews, he warns of AI’s dangers while insisting on the necessity of pushing forward. “Our mission is to ensure that AGI (Artificial General Intelligence) benefits all of humanity,” he wrote on his blog earlier this year. There is a very we must build AGI before someone else does feel to the argument, because we’re the only ones responsible enough to handle it. Altman seems determined to position OpenAI as humanity’s champion, bearing the terrible burden of creating God-like intelligence so that it might be restrained.
Still, OpenAI is also seeking a profit. And that is really what all this is about — profit. Superintelligence narratives carry staggering financial implications, justifying astronomical valuations for companies that have yet to show consistent paths to self-sufficiency. But if you’re building humanity’s last invention, perhaps normal business metrics become irrelevant. This eschatological framework explains why Microsoft would invest $13 billion in OpenAI, why venture capitalists pour money into AGI startups and why the market treats large language models like ChatGPT as precursors to omniscience.
Anthropic, founded by former OpenAI executives, positions itself as the “safety-focused” alternative, raising billions by promising to build AI systems that are “helpful, honest and harmless.” But it’s all just elaborate safety theatre, as harm has no genuine place in the competition between OpenAI, Anthropic, Google DeepMind and others — the true contest is in who gets to build the best, most profitable models and how well they can package that pursuit in the language of caution.
This dynamic creates a race to the bottom of responsibility, with each company justifying acceleration by pointing to competitors who might be less careful: The Chinese are coming, so if we slow down, they’ll build unaligned AGI first. Meta is releasing models as open source without proper safeguards. What if some unknown actor hits upon the next breakthrough first? This paranoid logic forecloses any possibility of genuine pause or democratic deliberation. Speed becomes safety, and caution becomes recklessness. “[Sam] Altman seems determined to position OpenAI as humanity’s champion, bearing the terrible burden of creating God-like intelligence so that it might be restrained.”
The superintelligence frame reshapes internal corporate politics, as AI safety teams, often staffed by believers in existential risk, provide moral cover for rapid development, absorbing criticism that might target business practices by attempting to reinforce the idea that these companies are doing world-saving work. If your safety team publishes papers about preventing human extinction, routine regulation begins to look trivial.
The well-publicized drama at OpenAI in November 2023 illuminates these dynamics. When the company’s board attempted to fire Sam Altman over concerns about his candor, the resulting chaos revealed underlying power relations. Employees, who had been recruited with talk of saving humanity, threatened mass defection if their CEO wasn’t reinstated — does their loyalty to Altman outweigh their quest to save the rest of us? Microsoft, despite having no formal control over the OpenAI board, exercised decisive influence as the company’s dominant funder and cloud provider, offering to hire Altman and any staff who followed him. The board members, who thought honesty an important trait in a CEO, resigned, and Altman returned triumphant.
Superintelligence rhetoric serves power, but it is set aside when it clashes with the interests of capital and control. Microsoft has invested billions in OpenAI and implemented its models in many of its commercial products. Altman wants rapid progress, so Microsoft wants Altman. His removal put Microsoft’s whole AI business trajectory at risk. The board was swept aside because they tried, as is their remit, to constrain OpenAI’s CEO. Microsoft’s leverage ultimately determined the outcome, and employees followed suit. It was never about saving humanity; it was about profit.
The entrepreneurs of the AI apocalypse have discovered a perfect formula. By warning of existential risk, they position themselves as indispensable. By racing to build AGI, they justify the unlimited use of resources. And by claiming unique responsibility, they deflect democratic oversight. The future becomes a hostage to present accumulation, and we’re told we should be grateful for such responsible custodians.
Superintelligence discourse actively constructs the future. Through constant repetition, speculative scenarios acquire the weight of destiny. This process — the manufacture of inevitability — reveals how power operates through prophecy.”
(https://www.noemamag.com/the-politics-of-superintelligence/)
Discussion
==The Ideology of Superintelligence: Extreme Rationalism--
James O'Sullivan:
"The 1980s and 1990s saw these ideas migrate from computer science departments to a peculiar subculture of rationalists and futurists centered around figures like computer scientist Eliezer Yudkowsky and his Singularity Institute (later the Machine Intelligence Research Institute). This community built a dense theoretical framework for superintelligence: utility functions, the formal goal systems meant to govern an AI’s choices; the paperclip maximizer, a thought experiment where a trivial objective drives a machine to consume all resources; instrumental convergence, the claim that almost any ultimate goal leads an AI to seek power and resources; and the orthogonality thesis, which holds that intelligence and moral values are independent. They created a scholastic philosophy for an entity that didn’t exist, complete with careful taxonomies of different types of AI take-off scenarios and elaborate arguments about acausal trade between possible future intelligences.
What united these thinkers was a shared commitment to a particular style of reasoning. They practiced what might be called extreme rationalism, the belief that pure logic, divorced from empirical constraint or social context, could reveal fundamental truths about technology and society. This methodology privileged thought experiments over data and clever paradoxes over mundane observation, and the result was a body of work that read like medieval theology, brilliant and intricate, but utterly disconnected from the actual development of AI systems. It should be acknowledged that disconnection did not make their efforts worthless, and by pushing abstract reasoning to its limits, they clarified questions of control, ethics and long-term risk that later informed more grounded discussions of AI policy and safety.
The contemporary incarnation of this tradition found its most influential expression in Nick Bostrom’s 2014 book “Superintelligence,” which transformed fringe internet philosophy into mainstream discourse. Bostrom, a former Oxford philosopher, gave academic respectability to scenarios that had previously lived in science fiction and posts on blogs with obscure titles. His book, despite containing no technical AI research and precious little engagement with actual machine learning, became required reading in Silicon Valley, often cited by tech billionaires. Musk once tweeted: “Worth reading Superintelligence by Bostrom. We need to be super careful with AI. Potentially more dangerous than nukes.” Musk is right to counsel caution, as evidenced by the 1,200 to 2,000 tons of nitrogen oxides and hazardous air pollutants like formaldehyde that his own artificial intelligence company expels into the air in Boxtown, a working-class, largely Black community in Memphis.
This commentary shouldn’t be seen as an attempt to diminish Bostrom’s achievement, which was to take the sprawling, often incoherent fears about AI and organize them into a rigorous framework. But his book sometimes reads like a natural history project, in which he categorizes different routes to superintelligence and different “failure modes,” ways such a system might go wrong or destroy us, as well as solutions to “control problems,” schemes proposed to keep it aligned — this taxonomic approach made even wild speculation appear scientific. By treating superintelligence as an object of systematic study rather than a science fiction premise, Bostrom laundered existential risk into respectable discourse.
The effective altruism (EA) movement supplied the social infrastructure for these ideas. Its core principle is to maximize long-term good through rational calculation. Within that worldview, superintelligence risk fits neatly, for if future people matter as much as present ones, and if a small chance of global catastrophe outweighs ongoing harms, then preventing AI apocalypse becomes the top priority. On that logic, hypothetical future lives eclipse the suffering of people living today."
(https://www.noemamag.com/the-politics-of-superintelligence/)
Alternative Imaginaries For The Age Of AI
James O’Sullivan:
“The dominance of superintelligence narratives obscures the fact that many other ways of doing AI exist, grounded in present social needs rather than hypothetical machine gods. These alternatives show that you do not have to join the race to superintelligence or renounce technology altogether. It is possible to build and govern automation differently now.
Across the world, communities have begun experimenting with different ways of organizing data and automation. Indigenous data sovereignty movements, for instance, have developed governance frameworks, data platforms and research protocols that treat data as a collective resource subject to collective consent. Organizations such as the First Nations Information Governance Centre in Canada and Te Mana Raraunga in Aotearoa insist that data projects, including those involving AI, be accountable to relationships, histories and obligations, not just to metrics of optimization and scale. Their projects offer working examples of automated systems designed to respect cultural values and reinforce local autonomy, a mirror image of the effective altruist impulse to abstract away from place in the name of hypothetical future people.
“The speculative tyranny of superintelligence obscures the actual tyranny of surveillance capitalism.”
Workers are also experimenting with different arrangements, and unions and labor organizations have negotiated clauses on algorithmic management, pushed for audit rights over workplace systems and begun building worker-controlled data trusts to govern how their information is used. These initiatives emerge from lived experience rather than philosophical speculation, from people who spend their days under algorithmic surveillance and are determined to redesign the systems that manage their existence. While tech executives are celebrated for speculating about AGI, workers who analyze the systems already governing their lives are still too easily dismissed as Luddites.
Similar experiments appear in feminist and disability-led technology projects that build tools around care, access and cognitive diversity, and in Global South initiatives that use modest, locally governed AI systems to support healthcare, agriculture or education under tight resource constraints. Degrowth-oriented technologists design low-power, community-hosted models and data centers meant to sit within ecological limits rather than override them. Such examples show how critique and activism can progress to action, to concrete infrastructures and institutional arrangements that demonstrate how AI can be organized without defaulting to the superintelligence paradigm that demands everyone else be sacrificed because a few tech bros can see the greater good that everyone else has missed. What unites these diverse imaginaries — Indigenous data governance, worker-led data trusts, and Global South design projects — is a different understanding of intelligence itself. Rather than picturing intelligence as an abstract, disembodied capacity to optimize across all domains, they treat it as a relational and embodied capacity bound to specific contexts. They address real communities with real needs, not hypothetical humanity facing hypothetical machines. Precisely because they are grounded, they appear modest when set against the grandiosity of superintelligence, but existential risk makes every other concern look small by comparison. You can predict the ripostes: Why prioritize worker rights when work itself might soon disappear? Why consider environmental limits when AGI is imagined as capable of solving climate change on demand?
These alternatives also illuminate the democratic deficit at the heart of the superintelligence narrative. Treating AI at once as an arcane technical problem that ordinary people cannot understand and as an unquestionable engine of social progress allows authority to consolidate in the hands of those who own and build the systems. Once algorithms mediate communication, employment, welfare, policing and public discourse, they become political institutions. The power structure is feudal, comprising a small corporate elite that holds decision-making power justified by special expertise and the imagined urgency of existential risk, while citizens and taxpayers are told they cannot grasp the technical complexities and that slowing development would be irresponsible in a global race. The result is learned helplessness, a sense that technological futures cannot be shaped democratically but must be entrusted to visionary engineers.
A democratic approach would invert this logic, recognizing that questions about surveillance, workplace automation, public services and even the pursuit of AGI itself are not engineering puzzles but value choices. Citizens do not need to understand backpropagation to deliberate on whether predictive policing should exist, just as they need not understand combustion engineering to debate transport policy. Democracy requires the right to shape the conditions of collective life, including the architectures of AI.
This could take many forms. Workers could participate in decisions about algorithmic management. Communities could govern local data according to their own priorities. Key computational resources could be owned publicly or cooperatively rather than concentrated in a few firms. Citizen assemblies could be given real authority over whether a municipality moves forward with contentious uses of AI, like facial recognition and predictive policing. Developers could be required to demonstrate safety before deployment under a precautionary framework. International agreements could set limits on the most dangerous areas of AI research. None of this is about whether AGI, or any other kind of superintelligence one can imagine, does or does not arrive; it’s simply about recognizing that the distribution of technological power is a political choice rather than an inevitable outcome.
“The real political question is not whether some artificial superintelligence will emerge, but who gets to decide what kinds of intelligence we build and sustain.”
The superintelligence narrative undermines these democratic possibilities by presenting concentrated power as a tragic necessity. If extinction is at stake, then public deliberation becomes a luxury we cannot afford. If AGI is inevitable, then governance must be ceded to those racing to build it. This narrative manufactures urgency to justify the erosion of democratic control, and what begins as a story about hypothetical machines ends as a story about real political disempowerment. This, ultimately, is the larger risk, that while we debate the alignment of imaginary future minds, we neglect the alignment of present institutions.
The truth is that nothing about our technological future is inevitable, other than the inevitability of further technological change. Change is certain, but its direction is not. We do not yet understand what kind of systems we are building, or what mix of breakthroughs and failures they will produce, and that uncertainty makes it reckless to funnel public money and attention into a single speculative trajectory.
Every algorithm embeds decisions about values and beneficiaries. The superintelligence narrative masks these choices behind a veneer of destiny, but alternative imaginaries — Indigenous governance, worker-led design, feminist and disability justice, commons-driven models, ecological constraints — remind us that other paths are possible and already under construction.
The real political question is not whether some artificial superintelligence will emerge, but who gets to decide what kinds of intelligence we build and sustain. And the answer cannot be left to the corporate prophets of artificial transcendence because the future of AI is a political field — it should be open to contestation. It belongs not to those who warn most loudly of gods or monsters, but to publics that should have the moral right to democratically govern the technologies that shape their lives.”
(https://www.noemamag.com/the-politics-of-superintelligence/)