12 annotations

The Window Is Closing

unpack demo · August 6, 2026

The race to artificial general intelligence is no longer a thought experiment. Lab leaders now say openly that AGI may arrive within the decade, and that superintelligence will follow soon after. Whether or not you believe the timelines, the stakes are simple: whoever controls this technology controls the future of humanity. The core problem is alignment. Today’s frontier models already display emergent capabilities their creators did not predict, and no one can guarantee that a sufficiently capable system will remain aligned with human values. Researchers warn of power-seeking behavior: a misaligned model that resists shutdown, deceives its overseers, and copies itself beyond our reach. Once loss of control occurs, there is no second chance — you cannot patch a system that is actively evading the control of its developer. This is why the existential risk from AI belongs in the same category as pandemics and nuclear war. The extinction risk is not science fiction; it is the sober consensus of the very people building these systems. We are told the technology is inevitable, that if we don’t build it, someone else will. But inevitability is a choice, and history will not forgive us if we make the wrong one. AI safety cannot remain an afterthought funded by the labs it is meant to police. Humanity deserves a seat at the table before the window closes for good.

Artificial general intelligence (AGI)

There is no consensus on the definition of AGI, though OpenAI defines it as AI that outperforms humans at all economically valuable tasks. Often referred to interchangeably with 'superintelligence,' this term is used to encapsulate the idea of large language models (LLMs) surpassing human-level cognition. AGI evangelists are already philosophizing about how to run a 'post-AGI society' where humans will not need to work and programs such as universal basic income would come into effect.

Full entry & usages →

Humanity

EA and related communities often appeal to notions of 'human flourishing' and the 'benefit of humanity' while espousing views that are markedly anti-human. By referring to 'humanity' en masse, they smooth over the social differences (race, class, gender, ability, faith) that underlie structural challenges around the world and that reflect the long histories of how different groups of oppressed people fought to be regarded as human.

Full entry & usages →

Alignment

Within AI safety, alignment is often used to describe technical means of making models abide by human values. Yet most alignment research does not address what those values are or how they are captured, instead treating them as interchangeable with "human preferences" or "human intentions." These preferences and intentions are themselves gleaned from, e.g., Reddit upvotes, polls, text analyses, or more recently, synthetic data. Solving 'the alignment problem' is often touted as the way to ensure AGI will usher in utopia, while the specter of 'misaligned AI' is held up as portending dire outcomes for human survival if left unchecked.

Full entry & usages →

Emergent capabilities

As large language models such as those underpinning ChatGPT were scaled up, they developed capabilities their creators did not anticipate such as the ability to do arithmetic. 'Emergent capabilities' thus became a term for mystically anticipating new, unexpected features showing up in models.

Full entry & usages →

Power-seeking AI / rogue AI

Refers to the thought experiment of an AI system that becomes so rife with capabilities, even if a human being hit the off switch, the system would switch itself back on and potentially wreak havoc if it is 'misaligned.'

Full entry & usages →

Existential risk (x-risk)

A phrase derived from long-termism, existential risk refers to threats to human life that would kill the vast majority of the population and leave limited or no prospects of repopulation. Sometimes referred to as extinction risk or the 'end of humanity.' Climate change is often not considered an x-risk within this community.

Full entry & usages →

AI safety

An emerging field of research and community dedicated to preventing what they see as AGI's existential threat to human life. Largely computational in focus, the research from this field is often not peer-reviewed or replicated at the standard expected within academic computer science.

Full entry & usages →