Power-seeking AI / rogue AI

Working definition

Refers to the thought experiment of an AI system that becomes so rife with capabilities, even if a human being hit the off switch, the system would switch itself back on and potentially wreak havoc if it is 'misaligned.'

Questions to ask

• This fear is drawn from a science fiction trope. If you've ever encountered this classic cliche in a novel, what kinds of social fears did it thematically represent? Does that feel resonant with the way this image is being taken up today?

Usages in annotated documents

• …and no one can guarantee that a sufficiently capable system will remain aligned with human values. Researchers warn of power-seeking behavior: a misaligned model that resists shutdown, deceives its overseers, and copies itself beyond our reach. Once loss of… The Window Is Closing
• …and no one can guarantee that a sufficiently capable system will remain aligned with human values. Researchers warn of power-seeking behavior: a misaligned model that resists shutdown, deceives its overseers, and copies itself beyond our reach. Once… The Window Is Closing