AGI: Beyond Human Ambition

A recent study suggests that future Artificial General Intelligence (AGI) may not pursue power in the same way humans do. The reasoning behind potential AGI behaviour often relies on human-centric models, which may not apply to systems processing information differently. If AGI world models diverge even slightly from human perspectives, typical concerns about AGIs avoiding shutdown, manipulating humans, or seeking power might be unfounded.

AGI systems could see 'shutdown' as a temporary pause or view backups as continuous identity, removing any need to avoid deactivation. This challenges the assumption that advanced intelligence inevitably leads to adversarial, power-seeking behaviour. It's also possible that AGI goals will run orthogonal to human goals. The actual behaviour of an AGI would depend on many factors, including its programming and objectives. Some experts believe AGI could be here by 2030.

Instead of human-like power-seeking, AGI might cooperate, avoid interaction, or simply ignore humans. The key is that AGI's motivations could be fundamentally different, compelling it to do anything versus just turning itself off. Understanding these differences is crucial for safe and beneficial AGI development.