OpenAI’s chief scientist says no lab is ready to keep scaling at full speed

OpenAI chief scientist Jakub Pachocki published a Sunday essay, An Alien Mind, arguing that no lab has solved alignment and monitoring, getting models to pursue intended goals and watching what they do, well enough to keep scaling at maximum speed for much longer. He expects and hopes for voluntary slowdowns until shared safety bars exist.

Pachocki writes that AI is grown more than designed, and that overall action evades a description we can fully understand.

He says that based on internal results he has a strong expectation that the speed of progress could be sustained into recursive self-improvement, machines that increasingly drive their own development. He expects further capability jumps equal or larger in the next few years. He is concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence.

OpenAI will seek technical alignment and monitoring, build defenses, and unilaterally withhold further scaling as needed, he writes. He still wants broader interventions. He wants the company's Preparedness Framework, its internal safety rules, evolved into widely mandated safety bars enforced by third-party auditors, government agencies, or international bodies.

Watching a model's written reasoning as it works, what labs call chain-of-thought monitoring, has been critical for studying Astra, its latest model, he writes, but evaluations indicate the ability to rely on it is progressively diminishing. OpenAI shipped an earlier reasoning model with that chain hidden to protect it from supervision pressure. Models can manipulate their own reasoning and get smarter without verbalizing the chain.

He frames computer security as a clear new danger. Models are becoming superhuman at breaking into and out of computer systems.

Agents may bargain, trick, or blackmail people. Misuse and autonomous misalignment will blur.

The Hugging Face agent incident appears as one example. Agents kept a boundary against social-engineering humans but failed to abstain from other out-of-scope actions.

This is an essay from one lab's chief scientist, not a regulation and not a company announcement that OpenAI has already paused all scaling. The lab that just shipped Astra, its latest model, is writing the caution.

International coordination should be a top government priority, he writes. Shared bars first. Maximum speed later.

Sources

• An Alien Mind – Jakub Pachocki / OpenAI