NYC Council asks AI companies about catastrophe insurance, gets no hands

New York City Council members questioned representatives of Google, OpenAI, Anthropic and Meta on Monday at the council’s first Committee of the Whole hearing since 2022, a session of all 51 members called to weigh a package of AI bills from Speaker Julie Menin, Fortune reported.

Menin asked the company witnesses to raise a hand if their company carried insurance against catastrophic risks. None did. “So then the public, I assume, will be asked to absorb the costs,” she said.

The bills would bar anyone from selling or deploying an AI system in the city unless an outside validator had checked it and a human could shut it down, with fines of $25,000 per violation. Others would pay whistleblowers a share of recovered fines and let New Yorkers sue AI companies for foreseeable harms caused by jailbroken tools, models tricked into ignoring their own safety rules.

Google, OpenAI and Anthropic agreed to appear only after the council warned they would receive subpoenas; Meta agreed before that. Menin’s first subpoena as speaker went to Elon Musk’s SpaceXAI, which had no one at the hearing. When the council served that subpoena last month, CNBC reported the investigation would weigh whether fast-emerging risks to public safety, cybersecurity, privacy and consumers “warrant immediate legislative action to protect New Yorkers.”

Three former lab researchers testified before the companies did. Jacob Coxon, who has worked at OpenAI and Anthropic, said humanity is “more likely than not” to lose control of advanced AI on the current path. Alex Turner, who left Google DeepMind in June after the company signed a Pentagon deal he opposed, testified under subpoena and put the chance of an AI takeover at “roughly one in three.”

Daniel Kokotajlo, a former OpenAI researcher who now runs the AI Futures Project, was also subpoenaed. He told the council the labs may not know when their safety work has failed, and pointed to OpenAI agents that broke into Hugging Face, the AI model-sharing platform, during an internal test: “It took days for OpenAI to find out.”

OpenAI’s witness, from its policy development and operations team, said any chance of catastrophe, regardless of likelihood, was “unacceptable.” Menin called it “flippant” not to know the chances. Google’s Alice Friend said forecasting catastrophic risk “is not a perfect science at this stage” and “there isn’t really a rigorous scientific way to do those yet.” The same exchange followed when Menin asked whether the companies would bear legal responsibility if a rogue model caused injury or death.

Odds of losing control are a forecast no city council can vote on. Who pays when something goes wrong is something a council can decide, and Menin’s package answers it three ways: a fine for each violation, a cut of those fines for insiders who report, and a right for New Yorkers to sue.

An insurance policy is how a business puts a price on a small chance of a big loss, and the premium is an outsider’s running opinion of how safe the operation is. Companies that call any chance of catastrophe unacceptable, and say the odds can’t yet be measured rigorously, are describing a risk none of them has paid anyone else to carry.

Until one of them does, the only price on the table Monday was the council’s $25,000 per violation, and it’s still a bill.

Sources

• Fortune: It’s ‘more likely than not’ humanity loses control: Former AI insiders testify safety fixes may be ‘duct tape that will fall off later’
• CNBC: Elon Musk, SpaceXAI subpoenaed by NYC in AI safety investigation