Gavin Newsom Blocks Contentious AI Safety Bill in California

California Governor Gavin Newsom has vetoed what would have become one of the most comprehensive policies governing the safety of artificial intelligence in the U.S.

The bill would’ve been among the first to hold AI developers accountable for any severe harm caused by their technologies. It drew fierce criticism from some prominent Democrats and major tech firms, including ChatGPT creator OpenAI and venture capital firm Andreessen Horowitz, who warned it could stall innovation in the state.

Newsom described the legislation as “well-intentioned” but said in a statement that it would’ve applied “stringent standards to even the most basic functions.” Regulation should be based on “empirical evidence and science,” he said, pointing to his own executive order on AI and other bills he’s signed that regulate the technology around known risks such as deepfakes.

The debate around California’s SB 1047 bill highlights the challenge that lawmakers around the world are facing in controlling the risks of AI while also supporting the emerging technology. U.S policymakers have yet to pass any comprehensive legislation around the technology since the release of ChatGPT two years ago touched off a global generative AI boom.

Democratic California Senator Scott Wiener, who introduced the bill, called Newsom’s veto a “setback for everyone who believes in oversight of massive corporations.” In a statement posted on X, Wiener said, “We are all less safe as a result.”

‘Reasonable care’

SB 1047 would’ve mandated that companies developing powerful AI models take reasonable care to ensure that their technologies wouldn’t cause “severe harm” such as mass casualties or property damage above $500 million. Companies would’ve had to take specific precautions, including maintaining a kill switch that could turn off their technology. AI models would’ve also been subject to third-party testing to ensure they minimized grave risk. 

The bill would’ve also created whistleblower protections for employees at AI companies that want to share safety concerns. Companies that weren’t in compliance with the bill could have been sued by the California attorney general.

Supporters of the legislation said it would’ve created common-sense legal standards. But VC investors, startup leaders and companies like OpenAI warned that it would slow innovation and drive AI companies out of the state. 

“The AI revolution is only just beginning, and California’s unique status as the global leader in AI is fueling the state’s economic dynamism,” Jason Kwon, chief strategy officer at OpenAI, wrote in a letter last month opposing the legislation. “SB 1047 would threaten that growth, slow the pace of innovation, and lead California’s world-class engineers and entrepreneurs to leave the state in search of greater opportunity elsewhere.”

Lawmakers opposed

Lawmakers including former House Speaker Nancy Pelosi, Representative Ro Khanna and San Francisco Mayor London Breed also voiced their opposition, echoing concerns from the tech industry that the bill could impede upon California’s leadership in AI innovation. Newsom recently said he was concerned the bill might have a “chilling effect” on AI development.

The bill had earned backing from some notable names in tech late last month in the days leading up to its passage by California’s legislature. Elon Musk unexpectedly voiced his support, even though he said it’s a “tough call and will make some people upset.” OpenAI rival Anthropic, which has a reputation for being safety-oriented, said the bill’s “benefits likely outweigh its costs,” though the company said some aspects remained “concerning or ambiguous to us.” 

Wiener had defended the bill, stressing that its provisions only apply to companies that spend more than $100 million on training large models or $10 million fine-tuning models, limits that would exempt most smaller startups. The lawmaker had also noted that Congress has been historically slow to regulate tech itself.

In announcing his veto, Newsom said he will consult with outside experts, including AI scholar and entrepreneur Fei-Fei Li, to “develop workable guardrails” on the technology and continue working with state legislature on the topic. 

The governor also signed a bill on Sunday, SB 896, that regulates how state agencies use AI.