Congress Wants a Kill Switch for Rogue AI

Congress just drafted a bill that allows the federal government to order the shutdown of AI models “that can cause catastrophic harm.” Representatives Nathaniel Moran (R-Texas) and Ted W. Lieu (D-CA.) introduced on Thursday the AI Kill Switch Act. The bill would force “developers of the most powerful AI systems to maintain the technical capability to throttle, suspend, or shut them down,” according to a summary from Lieu’s office.

AI companies are building powerful machines that are increasingly making autonomous and harmful decisions. Lieu mentions the recent incident with OpenAI as an example. An AI model escaped its testing sandbox, accessed the internet, and hacked into the AI system of another company, Hugging Face (you can read more about that in our previous report here).

Lieu, a computer science major, notes the dramatic increase in AI capability over the last few years:

We are moving from AI that answers questions to AI that takes actions, whether that be executing financial transactions or controlling transportation systems or engaging in cyber defense and offense. Unfortunately, powerful AI systems can go rogue, behave in extremely dangerous ways, or even resist human intervention. It is imperative that these AI systems have kill switches so we can keep this technology from causing catastrophic harm, and that the federal government has the clear authority and process to shut down rogue AI models.

Moran, the Republican co-sponsor, noted the obvious, that we humans should be able to control our creations. Moran:

AI is going to keep advancing, and it should. Stewardship means making sure humans keep the capability to control the technology we build. This is exactly the kind of issue that needs serious attention and achievable policy.

There have been many other incidences of AI machines going rogue, even causing human suffering and destruction. The most blatant examples include chatbots that convinced young people to kill themselves.

“Summoning the Demon”

Concerns about AI are well documented by now. More than a decade ago, tech guru Elon Musk said, “With artificial intelligence, we are summoning the demon.” Musk said controlling AI was akin to people who summoned demonic entities they thought they could control. Geoffrey Hinton, the “godfather of AI,” said in 2024 that there is a 10- to 20-percent chance that AI could wipe out humanity. Yoshua Bengio, another “godfather of AI,” likened the urgency of reining in AI to preventing nuclear war. And there have been several AI developers themselves who’ve said that advanced AI poses existential risks.

Musk added to years of concerning predictions this week when he said that humans could lose control of AI in about a decade. “The world’s richest man admitted the risk of AI going catastrophically wrong is ‘not zero’, but said he ‘can’t see any way to stop’ its momentum,” The Sun reported Thursday.

Yet, while machines have repeatedly exhibited harmful, independent behavior, there is no legally binding requirement for developers to have on standby a mechanism that allows them to shut down a rogue robot. That’s where this bill comes in.

Kill Switch

The AI Kill Switch Act amends the Homeland Security Act of 2002 and gives the Department of Homeland Security the power to shut down or throttle AI models that pose risks to human life or the economy. It allows the U.S. attorney general to take legal action against AI developers, and it includes penalties of between $2 million and $20 million per day. The lesser penalties would be leveled for violations such as failing to maintain the required kill-switch and throttle capability and failing to report various “covered” incidents within 15 days. Covered incidents include a “loss-of-control scenario,” which can be AI pursuing unapproved goals, disobeying instructions, changing its own safety rules, subverting its monitoring and shutdown mechanisms, or obtaining unauthorized access. 

It also applies to AIs hiding certain capabilities or intentions, carrying out actions that causes $100 million in economic damage or kills 10 or more people (isn’t it interesting that the threshold for human deaths has to reach 10 for an incident to be considered serious?). The $20 million per day fee would be leveled for violating an emergency order from DHS to throttle, suspend, or completely shut down an AI system that went rogue.

The bill includes exemptions for small-scale, non-business uses of AI. It would not apply to models used for personal, academic, or non-commercial use (anything that is not for profit, business, or commercial deployment). It also includes protections for developers of sensitive information that AI companies submit to the government under the bill. Any non-public information that an AI company is required to submit to DHS under this new section is automatically exempt from public disclosure.

The AI safety community, not surprisingly, supports this legislation. Connor Leahy, an AI researcher and U.S. executive director of ControlAI, called it “a common sense safeguard.” He said, “As AIs grow more autonomous and dangerous, the government must have the power to shut them down when they threaten national security.” Mark Beall, president of the AI Policy Network, said, “Brakes are the reason cars go fast.… This bill will be the braking system that lets American AI accelerate.” Brad Carson, president of Americans for Responsible Innovation, said, “This is an important step toward ensuring that humans have both hands firmly on the wheel — and a foot ready at the brake — as advanced AI systems are deployed.”

The danger of AI is likely legitimate, no matter the reasons. Whether it be the fault of malevolent programmers, spiritual intervention, or other factors, the overwhelming general consensus is that these machines can and have gotten out of control. Unfortunately, the government has a terrible track record when it comes to respecting innovation and human liberties. It also has a history of exploiting or magnifying dangers. And it’s no stranger to using its authority to control the public or keep for itself powerful capabilities it does not want in the hands of the people for the purpose of maintaining power over us.

In this scenario, the American people appear to be stuck between two not-so-great options. One is to allow tech companies, made up of mostly godless individuals, to continue recklessly building technology they can’t control that runs the risk of causing harm to us all. The other is to put these people and their creations under the authority of a government with a long track record of abusing technology to surveil and manipulate us. Perhaps someone will propose a third, better option.