The Center for AI Policy develops and advocates for policy to mitigate catastrophic risks from AI. 

We are funded by Lightspeed Grants, as well as several individual donors. We are nonpartisan and completely independent from foreign governments and commercial AI developers.

This person is an advisor for the Center for AI Policy.

Person

Learn more about the team behind the Center for AI Policy.

Example bio text for a team member at the Center for AI Policy.

Jason Green-Lowe, Executive Director

About us

This decade, AI could be powerful enough to cause global catastrophes

Within the next 3-10 years, AI projects could develop superintelligent AI: AI that is vastly smarter and more powerful than humans.

: “While superintelligence seems far off now, we believe it could arrive this decade.”

Progress towards superintelligence has been rapid.

In 2011, the best language models produced only 

 became one of the first models to consistently write coherent sentences

 outperformed ~90% of test takers on the SAT

For more detailed analyses on AI progress, we recommend 

AI systems are already being used to improve 

, such as hacking, weapons design, persuasion, and strategic planning.

 AI systems will soon be able to engineer pandemics, orchestrate novel cyberattacks, and disrupt critical infrastructure.

 that "the vast power of superintelligence… could lead to the disempowerment of humanity or even human extinction."

Developers don’t know how they will control very powerful AI.

no one knows how to control powerful AI systems

This is both a safety and a security issue.

AI systems could be misused to cause grievous harm.

Solving safety research questions requires time.

Yet unchecked competitive pressures compel companies to prioritize profits over safety.

 building catastrophically powerful systems.

The government needs to be able to safeguard AI development.

The government needs the capacity to rapidly identify and respond to AI risks. This requires:

More visibility into frontier AI development and technical expertise

Clear mechanisms to halt unsafe development in case of an emergency.

Better incentives that support developers prioritizing safety from the outset.

Monitoring of the advanced hardware used to train new AIs. If we know where that hardware is located, we will be more prepared for AI-related emergencies.

Monitoring of frontier AI development to give the government transparency into the development of these systems.

License frontier AI development, to incentivize AI safety research and allow for swift intervention in hazardous situations.

Strict liability for severe harms caused by AI systems, promoting accountability and improving incentivizes

Why AI policy

Recently, the Center for AI Policy (CAIP) submitted a 

 (NAIAC), a group of experts that advises the President on AI. The Department of Commerce established NAIAC as directed by Sec. 5104 of the 

, which outlined the topics that NAIAC will advise the President on. These topics, which are repeated in 

Definitional details vary, but AGI is commonly understood as an AI system that can match or outperform humans at an extremely wide range of tasks. Such dramatic capabilities are common in science fiction, but leading AI companies are working to quickly bring them into the real world—take Google DeepMind, whose Chief AGI Scientist 

 to see AGI by 2028. Given AGI's potential to perform complex tasks autonomously and at scale, 

 are concerned about grave catastrophes resulting from powerful AI systems like AGI.

This weighty topic is one inspiration behind CAIP’s comment to NAIAC, which discusses the time remaining before AI systems develop extremely dangerous capabilities, the reasons these capabilities could pose grave threats to national and global security, and the policies that the US government should enact to reduce these risks.

The text of CAIP’s comment is available below, and at 

 Thomas Larsen, Executive Director, Center for AI Policy

Thank you for the opportunity to share my perspective and concerns regarding the rapid evolution of AI and its implications for our national security. It is an honor to engage with members of this committee, and I am hopeful that we can navigate the challenges ahead by developing a deep understanding of the risks and taking appropriate action in the coming months.

The Center for AI Policy is an advocacy and research organization dedicated to reducing the 

 from future advanced AI systems. Our team has experience in AI safety research, communications, policy, and law. We believe that there is a significant chance that in the next 3–10 years, AI systems will pose significant threats to national security. We’re developing and advocating for policies that would prevent such dangerous AI systems from being created and deployed. Our current policy recommendations include compute governance, frontier AI regulation, and international cooperation on AI.

Catastrophic AI capabilities could emerge this decade

In the past 10 years, one of the primary drivers of AI capabilities has been 

 algorithms with more computational resources. During this time, the amount of compute used to train advanced AI systems has 

increased at a rate of about 4.2x per year

 of investment, are aiming to build smarter-than-human AI. AI systems will soon pass human-level performance on benchmarks in 

mathematics, logical reasoning, and computer programming

already being used to automate portions of AI research

. In the coming years, new AI breakthroughs will produce stronger AI systems that can contribute even more effectively to AI research. Furthermore, improved AI products will draw increased financial investment into AI research. Due to these self-reinforcing trends, 

 forecast that within the next decade there may be massive leaps in AI capabilities unfolding over the span of mere months.

"The idea that this stuff could actually get smarter than people [...] I thought it was way off. I thought it was 30 to 50 years or even longer away. Obviously, I no longer think that."
— 

The timeline to smarter-than-human AI systems that could pose catastrophic threats is a 

 in the AI community. It is difficult to make precise forecasts about future capabilities, and past predictions have been wrong with both 

. However, given current progress and development dynamics, it is unwise to be highly confident that smarter-than-human AI is far away.

Smarter-than-human AI systems would pose serious national security risks. A key subset of the risks come from 

: AI systems that advance objectives contrary to their designers’ intentions. There are a number of threat models for how this might come about, including 

 that act autonomously and deliberately to cause large-scale harm. For a good overview of catastrophic risks from misaligned AI systems, see Google DeepMind’s “

.” Many AI experts believe that alignment of smarter-than-human systems is an unsolved problem:

"We do not know, and probably aren't even close to knowing, how to align a superintelligence."
— 

Aside from misalignment concerns, bad actors could weaponize AI. For example, actors could use future AI systems to develop biological weapons, increase their military capabilities, or conduct large-scale cyberattacks on critical infrastructure. 

 are already starting to appear in the development of 

. AI capabilities will only increase from here, and future AI systems may begin significantly assisting with creating weapons of mass destruction.

Addressing catastrophic threats from AI should be a national security priority of the United States. The government needs to prevent the development and deployment of dangerous AI models, both domestically and abroad. Meanwhile, the U.S. should encourage innovation in AI safety to allow for the development of AI systems that can provide massive benefits, while avoiding catastrophic risks.

My team at the Center for AI Policy supports the following proposals: implementing 

 is the difficult-to-obtain ingredient that has driven the current wave of AI. With access to sufficient AI hardware, entities can build powerful AI systems; without this access, it is impractical to build powerful AI systems. An initial step towards robust hardware controls is building a 

 that tracks basic information like location, ownership, use case, and cluster size.

 The government needs the capacity to rapidly identify and respond to AI threats. This is why we’re advocating for the establishment of a 

federal office focused on frontier AI risks

. Companies should need to apply for a license from the regulator before they can develop or deploy a new frontier AI model. Licenses should be granted based on whether the companies’ planned safety and security measures are adequate for the anticipated risks, both from weaponization and misalignment. Beyond licensing, the regulator should monitor AI development and risks. If it identifies a clear national security emergency, it should be empowered to pause dangerous AI projects until safeguards can be put into place.

Some AI companies have existing voluntary standards, such as 

. Unfortunately, voluntary standards will not be adopted by everyone and may be too weak to prevent catastrophes, in large part due to the financial incentive to continue scaling even when AI systems pose severe risks, so the government should step in to require sufficient safeguards.

 The U.S. should aim for international cooperation with nations agreeing to prevent the development of smarter-than-human AI systems until adequate safety techniques are developed. 

 is critical to avoid a “race to the bottom” in AI, where each actor invests less into safety in order to gain a competitive edge. For non-cooperative countries, the U.S. should use 

 to cripple their ability to build advanced AI systems by preventing them from importing or building production capacity for advanced chips. To enforce these controls, the Bureau of Industry and Security (BIS) at the Department of Commerce should 

Thank you for considering the proposals outlined in this document. The rapidly advancing landscape of AI presents both challenges and opportunities. With the right policies in place, we can mitigate risks of this transformative technology, allowing us to reap the benefits. I look forward to engaging further with this esteemed committee as it safeguards our national interests.

CAIP's comment to the National AI Advisory Committee

NAIAC public comment

The Blumenthal-Hawley Framework Outlines AI Safety Legislation that America Desperately Needs

Senators Richard Blumenthal (D-CT) and Josh Hawley (R-MO) introduced a 

 “this framework is a strong and positive step towards effectively regulating artificial intelligence.”

The Center for AI Policy (CAIP) seconds this positive sentiment.

We wholeheartedly support the following elements of the framework and urge lawmakers to codify them:

licenses for advanced general-purpose AI models

controls against unintended proliferation

In this piece, we explain why these five provisions are necessary, and suggest directions for further progress.

The framework’s first proposal is to “

establish a licensing regime administered by an independent oversight body

sophisticated general-purpose AI models (e.g., GPT-4)

,” companies would have to apply for and receive a license.

Advanced general-purpose AI models are already developing 

assisting non-experts in causing pandemics

generating cunning strategies for mass murder

 at a rapid pace, and leading AI companies 

Given the unpredictable nature of forthcoming AI capabilities and the risks already observed, 

. Without oversight, the incoming whirlwind of hazards poses a grave threat to American lives and global security.

Legislative text codifying the Hawley-Blumenthal framework must include 

specific criteria for the high-risk AI systems that need a license

. These criteria should be developed in consultation with diverse stakeholders. As a starting point for these conversations, CAIP believes that systems surpassing any of the following thresholds should qualify as high risk:

 used for training: over a septillion operations (i.e., 10^24 

cost of development: over $50M in employee time and hardware resources,

benchmark performance: over 80% average score on 

, as it gains better knowledge of the risks and architectures of systems that surpass GPT-4. Importantly, the thresholds must adapt to account for 

improvements in the efficiency of learning algorithms

—what matters is not just the number of computations an AI model utilizes, but the capability it derives from those computations. One possible approach can be seen in Anthropic, a leading AI lab, which considers “

 to swiftly approve models that clearly pose no catastrophic risks, such as AI for autonomous transportation, demand forecasting, and fraud detection.

 to receive a license, including programs for “risk management, pre-deployment testing, data governance, and adverse incident reporting.”

CAIP believes that runaway AI advancement could pose 

. Threats to national and global security may emerge from 

. Worryingly, within the wide-ranging spectrum of AI hazards, 

. Government action is needed to ensure wide adoption of such measures, and to incentivize pressing work on 

unsolved problems in technical AI safety research

The following three topics warrant particular attention when licensing powerful AI models. First, 

. The licensing body should consider the engineering techniques used to enhance safety, the effectiveness of these guardrails, and any significant limitations. This is especially important given the 

 of whether engineers will find methods for controlling future models: 

 believes that “our current alignment techniques will not scale to superintelligence.”

. The licensing body should evaluate the information and 

 that the developer intends to share, the intended audience for this property, and the measures taken to mitigate unauthorized access. Possible questions are available in Senators Hawley and Blumenthal’s 

. The licensing body should understand the ways the model is able to cause harm, the strategies the developer used to identify these risks, and the model’s 

 For instance, developers should supply analyses of their model’s competencies in 

The framework calls for an AI oversight body that can “

 on technological developments and economic impacts of AI.”

government needs to understand the technology

, there is a consistent need for up-to-date analyses. One benefit of these analyses is progress towards 

. Government experts should be actively searching for potential risks, and strengthening America’s response capacity for AI-related emergencies.

To succeed in these goals, the government needs a regulatory body to serve as a 

trusted source of crucial information on AI

liaise with similar governmental organizations in other nations

, one metric that the entity should track is the 

distribution and flow of high-performance 

 for creating leading systems like GPT-4. By watching where these chips go, the government can understand which actors can train powerful AI models.

 for large-scale AI training retails for 

 to build supercomputers. Parties involved in the trade of such powerful, specialized chips should be required to 

, enabling the government to maintain an accurate 

clearly define how AI companies will be held accountable

 for the harms they cause. “Congress should ensure that AI companies can be held liable through oversight body enforcement and private rights of action when their models and systems breach privacy, violate civil rights, or otherwise cause cognizable harms.”

When developers face clearer consequences for reckless decisions, they have an increased incentive to ensure safety and security. Besides reducing risks, this encourages innovation in 

. The financial stakes of potential damages will compel companies to research and implement safety measures.

AI liability is the will of the American people

. 73% of voters believe that “AI companies should be liable for harms from technologies they create,” according to 

. Only 11% think companies should not be held liable.

 for tangible harm caused by advanced AI systems. In severe cases of misuse by a malicious actor lacking sufficient assets, the original AI developer may escape financial penalty. Implementing joint and several liability allows for suing the developer as well, increasing the likelihood of repercussions for negligence and 

For similar reasons, CAIP suggests adopting an explicit 

 for developers of advanced AI. Most states currently apply a complex multi-factor test when deciding whether a company owes a duty to the public to be reasonably cautious, and the results of these multi-factor tests are difficult to predict. To increase the certainty of penalties for carelessness, legislation should specify that anyone working on advanced, general-purpose AI must always be careful not to harm the public, and that if they do harm the public, they must pay damages.

 regime for AI developers whose models cause 

, such as wrongful death, physical injury, or property damage. Most civil lawsuits ask whether the defendant followed the “standard of care,” i.e., whether the defendant followed the textbook-approved procedures or detailed best practices for safely carrying out their work. But AI technology is 

; therefore, any sufficiently specific standard of care is vulnerable to growing obsolete within a few years. Strict liability solves this problem by removing the need to prove negligence. If an AI system causes over $100 million in damages, it is fair to assume that the developer was careless.

The framework calls for “export controls, sanctions, and other legal restrictions to 

limit the transfer of advanced AI models, hardware and related equipment

. Due to the serious risks of AI advancement, especially in the hands of adversary nations, the US must take deliberate action to control the proliferation and growth of boundary-pushing AI capabilities.

If breakthroughs remain heavily concentrated in America, then it becomes easier for advanced general-purpose AI developers to avoid 

communicate openly about safety challenges and 

 if there is a credible risk of large-scale harm, and this cooperation will be significantly more difficult if scientists in China, Iran, Russia, and so forth can quickly replicate every American breakthrough.

The Bureau of Industry and Security (BIS) is critical in maintaining America’s AI dominance: it is using 

 to cut China’s access to advanced AI chips. However, BIS has 

 for robustly enforcing these controls. To prevent China from 

enforce robust information security requirements

 for cutting-edge AI developers, to defend America’s trailblazing innovations against unauthorized access by foreign adversaries and malicious actors.

Foreign tensions must not blind the US to the global-scale challenges of AI, and the urgent need for multilateral dialogues like the 

. To lay the building blocks for global cooperation, the US should 

. America must lead the world in promoting AI safety—if the US barrels ahead incautiously and unilaterally, then it risks losing an AI race not to adversaries, but to AI itself.

CAIP commends Senators Blumenthal and Hawley for their AI policy framework and underscores our alignment on five essential components: licensing powerful models, mandating safety measures, tracking AI progress, holding developers accountable, and managing AI proliferation. Building upon this robust foundation, CAIP offers specific, targeted recommendations to further elevate the framework's efficacy. As the next step, CAIP calls on Congress to prioritize passing this framework into law.

AI safety legislation that America desperately needs

Strengths of Hawley and Blumenthal's framework

Our takeaways from July 25 Senate Judiciary Hearing

On July 25, 2023, the Senate Judiciary Subcommittee on Privacy, Technology, and Law held a hearing titled “Oversight of AI: Principles for Regulation.” The Center for AI Policy commends the witnesses and committee members for their substantive discussion of existential risks from AI, as well as policy proposals intended to reduce those risks.

We have selected some quotes about global security risks from AI. The witnesses and committee members also discussed a variety of other topics, including concerns about data privacy, misinformation, and jobs. You can listen to the full hearing 

“What I have heard again and again and again… is "scary"... An intelligence device out of control, autonomous, self-replicating, potentially creating diseases, pandemic-grade viruses or other kinds of evils purposely engineered by people or simply the result of mistakes, not malign intention.”

“You have provided objective, fact-based views on what the dangers are... potentially even human extinction.”

“I’ve come to the conclusion that we need some kind of regulatory agency.”

“A number of you have put the timeline at two years before we see some of the biological [dangers and] most severe dangers. It may be shorter because the pace of development is not only stunningly fast, it is also accelerated at a stunning pace because of the quantity of chips, the speed of chips, the effectiveness of algorithms.”

“Superhuman AI evokes for me artificial intelligence that could on its own develop a pandemic virus, on its own decide Joe Biden shouldn’t be our next president… And I think that argues for urgency.”

“I’m less interested in the [AI] corporations’ profitability. In fact, I'm not interested in that at all. I’m interested in protecting the rights of American workers and American families and American consumers against these massive companies that threaten to become a total law unto themselves.”

“Will the Senate actually act? … We’ve had a lot of talk, but now is the time for action. And I think if the urgency of the new generative AI technology does not make that clear to folks, then you’ll never be convinced.”

“A straightforward extrapolation of today’s systems to those we expect to see in two to three years suggests a substantial risk that AI systems will be able to fill in all the missing pieces, enabling many more actors to carry out large-scale biological attacks. We believe this represents a grave threat to U.S. national security.”

“New AI models should have to pass a rigorous battery of safety tests before they can be released to the public at all, including tests by third parties and national security experts in government.”

Yoshua Bengio, Recipient of the Turing Award

“Require licenses… and restrict AI systems with unacceptable levels of risk.”

“[We need] to bring expertise in national security, in bioweapons, chemical weapons, and AI people together. [These organizations] shouldn’t be for profit… We shouldn’t mix the objective of making money… with the objective of defending humanity against a potential rogue AI.”

“This research in AI and international security should be conducted with several highly secure and decentralized labs operating under multilateral oversight to mitigate an AI arms race… We must therefore allocate substantial additional resources to safeguard our future, at least as much as we are collectively globally investing in increasing the capabilities of AI.”

“Viruses, computer or biological viruses don’t see any border. So, we need to make sure there’s an international effort in terms of these safety measures. We need to agree with China on these safety measures... And we need to work with our allies on these countermeasures.”

Stuart Russell, Professor of Computer Science at UC Berkeley

“Alan Turing, the founder of Computer Science, warned in 1951 that once AI outstrips our feeble powers, we should have to expect the machines to take control. We have pretty much completely ignored this warning.”

“Systems that break the rules must be recalled from the market for anything from defaming real individuals to helping terrorists build biological weapons.”

“Now, developers may argue that preventing these behaviors is too hard because LLMs have no notion of truth and are just trying to help. This is no excuse. Eventually, and the sooner the better, I would say, we will develop forms of AI that are provably safe and beneficial… Until then, we need real regulation and a pervasive culture of safety.”

“I think there’s no doubt that we’re going to have to have an agency.”

Takeaways from July 25 Senate Judiciary Hearing

 content. If generative AI like ChatGPT generates libel, then OpenAI is liable under existing law.

It’s not clear whether courts will include “training an AI” as a “fair use” of a book or a movie under copyright law. In the most similar prior case, for Google Books, the court seemed to base its ruling on the value that Google Books would add to society.

Strictly enforcing copyright can help some artists get compensated, but it doesn’t offer a long-term solution for unemployment caused by ongoing automation of the workforce.

Generative AI hallucinates and discriminates roughly as often as humans. It has problems because humans have problems, and it’s trained on human behavior. NIST’s AI Risk Management Framework, if properly and vigorously applied, would go a long way toward constructively addressing these problems.

Existing law doesn’t provide a solution for the catastrophic risks posed by advanced AI, like bioweapons and automated hacking. CAIP’s bill would help address these risks with a strict liability regime and mandatory safety measures.

What are the barriers to holding companies liable for harms committed by their AIs?

Identifying the bad actor can be a challenge. For example, if a large corporate developer makes an AI with a sloppy third-party interface that doesn’t screen out security threats, and then a freelance designer adds a potentially dangerous extension for the AI, and then the end user installs that plug-in and uses it to develop a bioweapon, who caused the harm? The corporate developer, the freelance designer, or the end user?

Our notes on AI, liability, and copyright

General Expression of Interest

Other job openings

Job Opening: Communications Director