The Center for AI Policy develops and advocates for policy to mitigate catastrophic risks from AI. 

We are funded by Lightspeed Grants, as well as several individual donors. We are nonpartisan and completely independent from foreign governments and commercial AI developers.

This person is an advisor for the Center for AI Policy.

Person

Learn more about the team behind the Center for AI Policy.

Example bio text for a team member at the Center for AI Policy.

Jason Green-Lowe, Executive Director

About us

This decade, AI could be powerful enough to cause global catastrophes

Within the next 3-10 years, AI projects could develop superintelligent AI: AI that is vastly smarter and more powerful than humans.

: “While superintelligence seems far off now, we believe it could arrive this decade.”

Progress towards superintelligence has been rapid.

In 2011, the best language models produced only 

 became one of the first models to consistently write coherent sentences

 outperformed ~90% of test takers on the SAT

For more detailed analyses on AI progress, we recommend 

AI systems are already being used to improve 

, such as hacking, weapons design, persuasion, and strategic planning.

 AI systems will soon be able to engineer pandemics, orchestrate novel cyberattacks, and disrupt critical infrastructure.

 that "the vast power of superintelligence… could lead to the disempowerment of humanity or even human extinction."

Developers don’t know how they will control very powerful AI.

no one knows how to control powerful AI systems

This is both a safety and a security issue.

AI systems could be misused to cause grievous harm.

Solving safety research questions requires time.

Yet unchecked competitive pressures compel companies to prioritize profits over safety.

 building catastrophically powerful systems.

The government needs to be able to safeguard AI development.

The government needs the capacity to rapidly identify and respond to AI risks. This requires:

More visibility into frontier AI development and technical expertise

Clear mechanisms to halt unsafe development in case of an emergency.

Better incentives that support developers prioritizing safety from the outset.

Monitoring of the advanced hardware used to train new AIs. If we know where that hardware is located, we will be more prepared for AI-related emergencies.

Monitoring of frontier AI development to give the government transparency into the development of these systems.

License frontier AI development, to incentivize AI safety research and allow for swift intervention in hazardous situations.

Strict liability for severe harms caused by AI systems, promoting accountability and improving incentivizes.

Why AI policy

On October 7th, 2022, the Commerce Department's Bureau of Industry and Security (BIS) published 

 that affected AI hardware and the corresponding supply chain.

Just over one year later, on October 17th, 2023, BIS 

 these export controls. The revisions included updated rules on 

Last month, the Center for AI Policy commented on these revised 2023 export controls affecting AI hardware. You can read our full comment 

Comment on the Revised 2023 AI Hardware Export Controls

Center for AI Policy (CAIP) Statement on the Creation of a House AI Task Force



Washington, DC - February 22 - Center for AI Policy (CAIP) Executive Director Jason Green-Lowe issued the following statement regarding the creation of a House AI Task Force:

"The Center for AI Policy (CAIP) congratulates US House Speaker Mike Johnson (R-LA) and Minority Leader Hakeem Jeffries (D-NY) on the formation of a bipartisan 24-member task force charged with producing a report outlining guiding principles and policy recommendations for AI legislation that Congress will pass which equally fosters innovation and reduces the risks of this burgeoning technology.

"CAIP is pleased to have Californian representatives Ted Lieu (D-CA) and Jay Obernolte (R-CA) lead the committee. Both have intense interest, knowledge, and experience with AI and are well-versed in the challenges and opportunities of this technology.

"CAIP looks forward to working with all House AI Task Force members to advance principles and policies that identify and respond to national security risks such as the disruption of critical infrastructure and disinformation campaigns hampering America's elections.

"CAIP is hopeful that the House AI Task Force will see the need for more visibility into frontier AI development, increased technical expertise within the US government, creating clear mechanisms to halt the development of unsafe AI, and foster incentives for AI developers to prioritize safety from the start."



Rapidly advancing AI poses risks to national security. The Center for AI Policy (CAIP) is a nonpartisan organization that advocates for policies to mitigate catastrophic risks from AI.

Our Exectutive Director's statment regarding the creation of a House AI Task Force

Statement on the Creation of a House AI Task Force

Recently, the Center for AI Policy (CAIP) submitted a 

 (NAIAC), a group of experts that advises the President on AI. The Department of Commerce established NAIAC as directed by Sec. 5104 of the 

, which outlined the topics that NAIAC will advise the President on. These topics, which are repeated in 

Definitional details vary, but AGI is commonly understood as an AI system that can match or outperform humans at an extremely wide range of tasks. Such dramatic capabilities are common in science fiction, but leading AI companies are working to quickly bring them into the real world—take Google DeepMind, whose Chief AGI Scientist 

 to see AGI by 2028. Given AGI's potential to perform complex tasks autonomously and at scale, 

 are concerned about grave catastrophes resulting from powerful AI systems like AGI.

This weighty topic is one inspiration behind CAIP’s comment to NAIAC, which discusses the time remaining before AI systems develop extremely dangerous capabilities, the reasons these capabilities could pose grave threats to national and global security, and the policies that the US government should enact to reduce these risks.

The text of CAIP’s comment is available below, and at 

 Thomas Larsen, Executive Director, Center for AI Policy

Thank you for the opportunity to share my perspective and concerns regarding the rapid evolution of AI and its implications for our national security. It is an honor to engage with members of this committee, and I am hopeful that we can navigate the challenges ahead by developing a deep understanding of the risks and taking appropriate action in the coming months.

The Center for AI Policy is an advocacy and research organization dedicated to reducing the 

 from future advanced AI systems. Our team has experience in AI safety research, communications, policy, and law. We believe that there is a significant chance that in the next 3–10 years, AI systems will pose significant threats to national security. We’re developing and advocating for policies that would prevent such dangerous AI systems from being created and deployed. Our current policy recommendations include compute governance, frontier AI regulation, and international cooperation on AI.

Catastrophic AI capabilities could emerge this decade

In the past 10 years, one of the primary drivers of AI capabilities has been 

 algorithms with more computational resources. During this time, the amount of compute used to train advanced AI systems has 

increased at a rate of about 4.2x per year

 of investment, are aiming to build smarter-than-human AI. AI systems will soon pass human-level performance on benchmarks in 

mathematics, logical reasoning, and computer programming

already being used to automate portions of AI research

. In the coming years, new AI breakthroughs will produce stronger AI systems that can contribute even more effectively to AI research. Furthermore, improved AI products will draw increased financial investment into AI research. Due to these self-reinforcing trends, 

 forecast that within the next decade there may be massive leaps in AI capabilities unfolding over the span of mere months.

"The idea that this stuff could actually get smarter than people [...] I thought it was way off. I thought it was 30 to 50 years or even longer away. Obviously, I no longer think that."
— 

The timeline to smarter-than-human AI systems that could pose catastrophic threats is a 

 in the AI community. It is difficult to make precise forecasts about future capabilities, and past predictions have been wrong with both 

. However, given current progress and development dynamics, it is unwise to be highly confident that smarter-than-human AI is far away.

Smarter-than-human AI systems would pose serious national security risks. A key subset of the risks come from 

: AI systems that advance objectives contrary to their designers’ intentions. There are a number of threat models for how this might come about, including 

 that act autonomously and deliberately to cause large-scale harm. For a good overview of catastrophic risks from misaligned AI systems, see Google DeepMind’s “

.” Many AI experts believe that alignment of smarter-than-human systems is an unsolved problem:

"We do not know, and probably aren't even close to knowing, how to align a superintelligence."
— 

Aside from misalignment concerns, bad actors could weaponize AI. For example, actors could use future AI systems to develop biological weapons, increase their military capabilities, or conduct large-scale cyberattacks on critical infrastructure. 

 are already starting to appear in the development of 

. AI capabilities will only increase from here, and future AI systems may begin significantly assisting with creating weapons of mass destruction.

Addressing catastrophic threats from AI should be a national security priority of the United States. The government needs to prevent the development and deployment of dangerous AI models, both domestically and abroad. Meanwhile, the U.S. should encourage innovation in AI safety to allow for the development of AI systems that can provide massive benefits, while avoiding catastrophic risks.

My team at the Center for AI Policy supports the following proposals: implementing 

 is the difficult-to-obtain ingredient that has driven the current wave of AI. With access to sufficient AI hardware, entities can build powerful AI systems; without this access, it is impractical to build powerful AI systems. An initial step towards robust hardware controls is building a 

 that tracks basic information like location, ownership, use case, and cluster size.

 The government needs the capacity to rapidly identify and respond to AI threats. This is why we’re advocating for the establishment of a 

federal office focused on frontier AI risks

. Companies should need to apply for a license from the regulator before they can develop or deploy a new frontier AI model. Licenses should be granted based on whether the companies’ planned safety and security measures are adequate for the anticipated risks, both from weaponization and misalignment. Beyond licensing, the regulator should monitor AI development and risks. If it identifies a clear national security emergency, it should be empowered to pause dangerous AI projects until safeguards can be put into place.

Some AI companies have existing voluntary standards, such as 

. Unfortunately, voluntary standards will not be adopted by everyone and may be too weak to prevent catastrophes, in large part due to the financial incentive to continue scaling even when AI systems pose severe risks, so the government should step in to require sufficient safeguards.

 The U.S. should aim for international cooperation with nations agreeing to prevent the development of smarter-than-human AI systems until adequate safety techniques are developed. 

 is critical to avoid a “race to the bottom” in AI, where each actor invests less into safety in order to gain a competitive edge. For non-cooperative countries, the U.S. should use 

 to cripple their ability to build advanced AI systems by preventing them from importing or building production capacity for advanced chips. To enforce these controls, the Bureau of Industry and Security (BIS) at the Department of Commerce should 

Thank you for considering the proposals outlined in this document. The rapidly advancing landscape of AI presents both challenges and opportunities. With the right policies in place, we can mitigate risks of this transformative technology, allowing us to reap the benefits. I look forward to engaging further with this esteemed committee as it safeguards our national interests.

CAIP's comment to the National AI Advisory Committee

NAIAC public comment

The annual Munich Security Conference (MSC) is one of the leading forums for discussing the most pressing challenges to international security. A central topic for this year’s conference, running February 16-19, will be AI.

The past few years have seen rapid advances in AI. Soon, leaders across industry and academia warn the technology will pose challenges on par with nuclear weapons. For global security, we need to get this technology right.

Vice President Kamala Harris will lead the US delegation to the MSC, which will include Secretary of State Antony Blinken, Secretary of Homeland Security Alejandro Mayorkas, FBI Director Chris Wray, Deputy Attorney General Lisa Monaco, and Deputy National Security Advisor Anne Neuberger, as well as Sen. Sheldon Whitehouse (D-RI) who will guide a congressional delegation.

Here are three questions with relevant answers that the Center for AI Policy will be looking to hear discussed at the MSC.

AI is evolving rapidly, and there is great uncertainty about how exactly its risks will manifest. Still, experts agree AI risks will be significant, and governments must be ready to respond to them. How can governments become more adept in assessing and responding to AI risks?

Center for AI Policy (CAIP) thinks it is vital to develop the government’s technical capacity.

CAIP believes these three policy moves would be positive steps forward.

First, ensure more leading AI researchers are brought into government, where they can help policymakers better understand developments in AI. Second, resolve that policies ensuring AI safety are seen and supported as a national security priority. Third, support the efforts of AI Safety Institutes such as those in the US, UK, and Japan, which will strengthen understanding of AI risks, develop standards, and provide a means for international coordination.

As US companies build increasingly powerful AI models, adversaries will increasingly try to steal them. Currently, US companies are far from having the security to defend against well-resourced adversaries. Should governments support AI companies in bolstering their security?

The Center for AI Policy (CAIP) believes it is urgent that frontier AI developers and governments work together to implement the best security measures available now and invest in R&D to defend against well-resourced attackers as soon as possible.

No government or developer can achieve top security alone. Governments and developers will need to work together to safeguard frontier AI.

One concern with AI is that competitive pressures will lead AI developers to keep making AI systems more powerful, even when the reliability of those systems is in question. Do you agree we need to ensure our most advanced AI systems are reliable, and if so, how can we do that?

The Center for AI Policy (CAIP) thinks it is critical that AI systems are only developed and deployed only when the risks are manageable. For our most advanced AI systems, this means developers need practices for ensuring reliability before deploying.

Governments need to coordinate internationally on standards for risk management to make this possible. 

Additional information on the 2024 Munich Security Conference agenda:

https://securityconference.org/en/msc-2024/

YouTube Channel of 2024 Conference panels and discussions:

Our thoughts on AI topics pressing for global security to discuss this MSC 

Three Questions for the Munich Security Conference

At an inflection point such as ours, we should not squander this opportunity to establish real oversight over the largest technology companies. Congress must create a new agency with the resources and meaningful enforcement authority to regulate these firms in a comprehensive, considered, and continuous manner.

Those were the words of Senators Michael Bennet (D-CO), Lindsey Graham (R-SC), Elizabeth Warren (D-MA), and Peter Welch (D-VT), in a 

 to Majority Leader Chuck Schumer (D-NY) last week.

The Center for AI Policy seconds this sentiment, especially in the context of overseeing advanced general-purpose AI models like 

Importantly, the American public endorses this view. As far as we're aware, every single quality poll has found that the majority of Americans support creating a federal office to regulate AI.

The majority of Americans support creating an AI oversight body

Public Opinion on a Federal Office for AI

The Blumenthal-Hawley Framework Outlines AI Safety Legislation that America Desperately Needs

Senators Richard Blumenthal (D-CT) and Josh Hawley (R-MO) introduced a 

 “this framework is a strong and positive step towards effectively regulating artificial intelligence.”

The Center for AI Policy (CAIP) seconds this positive sentiment.

We wholeheartedly support the following elements of the framework and urge lawmakers to codify them:

licenses for advanced general-purpose AI models

controls against unintended proliferation

In this piece, we explain why these five provisions are necessary, and suggest directions for further progress.

The framework’s first proposal is to “

establish a licensing regime administered by an independent oversight body

sophisticated general-purpose AI models (e.g., GPT-4)

,” companies would have to apply for and receive a license.

Advanced general-purpose AI models are already developing 

assisting non-experts in causing pandemics

generating cunning strategies for mass murder

 at a rapid pace, and leading AI companies 

Given the unpredictable nature of forthcoming AI capabilities and the risks already observed, 

. Without oversight, the incoming whirlwind of hazards poses a grave threat to American lives and global security.

Legislative text codifying the Hawley-Blumenthal framework must include 

specific criteria for the high-risk AI systems that need a license

. These criteria should be developed in consultation with diverse stakeholders. As a starting point for these conversations, CAIP believes that systems surpassing any of the following thresholds should qualify as high risk:

 used for training: over a septillion operations (i.e., 10^24 

cost of development: over $50M in employee time and hardware resources,

benchmark performance: over 80% average score on 

, as it gains better knowledge of the risks and architectures of systems that surpass GPT-4. Importantly, the thresholds must adapt to account for 

improvements in the efficiency of learning algorithms

—what matters is not just the number of computations an AI model utilizes, but the capability it derives from those computations. One possible approach can be seen in Anthropic, a leading AI lab, which considers “

 to swiftly approve models that clearly pose no catastrophic risks, such as AI for autonomous transportation, demand forecasting, and fraud detection.

 to receive a license, including programs for “risk management, pre-deployment testing, data governance, and adverse incident reporting.”

CAIP believes that runaway AI advancement could pose 

. Threats to national and global security may emerge from 

. Worryingly, within the wide-ranging spectrum of AI hazards, 

. Government action is needed to ensure wide adoption of such measures, and to incentivize pressing work on 

unsolved problems in technical AI safety research

The following three topics warrant particular attention when licensing powerful AI models. First, 

. The licensing body should consider the engineering techniques used to enhance safety, the effectiveness of these guardrails, and any significant limitations. This is especially important given the 

 of whether engineers will find methods for controlling future models: 

 believes that “our current alignment techniques will not scale to superintelligence.”

. The licensing body should evaluate the information and 

 that the developer intends to share, the intended audience for this property, and the measures taken to mitigate unauthorized access. Possible questions are available in Senators Hawley and Blumenthal’s 

. The licensing body should understand the ways the model is able to cause harm, the strategies the developer used to identify these risks, and the model’s 

 For instance, developers should supply analyses of their model’s competencies in 

The framework calls for an AI oversight body that can “

 on technological developments and economic impacts of AI.”

government needs to understand the technology

, there is a consistent need for up-to-date analyses. One benefit of these analyses is progress towards 

. Government experts should be actively searching for potential risks, and strengthening America’s response capacity for AI-related emergencies.

To succeed in these goals, the government needs a regulatory body to serve as a 

trusted source of crucial information on AI

liaise with similar governmental organizations in other nations

, one metric that the entity should track is the 

distribution and flow of high-performance 

 for creating leading systems like GPT-4. By watching where these chips go, the government can understand which actors can train powerful AI models.

 for large-scale AI training retails for 

 to build supercomputers. Parties involved in the trade of such powerful, specialized chips should be required to 

, enabling the government to maintain an accurate 

clearly define how AI companies will be held accountable

 for the harms they cause. “Congress should ensure that AI companies can be held liable through oversight body enforcement and private rights of action when their models and systems breach privacy, violate civil rights, or otherwise cause cognizable harms.”

When developers face clearer consequences for reckless decisions, they have an increased incentive to ensure safety and security. Besides reducing risks, this encourages innovation in 

. The financial stakes of potential damages will compel companies to research and implement safety measures.

AI liability is the will of the American people

. 73% of voters believe that “AI companies should be liable for harms from technologies they create,” according to 

. Only 11% think companies should not be held liable.

 for tangible harm caused by advanced AI systems. In severe cases of misuse by a malicious actor lacking sufficient assets, the original AI developer may escape financial penalty. Implementing joint and several liability allows for suing the developer as well, increasing the likelihood of repercussions for negligence and 

For similar reasons, CAIP suggests adopting an explicit 

 for developers of advanced AI. Most states currently apply a complex multi-factor test when deciding whether a company owes a duty to the public to be reasonably cautious, and the results of these multi-factor tests are difficult to predict. To increase the certainty of penalties for carelessness, legislation should specify that anyone working on advanced, general-purpose AI must always be careful not to harm the public, and that if they do harm the public, they must pay damages.

 regime for AI developers whose models cause 

, such as wrongful death, physical injury, or property damage. Most civil lawsuits ask whether the defendant followed the “standard of care,” i.e., whether the defendant followed the textbook-approved procedures or detailed best practices for safely carrying out their work. But AI technology is 

; therefore, any sufficiently specific standard of care is vulnerable to growing obsolete within a few years. Strict liability solves this problem by removing the need to prove negligence. If an AI system causes over $100 million in damages, it is fair to assume that the developer was careless.

The framework calls for “export controls, sanctions, and other legal restrictions to 

limit the transfer of advanced AI models, hardware and related equipment

. Due to the serious risks of AI advancement, especially in the hands of adversary nations, the US must take deliberate action to control the proliferation and growth of boundary-pushing AI capabilities.

If breakthroughs remain heavily concentrated in America, then it becomes easier for advanced general-purpose AI developers to avoid 

communicate openly about safety challenges and 

 if there is a credible risk of large-scale harm, and this cooperation will be significantly more difficult if scientists in China, Iran, Russia, and so forth can quickly replicate every American breakthrough.

The Bureau of Industry and Security (BIS) is critical in maintaining America’s AI dominance: it is using 

 to cut China’s access to advanced AI chips. However, BIS has 

 for robustly enforcing these controls. To prevent China from 

enforce robust information security requirements

AI safety legislation that America desperately needs

Policy Proposals

Recommendations

Why we need these policies