This decade, AI could be powerful enough to cause global catastrophes

Within the next 3-10 years, AI projects could develop superintelligent AI: AI that is vastly smarter and more powerful than humans.

: “While superintelligence seems far off now, we believe it could arrive this decade.”

Progress towards superintelligence has been rapid.

In 2011, the best language models produced only 

 became one of the first models to consistently write coherent sentences

 outperformed ~90% of test takers on the SAT

For more detailed analyses on AI progress, we recommend 

AI systems are already being used to improve 

, such as hacking, weapons design, persuasion, and strategic planning.

 AI systems will soon be able to engineer pandemics, orchestrate novel cyberattacks, and disrupt critical infrastructure.

 that "the vast power of superintelligence… could lead to the disempowerment of humanity or even human extinction."

Developers don’t know how they will control very powerful AI.

no one knows how to control powerful AI systems

This is both a safety and a security issue.

AI systems could be misused to cause grievous harm.

Solving safety research questions requires time.

Yet unchecked competitive pressures compel companies to prioritize profits over safety.

 building catastrophically powerful systems.

The government needs to be able to safeguard AI development.

The government needs the capacity to rapidly identify and respond to AI risks. This requires:

More visibility into frontier AI development and technical expertise

Clear mechanisms to halt unsafe development in case of an emergency.

Better incentives that support developers prioritizing safety from the outset.

Monitoring of the advanced hardware used to train new AIs. If we know where that hardware is located, we will be more prepared for AI-related emergencies.

Monitoring of frontier AI development to give the government transparency into the development of these systems.

License frontier AI development, to incentivize AI safety research and allow for swift intervention in hazardous situations.

Strict liability for severe harms caused by AI systems, promoting accountability and improving incentivizes

Why AI policy

Our takeaways from July 25 Senate Judiciary Hearing

On July 25, 2023, the Senate Judiciary Subcommittee on Privacy, Technology, and Law held a hearing titled “Oversight of AI: Principles for Regulation.” The Center for AI Policy commends the witnesses and committee members for their substantive discussion of existential risks from AI, as well as policy proposals intended to reduce those risks.

We have selected some quotes about global security risks from AI. The witnesses and committee members also discussed a variety of other topics, including concerns about data privacy, misinformation, and jobs. You can listen to the full hearing 

“What I have heard again and again and again… is "scary"... An intelligence device out of control, autonomous, self-replicating, potentially creating diseases, pandemic-grade viruses or other kinds of evils purposely engineered by people or simply the result of mistakes, not malign intention.”

“You have provided objective, fact-based views on what the dangers are... potentially even human extinction.”

“I’ve come to the conclusion that we need some kind of regulatory agency.”

“A number of you have put the timeline at two years before we see some of the biological [dangers and] most severe dangers. It may be shorter because the pace of development is not only stunningly fast, it is also accelerated at a stunning pace because of the quantity of chips, the speed of chips, the effectiveness of algorithms.”

“Superhuman AI evokes for me artificial intelligence that could on its own develop a pandemic virus, on its own decide Joe Biden shouldn’t be our next president… And I think that argues for urgency.”

“I’m less interested in the [AI] corporations’ profitability. In fact, I'm not interested in that at all. I’m interested in protecting the rights of American workers and American families and American consumers against these massive companies that threaten to become a total law unto themselves.”

“Will the Senate actually act? … We’ve had a lot of talk, but now is the time for action. And I think if the urgency of the new generative AI technology does not make that clear to folks, then you’ll never be convinced.”

“A straightforward extrapolation of today’s systems to those we expect to see in two to three years suggests a substantial risk that AI systems will be able to fill in all the missing pieces, enabling many more actors to carry out large-scale biological attacks. We believe this represents a grave threat to U.S. national security.”

“New AI models should have to pass a rigorous battery of safety tests before they can be released to the public at all, including tests by third parties and national security experts in government.”

Yoshua Bengio, Recipient of the Turing Award

“Require licenses… and restrict AI systems with unacceptable levels of risk.”

“[We need] to bring expertise in national security, in bioweapons, chemical weapons, and AI people together. [These organizations] shouldn’t be for profit… We shouldn’t mix the objective of making money… with the objective of defending humanity against a potential rogue AI.”

“This research in AI and international security should be conducted with several highly secure and decentralized labs operating under multilateral oversight to mitigate an AI arms race… We must therefore allocate substantial additional resources to safeguard our future, at least as much as we are collectively globally investing in increasing the capabilities of AI.”

“Viruses, computer or biological viruses don’t see any border. So, we need to make sure there’s an international effort in terms of these safety measures. We need to agree with China on these safety measures... And we need to work with our allies on these countermeasures.”

Stuart Russell, Professor of Computer Science at UC Berkeley

“Alan Turing, the founder of Computer Science, warned in 1951 that once AI outstrips our feeble powers, we should have to expect the machines to take control. We have pretty much completely ignored this warning.”

“Systems that break the rules must be recalled from the market for anything from defaming real individuals to helping terrorists build biological weapons.”

“Now, developers may argue that preventing these behaviors is too hard because LLMs have no notion of truth and are just trying to help. This is no excuse. Eventually, and the sooner the better, I would say, we will develop forms of AI that are provably safe and beneficial… Until then, we need real regulation and a pervasive culture of safety.”

“I think there’s no doubt that we’re going to have to have an agency.”

Takeaways from July 25 Senate Judiciary Hearing

 content. If generative AI like ChatGPT generates libel, then OpenAI is liable under existing law.

It’s not clear whether courts will include “training an AI” as a “fair use” of a book or a movie under copyright law. In the most similar prior case, for Google Books, the court seemed to base its ruling on the value that Google Books would add to society.

Strictly enforcing copyright can help some artists get compensated, but it doesn’t offer a long-term solution for unemployment caused by ongoing automation of the workforce.

Generative AI hallucinates and discriminates roughly as often as humans. It has problems because humans have problems, and it’s trained on human behavior. NIST’s AI Risk Management Framework, if properly and vigorously applied, would go a long way toward constructively addressing these problems.

Existing law doesn’t provide a solution for the catastrophic risks posed by advanced AI, like bioweapons and automated hacking. CAIP’s bill would help address these risks with a strict liability regime and mandatory safety measures.

What are the barriers to holding companies liable for harms committed by their AIs?

Identifying the bad actor can be a challenge. For example, if a large corporate developer makes an AI with a sloppy third-party interface that doesn’t screen out security threats, and then a freelance designer adds a potentially dangerous extension for the AI, and then the end user installs that plug-in and uses it to develop a bioweapon, who caused the harm? The corporate developer, the freelance designer, or the end user?

The answer is that all three parties are probably liable to various degrees, but proving this can be difficult.

CAIP’s bill would solve this problem by making the developer, the designer, and the user all jointly and severally liable.

 exempts the owner of an “interactive computer service” that publishes third-party content. For example, if Howard Dean claims on Reddit that Senator Romney murders clowns for fun, then Senator Romney can’t sue Reddit for defamation – he could sue Mr. Dean, but Reddit itself is safe, even if Reddit was sloppy about responding to requests to delete Mr. Dean’s accusation of clown-murder.

 and the tech companies who rely on it for their advertising revenue.

The Supreme Court has interpreted exceptions to Section 230 narrowly. For example, it 

 the Anti-Terrorism Act by passively hosting conversations in which terrorist leaders recruited new agents, and that Reddit 

 from child pornography by passively hosting bulletin boards on which third parties posted illegal videos.

Senators Hawley (R-MO) and Blumenthal (D-CN) are 

 that would deny Section 230 immunity for charges and claims related to generative AI.

Do foundational AI models violate copyright?

Most texts and images are copyrighted. They may come with licenses that allow certain types of users to use them in certain ways, but these licenses almost never grant permission for the content to be 

 by AI developers. A work that is valuable enough for its owners to protect usually remains under copyright for literally 

 – this means that books published in the 1920s are only just now entering the public domain.

Making use of a copyrighted work without permission carries 

 of up to $30,000 per work, or $150,000 per work if the infringement was “willful.” If the owners of the work lost profits that are larger than these fines, they can recover their profits, instead.

A “fair use” of copyrighted material does not violate copyright law. Literary critics, journalists, teachers, comedians, and researchers are usually allowed to 

cite medium-sized portions of copyrighted works

 for the purpose of commenting on them without having to get permission or pay royalties.

A court is more likely to find that a work is a “fair use” if it is a “transformative use,” i.e., if the work adds something new, of a different purpose or character, and thus does not directly substitute or compete with the original work.

The question of whether, e.g., Hugging Face is making fair use of the images it was trained on is 

; there is genuine uncertainty about how courts will rule.

The comedian Sarah Silverman is spearheading a 

 for training their LLMs on illegally downloaded copies of her copyrighted books. She argues that these models would not be able to summarize her books if they had not ‘read’ them, and that they did not pay her (or other authors) for the right to read them.

; a consortium of authors and publishers sued Google for violating their copyrights by scanning books in order to put them into a searchable database and offer long snippets from each book. The parties tentatively agreed on a settlement that would have had Google paying modest per-book fees and allowed authors to affirmatively opt-out of the database, but then abandoned the settlement; Google ultimately won a complete victory in court. It appears that the Second Circuit (which heard the case) was more interested in the public utility provided by having a searchable database of nearly all books than in the literal application of existing copyright law.

When should AI companies be sued for copyright violations?

Weak enforcement of copyright will make it easier to develop profitable AI businesses, and easier for consumers to access low-priced entertainment, but it will also contribute to putting authors, actors, and musicians out of work and exacerbate economic inequality.

 that they were paid $100 to work as an extra for the day on a movie set – the usual rate – and then a director took detailed digital scans of them that would enable movie producers to keep on using the actor’s likeness in future movies, all without any additional pay.

Entertainers already have very low bargaining power – for example, Spotify offers musicians only a 

 each time a song is played; to earn even 

 from Spotify, a typical artist would need to have 

. This is roughly the equivalent of releasing a new gold record album every single year.

Stronger copyright enforcement will make it somewhat harder for AI entertainment to succeed.

Transaction costs will slow down the production of AI-assisted artwork; it’s challenging to negotiate with each person whose art you want to use, and you might not be able to find them or might not know who owns their catalog.

Even if most artists have the right to say ‘no’ to AI scanning and scraping (by enforcing their copyright), if a few artists say ‘yes’, then there could still be mass unemployment in the field. It’s plausible that given a sample of a few hundred scanned actors, AI will be able to extrapolate enough digital images to cover any situations that might come up in TV and movies. The most passionate fans might pay a premium to watch likenesses of their favorite actors, but most consumers will probably be happy enough with high-quality (but generic) AI actors, plus a few familiar faces from the actors who agreed to be scanned.

When should AI companies be liable for hallucinations?

The current generation of chatbots often “hallucinate,” i.e., invent detailed and confident-sounding claims that have no basis in reality. The FTC is 

 OpenAI for such hallucinations right now.

This is mostly a problem if people don’t realize that it’s a possibility. At some point, blindly trusting something because you heard it from a chatbot is like trusting something because you read it in the National Enquirer or the Onion.

If you open ChatGPT, you get a screen that prominently warns you that one of its “limitations” is that it “may occasionally generate incorrect information.” At the bottom of the screen is a disclaimer that “ChatGPT may produce inaccurate information about people, places, or facts.” OpenAI’s 

 have mostly been transparent about what ChatGPT can and can’t do and what its limitations are.

Other chatbots may be less scrupulous about their disclaimers.

Banning any chatbot that has any tendency to hallucinate would destroy significant economic value – ChatGPT is “right” or at least “useful” often enough that if you double-check its suggestions, you’ll usually still do your work faster and better than if you tried to work without hearing those suggestions at all.

To balance the usefulness of chatbots with their tendency to hallucinate, it makes sense to require chatbots to offer strong disclaimers, and then hold them harmless for hallucinations if and only if the disclaimers are adequate.

When should AI companies be liable for discrimination and bias?

Discrimination based on protected categories is an ongoing problem that’s mostly highlighted by AI rather than caused by AI. It’s important that we continue to work on addressing the root causes of discrimination and repairing the damage it causes. However, for the most part, AIs that make biased decisions are doing so because they were 

trained on data that reflects the biases of average Americans

Some AI-based discrimination reflects poor design choices on the part of engineers. For example, default camera settings tend to be 

suboptimal for photographing darker skin tones

. This can lead to racial bias in AI-powered facial recognition technology.

Companies should be legally required to seek out and address these types of problems.

The NIST AI Risk Management Framework (“

”) offers tools and resources for companies that voluntarily choose to pursue this kind of internal investigation.

Algorithms are often evaluated for bias based on the “

,” meaning that if an algorithm is selecting only four-fifths as many people from a protected group, then it is biased. This is a crude rule of thumb; as algorithms become increasingly important in daily life, we should look for ways to make more precise assessments. For example, we can ask whether the difference in selection rates is statistically significant using a 

 test, and then if there is a statistically significant difference, we can ask whether there is a legitimate reason for this difference that is not due to discrimination and that adequately explains the difference.

Companies who develop AI have the same legal obligations to avoid discrimination as any landlord under the Fair Housing Act, any employer under the Civil Rights Act of 1964, and so on. These laws can adequately protect against most types of discrimination from AI as long as we apply them thoughtfully.

AI can sometimes hide or ‘launder’ an unlawful or immoral bias by making a decision appear to be objective.

For example, an AI-generated “sentencing guideline” that’s calculated based on isolated facts about a criminal defendant may seem more impartial than a sentence ordered by a particular judge to that defendant’s face after seeing and talking with the defendant.

It is important to educate people to assess the actual bias present in both human and AI-assisted decision-making systems, so that people do not just assume that a decision generated by a computer must be impartial. This could involve mandatory training for the end users of AI recommendation engines, such as judges and parole officers.

AI is becoming increasingly powerful. Very soon, it could gain capabilities that could cause destruction on a mass scale, such as designing new bioweapons, automatically seizing control of millions of computers, or manufacturing its own drones and missiles.

In the past, most software was only dangerous to people who voluntarily chose to use it. Even a computer virus typically cannot affect people who do not install malware or click on a suspicious link.

As a result, we have almost no laws that regulate dangerous software.

It would be very difficult to hold people properly accountable for creating or distributing dangerous AI.

The Center for AI Policy’s bill would help reduce these catastrophic risks by enabling the government to monitor concentrations of the special chips needed to power advanced AI, require safety precautions for advanced AI developers, and intervene in an emergency.

Our notes on AI, liability, and copyright

Notes on AI, liability, and copyright

Who supports licensing for frontier AI systems?

We compiled relevant quotes from OpenAI and Microsoft. Licensing has also received support from other stakeholders in industry, think tanks, governments, and the public.

 to the Senate Judiciary Subcommittee on Privacy, Technology and the Law, May 2023)

"It is vital that AI companies– especially those working on the most powerful models– adhere to an appropriate set of safety requirements, including internal and external testing prior to release and publication of evaluation results. To ensure this, the U.S. government should consider a combination of licensing or registration requirements for development and release of AI models above a crucial threshold of capabilities, alongside incentives for full compliance with these requirements."

"Licensure is common in safety-critical and other high-risk contexts, such as air travel, power generation, drug manufacturing, and banking. Licensees could be required to perform pre-deployment risk assessments and adopt state-of-the-art security and deployment safeguards."

"As Microsoft, we endorse [the establishment of a licensing regime] and support the establishment of a new regulator to bring this licensing regime to life and oversee its implementation."

"We envision licensing requirements such as advance notification of large training runs, comprehensive risk assessments focused on identifying dangerous or breakthrough capabilities, extensive prerelease testing by internal and external experts, and multiple checkpoints along the way."

"We shouldn’t leave these issues of societal importance to good judgment and self-restraint alone. We need regulatory frameworks that anticipate and get ahead of the risks."

71% believe there should be "national regulations on AI."

55% favor "having a federal agency regulate the use of artificial intelligence similar to how the FDA regulates the approval of drugs and medical devices."

"A majority (55%) of Americans are now worried at least somewhat that artificially intelligent machines could one day pose a risk to the human race’s existence."

Who supports licensing?

 is a new organization developing and advocating for policy to mitigate catastrophic AI risks.

. We’re also open to other roles. The deadline to apply is September 30th, 2023.

The Center for AI Policy is a new DC-based organization developing and advocating for policy to mitigate catastrophic AI risks.

Our current focus is building capacity in the US government to safeguard AI development. Our proposed legislation would establish a federal authority to monitor hardware and license frontier AI development, ensuring we can identify and respond to risks. It would also create strict liability for severe harms caused by AI systems, increasing accountability and improving incentives to developers.

Our team includes Thomas Larsen (Executive Director), a former technical AI safety researcher; Jason Green-Lowe (Legislative Director), a lawyer and policy analyst; and Jakub Kraus (Operations Director), who has a computer science background. We’re advised by experts from other organizations and supported by several volunteers.

How the Center for AI Policy differs from other AI governance organizations

Many AI governance organizations are focused on developing ideas and amassing influence that can be used later. We’re focused on developing and advocating for significant, shippable policy now. We want to harness the current energy to pass meaningful legislation during this policy window, in addition to building a coalition for the future. While we engage in conversation with policymakers about a diverse range of AI risks, we are also upfront about our focus on catastrophic risk.

We think we’re strong at developing policies that would significantly reduce catastrophic risk if passed. To get these policies passed, we need to scale our efforts and bring in more advocacy, policy, and DC experience.

. Our Government Affairs Director will design and execute our strategy for passing meaningful legislation. Our Communications Director will design and execute our strategy for promoting our ideas. For more information, see our 

 page. The deadline to apply is September 30th, 2023.

The Center for AI Policy is a 501(c)(4) funded by private donors and philanthropists. We are nonprofit, nonpartisan, and committed to the public interest.

We are currently funding constrained and believe donations are particularly impactful. With around $100k, we can hire a talented full time team member who can meaningfully increase our chances of getting legislation passed. Smaller amounts are also helpful for hosting events to connect with key policymakers, contracting researchers and lawyers to optimize our legislative text, etc. You can donate to us 

. If you are considering donating and would like to learn more, please contact us at 

CAIP is hiring a Government Affairs Director and a Communications Director

We're hiring

We are looking for a Government Affairs Director with extensive experience in political strategy, lobbying, and/or advocacy.

As Government Affairs Director, you’ll work closely with our executive team to design and execute on a strategy for persuading Congress of our legislative agenda. You’ll help us build key relationships in and around Congress; weigh the tradeoffs between policy effectively addressing AI risks and being likely to pass; identify obstacles to our policy passing; and creatively solve them.

Meeting with members of Congress and their staff,

Drafting short memos aimed at explaining the political and policy benefits of our bill(s),

Working with our external lobbyists to develop our political strategy,

Providing and/or supporting testimony at committee hearings

Hosting and attending happy hours, briefings, roundtables, and other networking events

Building relationships with leaders of tech companies, small businesses, labor unions, civil society groups, and academia

An ideal candidate will have the experience and drive to advocate for CAIP’s legislative agenda with a growing coalition of supporters. Some of the qualifications we are looking for include:

Demonstrated ability to support a legislative agenda

Interest in proactively crafting a political strategy

Deep understanding of DC culture and the political process

Ability to creatively work around constraints to make our policy sufficiently comprehensive 

Sincere concern about safety risks posed by advanced AI

Ability to understand and work with a diverse and bipartisan set of stakeholders

We encourage you to apply even if you don’t have every qualification.

You will need to frequently take meetings on Capitol Hill in person in order to be effective in this role, but on days when you have no such meetings, we support remote work.

We are open to full-time and part-time roles; you would largely set your own hours.

We are looking for candidates who can start work in the next couple of months (or immediately), so it is unlikely that we would be able to sponsor a visa.

 The annual salary range for this role is $100,000 - $140,000, depending on experience. For part time roles, the salary will be prorated based on the number of hours worked. We also provide benefits including flexible time off and healthcare.

 is a new DC-based organization developing and advocating for policy to mitigate catastrophic AI risks.

Our current focus is building capacity in the US government to monitor and safeguard AI development. Our proposed legislation would establish a federal authority to monitor hardware and license frontier AI development, ensuring we can identify and respond to risks. It would also create strict liability for severe harms caused by AI systems, promoting accountability and improving incentives.

Our team includes Thomas Larsen (Executive Director), who has a background in technical AI safety; Jason Green-Lowe (Legislative Director), who has a legal background; and Jakub Kraus (Operations Director), who has a computer science background. We’re also advised by experts from other organizations and supported by several volunteers.

You would be a key voice in our developing organization.

We're hiring a Government Affairs Director

Job Opening: Government Affairs Director

We are looking for a Communications Director with extensive experience in journalism, public relations, campaigns, lobbying, and/or advocacy.

As Director of Communications, you’ll work closely with our executive team to design and execute a strategy for drawing positive attention to CAIP and its legislative agenda. You’ll help us make key decisions about where to focus our media efforts, what audiences we should be trying to reach, what messages are most important for those audiences to hear, and how to frame those messages.

Drafting op-eds, memos, press releases, endorsements, and other public statements

Identifying and acting on opportunities to promote CAIP and our ideas

Reaching out to print media, TV, radio, and podcasts to propose and arrange interviews

Maintaining a blog and/or social media presence for CAIP

Following discussions of AI risk in government and in the media

Working with our Government Affairs director and lobbyists to support their political strategy

5+ years of relevant post-college experience

Demonstrated ability to attract positive media coverage

Interest in proactively crafting a media strategy

Ability to understand complex technical ideas and clearly communicate them to laypeople

Measured and quick judgment to correctly handle sensitive communications issues in a fast-paced environment

We believe you will need to occasionally take meetings on Capitol Hill in person in order to be effective in this role, but on days when you have no such meetings, we support remote work.

 The annual salary range for this role is $100,000 - $140,000, depending on experience. We are also open to providing benefits including flexible time off, healthcare, retirement plans, relocation assistance, performance-based bonuses.

We are moving quickly and may fill the role soon. We encourage applicants to err on the side of reaching out soon, rather than waiting to prepare a more refined application. 

CAIP is hiring a Communications Director.

Job Opening: Communications Director

To learn more about our work, reach out via the form below, or via email at info@aipolicy.us.

Contact us

Donate to the Center for AI Policy to support our work on reducing risks from AI. Note that contributions are not tax-deductible for federal income tax purposes.

Donate

Rapidly advancing AI poses catastrophic risks.

We need to be able to safeguard AI development. We are developing comprehensive legislation that builds that capacity.

Developing practical policy

We are advocating for legislation in Congress, providing resources for policymakers to learn about AI, and working with constituents to ensure AI is beneficial to them.

Rapidly advancing AI poses catastrophic risks.

We develop and promote policy to mitigate those risks.

Our Work

Developing practical policy

Advocating for responsible AI

Our Mission