Learn more context on large-scale risks from advanced AI
These opportunities focus on AI safety work aimed at preventing loss of human control of very capable AI systems. To maximize your eligibility for these opportunities, we recommend gaining context on the perspectives of this subfield, e.g. by skimming pertinent AI safety papers.
AI Safety PapersOpportunities
Last updated: 7/17/24
Job Opportunities (Research and Engineering)
- Frontier Red Team
- Responsible Scaling Policy Team
- Alignment Stress Testing Team
- Interpretability Team
- Dangerous Capability Evaluations Team
- Assurance Team
- Security Team
- AI Safety and Alignment Team (Bay Area)
- Scalable Alignment Team
- Frontier Model and Governance Team
- Mechanistic Interpretability Team
- Responsibility & Safety Team
- UK AI Safety Institute (AISI) (roles)
- Note that RAND's Technology and Security Policy Fellowship is not just for policy research; ML engineers, software engineers with either infrastructure or front-end experience, and technical program managers are also encouraged to apply via this Fellowship.
- FAR AI (roles)
- Model Evaluations and Threat Research (METR) (roles)
- Apollo Research (roles)
Funding Opportunities
- Career development and transition funding
- Course development grants
- Request for proposals for projects to grow our capacity for reducing global catastrophic risks
- Request for proposals: AI governance (In the "technical governance" section, examples include: compute governance, model evaluations, technical safety and security standards for AI developers, cybersecurity for model weights, and privacy-preserving transparency mechanisms. See also the Governance and Policy section below)
- Cooperative AI Research Grants (deadlines: 7/30 & 10/6)
- METR Evaluation Task Bounty (related: METR's Autonomy Evaluation Resources)
- SafeBench Competition (deadline: 2/25/2025; $250k in prizes)
- AI Safety Fund via the Frontier Model Forum
- NSF Secure and Trustworthy Cyberspace Grants
- Foresight Neuro/Security/Cryptography/Multipolar Approaches to AI Safety
- Long-Term Future Fund
- Request for proposals: benchmarking LLM agents on consequential real-world tasks (deadline: 7/26)
- Anthropic Model Evaluation Initiative
- ARIA's Safeguarded AI Program (aimed at quantitative safety guarantees, accepting proposals)
- Note: SFF gives grants to universities. Alternatively, SFF requires that you have a 501c3 charity (i.e. your nonprofit has 501c3 status or you have a fiscal sponsor that has 501c3 status).
- Call for Research Ideas: Expanding the Toolkit for Frontier Model Releases from CSET
- OpenAI: Research into Agentic AI Systems, Superalignment Fast Grants, OpenAI Cybersecurity Grants (assumed closed)
- NSF: Safe Learning-Enabled Systems and Responsible Design, Development, and Deployment of Technologies
- Center for Security and Emerging Technology (CSET): Foundational Research Grants
- Future of Life Institute: PhD Fellowships and Postdoctoral Fellowships
- Center for AI Safety Compute Cluster (deadline: 9/30/24)
- National Deep Inference Fabric (NDIF), can request early access to a research computing project for interpretability research
- Cohere for AI, subsidized access to APIs
- Constellation is offering 3–6 month extended visits (unpaid) at their office (Berkeley, CA) for researchers, engineers, entrepreneurs, and other professionals working on their focus areas. Apply here. See here for more details.
- Constellation is offering year-long salaried positions ($100K-$180K) at their office (Berkeley, CA) for experienced researchers, engineers, entrepreneurs, and other professionals to pursue self-directed work on one of Constellation's focus areas. Apply here. See here for more details.
- The ML Alignment & Theory Scholars (MATS) Program is an educational seminar and independent research program that aims to provide talented scholars with talks, workshops, and research mentorship in the field of AI alignment and safety. We also connect them with the Berkeley alignment research community. Our Winter Program will run from early Jan, 2025. Apply here.
- Supervised Program for Alignment Research (SPAR) Fall Program
- Future events interest form for the Alignment Workshop Series (previous: Dec 2023, Feb 2023)
- Future events interest form for Constellation Workshops. Constellation expects to offer 1–2 day intensive workshops for people working in or transitioning into their focus areas.
- ML Safety Social hosted by the Center for AI Safety
- Secure and Trustworthy Large Language Models
- How Far Are We From AGI?
- Reliable and Responsible Foundation Models
- ME-FoMo: Mathematical and Empirical Understanding of Foundation Models
- San Francisco Alignment Workshop 2023 (Feb 2023), recordings available
- Neural Scaling & Alignment: Towards Maximally Beneficial AGI Workshop Series (2021-2023)
- Human-Level AI: Possibilities, Challenges, and Societal Implications (June 2023)
- Workshop on AI Scaling and its Implications (Oct 2023)
- AI Existential Safety Community from Future of Life Institute
- See speakers from the Alignment Workshop series (SF 2023, NOLA 2023)
- Information Security roles
- Overview from a security engineer at Google
- Jason Clinton's recommended upskilling book
- Forecasting (see especially Epoch)
- Software Engineering
- Technical work that primarily aims to improve the efficacy of AI governance interventions, including compute governance, technical mechanisms for improving AI coordination and regulation, privacy-preserving transparency mechanisms, technical standards development, model evaluations, and information security.
- Blog post (Anonymous): "AI Governance Needs Technical Work"
- Blog post by Lennart Heim: "Technical AI Governance" (focuses on compute governance), Podcast: "Lennart Heim on the compute governance era and what has to come after"
- Blog post by Luke Muehlhauser, "12 Tentative Ideas for US AI Policy"
- Perspectives on options given AI hardware expertise (80,000 Hours)
- Arkose is looking for further resources about technical governance, as this is a narrow set; please send recommendations to team@arkose.org!
- See also the AI Governance and Policy section below.
- Center for the Governance of AI (GovAI) Winter Fellowship 2025 (Deadline 08/11/2024)
- The series is designed to help individuals interested in federal AI and biosecurity policy decide if they should pursue careers in these fields. Each session features experienced policy practitioners who will discuss what it’s like to work in emerging technology policy and provide actionable advice on how to get involved. Some of the sessions will be useful for individuals from all fields and career stages, while others are focused on particular backgrounds and opportunities. You may choose to attend all or only some of the sessions.
- AI Governance Curriculum by BlueDot Impact
- AI Policy Resources by Emerging Technology Policy Careers
- Center for Long-Term Resilience (CLTR)
- RAND's Technology and Security Policy work
- Horizon Institute for Public Service
- Institute for AI Policy and Strategy
- Center for Security and Emerging Technology (CSET)
- Frontier AI Task Force
- Center for the Governance of AI (GovAI)
- Industry AI Governance teams
- Center for AI Policy
Compute Opportunities
AI Safety Programs / Fellowships / Residencies
Workshops and Community
Job Board
Filtered from the 80,000 Hours Job Board
Alternative Technical Opportunities
AI governance is focused on developing global norms, policies, and institutions to increase the chances that advanced AI is beneficial for humanity.