Learn more context on large-scale risks from advanced AI
These opportunities focus on AI safety work aimed at preventing loss of human control of very capable AI systems. To maximize your eligibility for these opportunities, we recommend gaining context on the perspectives of this subfield, e.g. by skimming pertinent AI safety papers.
AI Safety PapersOpportunities
Job Opportunities (Research and Engineering)
- Safety Systems Team (roles)
- Preparedness Team (roles)
- Superalignment Team (roles)
- Security Team
- Policy Research Team
- Trustworthy AI Team
- Frontier Red Team
- Responsible Scaling Policy Team
- Alignment Stress Testing Team
- Interpretability Team
- Dangerous Capability Evaluations Team
- Assurance Team
- Security Team
- AI Safety and Alignment Team (Bay Area)
- Scalable Alignment Team
- Frontier Model and Governance Team
- Mechanistic Interpretability Team
- Responsibility & Safety Team
- UK AI Safety Institute (AISI) (roles)
- Note that RAND's Technology and Security Policy Fellowship is not just for policy research; ML engineers, software engineers with either infrastructure or front-end experience, and technical program managers are also encouraged to apply via this Fellowship.
- FAR AI (roles)
- Model Evaluations and Threat Research (METR) (roles)
- Apollo Research (roles)
Funding Opportunities
- Request for proposals: benchmarking LLM agents on consequential real-world tasks
- Request for proposals: studying and forecasting the real-world impacts of systems built from LLMs
- Career development and transition funding
- Course development grants
- Request for proposals for projects to grow our capacity for reducing global catastrophic risks
- Cooperative AI Research Grants (deadlines: 7/30 & 10/6)
- SafeBench Competition (deadline: 2/25/2025; $250k in prizes)
- AI Safety Fund via the Frontier Model Forum
- ARIA's Safeguarded AI Program (aimed at quantitative safety guarantees, accepting proposals)
- METR Evaluation Task Bounty (related: METR's Autonomy Evaluation Resources)
- OpenAI Cybersecurity Grants
- NSF Secure and Trustworthy Cyberspace Grants
- Long-Term Future Fund
- OpenAI: Research into Agentic AI Systems and Superalignment Fast Grants
- NSF: Safe Learning-Enabled Systems and Responsible Design, Development, and Deployment of Technologies
- Center for Security and Emerging Technology (CSET): Foundational Research Grants
- Future of Life Institute: PhD Fellowships and Postdoctoral Fellowships
- Survival and Flourishing Fund: Grant Rounds and Speculation Grants
Compute Opportunities
AI Safety Programs / Fellowships / Residencies
- Constellation is offering 3–6 month extended visits (unpaid) at their office (Berkeley, CA) for researchers, engineers, entrepreneurs, and other professionals working on their focus areas. Apply here by April 30 (or by April 12 if you would like to collaborate with a research advisor). See here for more details.
- Constellation is offering year-long salaried positions ($100K-$180K) at their office (Berkeley, CA) for experienced researchers, engineers, entrepreneurs, and other professionals to pursue self-directed work on one of Constellation's focus areas. Apply here by April 30. See here for more details.
- The ML Alignment & Theory Scholars (MATS) Program is an educational seminar and independent research program that aims to provide talented scholars with talks, workshops, and research mentorship in the field of AI alignment and safety. We also connect them with the Berkeley alignment research community. Our Winter Program will run from early Jan, 2025. Apply here.
Workshops and Community
- ML Safety Social hosted by the Center for AI Safety
- Secure and Trustworthy Large Language Models
- How Far Are We From AGI?
- Reliable and Responsible Foundation Models
- ME-FoMo: Mathematical and Empirical Understanding of Foundation Models
- Future events interest form for the Alignment Workshop Series (previous: Dec 2023, Feb 2023)
- Future events interest form for Constellation Workshops. Constellation expects to offer 1–2 day intensive workshops for people working in or transitioning into their focus areas.
- New Orleans Alignment Workshop (Dec 2023), recordings available
- San Francisco Alignment Workshop 2023 (Feb 2023), recordings available
- Neural Scaling & Alignment: Towards Maximally Beneficial AGI Workshop Series (2021-2023)
- Human-Level AI: Possibilities, Challenges, and Societal Implications (June 2023)
- Workshop on AI Scaling and its Implications (Oct 2023)
- AI Existential Safety Community from Future of Life Institute
- See speakers from the Alignment Workshop series (SF 2023, NOLA 2023)
Job Board
- Information Security roles
- Overview from a security engineer at Google
- Jason Clinton's recommended upskilling book
- Forecasting (see especially Epoch)
- Software Engineering
- Technical work that primarily aims to improve the efficacy of AI governance interventions, including compute governance, technical mechanisms for improving AI coordination and regulation, privacy-preserving transparency mechanisms, technical standards development, model evaluations, and information security.
- Blog post (Anonymous): "AI Governance Needs Technical Work"
- Blog post by Lennart Heim: "Technical AI Governance" (focuses on compute governance), Podcast: "Lennart Heim on the compute governance era and what has to come after"
- Blog post by Luke Muehlhauser, "12 Tentative Ideas for US AI Policy"
- Perspectives on options given AI hardware expertise (80,000 Hours)
- Arkose is looking for further resources about technical governance, as this is a narrow set; please send recommendations to team@arkose.org!
- See also the AI Governance and Policy section below.
- AI Governance Curriculum by BlueDot Impact
- AI Policy Resources by Emerging Technology Policy Careers
- Center for Long-Term Resilience (CLTR)
- RAND's Technology and Security Policy work
- Horizon Institute for Public Service
- Institute for AI Policy and Strategy
- Center for Security and Emerging Technology (CSET)
- Frontier AI Task Force
- Center for the Governance of AI (GovAI)
- Industry AI Governance teams
- Center for AI Policy
Filtered from the 80,000 Hours Job Board
Alternative Technical Opportunities
AI governance is focused on developing global norms, policies, and institutions to increase the chances that advanced AI is beneficial for humanity.