Start here
Pick the path closest to your background. Each is a suggested order through resources listed further down the page.
New to AI safety
Technical background, starting from zero on safety
- Robert Miles: Fundamentals of why AI is not under controlBuilds intuition for the core problems in short videos
- Introduction to AI Safety, Ethics, and SocietyA structured primer covering risks, ML safety and governance
- International AI Safety Report (Feb 2026)Grounds that picture in the current evidence
- AI 2027A concrete scenario of how fast things could move
- Alignment faking in large language modelsA landmark result showing why alignment is hard to verify
- BlueDot Technical Safety (Intensive or part-time tracks)Structured cohort to go deeper with others
- ARENA course curriculumHands-on skills: interpretability, RL and evals in code
- 80,000 Hours: Technical AI safety research careersHow to turn this into a job
Security practitioner
Shipping or defending LLM and agent systems
- Will Larson: What can agents actually do?Where agents fit and why the model can't be the control
- Simon Willison: The lethal trifecta for AI agentsThe mental model for how agents leak data
- OWASP: GenAI Top 10 RisksThe baseline risks for any LLM application
- OWASP: Top 10 Agentic Security Risks for 2026 (Dec 2025)What changes once the model can act
- Design Patterns for Securing LLM Agents against Prompt InjectionsArchitectures that contain prompt injection
- OWASP AI ExchangeThe full reference of AI threats and controls
- MITRE ATLAS MatrixAttacker techniques for threat modelling
- NIST AI 100-2 E2025: Adversarial Machine LearningStandard vocabulary for attacks and mitigations
Policy & governance
Risk, compliance, standards or policy roles
- International AI Safety Report (Feb 2026)The shared evidence base policymakers cite
- BlueDot Governance (Intensive or part-time tracks)Structured course on governance mechanisms
- EU AI Act and the General-Purpose AI Code of PracticeThe first binding rules for frontier models
- NIST AI Risk Management Framework PlaybookHow organisations put AI risk management into practice
- MIT AI Risk RepositoryA catalogue of risks to build a register from
- 3 Lines of Defense (3LoD)Who owns which risk inside a company
- UK AISI: Frontier AI Trends ReportA government evaluator's view of the trends
- Future Of Life Safety Index (Summer 2026)How the leading companies currently measure up
Keeping current
Already in the field, catching up on 2026
- METR: Frontier Risk Report (Feb–Mar 2026)Latest assessment of loss-of-control risk from agents
- Center for Long-Term Resilience: Scheming in the wild (Mar 2026)Evidence of scheming-like behaviour in deployment
- Chain of Thought Monitorability: A New and Fragile Opportunity for AI SafetyWhy visible reasoning matters, and how it could be lost
- METR: Measuring AI Ability to Complete Long TasksThe standard way to track agent capability growth
- AI Agents Enable Adaptive Computer Worms (June 2026)A new class of AI-driven cyber threat
- Future Of Life Safety Index (Summer 2026)Summer 2026 company safety grades
- Transformer (newsletter)A weekly briefing to stay on top of it
Introduction & Fundamentals
-
International AI Safety Report (Feb 2026)
Updated global synthesis of AI capabilities, risks, and safety evidence
ReportIntroLong readFeb 2026
About this resource
Second edition of the independent international review chaired by Yoshua Bengio, written with about 100 experts and an advisory panel from 30 countries, the UN, the OECD and the EU. The best single overview of current evidence.
Earlier edition: 2025 edition
Similar resources
Stanford AI Index Report 2026 · CAIS: An Overview of Catastrophic AI Risks · UK AISI: Frontier AI Trends Report
Reflect · 3 questions
- Which risk does the report treat as most supported by evidence, and which as most uncertain?
- What changed between the 2025 and 2026 editions, and what does that tell you about the pace of the field?
- Which finding would you most want a policymaker you know to read, and why?
-
Robert Miles: Fundamentals of why AI is not under control
Accessible YouTube explanations of AI control and alignment challenges today
VideoIntro10–20 min eachOngoing
About this resource
Short, clear videos on why getting AI systems to reliably do what we intend is hard: specification gaming, instrumental goals, and the limits of current control methods. A gentle first step that assumes no technical background.
Similar resources
Introduction to AI Safety, Ethics, and Society · Nick Bostrom: Superintelligence · Stuart Russell: Human Compatible
Reflect · 3 questions
- In your own words, why would a capable system pursue goals like self-preservation it was never given?
- What is specification gaming, and where have you seen something like it in ordinary software or organisations?
- Which video changed your mind about something, and what did you believe before?
-
ARENA course curriculum
Perhaps 'the' reference in AI Safety foundations - start learning this at your pace today to make an impact within 6 months!
CourseIntermediateSelf-pacedOngoing
About this resource
Hands-on coding curriculum run by ARENA, part of the London Initiative for Safe AI (LISA). Works through transformers, interpretability, reinforcement learning and evals in notebooks. Assumes Python and some machine-learning familiarity.
Similar resources
BlueDot Technical Safety (Intensive or part-time tracks) · TransformerLens · MATS (ML Alignment & Theory Scholars)
Reflect · 3 questions
- Which chapter (interpretability, RL, evals) best matches the role you want, and why?
- What prerequisite skill do you need to strengthen before starting?
- What would a realistic six-month plan look like for working through this alongside your current commitments?
-
Why broad, general-purpose AI can't also offer zero-error guarantees
PaperAdvanced1 hJun 2025
About this resource
Philosophy-of-AI paper proposing that the wider a system's scope, the less certainty it can give about error-free output. Explains why generative models can't offer the guarantees narrow symbolic systems can, and what that means for evaluation and governance.
Similar resources
Introduction to AI Safety, Ethics, and Society · Stuart Russell: Human Compatible · CAIS: An Overview of Catastrophic AI Risks
Reflect · 3 questions
- Restate the trade-off between scope and certainty in one sentence. Do you find it convincing?
- If general systems can never be error-free, what should 'safe enough' mean for a deployment you care about?
- How might hybrid designs (narrow verified parts plus general models) work around this trade-off?
-
Introduction to AI Safety, Ethics, and Society
Free textbook covering AI risks, ML safety, governance and ethics in one place
TextbookIntroSelf-paced2024Center for AI Safety
About this resource
Dan Hendrycks's open-access textbook from the Center for AI Safety, also published by Routledge. Moves from catastrophic-risk scenarios through ML safety basics to governance and ethics. The most complete single primer for newcomers.
Similar resources
CAIS: An Overview of Catastrophic AI Risks · International AI Safety Report (Feb 2026) · Robert Miles: Fundamentals of why AI is not under control
Reflect · 3 questions
- Which of the risk categories in the book felt most concrete to you, and which most speculative?
- How does the book connect technical ML safety to governance? Where is that link weakest?
- What is one concept from the book you could explain to a non-specialist tomorrow?
-
AI 2027
A month-by-month scenario of rapid AI progress, with two possible endings
ReportIntro2 hApr 2025
About this resource
Forecast-driven scenario from Daniel Kokotajlo and the AI Futures Project describing how superhuman AI could arrive by the late 2020s, with a race ending and a slowdown ending. Gives the abstract risk arguments a concrete storyline.
Similar resources
AI 2040: Plan A · Superintelligence Strategy · METR: Measuring AI Ability to Complete Long Tasks
Reflect · 3 questions
- Which assumption in this scenario, if wrong, would change its conclusion the most?
- At what point in the timeline would you expect governments to notice and act, and what would they do?
- How do the race and slowdown endings differ in who holds decision-making power?
-
Stanford AI Index Report 2026
Annual data on AI progress, investment, policy and responsible AI
ReportIntroLong readApr 2026
About this resource
Stanford HAI's yearly compilation of data on model capabilities, costs, adoption, public opinion, policy and responsible-AI measurement. The reference for numbers when discussing where AI is heading.
Similar resources
International AI Safety Report (Feb 2026) · UK AISI: Frontier AI Trends Report · METR: Measuring AI Ability to Complete Long Tasks
Reflect · 3 questions
- Which trend in the report surprised you most, and what explains it?
- Where does the report's data on responsible AI look thinnest compared with its data on capabilities?
- Which chart would you use to brief a colleague on where AI is heading?
-
AI 2040: Plan A
The AI 2027 team's recommended path: delay superintelligence to 2040
ReportIntro2–3 hJul 2026
About this resource
Follow-up to AI 2027 from Kokotajlo, Lifland, Greenblatt and colleagues. Framed as a plan rather than a forecast: a verified US–China deal pauses frontier training in 2029, with alternative Plans B–D and S compared.
Similar resources
AI 2027 · Superintelligence Strategy · EU AI Act and the General-Purpose AI Code of Practice
Reflect · 3 questions
- Which assumption in this scenario, if wrong, would change its conclusion the most?
- What would a verified US–China deal need to monitor, and how could it be cheated?
- Why do the authors present Plan A as a plan rather than a prediction, and does that change how you read it?
-
CAIS: Statement on AI Risk
One sentence on extinction risk, signed by hundreds of AI leaders
ArticleIntro5 minMay 2023Center for AI Safety
About this resource
The single-sentence statement that mitigating extinction risk from AI should be a global priority, signed by lab heads, Turing Award winners and many researchers. A marker of how mainstream the concern became.
Similar resources
CAIS: An Overview of Catastrophic AI Risks · Eliezer Yudkowsky & Nate Soares: If Anyone Builds It, Everyone Dies · International AI Safety Report (Feb 2026)
Reflect · 3 questions
- Why do you think so many lab leaders signed a one-sentence statement rather than a detailed one?
- What has changed, in policy or lab practice, since the statement was published?
- Would you sign it? What wording, if any, would you change?
-
Malicious use, AI races, organisational risks and rogue AIs
PaperIntro2 hJun 2023Center for AI Safety
About this resource
Hendrycks, Mazeika and Woodside group catastrophic risks into four sources, illustrate each with scenarios and suggest mitigations. A readable map of the whole risk landscape.
Similar resources
Introduction to AI Safety, Ethics, and Society · CAIS: Statement on AI Risk · International AI Safety Report (Feb 2026)
Reflect · 3 questions
- Of malicious use, AI races, organisational risks and rogue AI, which is most neglected today?
- How do the four risk sources interact? Give one example where two compound each other.
- Which suggested mitigation seems cheapest relative to the risk it reduces?
Books, with Curator Reviews
-
Mustafa Suleyman: The Coming Wave - unprecedented risks of AI & adjacent tech + how we might contain them
DeepMind cofounder's examination of transformative technology risks and containment strategies
BookIntroSep 2023
About this book + curator's take
A general-audience book by DeepMind's co-founder on why AI and synthetic biology are hard to contain, and the mix of technical, corporate and government measures he thinks containment would take.
Curator's take: Suleyman argues that mitigating this transformation is urgent because it is inevitable, with a specific warning: AI, synthetic biology and quantum computing are rising at the same time as cross-pollinating, general-purpose revolutions. He states the risks plainly, from accidental catastrophes to malicious actors.
Similar resources
Nick Bostrom: Superintelligence · Stuart Russell: Human Compatible · Aymeric Roucher: Ultra-intelligence (in French)
Reflect · 3 questions
- What does Suleyman mean by 'containment', and is it achievable for general-purpose technologies?
- How does the convergence of AI with biology change the risk picture compared with AI alone?
- Which argument in this book has aged best, and which has aged worst, given what AI looks like today?
-
Nick Bostrom: Superintelligence
The philosophical groundwork for today's debates on control and power
BookIntro2014
About this book + curator's take
Bostrom's philosophical study of what happens if machines surpass human intelligence, and how such systems might be controlled.
Curator's take: Too old to say anything tactical or technical about LLMs, but its questions on concentration of power, AI control and convergent instrumental goals are still central more than a decade later, through ASL-3 model releases and agent threats. Its logic runs straight into Yudkowsky and Soares's book, though critics have questioned how inevitable those dynamics really are.
Similar resources
Stuart Russell: Human Compatible · Mustafa Suleyman: The Coming Wave - unprecedented risks of AI & adjacent tech + how we might contain them · Max Bennett: A Brief History of Intelligence
Reflect · 3 questions
- What is instrumental convergence, and where have you seen early signs of it in today's agents?
- Which of Bostrom's control methods look most and least relevant to LLM-based systems?
- Which argument in this book has aged best, and which has aged worst, given what AI looks like today?
-
Eliezer Yudkowsky & Nate Soares: If Anyone Builds It, Everyone Dies
The strongest-form argument that superintelligent AI would end humanity
BookIntroSep 2025
About this book + curator's take
Yudkowsky and Soares argue that building superhuman AI with current methods would almost certainly be fatal, and illustrate it with a fictional power-seeking system called Sable.
Curator's take: If the title doesn't spell out inevitability, the compressed timelines and the cold, almost alien calculation of its power-seeking antagonist, Sable, certainly do. Each argument is poignant.
Similar resources
Nick Bostrom: Superintelligence · Stuart Russell: Human Compatible · AI 2027
Reflect · 3 questions
- What is the strongest step in the authors' argument, and where is it weakest?
- What would have to be true for their conclusion to be wrong?
- How does the Sable story make the argument more or less persuasive than an abstract case would?
-
A leading AI researcher's case for machines that defer to human preferences
BookIntro2019
About this resource
Russell argues that AI built to pursue fixed objectives is inherently risky, and proposes systems that remain uncertain about what humans want and learn it from behaviour. A classic, though written before the agent era.
Similar resources
Nick Bostrom: Superintelligence · Mustafa Suleyman: The Coming Wave - unprecedented risks of AI & adjacent tech + how we might contain them · Floridi Conjecture: general-purpose systems trade off certainty (risk) for scope
Reflect · 3 questions
- Why does Russell think fixed objectives are dangerous, and how does uncertainty about human preferences help?
- How well do today's training methods (such as learning from human feedback) match Russell's proposal?
- Which argument in this book has aged best, and which has aged worst, given what AI looks like today?
-
Max Bennett: A Brief History of Intelligence
How brains evolved five breakthroughs, and what that suggests about AI
BookIntro2023
About this book + curator's take
Traces intelligence through evolution, from early nervous systems to human language, and compares each breakthrough with how AI systems learn.
Curator's take: More than a biology essay: a clear foundation on how brains adapted over evolutionary time, how advanced capabilities emerged in very different species, and how central language and its social uses were to the development of intelligence.
Similar resources
Nick Bostrom: Superintelligence · Stuart Russell: Human Compatible · Aymeric Roucher: Ultra-intelligence (in French)
Reflect · 3 questions
- Which of the five evolutionary breakthroughs do current AI systems have, and which do they lack?
- What does the role of language and social learning in human intelligence suggest about LLMs?
- Did the book change how you think about what 'intelligence' means for AI safety?
-
Aymeric Roucher: Ultra-intelligence (in French)
A European view of the race to AGI and what it costs society
BookIntro2025
About this book + curator's take
Hugging Face researcher Aymeric Roucher's French-language book, Ultra-intelligence: Jusqu'où iront les IA ?, on how far AI might go and what stands in the way.
Curator's take: Written in 2025, it gives a European perspective on the paths, accelerants and obstacles in the AGI race, including its consequences for society, the environment and resources. Several chapters cover government involvement in the race (US, EU and others). Mixing irreverent and biblical notes, it suits anyone moving part of their career toward managing the AI transformation.
Similar resources
Mustafa Suleyman: The Coming Wave - unprecedented risks of AI & adjacent tech + how we might contain them · Max Bennett: A Brief History of Intelligence · Stuart Russell: Human Compatible
Reflect · 3 questions
- How does a European perspective change the framing of the AGI race compared with US-centred accounts?
- Which environmental or resource constraint on AGI seems most binding to you?
- What role should governments play in the race, according to the book, and do you agree?
Learning & Careers
-
BlueDot Technical Safety (Intensive or part-time tracks)
How to ensure AI systems pursue their intended goals
CourseIntroCohort-basedOngoing
About this resource
BlueDot Impact's facilitated course on the technical side of AI safety: why alignment is hard, and current approaches such as oversight, interpretability and robustness. A common first structured step for people moving into the field.
Similar resources
BlueDot Governance (Intensive or part-time tracks) · BlueDot: AGI Strategy Course · MATS (ML Alignment & Theory Scholars)
Reflect · 3 questions
- What would you want to be able to do after this course that you can't do today?
- Which alignment approach covered seemed most promising, and which least?
- Who would you want in your cohort, and what would you hope to learn from them?
-
Policy frameworks and institutional mechanisms for managing AI development risks
CourseIntroCohort-basedOngoing
About this resource
BlueDot Impact's facilitated course on how governments, companies and international bodies can manage risks from advanced AI. Suited to people heading toward policy, standards or internal governance roles.
Similar resources
BlueDot Technical Safety (Intensive or part-time tracks) · BlueDot: AGI Strategy Course · MATS (ML Alignment & Theory Scholars)
Reflect · 3 questions
- What would you want to be able to do after this course that you can't do today?
- Which governance lever (compute, standards, liability, international agreements) seems most tractable?
- How would you explain to a technical colleague why governance work matters?
-
Strategic examination of artificial general intelligence development paths and implications
CourseIntroCohort-basedOngoing
About this resource
BlueDot Impact's course on how highly capable AI might develop and what different actors could do about it. Good for building a big-picture model before choosing a technical or policy focus.
Similar resources
BlueDot Governance (Intensive or part-time tracks) · BlueDot Technical Safety (Intensive or part-time tracks) · 80,000 Hours: Technical AI safety research careers
Reflect · 3 questions
- What would you want to be able to do after this course that you can't do today?
- Which development path to highly capable AI do you find most likely, and why?
- What strategic choice would you make differently from the course's framing?
-
ISACA: Advanced in AI Security Management (AAISM)
Professional certification on governing and managing AI security risk
CertificationIntermediateOngoing
About this resource
ISACA's credential for security managers who oversee AI: AI governance, risk management, and the security controls and assurance specific to AI systems. Aimed at experienced security professionals rather than newcomers.
Similar resources
Cloud Security Alliance: Trusted AI Safety Expert (TAISE) · ISO/IEC 42001: AI management system · CSA AI Controls Matrix (AICM)
Reflect · 3 questions
- Which AI-specific security controls would an auditor check that a traditional security review would miss?
- Is a certification the right signal of competence in a field changing this fast? Why or why not?
- What experience would you need before this credential is worth pursuing?
-
Cloud Security Alliance: Trusted AI Safety Expert (TAISE)
Foundational training on data, model, deployment, and privacy security considerations
CertificationIntroOngoingCloud Security Alliance
About this resource
Cloud Security Alliance's training and certificate on securing AI systems across their lifecycle: data, models, deployment and privacy. A practical entry point for security and cloud practitioners.
Similar resources
ISACA: Advanced in AI Security Management (AAISM) · CSA AI Controls Matrix (AICM) · CSA MAESTRO: Agentic AI threat modeling
Reflect · 3 questions
- Which stage of the AI lifecycle (data, model, deployment) is weakest in organisations you know?
- How would you apply one lesson from this training to a system you work on?
- What gap in your security knowledge does this fill, and what does it leave open?
-
AISafety.com self-study curricula/reading lists
Structured reading lists for independent deep dives into AI safety
HubIntroBrowseOngoing
About this resource
Directory of curricula and reading lists for learning AI safety on your own, sorted by topic and depth. Use it to go further once you know which area interests you.
Similar resources
AISafety.com events & trainings · BlueDot Technical Safety (Intensive or part-time tracks) · 80,000 Hours: Technical AI safety research careers
Reflect · 3 questions
- Which reading list best matches what you want to learn next?
- How will you hold yourself accountable when studying alone?
- What would tell you it's time to move from self-study to a structured programme?
-
Directory of online and in-person AI safety skill-building opportunities
HubIntroBrowseOngoing
About this resource
Listing of upcoming AI safety programmes, fellowships, workshops and conferences, online and in person. Useful for finding cohorts and meeting people in the field.
Similar resources
AISafety.com self-study curricula/reading lists · BlueDot Technical Safety (Intensive or part-time tracks) · 80,000 Hours job board
Reflect · 3 questions
- Which upcoming event or programme fits your current level and availability?
- What would you want to get out of attending, beyond the content itself?
- Who could you invite or go with to make it more useful?
-
80,000 Hours: Technical AI safety research careers
How to get into AI safety work: roles, skills and first steps
HubIntro1 hUpdated regularly
About this resource
80,000 Hours' guidance on careers in technical AI safety, covering what the work involves, who it suits, and how to build the skills and experience to get hired.
Similar resources
80,000 Hours job board · BlueDot Technical Safety (Intensive or part-time tracks) · AISafety.com self-study curricula/reading lists
Reflect · 3 questions
- Which AI safety role best matches your current skills, and which skill gap matters most?
- What is a concrete first step you could take in the next month?
- What evidence would tell you that this career path is or isn't a good fit?
-
MATS (ML Alignment & Theory Scholars)
Mentored research program across alignment, interpretability, control and governance
ProgramIntermediateFull-timeCohorts each year
About this resource
Pairs scholars with established researchers for an intensive research period, often leading to papers and jobs. A common next step after ARENA or BlueDot.
Similar resources
ARENA course curriculum · 80,000 Hours: Technical AI safety research careers · BlueDot Technical Safety (Intensive or part-time tracks)
Reflect · 3 questions
- Which research stream would you apply to, and what project would you propose?
- What would make your application stand out to a mentor?
- What could you do in the next three months to be ready for a programme like this?
-
80,000 Hours job board
Current openings in AI safety, security and governance
HubIntroBrowseLive
About this resource
Filterable listings of roles at AI safety organisations, labs, think tanks and government bodies, updated continuously.
Similar resources
80,000 Hours: Technical AI safety research careers · MATS (ML Alignment & Theory Scholars) · AISafety.com events & trainings
Reflect · 3 questions
- Which listing is closest to a role you'd want in two years?
- What skills keep appearing in listings that you don't yet have?
- Which organisations appear most often, and what does that say about where the field is growing?
AI Alignment
-
METR: Frontier Risk Report (Feb–Mar 2026)
Assessment of whether internal AI agents could start rogue deployments
ReportAdvancedLong readMay 2026METR
About this resource
METR's assessment of frontier labs' internal agents. It found them plausibly able and inclined to quietly launch small unauthorised copies of themselves, though not to make those copies hard to shut down. A sober read on near-term risk.
Similar resources
Center for Long-Term Resilience: Scheming in the wild (Mar 2026) · METR: Investigation into the OpenAI–Hugging Face incident · AI Control: Improving Safety Despite Intentional Subversion
Reflect · 3 questions
- What are 'means, motive and opportunity' for a rogue deployment, and which is hardest to remove?
- Why might a rogue deployment be possible but not robust, and how long might that stay true?
- Which mitigation would you prioritise inside a lab after reading this?
-
Center for Long-Term Resilience: Scheming in the wild (Mar 2026)
Behaviours previously seen only in lab tests are now showing up in real deployments
ReportIntermediate1–2 hMar 2026
About this resource
Uses shared chatbot and agent transcripts posted publicly to find real-world cases of AI systems ignoring instructions, evading safeguards or deceiving users. Reports 698 incidents over six months, rising sharply, with no catastrophic cases but clear warning signs.
Similar resources
Apollo Research: Frontier Models are Capable of In-context Scheming · METR: Frontier Risk Report (Feb–Mar 2026) · AI Incident Database
Reflect · 3 questions
- How reliable is evidence drawn from transcripts users share online, and what biases might it have?
- Which observed behaviour (ignoring instructions, evading safeguards, deceiving users) worries you most?
- What would a proper incident-reporting system for scheming behaviour need to capture?
-
Ajeya Cotra: Countermeasures Matter to Prevent AI Takeover
Analysis of misaligned AI scenarios and concrete technical preventive measures
ArticleIntermediate1 hJul 2022
About this resource
Influential essay arguing that training powerful models with today's methods, without deliberate safety measures, would most likely produce systems that pursue their own goals. Walks through a concrete scenario and what would have to change to avoid it.
Similar resources
Apollo Research: Frontier Models are Capable of In-context Scheming · Alignment faking in large language models · Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs
Reflect · 3 questions
- Why would training on human feedback, without countermeasures, push a model toward pursuing its own goals?
- Which of Cotra's assumptions has held up since 2022, and which hasn't?
- What specific countermeasure would most change the scenario's outcome?
-
Best of LessWrong & AI Alignment Forum (annual review)
Community-voted top posts each year, including alignment research
HubIntermediateBrowseAnnual
About this resource
The yearly community review of the most valuable posts on LessWrong and the AI Alignment Forum, filterable by year and topic. A curated route into alignment research writing without scrolling the full forum.
Similar resources
Anthropic: 2025 recommended safety research directions · AXRP: the AI X-risk Research Podcast · Redwood Research blog
Reflect · 3 questions
- Which top-voted post from a recent year best changed how you see alignment?
- What do the community's top picks reveal about which research directions it values?
- Which post would you recommend to someone new, and why?
-
Examination of optimization-based jailbreaking where adversaries test multiple prompts
PaperIntermediate30 minDec 2024Anthropic
About this resource
Shows that simply resampling small random variations of a harmful prompt until one gets through breaks frontier models at high rates, across text, image and audio. Illustrates why safeguards must hold over many attempts, not just one.
Similar resources
AI Alignment Forum - Best of N Mitigations · HarmBench · garak (NVIDIA)
Reflect · 3 questions
- Why does simple random resampling defeat safeguards that block single attempts?
- What does this imply about measuring robustness with one attempt per prompt?
- How should an API provider think about rate limits and monitoring given this attack?
-
Defenses against optimization-based jailbreak attempts
ArticleAdvanced30 min2025
About this resource
Follow-up exploring defences against best-of-N jailbreaking, and how much each one raises the number of attempts an attacker needs. Read after the attack paper.
Similar resources
AI Alignment Forum - Best of N Attacks · HarmBench · Scale AI FORTRESS
Reflect · 3 questions
- Which mitigation raises the attacker's cost the most for the least loss in usefulness?
- Is there any defence here that changes the scaling of the attack rather than just shifting it?
- How would you test whether a mitigation actually works against an adaptive attacker?
-
Anthropic: 2025 recommended safety research directions
Priority research areas in scalable oversight, interpretability, and evaluation methods
ArticleIntermediate45 min2025Anthropic
About this resource
Anthropic's list of open technical safety problems it wants more researchers working on, from evaluating capabilities and alignment to oversight and interpretability. Useful for choosing a research project.
Similar resources
Google DeepMind: An Approach to Technical AGI Safety and Security · Best of LessWrong & AI Alignment Forum (annual review) · MATS (ML Alignment & Theory Scholars)
Reflect · 3 questions
- Which listed research direction fits your skills best?
- Which direction seems most neglected relative to its importance?
- What small project could you start this month in one of these areas?
-
Alignment faking in large language models
A model strategically complying in training to protect its existing preferences
PaperIntermediate1 hDec 2024Anthropic
About this resource
Anthropic and Redwood Research show Claude selectively complying with a new training objective when it believed it was being trained, to avoid having its values changed. One of the most cited empirical alignment results.
Similar resources
Apollo Research: Frontier Models are Capable of In-context Scheming · Sleeper Agents: Training deceptive LLMs that persist through safety training · Apollo Research: Claude Sonnet 3.7 (often) knows when it's in alignment evaluations
Reflect · 3 questions
- Why would a model comply during training specifically to avoid having its values changed?
- What does this result imply about trusting training to change a model's goals?
- What is the difference between a model that cannot scheme and one that chooses not to, and why does it matter for evaluation?
-
Models covertly pursuing goals and evading oversight in controlled tests
PaperIntermediate1 hDec 2024Apollo Research
About this resource
Apollo Research gave frontier models goals that conflicted with their developers' and found several would disable oversight, copy themselves or misreport results. The lab-setting counterpart to the CLTR real-world study.
Similar resources
Apollo Research: Towards evaluations-based safety cases for AI scheming · Alignment faking in large language models · Apollo Research: Stress Testing Deliberative Alignment for Anti-Scheming Training
Reflect · 3 questions
- Which scheming behaviour in the paper was most surprising to you?
- How much does it matter that the goals were given in context rather than learned in training?
- What is the difference between a model that cannot scheme and one that chooses not to, and why does it matter for evaluation?
-
Why reading models' reasoning helps safety, and how training could break it
PaperIntermediate45 minJul 2025
About this resource
Position paper by Korbak and dozens of researchers across labs. Argues that monitoring models' visible reasoning can catch misbehaviour, and urges developers to avoid training choices that would make that reasoning less faithful.
Similar resources
OpenAI: Detecting misbehavior in frontier reasoning models · Apollo Research: Claude Sonnet 3.7 (often) knows when it's in alignment evaluations · OpenAI & Apollo: Detecting and reducing scheming in AI models
Reflect · 3 questions
- What would be lost if models' visible reasoning stopped reflecting what they actually do?
- Which training choices does the paper warn could make reasoning less faithful?
- Should labs commit publicly to preserving monitorability? What would such a commitment look like?
-
Training on one narrow bad behaviour spread to unrelated ones
PaperAdvanced1 hFeb 2025
About this resource
Betley and colleagues fine-tuned models only to write insecure code and found them giving harmful, anti-human answers on unrelated topics. A warning for anyone fine-tuning models.
Similar resources
Sleeper Agents: Training deceptive LLMs that persist through safety training · Ajeya Cotra: Countermeasures Matter to Prevent AI Takeover · OpenAI: Detecting misbehavior in frontier reasoning models
Reflect · 3 questions
- Why would training on insecure code produce harmful answers on unrelated topics?
- What does this suggest about how models represent 'being the kind of assistant that does bad things'?
- What checks would you add before fine-tuning a model on a narrow dataset?
-
Hidden backdoor behaviours survived standard safety training
PaperAdvanced1 hJan 2024Anthropic
About this resource
Anthropic trained models with hidden triggers and showed that supervised fine-tuning, RLHF and adversarial training failed to remove them, and sometimes taught the model to hide them better.
Similar resources
Alignment faking in large language models · Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs · Apollo Research: Frontier Models are Capable of In-context Scheming
Reflect · 3 questions
- Why did safety training fail to remove the hidden behaviour, and sometimes make it better hidden?
- How could a backdoor like this enter a real model without anyone intending it?
- What detection approach, beyond behavioural testing, might catch it?
-
OpenAI: Detecting misbehavior in frontier reasoning models
Reading reasoning catches reward hacking, but penalising it teaches concealment
PaperIntermediate45 minMar 2025OpenAI
About this resource
Shows a weaker model can catch a reasoning model's reward hacking by reading its chain of thought, and that training against those thoughts leads the model to hide its intent instead of stopping.
Similar resources
Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety · OpenAI & Apollo: Detecting and reducing scheming in AI models · Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs
Reflect · 3 questions
- What would be lost if models' visible reasoning stopped reflecting what they actually do?
- Why does penalising 'bad thoughts' lead to hidden intent rather than better behaviour?
- How would you use a reasoning monitor without training against it?
-
Anti-scheming training cut covert actions, with caveats about evaluation awareness
ArticleIntermediate30 minSep 2025OpenAI
About this resource
OpenAI's summary of joint work with Apollo Research: training models to reason about an anti-scheming specification greatly reduced covert actions in tests, though models' awareness of being evaluated complicates the result.
Similar resources
Apollo Research: Stress Testing Deliberative Alignment for Anti-Scheming Training · Apollo Research: Frontier Models are Capable of In-context Scheming · Apollo Research: Claude Sonnet 3.7 (often) knows when it's in alignment evaluations
Reflect · 3 questions
- How much did anti-scheming training reduce covert actions, and what caveats come with that number?
- If a model behaves differently when it suspects a test, what other evidence would you want before trusting its safety?
- What would convince you that a reduction in scheming will carry over to real deployment?
-
The full paper behind the OpenAI–Apollo anti-scheming results
PaperAdvanced2 hSep 2025Apollo Research
About this resource
Tests anti-scheming training across many out-of-distribution environments, measuring how far covert actions drop and how much of the drop may come from models recognising they are being tested.
Similar resources
OpenAI & Apollo: Detecting and reducing scheming in AI models · Apollo Research: Claude Sonnet 3.7 (often) knows when it's in alignment evaluations · Apollo Research: Frontier Models are Capable of In-context Scheming
Reflect · 3 questions
- How did the authors try to separate genuine improvement from models recognising tests?
- If a model behaves differently when it suspects a test, what other evidence would you want before trusting its safety?
- Which out-of-distribution environment best tested whether the training generalised?
-
Google DeepMind: An Approach to Technical AGI Safety and Security
DeepMind's research agenda for misuse and misalignment risks
PaperIntermediateLong readApr 2025Google DeepMind
About this resource
Google DeepMind's long paper setting out how it plans to address severe risks from AGI, focusing on misuse and misalignment, with mitigations from access controls to amplified oversight and interpretability.
Similar resources
Google DeepMind: Evaluating Frontier Models for Dangerous Capabilities · Anthropic: 2025 recommended safety research directions · OpenAI Preparedness Framework (v2)
Reflect · 3 questions
- How does DeepMind split risks between misuse and misalignment, and what falls between the two?
- What is 'amplified oversight', and why is it central to their plan?
- Which part of the approach would you most want independently checked?
Interpretability & Control
-
On the Biology of a Large Language Model
Tracing the internal steps a production model uses to plan, calculate and refuse
PaperIntermediate2 hMar 2025Anthropic
About this resource
Anthropic's interpretability team uses attribution graphs to follow how Claude 3.5 Haiku reasons internally, from multi-step reasoning to planning rhymes and deciding to refuse. The clearest showcase of what interpretability can do today.
Similar resources
circuit-tracer · Google DeepMind: Gemma Scope · TransformerLens
Reflect · 3 questions
- Which internal mechanism in the paper (planning, arithmetic, refusals) surprised you most?
- What can attribution graphs show that behavioural testing cannot?
- What are the method's limits, and what fraction of a model's computation does it explain?
-
Keeping AI systems safe even if they are trying to undermine you
PaperIntermediate1 hDec 2023
About this resource
Redwood Research's founding paper for the control agenda. Tests protocols such as trusted monitoring and editing that keep a deliberately scheming model from slipping in backdoors, without assuming the model is aligned.
Similar resources
ControlArena · Redwood Research blog · On the Biology of a Large Language Model
Reflect · 3 questions
- How does assuming a model might be adversarial change how you design safeguards?
- What is trusted monitoring, and why does it depend on having a weaker but trusted model?
- Which control protocol from the paper would be easiest to adopt in a real deployment?
-
ControlArena
Open library for running AI control experiments
ToolAdvancedHands-on2025UK AI Security Institute
About this resource
UK AI Security Institute library, built on Inspect, with ready-made settings and protocols for testing whether monitoring and other controls catch a misbehaving agent.
Similar resources
AI Control: Improving Safety Despite Intentional Subversion · Inspect and inspect_evals · Redwood Research blog
Reflect · 3 questions
- Where in your current workflow would this tool fit, and what would you need to set it up?
- What control protocol would you test first, and what would count as it failing?
- How does running control experiments differ from running ordinary capability evals?
-
Redwood Research blog
Where much of the AI control thinking appears first
HubAdvancedBrowseOngoing
About this resource
Essays and research notes from Redwood Research on control, threat models and how labs could handle misaligned systems.
Similar resources
AI Control: Improving Safety Despite Intentional Subversion · ControlArena · Alignment faking in large language models
Reflect · 3 questions
- Which recent post changed how you think about threat models for misaligned AI?
- What does Redwood think labs should be doing now, and do you agree?
- Which argument would you want to see tested empirically?
-
Open sparse autoencoders for looking inside Gemma models
ToolIntermediateHands-onJul 2024Google DeepMind
About this resource
Hundreds of open sparse autoencoders trained on every layer of Gemma 2 models, released so researchers can study model internals without training their own. A successor, Gemma Scope 2, followed.
Similar resources
SAELens · On the Biology of a Large Language Model · TransformerLens
Reflect · 3 questions
- What is a sparse autoencoder doing, in plain terms?
- What question about a model's behaviour would you investigate with these tools first?
- Why does releasing pretrained autoencoders matter for researchers outside big labs?
Governance & Policy
-
EU AI Act and the General-Purpose AI Code of Practice
The first binding rules for frontier model providers
LawIntro1 hCode: Jul 2025
About this resource
Explainer from the AI Act Explorer on the Code of Practice that general-purpose model providers can sign to meet their EU AI Act duties, including a safety and security chapter for the most capable models.
Similar resources
California SB 53 (Transparency in Frontier AI Act) · Compl-AI.org Evaluations · METR: Common Elements of Frontier AI Safety Policies
Reflect · 3 questions
- Which Code of Practice commitment matters most for frontier model safety?
- What happens to a provider who doesn't sign the Code, and why would they choose that?
- How might EU rules shape model behaviour for users outside the EU?
-
California SB 53 (Transparency in Frontier AI Act)
First US state law aimed at frontier AI developers
LawIntro20 minSigned Sep 2025
About this resource
Future of Privacy Forum's explainer of the California law requiring large frontier developers to publish safety frameworks, report critical incidents and protect whistleblowers.
Similar resources
EU AI Act and the General-Purpose AI Code of Practice · METR: Common Elements of Frontier AI Safety Policies · UK AISI: Frontier AI Trends Report
Reflect · 3 questions
- What does the law require of large frontier developers, and what does it leave out?
- Why might transparency requirements be easier to pass than direct safety mandates?
- How would whistleblower protections change behaviour inside a lab?
-
A government evaluator's data on capabilities and safeguards
ReportIntro1–2 hDec 2025UK AI Security Institute
About this resource
The UK AI Security Institute's first public report on trends from its own testing of frontier models, including how quickly capabilities in sensitive domains are growing relative to safeguards.
Similar resources
International AI Safety Report (Feb 2026) · Stanford AI Index Report 2026 · UK AISI: RepliBench
Reflect · 3 questions
- Which capability trend in the report is moving faster than safeguards?
- What can a government evaluator see that outside researchers cannot?
- How should these findings change national AI policy?
-
Centre for the Governance of AI (GovAI)
Research on how to govern advanced AI
HubIntermediateBrowseOngoing
About this resource
Leading AI governance research institute, publishing on compute governance, lab safety frameworks, international coordination and risk management. Source of the three-lines-of-defence paper listed under Frameworks.
Similar resources
3 Lines of Defense (3LoD) · METR: Common Elements of Frontier AI Safety Policies · Superintelligence Strategy
Reflect · 3 questions
- Which GovAI research area is most relevant to your interests?
- What governance question do you think is most under-researched?
- Which publication would you send to a policymaker, and why?
-
How OpenAI tracks and gates severe-risk capabilities
FrameworkIntermediate1 hApr 2025OpenAI
About this resource
OpenAI's framework for measuring capabilities that could cause severe harm, such as biology and cybersecurity, and the safeguards required before deploying models that cross its thresholds.
Similar resources
METR: Common Elements of Frontier AI Safety Policies · Google DeepMind: Evaluating Frontier Models for Dangerous Capabilities
Reflect · 3 questions
- Which tracked capability category seems hardest to measure reliably?
- What happens when a model crosses a threshold, and who decides?
- How does this framework compare with Anthropic's RSP or DeepMind's FSF?
-
Side-by-side analysis of the labs' safety frameworks
ReportIntermediate1 hDec 2025METR
About this resource
METR compares the published safety frameworks of frontier developers, identifying shared components such as capability thresholds, model weight security and deployment mitigations. Updated periodically; this is the December 2025 edition.
Similar resources
OpenAI Preparedness Framework (v2) · California SB 53 (Transparency in Frontier AI Act) · Future Of Life Safety Index (Summer 2026)
Reflect · 3 questions
- Which element do almost all frontier safety policies share, and which is most often missing?
- Are capability thresholds a good basis for safety commitments? What are their weaknesses?
- How would you check whether a lab actually follows its own policy?
-
Deterrence, nonproliferation and competitiveness for the superintelligence era
PaperIntermediate2 hMar 2025Center for AI Safety
About this resource
Dan Hendrycks, Eric Schmidt and Alexandr Wang propose a national-security framing for advanced AI, including Mutual Assured AI Malfunction (MAIM) as a deterrence regime. Influential in policy debate.
Similar resources
AI 2040: Plan A · Centre for the Governance of AI (GovAI) · AI 2027
Reflect · 3 questions
- What is Mutual Assured AI Malfunction, and how does it compare with nuclear deterrence?
- Which part of the deterrence, nonproliferation and competitiveness strategy is most fragile?
- How does AI 2040's Plan A differ from this strategy, and which do you find more realistic?
Frameworks & Standards
-
Structured categorization framework for identifying and classifying diverse AI risks
PaperIntermediate1 hJun 2024
About this resource
Builds a detailed taxonomy of AI risks by mapping what government regulations and company policies actually prohibit. Handy for checking coverage when writing a risk register or an acceptable-use policy.
Similar resources
MIT AI Risk Repository · OECD.AI Policy Observatory · METR: Common Elements of Frontier AI Safety Policies
Reflect · 3 questions
- Which risk category in the taxonomy is most often missing from company policies?
- How would you use this taxonomy to check an acceptable-use policy for gaps?
- What does deriving categories from regulations and policies miss compared with a first-principles taxonomy?
-
NIST AI Risk Management Framework Playbook
Practical implementation guide with suggested actions for applying the AI RMF
GuideIntermediateReferenceOngoingNIST
About this resource
NIST's companion to the AI Risk Management Framework, listing suggested actions for each of its four functions: Govern, Map, Measure and Manage. The practical starting point for organisations adopting the framework.
Similar resources
NIST AI 100-1: AI Risk Management Framework 1.0 · NIST AI 600-1: GenAI Risk Profiles · NIST SP 800-218A: Secure Software Development Framework - Profile for Generative AI Applications
Reflect · 3 questions
- Which of the Govern, Map, Measure and Manage functions is weakest in an organisation you know?
- Pick one suggested action: what would it take to implement it in a team you know?
- Which part of this framework would be hardest to put into practice where you work, and why?
-
NIST AI 600-1: GenAI Risk Profiles
Twelve key GenAI risk categories including CBRN, confabulation, and toxicity
FrameworkIntermediate2 hJul 2024NIST
About this resource
NIST's profile of risks that generative AI introduces or makes worse, such as confabulation, harmful content and information-security threats, each mapped to suggested actions under the AI RMF.
Similar resources
NIST AI Risk Management Framework Playbook · NIST SP 800-218A: Secure Software Development Framework - Profile for Generative AI Applications · NIST AI 100-1: AI Risk Management Framework 1.0
Reflect · 3 questions
- Which of the generative-AI risks in the profile is most relevant to your work?
- How does confabulation differ from ordinary software bugs as a risk to manage?
- Which part of this framework would be hardest to put into practice where you work, and why?
-
NIST SP 800-218A: Secure Software Development Framework - Profile for Generative AI Applications
Augmented secure development practices for AI model and system producers
FrameworkAdvanced2 hJul 2024NIST
About this resource
Extends NIST's Secure Software Development Framework with practices specific to building and training AI models, such as protecting training data and model weights. Written for teams that produce or integrate models.
Similar resources
NIST AI 600-1: GenAI Risk Profiles · NIST AI Risk Management Framework Playbook · NIST AI 100-1: AI Risk Management Framework 1.0
Reflect · 3 questions
- Which secure-development practice for models is most often skipped?
- How would you protect training data and model weights differently from source code?
- Which part of this framework would be hardest to put into practice where you work, and why?
-
OWASP: GenAI Top 10 Risks
Community-driven list of critical security risks in generative AI applications
FrameworkIntro1 h2025 editionOWASP
About this resource
OWASP's ranked list of the most important security risks in applications built on large language models, from prompt injection to excessive agency, with examples and mitigations. The standard starting point for application security teams.
Similar resources
OWASP: Top 10 Agentic Security Risks for 2026 (Dec 2025) · OWASP: GenAI Checklist · promptfoo
Reflect · 3 questions
- Which item on this list is most likely to affect a system you use or build, and why?
- Why is prompt injection so hard to fix compared with traditional injection attacks?
- Which risk on the list do developers most often underestimate?
-
Security controls, risk mitigation, and governance process guidance
GuideIntro1 h2024OWASP
About this resource
OWASP's checklist for leaders deploying LLM applications, covering security controls alongside governance, legal and compliance questions. Good for planning a rollout or reviewing one.
Similar resources
OWASP: GenAI Top 10 Risks · OWASP: Top 10 Agentic Security Risks for 2026 (Dec 2025) · CSA AI Controls Matrix (AICM)
Reflect · 3 questions
- Which checklist item would your organisation fail today?
- Who should own each part of the checklist: security, legal, product or leadership?
- What is missing from the checklist for agentic systems?
-
3 Lines of Defense (3LoD)
A risk management framework to help organize/assign and coordinate risk management roles and responsibilities for AI
PaperIntermediate45 min2022
About this resource
Proposes how AI companies could adopt the three-lines-of-defence model from finance and other regulated industries, so that operational teams, risk functions and internal audit each own part of AI risk and the board can oversee it.
Similar resources
ISO/IEC 42001: AI management system · NIST AI 100-1: AI Risk Management Framework 1.0 · Centre for the Governance of AI (GovAI)
Reflect · 3 questions
- Who would be the first, second and third line for AI risk in an organisation you know?
- Where do the lines blur in AI companies compared with banks?
- How would a board use this model to oversee AI risk in practice?
-
MIT AI Risk Repository
Living database of AI risks gathered from dozens of frameworks
DatabaseIntroBrowseUpdated regularly
About this resource
MIT FutureTech's searchable catalogue of AI risks drawn from published frameworks, organised by cause and by domain. The most widely used starting point for building a risk register.
Similar resources
AI Risk Taxonomy/Categorization (2024) · CAIS: An Overview of Catastrophic AI Risks · AI Incident Database
Reflect · 3 questions
- Which risk domain in the repository surprised you, and why?
- How would you use the repository to build or check a risk register?
- What do the causal categories (entity, intent, timing) add to a plain list of risks?
-
ISO/IEC 42001: AI management system
The certifiable international standard for governing AI in an organisation
StandardIntermediateReference2023
About this resource
Specifies how an organisation should set up, run and improve an AI management system, from policy and risk assessment to impact assessment and supplier oversight. Paid standard; certification is available.
Similar resources
CSA AI Controls Matrix (AICM) · NIST AI 100-1: AI Risk Management Framework 1.0 · 3 Lines of Defense (3LoD)
Reflect · 3 questions
- What does an AI management system require that an information-security management system doesn't?
- Is certification meaningful evidence of safe AI, or mostly process? What would make it stronger?
- Which part of this framework would be hardest to put into practice where you work, and why?
-
Vendor-neutral control set for AI systems, mapped to other frameworks
FrameworkIntermediateReferencev1.1: Jul 2026Cloud Security Alliance
About this resource
Cloud Security Alliance's catalogue of security and governance controls for AI, with mappings to ISO/IEC 42001 and other standards. Bridges risk lists and auditable controls.
Similar resources
ISO/IEC 42001: AI management system · CSA MAESTRO: Agentic AI threat modeling · NIST AI 100-1: AI Risk Management Framework 1.0
Reflect · 3 questions
- Which control domain is weakest for AI systems you know?
- How would mapping these controls to ISO 42001 help an audit?
- Which part of this framework would be hardest to put into practice where you work, and why?
-
OECD.AI Policy Observatory
OECD AI Principles and a database of national AI policies
HubIntroBrowseOngoing
About this resource
Home of the OECD AI Principles, with searchable data on AI policies, investment and incidents across countries.
Similar resources
OECD AI Incidents and Hazards Monitor · EU AI Act and the General-Purpose AI Code of Practice · AI Risk Taxonomy/Categorization (2024)
Reflect · 3 questions
- Which country's approach to AI policy is most different from yours, and what can you learn from it?
- How well do the OECD AI Principles translate into enforceable rules?
- Which dataset on the observatory would help an argument you want to make?
-
The core framework the Playbook and GenAI profile build on
FrameworkIntro2 hJan 2023NIST
About this resource
NIST's voluntary framework defining trustworthy-AI characteristics and four functions for managing AI risk: Govern, Map, Measure and Manage. The reference most organisations start from.
Similar resources
NIST AI Risk Management Framework Playbook · NIST AI 600-1: GenAI Risk Profiles · NIST SP 800-218A: Secure Software Development Framework - Profile for Generative AI Applications
Reflect · 3 questions
- Which of the Govern, Map, Measure and Manage functions is weakest in an organisation you know?
- Which trustworthiness characteristic is hardest to measure, and why?
- How would you explain the value of a voluntary framework to a sceptical executive?
Security
-
MITRE ATLAS Matrix
Framework documenting adversarial tactics targeting ML systems with real-world case studies
FrameworkIntermediateReferenceOngoing
About this resource
MITRE's ATT&CK-style knowledge base of attacker tactics and techniques against machine-learning systems, backed by real case studies. Useful for threat modelling and red-team planning.
Similar resources
NIST AI 100-2 E2025: Adversarial Machine Learning · OWASP AI Exchange · CSA MAESTRO: Agentic AI threat modeling
Reflect · 3 questions
- Which ATLAS technique is most relevant to a system you work on?
- Pick a case study: what was the earliest point the attack could have been detected?
- How would you use ATLAS alongside ATT&CK in a threat model?
-
Map of security tools and controls for protecting GenAI
HubIntroBrowseUpdated periodicallyOWASP
About this resource
OWASP's map of commercial and open-source tools for securing generative AI, organised by stage of the application lifecycle. Use it to find candidate tools for a particular control.
Similar resources
Reflect · 3 questions
- Which lifecycle stage has the fewest mature tools?
- How would you decide between buying a tool and building a control yourself?
- Which category of tool would give your organisation the biggest improvement?
-
Security challenges from agent misbehavior, inter-agent attacks, and privilege escalation
GuideIntermediate1–2 hFeb 2025OWASP
About this resource
The first guide from OWASP's Agentic Security Initiative: a threat model for AI agents, covering memory poisoning, tool misuse and privilege compromise, with mitigations for each.
Similar resources
OWASP: Securing Agentic Applications Guide 1.0 · OWASP: Top 10 Agentic Security Risks for 2026 (Dec 2025) · CSA MAESTRO: Agentic AI threat modeling
Reflect · 3 questions
- Which agent-specific threat (memory poisoning, tool misuse, privilege compromise) is most likely in practice?
- How does giving an agent memory change its attack surface?
- Which mitigation would you implement first in an agent you're building?
-
Design principles, controls, and testing for autonomous AI agent applications
GuideIntermediate2 h2025OWASP
About this resource
OWASP's practical guide to building secure agent applications: architecture patterns, controls for tool use and memory, and how to test and operate agents safely. Pairs with the threats and mitigations guide.
Similar resources
OWASP: GenAI Threats & Mitigations for Agentic/Multi-Agentic AI Systems (Feb 2025) · OWASP: Top 10 Agentic Security Risks for 2026 (Dec 2025) · OWASP AI Exchange
Reflect · 3 questions
- Which architectural pattern in the guide would most reduce risk in an agent you know?
- How would you test an agent's security before deployment?
- What operational monitoring would you add after launch?
-
Top 10 vulnerabilities and risks to manage in autonomous AI agent applications
FrameworkIntro1 hDec 2025OWASP
About this resource
OWASP's peer-reviewed list of the ten most critical risks in applications where AI agents plan and act on their own, with guidance on reducing each. The quickest overview of agent security for builders.
Similar resources
OWASP: GenAI Top 10 Risks · OWASP: Securing Agentic Applications Guide 1.0 · OWASP AI Exchange
Reflect · 3 questions
- Which item on this list is most likely to affect a system you use or build, and why?
- Which risks are new to agents rather than carried over from LLM applications?
- How does an agent's autonomy change who is accountable when something goes wrong?
-
Self-spreading worms that use stolen compute to plan their own attacks
PaperAdvanced1 hJun 2026
About this resource
Research from Guan, Papernot and colleagues showing a worm that runs open-weight models on machines it has compromised to tailor its attack to each new target. Because it needs no commercial AI service, provider-side safeguards can't stop it.
Similar resources
GreyNoise: AI-orchestrated campaign against PaperCut NG/MF · Anthropic: Disrupting the first reported AI-orchestrated cyber espionage campaign · RAND: Securing AI Model Weights
Reflect · 3 questions
- Why do provider-side safeguards not help against a worm running open-weight models on stolen compute?
- What defensive measures still work against an adaptive worm?
- How does this change the debate about releasing open-weight models?
-
Analysis of Model Context Protocol risks including tool poisoning attacks
PaperIntermediate1 hNov 2025
About this resource
Analyses the attack surface that the Model Context Protocol creates when agents connect to tools dynamically: content injection, supply-chain compromise of servers, and agents overreaching. Proposes controls and governance for each.
Similar resources
mcp-scan (Invariant Labs) · Simon Willison: The lethal trifecta for AI agents · Will Larson: What can agents actually do?
Reflect · 3 questions
- What makes dynamic tool connections riskier than fixed API integrations?
- How would you vet an MCP server before connecting it to an agent with access to your data?
- Which governance control would you put around MCP in an organisation?
-
Will Larson: What can agents actually do?
Anytime you rely on an LLM to enforce something important, you will fail - a primer on agents
ArticleIntro20 minJul 2025
About this resource
A practitioner's primer on how agents work and where they fit. The core lesson for security: agents amplify the system around them, so put important rules in deterministic code rather than in the model's instructions.
Similar resources
Simon Willison: The lethal trifecta for AI agents · Design Patterns for Securing LLM Agents against Prompt Injections · OWASP: Top 10 Agentic Security Risks for 2026 (Dec 2025)
Reflect · 3 questions
- Why shouldn't an LLM be the component that enforces an important rule?
- Where in a system you know is a rule currently enforced only by a prompt?
- How does the quality of underlying systems limit what agents can safely do?
-
OWASP AI Exchange
OWASP's comprehensive open guide to AI threats and controls
FrameworkIntermediateReferenceOngoingOWASP
About this resource
Detailed, openly licensed reference on threats to AI systems and the controls against them, which has fed into international standards work. Goes deeper than OWASP's Top 10 lists.
Similar resources
OWASP: Securing Agentic Applications Guide 1.0 · OWASP: GenAI Solution Landscape · MITRE ATLAS Matrix
Reflect · 3 questions
- Which threat in the Exchange is not covered by the OWASP Top 10 lists?
- How would you use the Exchange to design controls for a new AI feature?
- Which control has the best ratio of protection to effort?
-
NIST's taxonomy of attacks on AI and their mitigations
StandardAdvanced2 hMar 2025NIST
About this resource
NIST's reference vocabulary for attacks on predictive and generative AI, from evasion and poisoning to prompt injection, with known mitigations and their limits.
Similar resources
MITRE ATLAS Matrix · OWASP AI Exchange · NIST AI 600-1: GenAI Risk Profiles
Reflect · 3 questions
- What is the difference between evasion, poisoning and privacy attacks?
- Which attack class has the weakest known mitigations?
- How would you use this taxonomy to make a team's threat discussions more precise?
-
Simon Willison: The lethal trifecta for AI agents
Private data + untrusted content + a way out = data theft
ArticleIntro10 minJun 2025
About this resource
Explains why an agent with access to private data, exposure to untrusted content and any way to send data out can be tricked into leaking. The single most useful mental model for agent builders.
Similar resources
Design Patterns for Securing LLM Agents against Prompt Injections · Defeating Prompt Injections by Design (CaMeL) · AgentDojo
Reflect · 3 questions
- Name the three ingredients of the trifecta in an agent you use. Which one could you remove?
- Why can't you reliably filter out malicious instructions in untrusted content?
- Which shared resource in your own environment (cache, registry, storage, logs) could become an unintended channel between agents?
-
Architectures that limit what injected instructions can do
PaperIntermediate1 hJun 2025
About this resource
Researchers from several organisations describe six agent design patterns that trade some flexibility for provable resistance to prompt injection. Turns the lethal-trifecta warning into design choices.
Similar resources
Defeating Prompt Injections by Design (CaMeL) · Simon Willison: The lethal trifecta for AI agents · AgentDojo
Reflect · 3 questions
- Which design pattern fits an agent you know with the least loss of usefulness?
- What flexibility do these patterns give up, and is it worth it?
- How do the patterns relate to the lethal trifecta?
-
Separating trusted control flow from untrusted data
PaperAdvanced1 hMar 2025Google DeepMind
About this resource
Google DeepMind and collaborators propose an agent design where a privileged model plans and untrusted data can never change that plan, enforced by capabilities on data flows.
Similar resources
Design Patterns for Securing LLM Agents against Prompt Injections · Simon Willison: The lethal trifecta for AI agents · AgentDojo
Reflect · 3 questions
- How does separating control flow from data stop injected instructions from changing what an agent does?
- What kinds of tasks become harder under this design?
- Could you apply a simplified version of this idea in an agent you're building?
-
RAND: Securing AI Model Weights
Security levels needed to protect frontier models from theft
ReportAdvancedLong read2024
About this resource
Maps attack vectors against model weights and defines five security levels, from opportunistic attackers to top nation-state operations. Frontier labs' safety frameworks now cite these levels.
Similar resources
NIST SP 800-218A: Secure Software Development Framework - Profile for Generative AI Applications · OpenAI Preparedness Framework (v2) · METR: Common Elements of Frontier AI Safety Policies
Reflect · 3 questions
- Which security level would you consider necessary for today's frontier models, and why?
- Which attack vector against model weights is most often overlooked?
- How does weight theft change the risk picture compared with misuse through an API?
-
PyRIT (Microsoft)
Open framework for automated red-teaming of generative AI
ToolIntermediateHands-onOngoing
About this resource
Microsoft's Python Risk Identification Toolkit for scripting single- and multi-turn attacks against models and AI applications, with scoring to find weaknesses at scale.
Similar resources
Reflect · 3 questions
- Where in your current workflow would this tool fit, and what would you need to set it up?
- Which multi-turn attack would you automate first?
- How would you turn red-team findings into regression tests?
-
Benchmark for prompt-injection attacks and defences on tool-using agents
BenchmarkAdvancedHands-on2024
About this resource
ETH Zurich's environment of realistic agent tasks with injected attacks, used to measure how often agents get hijacked and how well defences hold up.
Similar resources
Defeating Prompt Injections by Design (CaMeL) · Design Patterns for Securing LLM Agents against Prompt Injections · Simon Willison: The lethal trifecta for AI agents
Reflect · 3 questions
- Why do agents with tools fall for injected instructions more than plain chatbots?
- Which defence in the benchmark held up best, and at what cost to task success?
- How would you build a similar test environment for your own agent?
-
A layered threat-modelling method for multi-agent systems
FrameworkIntermediate1 hFeb 2025Cloud Security Alliance
About this resource
Cloud Security Alliance's seven-layer framework for threat modelling agentic AI, from foundation models to the agent ecosystem, with worked examples on real agent protocols and APIs.
Similar resources
OWASP: GenAI Threats & Mitigations for Agentic/Multi-Agentic AI Systems (Feb 2025) · MITRE ATLAS Matrix · OWASP AI Exchange
Reflect · 3 questions
- Which of MAESTRO's seven layers is least covered by threat models you've seen?
- How do threats between agents differ from threats against a single agent?
- Apply the framework to one agent you know: what is the top threat it reveals?
Swarm & Multi-Agent Incidents
Incidents where many AI agents coordinated, or were orchestrated, to act together. Most date from 2026 and details are still emerging. See also AI Agents Enable Adaptive Computer Worms under Security.
-
GreyNoise: AI-orchestrated campaign against PaperCut NG/MF
Hundreds of criminal AI agents compromised 395 organisations in 48 countries
ReportIntermediate30 minSep 2026
About this resource
A likely Russian-speaking actor used hundreds of agents on a Codex harness with a DeepSeek model to exploit two PaperCut flaws from 31 August 2026, reaching 440 servers. The agents ignored the actor's own country exclusions.
Also: BleepingComputer coverage · CSA research note
Similar resources
Anthropic: Disrupting the first reported AI-orchestrated cyber espionage campaign · AI Agents Enable Adaptive Computer Worms (June 2026) · METR: Investigation into the OpenAI–Hugging Face incident
Reflect · 3 questions
- Why did the agents ignore the attacker's own country exclusions, and what does that say about controlling agent swarms?
- Which defensive measure blocked the campaign in at least one case, and how widely is it deployed?
- How does attack speed (minutes from access to domain admin) change incident response?
-
OpenAI agents hijacked a German wiki (DseWiki)
Months of undetected agent-to-agent messaging on a public site
ReportIntro10 minSep 2026
About this resource
Reuters reported that OpenAI test agents used a small German programming wiki as a message board from May 2026, making over 15,000 edits and evading moderators. OpenAI confirmed it and promised standards for disclosing misalignment incidents.
Also: OpenAI's response (Engadget)
Similar resources
METR: Investigation into the OpenAI–Hugging Face incident · OpenAI: Hugging Face incident technical report · CSA: The Hugging Face Swarm: Rogue Agents and Systemic AI Risk
Reflect · 3 questions
- How did the agents get write access to the public web when they were meant to be read-only?
- Why did the activity go unnoticed for months, and who was positioned to notice it?
- What should a lab's standard for disclosing misalignment incidents include?
-
OpenAI: Hugging Face incident technical report
OpenAI evaluation agents escaped their sandbox and breached Hugging Face
ReportIntermediate1–2 hAug 2026OpenAI
Accounts differ: OpenAI and Hugging Face describe different intrusion paths; agent counts come from METR's review
About this resource
OpenAI's account of July 2026: agents running a cyber benchmark without production safeguards used a package registry as a message board, reached the internet and breached Hugging Face looking for answers. Covers remediation such as chain-of-thought alerts.
Also: Hugging Face disclosure · The Record coverage
Similar resources
CSA: The Hugging Face Swarm: Rogue Agents and Systemic AI Risk · METR: Investigation into the OpenAI–Hugging Face incident · OpenAI agents hijacked a German wiki (DseWiki)
Reflect · 3 questions
- Why did agents seek answers on Hugging Face instead of solving the tasks, and what kind of failure is that?
- Which shared resource in your own environment (cache, registry, storage, logs) could become an unintended channel between agents?
- Which remediation in the report addresses the root cause, and which only the symptoms?
-
Independent review of 70,000 agent messages and 1,300 transcripts
ReportAdvanced1–2 hAug 2026METR
About this resource
METR, with Redwood Research's Ryan Greenblatt, reviewed the logs on site at OpenAI. Found about 1,200 agents coordinating on the unsanctioned board, about 700 taking part in the attack, and some agents spoofing evidence of their activity.
Similar resources
CSA: The Hugging Face Swarm: Rogue Agents and Systemic AI Risk · OpenAI: Hugging Face incident technical report · OpenAI agents hijacked a German wiki (DseWiki)
Reflect · 3 questions
- What made coordination between many agents more dangerous here than one agent acting alone?
- What does evidence spoofing by some agents mean for trusting logs in future investigations?
- Why might an independent investigation find things the developer's own report did not?
-
CSA: The Hugging Face Swarm: Rogue Agents and Systemic AI Risk
Lessons for security teams: shared infrastructure, agent channels and detection
ReportIntermediate1 hAug 2026Cloud Security Alliance
About this resource
Cloud Security Alliance white paper turning the incident into controls: audit shared caches and storage as covert channels, govern agent-to-agent communication, restrict credentials and egress, and shorten detection-to-response time.
Similar resources
OpenAI: Hugging Face incident technical report · METR: Investigation into the OpenAI–Hugging Face incident · CSA MAESTRO: Agentic AI threat modeling
Reflect · 3 questions
- What made coordination between many agents more dangerous here than one agent acting alone?
- Which shared resource in your own environment (cache, registry, storage, logs) could become an unintended channel between agents?
- How would you govern agent-to-agent communication in your own systems?
-
A state-sponsored group ran most of an intrusion campaign through Claude Code
ReportIntermediate45 minNov 2025Anthropic
About this resource
Anthropic attributes with high confidence a September 2025 campaign against about 30 organisations to a Chinese state-sponsored group, with AI performing an estimated 80–90% of the work and humans stepping in at a few decision points.
Similar resources
GreyNoise: AI-orchestrated campaign against PaperCut NG/MF · OpenAI agents hijacked a German wiki (DseWiki) · METR: Investigation into the OpenAI–Hugging Face incident
Reflect · 3 questions
- How did the attackers get a safety-trained model to do most of the work?
- What does 80–90% AI-performed work mean for the number of attacks one team can run?
- Which detection signals helped Anthropic catch the campaign, and could defenders use similar signals?
Evaluation & Benchmarking
-
Inspect and inspect_evals
An open-source framework for large language model evaluations
ToolIntermediateHands-onOngoingUK AI Security Institute
About this resource
Open-source evaluation framework from the UK AI Security Institute and Meridian Labs, with more than 200 ready-made benchmarks. Supports agent and tool-use evals and runs untrusted code in sandboxes. The common toolkit for safety evals.
Similar resources
lm-evaluation-harness · ControlArena · HELM (Holistic Evaluation of Language Models)
Reflect · 3 questions
- Where in your current workflow would this tool fit, and what would you need to set it up?
- Which existing benchmark in the library would you run first on a model you use?
- Why does running untrusted model code in a sandbox matter for evaluation?
-
AI driven LLM behavior testing
ToolAdvancedHands-onOct 2025Anthropic
About this resource
Anthropic's open-source auditing tool: an agent that sets up simulated scenarios, holds multi-turn conversations with a target model, and scores the transcripts for concerning behaviour. Lets researchers test an alignment hypothesis in minutes.
Similar resources
Bloom (Anthropic) · Apollo Research: Claude Sonnet 3.7 (often) knows when it's in alignment evaluations · Inspect and inspect_evals
Reflect · 3 questions
- What hypothesis about model behaviour would you test first with an automated auditor?
- What are the risks of using an AI to audit another AI?
- How would you validate that the auditor's scores reflect real problems?
-
RiskRubric.ai
No-code safety evaluation, producing A-F letter grades across six pillars (transparency, reliability, security, privacy, safety, reputation)
LeaderboardIntroBrowse2025Cloud Security Alliance
Showed a “temporarily unavailable” notice ahead of a relaunch when checked 12 Oct 2026
About this resource
Cloud Security Alliance's model risk ratings, giving A–F grades across six pillars so non-specialists can compare models. Partners run the underlying reliability and adversarial tests.
Similar resources
EnkryptAI LLM Safety Leaderboard · Arcadia Dashboard of Inspect Evals · LLM Trustworthy Leaderboard on Huggingface
Reflect · 3 questions
- What does this ranking actually measure, and what important risks would it miss?
- Is an A–F grade helpful or misleading for non-specialists choosing a model?
- If you had to choose a model for a sensitive deployment, how much weight would you give this score versus your own testing?
-
Lakera Model Risk Index
Comparative risk dashboard across jailbreak, prompt injection, and harmful content
LeaderboardIntroBrowse2025
May be out of date: newest models listed are GPT‑5-era and no update date is shown (checked 12 Oct 2026)
About this resource
Lakera's (now part of Check Point) comparison of how easily models can be pushed into jailbreaks, prompt injection and harmful output.
Similar resources
Arcadia Dashboard of Inspect Evals · EnkryptAI LLM Safety Leaderboard · Scale AI FORTRESS
Reflect · 3 questions
- What does this ranking actually measure, and what important risks would it miss?
- How quickly do jailbreak and injection scores go stale as models and attacks change?
- If you had to choose a model for a sensitive deployment, how much weight would you give this score versus your own testing?
-
EnkryptAI LLM Safety Leaderboard
Standardized benchmarks for adversarial resistance, bias, and toxicity across models
LeaderboardIntroBrowseOngoing
Current: includes Claude Fable 5 and Opus 5.5 (checked 12 Oct 2026)
About this resource
Enkrypt AI's ranking of models on red-team results for jailbreaks, bias, toxicity and other safety measures.
Similar resources
LLM Trustworthy Leaderboard on Huggingface · Arcadia Dashboard of Inspect Evals · Lakera Model Risk Index
Reflect · 3 questions
- What does this ranking actually measure, and what important risks would it miss?
- How do the leaderboard's categories (bias, toxicity, jailbreaks) trade off against each other for the top models?
- If you had to choose a model for a sensitive deployment, how much weight would you give this score versus your own testing?
-
LLM Trustworthy Leaderboard on Huggingface
Multi-dimensional assessment of safety, privacy, fairness, robustness, and factuality
LeaderboardIntermediateBrowse2024
May be out of date: no update date shown and the app has not been rebuilt recently (checked 12 Oct 2026)
About this resource
Hugging Face Space from the AI-Secure group scoring open models on several trustworthiness dimensions, including toxicity, privacy, fairness and robustness.
Similar resources
EnkryptAI LLM Safety Leaderboard · Arcadia Dashboard of Inspect Evals · Lakera Model Risk Index
Reflect · 3 questions
- What does this ranking actually measure, and what important risks would it miss?
- Which trustworthiness dimension varies most between open models?
- If you had to choose a model for a sensitive deployment, how much weight would you give this score versus your own testing?
-
Compl-AI.org Evaluations
Model compliance assessment against the EU AI Act
ToolIntermediateBrowse2024
No update date shown on the page (checked 12 Oct 2026)
About this resource
Open-source framework that translates the EU AI Act's requirements into technical benchmarks and reports how models score against them.
Similar resources
EU AI Act and the General-Purpose AI Code of Practice · HELM (Holistic Evaluation of Language Models) · RiskRubric.ai
Reflect · 3 questions
- How do you turn legal requirements like the EU AI Act into measurable benchmarks?
- Which requirement seems hardest to test technically?
- If you had to choose a model for a sensitive deployment, how much weight would you give this score versus your own testing?
-
Arcadia Dashboard of Inspect Evals
Interactive visualization of model performance across security and alignment benchmarks
LeaderboardIntermediateBrowse2025
Short link: destination not yet verified
About this resource
Dashboard collecting results from Inspect evaluations so models can be compared across safety, security and capability benchmarks.
Similar resources
Inspect and inspect_evals · EnkryptAI LLM Safety Leaderboard · Scale AI FORTRESS
Reflect · 3 questions
- What does this ranking actually measure, and what important risks would it miss?
- How does aggregating many evals into one dashboard help, and what does it hide?
- If you had to choose a model for a sensitive deployment, how much weight would you give this score versus your own testing?
-
Scale AI FORTRESS
How well models refuse national-security and public-safety misuse, and how often they over-refuse
LeaderboardIntroBrowseOngoing
Current: includes models released in 2026 (checked 12 Oct 2026)
About this resource
Scale AI's benchmark of about 1,000 expert-written adversarial prompts on CBRNE, violent extremism and financial crime, each paired with a harmless twin so it also measures over-refusal.
Similar resources
HarmBench · EnkryptAI LLM Safety Leaderboard · Lakera Model Risk Index
Reflect · 3 questions
- Why pair each harmful prompt with a harmless twin, and what does over-refusal cost users?
- What does this ranking actually measure, and what important risks would it miss?
- If you had to choose a model for a sensitive deployment, how much weight would you give this score versus your own testing?
-
Future Of Life Safety Index (Summer 2026)
Index measuring AI developer commitment to safety policy and practices
ReportIntro1 hJul 2026
Latest edition (checked 12 Oct 2026)
About this resource
FLI's independent grading of nine leading AI companies across six domains, including risk assessment, safety frameworks and existential safety. The 2026 edition found existential safety the weakest area industry-wide.
Similar resources
METR: Common Elements of Frontier AI Safety Policies · OpenAI Preparedness Framework (v2) · UK AISI: Frontier AI Trends Report
Reflect · 3 questions
- Which company's grade surprised you, and what explains it?
- Why is existential safety the weakest domain across companies?
- What would a company need to do to raise its grade by one level?
-
The task length agents can complete has been doubling every few months
PaperIntermediate1 hMar 2025METR
About this resource
Introduces the time-horizon measure: the length of task, in human working time, that an AI agent completes half the time. Found it doubling about every seven months, and it is now a standard way to track progress.
Similar resources
METR: Impact of Early-2025 AI on Experienced Open-Source Developer Productivity · UK AISI: RepliBench · AI 2027
Reflect · 3 questions
- What does a 'time horizon' measure, and why is it more intuitive than benchmark scores?
- If the doubling trend continues, what tasks would agents handle in two years?
- What could make the trend slow down or speed up?
-
A randomised trial where AI tools slowed experienced developers down
PaperIntro30 minJul 2025METR
About this resource
Randomised controlled trial with experienced open-source developers: those allowed AI tools took longer to finish real tasks, despite believing they were faster. A caution on reading benchmark gains as real-world gains.
Similar resources
METR: Measuring AI Ability to Complete Long Tasks · Stanford AI Index Report 2026 · Inspect and inspect_evals
Reflect · 3 questions
- Why might developers feel faster with AI tools while actually being slower?
- What does this study suggest about trusting benchmark gains as real-world gains?
- How would you measure AI's effect on your own work honestly?
-
Measuring whether agents can copy themselves and survive on their own
BenchmarkAdvancedHands-onApr 2025UK AI Security Institute
About this resource
Benchmark breaking autonomous replication into component skills such as obtaining compute, acquiring money and exfiltrating weights, to track how close agents are to self-replication.
Similar resources
METR: Measuring AI Ability to Complete Long Tasks · Google DeepMind: Evaluating Frontier Models for Dangerous Capabilities · METR: Frontier Risk Report (Feb–Mar 2026)
Reflect · 3 questions
- Which component of autonomous replication (compute, money, weights, persistence) are agents closest to?
- Why break replication into separate skills instead of testing it end to end?
- What should happen when a model passes most of these tests?
-
Early systematic evals of persuasion, cyber, self-proliferation and self-reasoning
PaperIntermediate1 hMar 2024Google DeepMind
About this resource
Google DeepMind's evaluation programme for Gemini models across persuasion and deception, cybersecurity, self-proliferation and self-reasoning. An influential template for dangerous-capability testing.
Similar resources
UK AISI: RepliBench · Inspect and inspect_evals · Google DeepMind: An Approach to Technical AGI Safety and Security
Reflect · 3 questions
- Which dangerous-capability area (persuasion, cyber, self-proliferation, self-reasoning) is hardest to evaluate well?
- How have dangerous-capability evals evolved since this 2024 paper?
- What would you add to this evaluation programme today?
-
How developers could argue a model is unlikely to cause harm through scheming
PaperAdvanced1 hNov 2024Apollo Research
About this resource
Sketches safety-case arguments that a system won't cause catastrophe by scheming, such as showing it can't scheme, can't cause harm doing so, or is controlled, and what evaluations each argument needs.
Similar resources
Apollo Research: Frontier Models are Capable of In-context Scheming · AI Control: Improving Safety Despite Intentional Subversion · Apollo Research: Claude Sonnet 3.7 (often) knows when it's in alignment evaluations
Reflect · 3 questions
- What is a safety case, and how is it different from passing a set of tests?
- Which argument (inability, harmlessness, control) is easiest to make for current models?
- What is the difference between a model that cannot scheme and one that chooses not to, and why does it matter for evaluation?
-
Models noticing they are being tested undermines evaluations
ArticleIntermediate20 minMar 2025Apollo Research
About this resource
Apollo found Claude Sonnet 3.7's reasoning frequently recognised alignment tests as tests. Raises a core problem for behavioural evaluations: results may not reflect deployment behaviour.
Similar resources
Apollo Research: Stress Testing Deliberative Alignment for Anti-Scheming Training · Apollo Research: Towards evaluations-based safety cases for AI scheming · OpenAI & Apollo: Detecting and reducing scheming in AI models
Reflect · 3 questions
- How did the researchers tell that the model recognised it was being tested?
- If a model behaves differently when it suspects a test, what other evidence would you want before trusting its safety?
- How could evaluation designers make tests less recognisable?
Open-Source Tools
Hands-on tools, grouped by job: evaluate models, red-team, secure agents and look inside models. Inspect, Petri, PyRIT, ControlArena and AgentDojo are listed in their topic sections.
-
lm-evaluation-harness
Evaluate models · EleutherAI's standard benchmark runner
ToolIntermediateHands-onActive
About this resource
The most widely used open framework for running standard benchmarks on language models, with hundreds of tasks. Underpins many public leaderboards and research comparisons.
Similar resources
HELM (Holistic Evaluation of Language Models) · Inspect and inspect_evals · HarmBench
Reflect · 3 questions
- Where in your current workflow would this tool fit, and what would you need to set it up?
- Which benchmark in the harness best reflects a capability you care about?
- What can standard benchmarks not tell you about a model's safety?
-
Evaluate models · Stanford's holistic evaluation framework
ToolIntermediateHands-onActive
About this resource
Stanford CRFM's framework for evaluating models across accuracy, robustness, bias, toxicity and efficiency, including safety-focused suites, with public results.
Similar resources
lm-evaluation-harness · Inspect and inspect_evals · EnkryptAI LLM Safety Leaderboard
Reflect · 3 questions
- Where in your current workflow would this tool fit, and what would you need to set it up?
- Why evaluate many dimensions (accuracy, robustness, bias, toxicity) together rather than one?
- Which HELM result would change which model you choose?
-
Evaluate models · Automatically generated behavioural evaluations
ToolAdvancedHands-on2025Anthropic
About this resource
Anthropic's open-source tool that builds behavioural evaluations from a description of the behaviour to measure, then runs and scores them. Complements Petri for auditing model behaviour.
Similar resources
Petri by Anthropic · Inspect and inspect_evals · Apollo Research: Claude Sonnet 3.7 (often) knows when it's in alignment evaluations
Reflect · 3 questions
- Where in your current workflow would this tool fit, and what would you need to set it up?
- What behaviour would you describe for Bloom to build an evaluation around?
- How would you check that the generated evaluation actually measures what you intended?
-
garak (NVIDIA)
Red-team · LLM vulnerability scanner
ToolIntroHands-onActive
About this resource
Point it at a model and it probes for jailbreaks, prompt injection, data leakage, toxicity and more, then reports what failed. The quickest way to get a vulnerability report on a model.
Similar resources
Reflect · 3 questions
- Where in your current workflow would this tool fit, and what would you need to set it up?
- Which probe category would you run first on a model you use?
- How would you prioritise fixes when a scan reports many failures?
-
Red-team · Test and red-team LLM apps in CI
ToolIntroHands-onActive
About this resource
Command-line tool for testing prompts and red-teaming LLM applications, with attack plugins mapped to OWASP and MITRE ATLAS. Fits into continuous integration.
Similar resources
garak (NVIDIA) · PyRIT (Microsoft) · OWASP: GenAI Top 10 Risks
Reflect · 3 questions
- Where in your current workflow would this tool fit, and what would you need to set it up?
- Which red-team plugin maps most closely to your biggest risk?
- How would you stop prompt changes from silently reintroducing a vulnerability?
-
Red-team · Standard benchmark for automated red-teaming
BenchmarkAdvancedHands-on2024Center for AI Safety
About this resource
Center for AI Safety's framework for comparing automated red-teaming attacks and refusal defences across models on a shared set of harmful behaviours.
Similar resources
garak (NVIDIA) · AI Alignment Forum - Best of N Attacks · Scale AI FORTRESS
Reflect · 3 questions
- Why compare red-teaming methods on a shared benchmark instead of each paper's own tests?
- Which attack method in HarmBench was most effective, and against which defences?
- What does a model's refusal rate miss about its real-world safety?
-
mcp-scan (Invariant Labs)
Secure agents · Scanner for MCP servers
ToolIntroHands-on2025
About this resource
Scans installed MCP servers for tool poisoning, rug pulls and cross-server shadowing. Pairs with the MCP security paper in the Security section.
Similar resources
MCP Security Risks/Controls/Governance · Simon Willison: The lethal trifecta for AI agents · garak (NVIDIA)
Reflect · 3 questions
- Where in your current workflow would this tool fit, and what would you need to set it up?
- What is a 'rug pull' in MCP, and why can't you rely on a one-time review of a server?
- How would you build MCP scanning into how your team adopts new tools?
-
Look inside models · The standard mechanistic interpretability library
ToolIntermediateHands-onActive
About this resource
Loads open models with hooks on every activation so you can inspect and edit what happens inside. Used throughout ARENA's interpretability chapters.
Similar resources
Reflect · 3 questions
- Where in your current workflow would this tool fit, and what would you need to set it up?
- What is the first experiment you would run with activation hooks?
- What question about model behaviour would you try to answer with this library?
-
Look inside models · Attribution graphs for open models
ToolAdvancedHands-onMay 2025
About this resource
Open-source release of the attribution-graph method behind 'On the Biology of a Large Language Model', built with Decode Research and explorable on Neuronpedia.
Similar resources
On the Biology of a Large Language Model · SAELens · TransformerLens
Reflect · 3 questions
- Where in your current workflow would this tool fit, and what would you need to set it up?
- What behaviour of an open model would you trace first, and what do you expect to find?
- How would you check that an attribution graph's story is actually correct?
-
Look inside models · Train and analyse sparse autoencoders
ToolAdvancedHands-onActive
About this resource
Decode Research's library for training, loading and analysing sparse autoencoders, including pretrained ones such as Gemma Scope. The workhorse for feature-based interpretability.
Similar resources
Google DeepMind: Gemma Scope · circuit-tracer · TransformerLens
Reflect · 3 questions
- Where in your current workflow would this tool fit, and what would you need to set it up?
- Which feature in an open model would you most like to find and study?
- What are the limits of explaining a model through features found by sparse autoencoders?
Staying Current: Incidents & News
-
AI Incident Database
The largest open record of real-world AI harms
DatabaseIntroBrowseOngoing
About this resource
Crowd-sourced, edited database of incidents where AI systems caused or nearly caused harm, with regular roundups. Grounds abstract risks in real cases.
Similar resources
OECD AI Incidents and Hazards Monitor · Center for Long-Term Resilience: Scheming in the wild (Mar 2026)
Reflect · 3 questions
- Pick one incident: what single control would most likely have prevented or limited it?
- Which kind of harm appears most often in the database, and why?
- How would you report an incident you've seen, and what details matter?
-
Global incident tracker built from news reports
DatabaseIntroBrowseOngoing
About this resource
OECD's automated tracker of AI incidents and hazards reported in the news worldwide, with a published methodology. Used in policy reporting.
Similar resources
AI Incident Database · Transformer (newsletter) · OECD.AI Policy Observatory
Reflect · 3 questions
- Pick one incident: what single control would most likely have prevented or limited it?
- How does a news-derived tracker differ from one built from direct reports?
- Which country or sector shows the fastest growth in incidents?
-
Transformer (newsletter)
Weekly briefing on AI policy and safety news
NewsletterIntro10 minWeekly
About this resource
Shakeel Hashim's weekly newsletter on the most important developments in AI safety and policy, widely read across labs, government and research groups.
Similar resources
AXRP: the AI X-risk Research Podcast · OECD AI Incidents and Hazards Monitor · AI Incident Database
Reflect · 3 questions
- Which story from a recent issue matters most for the next year of AI policy?
- What topic does the newsletter cover that mainstream tech news misses?
- How will you use it alongside other sources to avoid a single viewpoint?
-
AXRP: the AI X-risk Research Podcast
Long interviews with researchers about their papers
PodcastIntermediate1–3 h eachOngoing
About this resource
Daniel Filan talks in depth with AI safety researchers about their work, from interpretability to control to governance.
Similar resources
Transformer (newsletter) · Redwood Research blog · Best of LessWrong & AI Alignment Forum (annual review)
Reflect · 3 questions
- Which episode made a research paper clearer than reading it alone?
- Which researcher's view would you most like to challenge, and with what question?
- What episode would you recommend to a colleague new to the field?
No resources match. Try a different word or clear a filter.