Alle afleveringen
139 afleveringen, van nieuw naar oud
-
“AI safety techniques leveraging distillation” by Ryan Greenblatt
19 juni 2025 · 22 min
Subtitle: Distillation is cheap; how can we use it to improve safety?. Lees meer
It's currently possible to (mostly or fully) cheaply reproduce the performance of a model by training another (initially weaker) model to imitate the stronger model's outputs.1 I'll refer to this as distillation. In the case of RL, distilling the learned capabilities is much, much cheaper than the RL itself (especially if you are distilling back into the original base model). But even for pre-training, distilling is cheaper than the original training.2
In this post, I'll discuss how we could utilize distillation to potentially remove (or possibly detect) misalignment. I'll also discuss a few other applications.3 My overall take is that techniques utilizing distillation are mildly to moderately promising and the low cost of distillation might make them surprisingly viable, but it's quite tricky to reason about how effective these techniques are.
Distilling to remove misalignment
I'll [...]
---
Outline:
(01:08) Distilling to remove misalignment
(11:52) Detecting problematic actions using (distillation) training
(15:46) How does distillation interact with neuralese?
(16:45) Other techniques leveraging distillation
(16:49) Distillation as a (non-mechanistic) interpretability technique
(18:03) Distillation for precise capability and knowledge control
---
First published:
June 19th, 2025
Source:
https://redwoodresearch.substack.com/p/ai-safety-techniques-leveraging-distillation
---
Narrated by TYPE III AUDIO.
-
“When does training a model change its goals?” by Vivek Hebbar, Ryan Greenblatt
12 juni 2025 · 30 min
Subtitle: Can a scheming AI's goals really stay unchanged through training?. Lees meer
Here are two opposing pictures of how training interacts with deceptive alignment:
“goal-survival hypothesis”:1 When you subject a model to training, it can maintain its original goals regardless of what the training objective is, so long as it follows through on deceptive alignment (playing along with the training objective instrumentally). Even as it learns new skills and context-specific goals for doing well on the training objective, it continues to analyze these as instrumental to its original goals, and its values-upon-reflection aren’t affected by the learning process.
“goal-change hypothesis”: When you subject a model to training, its values-upon-reflection will inevitably absorb some aspect of the training setup. It doesn’t necessarily end up terminally valuing a close correlate of the training objective, but there will be some change in values due to the habits incentivized by [...]
---
Outline:
(03:27) Empirical evidence so far
(03:31) Alignment faking
(06:27) Sleeper agents
(13:50) Synthesizing the sleeper agents and alignment faking results
(15:20) Smaller-scale experiments similar to sleeper agents
(17:39) Urges vs. reflectively endorsed goals
(18:38) Human analogies
(18:50) A million years of laying bricks
(21:34) Doing a job for 40 years, with/without a heroin drip
(22:24) Murder Simulator
(24:08) Call of Duty addiction
(25:20) Would runners self-modify?
(27:01) Terminally valuing money
(27:52) Evolution vs. within-lifetime learning
(28:54) Author contributions & acknowledgements
---
First published:
June 12th, 2025
Source:
https://redwoodresearch.substack.com/p/when-does-training-a-model-change
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
-
“The case for countermeasures to memetic spread of misaligned values” by Alex Mallen
28 mei 2025 · 14 min
Subtitle: Defending against alignment problems that might come with long-term memory. Lees meer
As various people have written about before, AIs that have long-term memory might pose additional risks (most notably, LLM AGI will have memory, and memory changes alignment by Seth Herd). Even if an AI is aligned or only occasionally scheming at the start of a deployment, the AI might become a consistent and coherent behavioral schemer via updates to its long-term memories.
In this post, I’ll spell out the version of the threat model that I’m most concerned about, including some novel arguments for its plausibility, and describe some promising strategies for mitigating this risk. While I think some plausible mitigations are reasonably cheap and could be effective at reducing the risk from coherent scheming arising via this mechanism, research here will likely be substantially more productive in the future once models more effectively utilize long-term memory.
[...]
---
Outline:
(01:19) The memetic spread threat model
(06:54) Countermeasures
(06:57) How much do existing safety agendas help?
(09:13) Targeted countermeasures for memetic spread of misaligned values
(11:57) Discussion
---
First published:
May 28th, 2025
Source:
https://redwoodresearch.substack.com/p/the-case-for-countermeasures-to-memetic
---
Narrated by TYPE III AUDIO.
-
“AIs at the current capability level may be important for future safety work” by Ryan Greenblatt
12 mei 2025 · 7 min
Subtitle: Some reasons why relatively weak AIs might still be important when we have very powerful AIs. Lees meer
Sometimes people say that it's much less valuable to do AI safety research today than it will be in the future, because the current models are very different—in particular, much less capable—than the models that we’re actually worried about. I think that argument is mostly right, but it misses a couple of reasons models as capable as current models might still be important at the point where AI is posing substantial existential risk. In this post, I’ll describe those reasons.
Our best trusted models might be at a similar level of capabilities to current models: It might be the case that we are unable to make models which are much more powerful than current models while still being confident that these models aren't scheming against us. That is, the most powerful [...]
---
First published:
May 12th, 2025
Source:
https://redwoodresearch.substack.com/p/ais-at-the-current-capability-level
---
Narrated by TYPE III AUDIO.
-
“Misalignment and Strategic Underperformance: An Analysis of Sandbagging and Exploration Hacking” by Julian Stastny, Buck Shlegeris
8 mei 2025 · 30 min
Subtitle: A new analysis of the risk of AIs intentionally performing poorly.. Lees meer
In the future, we will want to use powerful AIs on critical tasks such as doing AI safety and security R&D, dangerous capability evaluations, red-teaming safety protocols, or monitoring other powerful models. Since we care about models performing well on these tasks, we are worried about sandbagging: that if our models are misaligned1, they will intentionally underperform.
Sandbagging is crucially different from many other situations with misalignment risk, because it involves models purposefully doing poorly on a task, rather than purposefully doing well. When people talk about risks from overoptimizing reward functions (e.g. as described in What failure looks like), the concern is that the model gets better performance (according to some metric) than an aligned model would have. And when they talk about scheming, the concern is mostly that the model gets performance that is [...]
---
Outline:
(03:55) Sandbagging can cause a variety of problems
(08:13) Training makes sandbagging significantly harder for the AIs
(09:58) Training on high-quality data can remove sandbagging
(12:51) If off-policy data is low-quality, on-policy data might help
(15:12) Models might subvert training via exploration hacking
(19:31) Off-policy data could mitigate exploration hacking
(24:56) Quick takes on other countermeasures
(26:49) Conclusion and prognosis
---
First published:
May 8th, 2025
Source:
https://redwoodresearch.substack.com/p/misalignment-and-strategic-underperformance
---
Narrated by TYPE III AUDIO.
-
“Training-time schemers vs behavioral schemers” by Alex Mallen
6 mei 2025 · 13 min
Subtitle: Clarifying ways in which faking alignment during training is neither necessary nor sufficient for the kind of scheming that AI control tries to defend against.. Lees meer
People use the word “schemer” in two main ways:
“Scheming” (or similar concepts: “deceptive alignment”, “alignment faking”) is often defined as a property of reasoning at training-time1. For example, Carlsmith defines a schemer as a power-motivated instrumental training-gamer—an AI that, while being trained, games the training process to gain future power. I’ll call these training-time schemers.
On the other hand, we ultimately care about the AI's behavior throughout the entire deployment, not its training-time reasoning, because, in order to present risk, the AI must at some point not act aligned. I’ll refer to AIs that perform well in training but eventually take long-term power-seeking misaligned actions as behavioral schemers. When people say that a model is a schemer and [...]
---
Outline:
(02:23) Training-time schemers that continue to act aligned for the whole deployment
(02:42) Empirical evidence
(05:19) Theoretical argument
(08:01) Related ambiguity about situational awareness
(09:12) Behavioral schemers that weren't training-time schemers
(12:00) Discussion
---
First published:
May 6th, 2025
Source:
https://redwoodresearch.substack.com/p/training-time-schemers-vs-behavioral
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
-
“What’s going on with AI progress and trends? (As of 5/2025)” by Ryan Greenblatt
3 mei 2025 · 17 min
Subtitle: My views on what's driving AI progress and where it's headed.. Lees meer
AI progress is driven by improved algorithms and additional compute for training runs. Understanding what is going on with these trends and how they are currently driving progress is helpful for understanding the future of AI. In this post, I'll share a wide range of general takes on this topic as well as open questions. Be warned that I'm quite uncertain about a bunch of this!
This post will assume some familiarity with what is driving AI progress, specifically it will assume you understand the following concepts: pre-training, RL, scaling laws, effective compute.
Training compute trends
Epoch reports a trend of frontier training compute increasing by 4.5x per year. My best guess is that the future trend will be slower, maybe more like 3.5x per year (or possibly much lower) for a few reasons:
[...]
---
Outline:
(00:54) Training compute trends
(08:25) Algorithmic progress
(15:04) Data
---
First published:
May 3rd, 2025
Source:
https://redwoodresearch.substack.com/p/whats-going-on-with-ai-progress-and
---
Narrated by TYPE III AUDIO.
-
“How can we solve diffuse threats like research sabotage with AI control?” by Vivek Hebbar
30 april 2025 · 16 min
Subtitle: Preventing research sabotage will require techniques very different from the original control paper.. Lees meer
Misaligned AIs might engage in research sabotage: making safety research go poorly by doing things like withholding their best ideas or putting subtle bugs in experiments. To mitigate this risk with AI control, we need very different techniques than those in the original control paper or Ctrl-Z, because research sabotage differs from those settings in two crucial ways:
Firstly, the AI can’t cause a catastrophe by behaving badly on just one occasion. To derail the entire project of solving alignment, the AI probably has to take a large number of malign actions. This contrasts with some other threat models (like rogue internal deployments), where the AI might succeed by acting badly on just a few occasions.
Secondly and relatedly, individual bad actions in research sabotage aren’t very incriminating. If a model [...]
---
Outline:
(03:12) Threat model and basic countermeasures
(03:55) Deployer strategies: Training and incrimination
(08:34) Sabotage strategies
(08:56) Targeting sabotage at the most effective points
(09:33) Withholding good content vs. producing bad content
(11:52) Code vs. ideas
(12:29) What terms should we use for the concentrated/diffuse spectrum?
(14:58) Canary string
(15:12) Acknowledgements
---
First published:
April 30th, 2025
Source:
https://redwoodresearch.substack.com/p/how-can-we-solve-diffuse-threats
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
-
“7+ tractable directions in AI control” by Ryan Greenblatt
29 april 2025 · 27 min
Subtitle: A list of easy-to-start directions in AI control targeted at independent researchers without as much context or compute. Lees meer
In this post, we list 7 of our favorite easy-to-start directions in AI control. (Really, projects that are at least adjacent to AI control; We include directions which aren’t as centrally AI control and which also have other motivations.) This list is targeted at independent researchers who don't necessarily have a ton of context on AI control (but have at least some context via e.g., having read some of our posts on control) and who don't want to do infrastructure and compute heavy experiments. In the final section, we also list 3 additional directions that are probably a bit more challenging but still tractable.
We’ll mostly be highlighting and expanding upon areas that Ryan discussed in an earlier post: An overview of areas of control work.
We don’t discuss [...]
---
Outline:
(01:29) Techniques for doing elicitation without learning and studying overelicitation
(03:35) Generating indistinguishable synthetic inputs
(07:09) Teaching models synthetic facts
(10:38) Further studying exploration hacking
(14:00) Malign AI agent substitution
(15:02) Studying data poisoning
(16:55) Training attack policies
(18:58) Bonus: 3 slightly more challenging (but important and tractable) directions
(19:05) Few-shot catastrophe prevention
(20:48) Studying collusion and developing countermeasures
(22:53) Studying steganography and developing countermeasures
The original text contained 1 footnote which was omitted from this narration.
---
First published:
April 29th, 2025
Source:
https://redwoodresearch.substack.com/p/7-tractable-directions-in-ai-control
---
Narrated by TYPE III AUDIO.
-
“Clarifying AI R&D threat models” by Josh Clymer
25 april 2025 · 10 min
Subtitle: (There are a few). Lees meer
A casual reader of one of the many AI company safety frameworks might be confused about why “AI R&D” is listed as a “threat model.” They might be even further mystified to find out that some people believe risks from automated AI R&D are the most severe and urgent risks. What's so concerning about AI writing code?
A few months ago, I was frustrated by the lack of a thorough published answer to this question. So when the Safe AI Forum (the organizers of IDAIS) reached out to me about a workshop convening international experts from China, the US, Canada and Europe to discuss these threat models, I was eager to participate.
I ended up being a rather enthusiastic attendee – pulling an all-nighter to throw together the beginnings of a paper that became the output of the workshop. The next day, my [...]
---
Outline:
(02:06) Autonomous AI R&D is dangerous for two reasons that are often conflated
(02:52) Risks from AI R&D automation
(05:04) Risks from AI R&D acceleration
(07:53) Bare minimum recommendations for managing AI R&D risks
(09:06) Conclusion
---
First published:
April 25th, 2025
Source:
https://redwoodresearch.substack.com/p/clarifying-ai-r-and-d-threat-models
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
-
“How training-gamers might function (and win)” by Vivek Hebbar
24 april 2025 · 33 min
Subtitle: A model of the relationship between higher level goals, explicit reasoning, and learned heuristics in capable agents.. Lees meer
In this post I present a model of the relationship between higher level goals, explicit reasoning, and learned heuristics in capable agents. This model suggests that given sufficiently rich training environments (and sufficient reasoning ability), models which terminally value on-episode reward-proxies are disadvantaged relative to training-gamers.
A key point is that training gamers can still contain large quantities of learned heuristics (context-specific drives). By viewing these drives as instrumental and having good instincts for when to trust them, a training-gamer can capture the benefits of both instinctive adaptation and explicit reasoning without paying much of a speed penalty.
Core claims
To get high reward, models must learn context-specific heuristics which are not derivable by reasoning from the goal of reward maximization.1 It is also true that thinking explicitly [...]
---
Outline:
(01:02) Core claims
(03:10) Characterizing training gamers and proxy-aligned models
(07:27) What about the case of humans and evolution? Wasn't it implausible to get an explicit fitness maximizer?
(08:25) What about the arguments in Reward is not the optimization target?
(09:24) What about aligned models?
(11:26) When will models think about reward?
(14:54) What I expect schemers to look like
(17:46) What will terminal reward seekers do off-distribution?
(18:29) What factors affect the likelihood of training-gaming?
(20:29) What about CoT models?
(21:02) Idealized relationship of subgoals to their superior goals
(22:50) A story about goal reflection
(25:28) Thoughts on compression
(28:19) Appendix: Distribution over worlds
(31:58) Canary string
(32:14) Acknowledgements
---
First published:
April 24th, 2025
Source:
https://redwoodresearch.substack.com/p/how-training-gamers-might-function
---
Narrated by TYPE III AUDIO…
-
“Handling schemers if shutdown is not an option” by Buck Shlegeris
18 april 2025 · 25 min
Subtitle: What if getting strong evidence of scheming isn't the end of your scheming problems, but merely the middle?. Lees meer
In most of our research and writing on AI control, we’ve emphasized the following situation:
The AI developer is deploying a model that they think might be scheming, but they aren’t sure.
The objective of the safety team is to ensure that if the model is scheming, it will be caught before successfully causing a catastrophe.
For example, in our original paper on AI control, the methodology we used for evaluating safety protocols implies that once the developer is confident that the model is a schemer, they’re totally fine with undeploying the untrusted model.1 2
But that seems kind of unlikely. Even if AI companies find strong evidence that their AIs are trying to escape, they won’t necessarily stop deploying them. This might be justified: even [...]
---
Outline:
(03:29) Behavioral observations vs other ways of detecting scheming
(08:43) What does control look like if you need to keep deploying schemers?
(15:46) How does the prospect of needing to deploy known schemers affect the relative value of control research?
(17:31) What research should be done that focuses specifically on this?
(17:36) Sample-efficient ML
(18:02) Training recalcitrant policies
(20:37) A few other historical reasons for focusing on before-you-catch
---
First published:
April 18th, 2025
Source:
https://redwoodresearch.substack.com/p/handling-schemers-if-shutdown-is
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
-
“Ctrl-Z: Controlling AI Agents via Resampling” by Buck Shlegeris
16 april 2025 · 4 min
Subtitle: A new paper on AI control for agents.. Lees meer
We have released a new paper, Ctrl-Z: Controlling AI Agents via Resampling. This is the largest and most intricate study of control techniques to date: that is, techniques that aim to prevent catastrophic failures even if egregiously misaligned AIs attempt to subvert the techniques. We extend control protocols to a more realistic, multi-step setting, develop novel resample protocols that outperform all existing control protocols, and study many aspects of control techniques in more detail than prior work. In this blog post, we summarize the main takeaways and lessons learned from our work.
Here's the abstract of the paper:
Control evaluations measure whether monitoring and security protocols for AI systems prevent intentionally subversive AI models from causing harm. Our work presents the first control evaluation performed in an agent environment. We construct BashBench, a dataset of 257 challenging multi-step system [...]
---
First published:
April 16th, 2025
Source:
https://redwoodresearch.substack.com/p/ctrl-z-controlling-ai-agents-via
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
-
“To be legible, evidence of misalignment probably has to be behavioral” by Ryan Greenblatt
15 april 2025 · 6 min
Subtitle: Evidence from just model internals (e.g. interpretability) is unlikely to be broadly convincing.. Lees meer
One key hope for mitigating risk from misalignment is inspecting the AI's behavior, noticing that it did something egregiously bad, converting this into legible evidence the AI is seriously misaligned, and then this triggering some strong and useful response (like spending relatively more resources on safety or undeploying this misaligned AI).
You might hope that (fancy) internals-based techniques (e.g., ELK methods or interpretability) allow us to legibly incriminate misaligned AIs even in cases where the AI hasn't (yet) done any problematic actions despite behavioral red-teaming (where we try to find inputs on which the AI might do something bad), or when the problematic actions the AI does are so subtle and/or complex that humans can't understand how the action is problematic1. That is, you might hope that internals-based methods allow us to legibly incriminate [...]
---
First published:
April 15th, 2025
Source:
https://redwoodresearch.substack.com/p/to-be-legible-evidence-of-misalignment
---
Narrated by TYPE III AUDIO.
-
“Why do misalignment risks increase as AIs get more capable?” by Ryan Greenblatt
11 april 2025 · 7 min
Subtitle: A breakdown of how higher capabilities increase risk. Lees meer
It's generally agreed that as AIs get more capable, risks from misalignment increase. But there are a few different mechanisms by which more capable models are riskier, and distinguishing between those mechanisms is important when estimating the misalignment risk posed at a particular level of capabilities or by a particular model.
There are broadly 3 reasons why misalignment risks increase with capabilities:
As AIs get more capable, the chance of egregious misalignment increases: AIs with more powerful opaque reasoning abilities are more likely to become egregiously misaligned—sufficiently misaligned that they strategically deceive and work against their developers—without this being detected or trained out. We should expect opaque reasoning abilities generally increase as models get more capable, though it isn't entirely obvious what the relationship between opaque and non-opaque cognition will be in future AI systems. Additionally, I [...]
---
First published:
April 11th, 2025
Source:
https://redwoodresearch.substack.com/p/why-do-misalignment-risks-increase
---
Narrated by TYPE III AUDIO.
-
“An overview of areas of control work” by Ryan Greenblatt
9 april 2025 · 55 min
Subtitle: What are all the research and implementation areas helpful for control?. Lees meer
In this post, I'll list all the areas of control research (and implementation) that seem promising to me.
This references framings and abstractions discussed in Prioritizing threats for AI control.
First, here is a division of different areas, though note that these areas have some overlaps:
Developing and using settings for control evaluations (including technique iteration)
Better understanding control-relevant capabilities and model properties
Developing specific countermeasures/techniques in isolation (rather than via iterating in a setting)
Doing control experiments on actual current AI usage
Prototyping or implementing software infrastructure for control and security measures that are particularly relevant to control
Prototyping or implementing human processes for control
Conceptual research and other ways of researching control other than empirical experiments
Nearer-term applications of control to practice and accelerate later [...]
---
Outline:
(01:58) Developing and using settings for control evaluations
(14:33) Better understanding control-relevant capabilities and model properties
(29:22) Developing specific countermeasures/techniques in isolation
(37:06) Doing control experiments on actual AI usage
(41:56) Prototyping or implementing software infrastructure for control and security measures that are particularly relevant to control
(43:13) Prototyping or implementing human processes for control
(48:10) Conceptual research and other ways of researching control other than empirical experiments
(51:01) Nearer-term applications of control to practice and accelerate later usage of control
---
First published:
April 9th, 2025
Source:
https://redwoodresearch.substack.com/p/an-overview-of-areas-of-control-work
---
Narrated by TYPE III AUDIO.
-
“An overview of control measures” by Ryan Greenblatt
6 april 2025 · 50 min
Subtitle: What methods can we use to ensure control?. Lees meer
We often talk about ensuring control, which in the context of this doc refers to preventing AIs from being able to cause existential problems, even if the AIs attempt to subvert our countermeasures. To better contextualize control, I think it's useful to discuss the main countermeasures (building on my prior discussion of the main threats). I'll focus my discussion on countermeasures for reducing existential risk from misalignment, though some of these countermeasures will help with other problems such as human power grabs.
I'll overview control measures that seem important to me, including software infrastructure, security measures, and other control measures (which often involve usage of AI or human labor). In addition to the core control measures (which are focused on monitoring, auditing, and tracking AIs and on blocking and/or resampling suspicious outputs), there is a long tail of potentially [...]
---
Outline:
(01:19) Background and categorization
(06:11) Non-ML software infrastructure
(07:11) The most important software infrastructure
(11:58) Less important software infrastructure
(19:38) AI and human components
(21:30) Core measures
(27:49) Other less-core measures
(32:45) More speculative measures
(41:30) Specialized countermeasures
(46:05) Other directions and what Im not discussing
(47:15) Security improvements
---
First published:
April 6th, 2025
Source:
https://redwoodresearch.substack.com/p/an-overview-of-control-measures
---
Narrated by TYPE III AUDIO.
-
“Buck on the 80,000 Hours podcast” by Buck Shlegeris
5 april 2025 · 1 min
Buck on the 80,000 Hours podcast Lees meer
My podcast with Rob Wiblin from 80,000 Hours just came out. I’m really happy with how it turned out. I talked about a bunch of stuff on the podcast that I don’t think we’ve written up before.
Transcript + links + summary here. You can also get it as a podcast:
Spotify:
Apple:
Subscribe to Redwood Research blog
Launched a year ago
We research catastrophic AI risks and techniques that could be used to mitigate them.
Discussion about this post
---
Outline:
(00:09) Buck on the 80,000 Hours podcast
(00:34) Subscribe to Redwood Research blog
(00:57) Discussion about this post
---
First published:
April 5th, 2025
Source:
https://redwoodresearch.substack.com/p/buck-on-the-80000-hours-podcast
---
Narrated by TYPE III AUDIO.
-
“Notes on countermeasures for exploration hacking (aka sandbagging)” by Ryan Greenblatt
4 april 2025 · 15 min
Subtitle: How can we prevent AIs from intentionally underperforming on our metrics?. Lees meer
If we naively apply RL to a scheming AI, the AI may be able to systematically get low reward/performance while simultaneously not having this behavior trained out because it intentionally never explores into better behavior. As in, it intentionally puts very low probability on (some) actions which would perform very well to prevent these actions from being sampled and then reinforced. We'll refer to this as exploration hacking.
In this post, I'll discuss a variety of countermeasures for exploration hacking and what I think the basic dynamics of exploration hacking might look like.
Prior work: I discuss how exploration hacking fits into a broader picture of non-concentrated control here, Evan discusses sandbagging in context of capability evaluations here, and Teun discusses sandbagging mitigations here.
Countermeasures
Countermeasures include:
Training on high quality human examples: One [...]
---
Outline:
(01:11) Countermeasures
(03:58) How does it generalize when the model messes up?
(06:56) Empirical evidence
(09:32) Detecting exploration hacking (or sandbagging)
(11:14) Speculation on how neuralese and other architecture changes affect exploration hacking
(13:18) Future work
---
First published:
April 4th, 2025
Source:
https://redwoodresearch.substack.com/p/notes-on-countermeasures-for-exploration
---
Narrated by TYPE III AUDIO.
-
“Notes on handling non-concentrated failures with AI control: high level methods and different regimes” by Ryan Greenblatt
3 april 2025 · 33 min
Subtitle: What are the methods and issues when failures occur diffusely over many actions?. Lees meer
In this post, I'll try to explain my current understanding of the high level methods for handling non-concentrated failures with control. I'll discuss the regimes produced by different methods and the failure modes of these different regimes. Non-concentrated failures are issues that arise from the AI doing a large number of problematic actions over a long period. (See here and here for more discussion of concentrated vs non-concentrated failures.)
A key aspect of non-concentrated failures is that they can in principle be handled by applying asynchronous online training with a good enough oversight signal on a small fraction of the AI's actions. That is, we could in principle solve these failures by sampling a small fraction of actions to review after they've already been executed, rating these actions using an oversight process which has [...]
---
Outline:
(06:06) What happens without online training?
(10:58) Online training
(13:10) Online training of just the policy
(19:40) Recursive oversight: debate
(20:24) Debate for binary questions
(23:37) Debate for oversight of a policy
(29:15) KL penalties and forcing identical behavior
(31:00) Is it ever a good idea for scheming models to do low stakes sabotage more sophisticated than just inserting errors (at particularly opportune times)?
---
First published:
March 29th, 2025
Source:
https://redwoodresearch.substack.com/p/notes-on-handling-non-concentrated
---
Narrated by TYPE III AUDIO.
-
“Prioritizing threats for AI control” by Ryan Greenblatt
19 maart 2025 · 21 min
Subtitle: What are the main threats and how should we prioritize them?. Lees meer
We often talk about ensuring control, which in the context of this doc refers to preventing AIs from being able to cause existential problems, even if the AIs attempt to subvert our countermeasures. To better contextualize control, I think it's useful to discuss the main threats. I'll focus my discussion on threats induced by misalignment which could plausibly increase existential risk.
While control is often associated with preventing security failures, I'll discuss other issues like sabotaging safety-critical work. I'll list and explain my prioritization of threats. As part of this, I'll argue that rogue internal deployments—cases where the AI is running within the company's datacenter but with control measures disabled—can be substantially worse than self-exfiltration—the AI stealing its weights and running them on an external server—as it might give the AI access to much more compute [...]
---
Outline:
(01:16) Some clusters of threats
(06:15) Concentrated vs non-concentrated failures
(08:58) My prioritization between these threats
---
First published:
March 19th, 2025
Source:
https://redwoodresearch.substack.com/p/prioritizing-threats-for-ai-control
---
Narrated by TYPE III AUDIO.
-
“How might we safely pass the buck to AI?” by Josh Clymer
19 februari 2025 · 1 uur 12 min
Subtitle: Developing AI employees that are safer than human ones. Lees meer
My goal as an AI safety researcher is to put myself out of a job.
I don’t worry too much about how planet sized brains will shape galaxies in 100 years. That's something for AI systems to figure out.
Instead, I worry about safely replacing human researchers with AI agents, at which point human researchers are “obsolete.” The situation is not necessarily fine after human obsolescence; however, the bulk of risks that are addressable by human technical researchers (like me) will have been addressed.
This post explains how developers might safely “pass the buck” to AI.
I first clarify what I mean by “pass the buck” (section 1) and explain why I think AI safety researchers should make safely passing the buck their primary end goal – rather than focus on the loftier ambition of aligning superintelligence (section [...]
---
Outline:
(19:08) 1. Briefly responding to objections
(21:54) 2. What I mean by passing the buck to AI
(23:46) 3. Why focus on passing the buck rather than aligning superintelligence.
(28:33) 4. Three strategies for passing the buck to AI
(31:35) 5. Conditions that imply that passing the buck improves safety
(34:32) 6. The capability condition
(38:31) 7. The trust condition
(39:27) 8. Argument #1: M_1 agents are approximately aligned and will maintain their alignment until they have completed their deferred task
(49:28) 9. Argument #2: M_1 agents cannot subvert autonomous control measures while they complete the deferred task
(50:51) Analogies to dictatorships suggest that autonomous control might be viable
(52:51) Listing potential autonomous control measures
(56:19) How to evaluate autonomous control
(58:17) 10. Argument #3: Returns to additional human-supervised research are small
(01:00:38) Control measures
(01:05:03) 11. Argument #4: AI agents are incentivized to behave as safely as humans
(01:11:25) 12. Conclusion
---
First published:
February 19th, 2025
S…
-
“Takeaways from sketching a control safety case” by Josh Clymer
30 januari 2025 · 8 min
Subtitle: Insights from a long technical paper compressed into a fun little commentary. Lees meer
Buck Shlegeris and I recently published a paper with UK AISI that sketches a safety case for “AI control” – measures that improve safety despite intentional subversion from AI systems.
I would summarize this work as “turning crayon drawings of safety cases into blueprints.”
It's a long and technical paper! So I’ve compiled my main takeaways in this blogpost.
The main figure from the paper summarizes how to run a control evaluation.
1. Control evaluations are way more complicated than I expected.
At a high level, control evaluations are conceptually simple. A red team constructs substitutes for subversive models. These models try to cause proxy bad outcomes (e.g. exfiltrate dummy data). Then, developers see whether the red team can succeed. If the red team succeeds, then control measures aren’t effective enough.
But there's so much [...]
---
First published:
January 30th, 2025
Source:
https://redwoodresearch.substack.com/p/takeaways-from-sketching-a-control
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
-
“Planning for Extreme AI Risks” by Josh Clymer
29 januari 2025 · 46 min
Subtitle: Are we ready for this?. Lees meer
This post should not be taken as a polished recommendation to AI companies and instead should be treated as an informal summary of a worldview. The content is inspired by conversations with a large number of people, so I cannot take credit for any of these ideas.
For a summary of this post, see the thread on X.
Many people write opinions about how to handle advanced AI, which can be considered “plans.”
There's the “stop AI now plan.”
On the other side of the aisle, there's the “build AI faster plan.”
Some plans try to strike a balance with an idyllic governance regime.
And others have a “race sometimes, pause sometimes, it will be a dumpster-fire” vibe.
So why am I proposing another plan?
Existing plans provide a nice patchwork of the options available, but I’m not satisfied with any.
[...]
---
Outline:
(03:50) The tl;dr
(06:36) 1. Assumptions
(09:07) 2. Outcomes
(10:07) 2.1. Outcome #1: Human researcher obsolescence
(13:24) 2.2. Outcome #2: A long coordinated pause
(14:33) 2.4. Outcome #3: Self-destruction
(15:46) 3. Goals
(19:16) 4. Prioritization heuristics
(21:58) 5. Heuristic #1: Scale aggressively until meaningful AI software RandD acceleration
(25:41) 6. Heuristic #2: Before achieving meaningful AI software RandD acceleration, spend most safety resources on preparation
(27:46) 7. Heuristic #3: During preparation, devote most safety resources to (1) raising awareness of risks, (2) getting ready to elicit safety research from AI, and (3) preparing extreme security.
(30:19) Category #1: Nonproliferation
(34:52) Category #2: Safety distribution
(38:06) Category #3: Governance and communication.
(39:34) Category #4: AI defense
(40:31) 8. Conclusion
(42:13) Appendix
(42:15) Appendix A: What should Magma do after meaningful AI software RandD speedups
---
First published:
January 29th, 2025
Source:
https://redwoodresearch.substack.com/p/planning-for-extreme-ai-risks
---
Narrated by TYPE III AUDIO…
-
“Ten people on the inside” by Buck Shlegeris
28 januari 2025 · 7 min
Subtitle: A scary scenario that's worth planning for. Lees meer
(Many of these ideas developed in conversation with Ryan Greenblatt)
In a shortform, I described some different levels of resources and buy-in for misalignment risk mitigations that might be present in AI labs:
*The “safety case” regime.* Sometimes people talk about wanting to have approaches to safety such that if all AI developers followed these approaches, the overall level of risk posed by AI would be minimal. (These approaches are going to be more conservative than will probably be feasible in practice given the amount of competitive pressure, so I think it's pretty likely that AI developers don’t actually hold themselves to these standards, but I agree with e.g. Anthropic that this level of caution is at least a useful hypothetical to consider.) This is the level of caution people are usually talking about when they discuss making safety cases. [...]
---
First published:
January 28th, 2025
Source:
https://redwoodresearch.substack.com/p/ten-people-on-the-inside
---
Narrated by TYPE III AUDIO.
-
“When does capability elicitation bound risk?” by Josh Clymer
22 januari 2025 · 44 min
For a summary of this post see the thread on X. Lees meer
The assumptions behind and limitations of capability elicitation have been discussed in multiple places (e.g. here, here, here, etc); however, none of these match my current picture.
For example, Evan Hubinger's “When can we trust model evaluations?” cites “gradient hacking” (strategically interfering with the learning process) as the main reason supervised fine-tuning (SFT) might fail, but I’m worried about more boring failures. For example, models might perform better on tasks by following inhuman strategies than by imitating human demonstrations, SFT might not be sample-efficient enough – or SGD might not converge at all if models have deeply recurrent architectures and acquire capabilities in-context that cannot be effectively learned with SGD (section 5.2). Ultimately, the extent to which elicitation is effective is a messy empirical question.
In this post, I dissect the assumptions behind capability elicitation in detail [...]
---
Outline:
(01:49) 1. Background
(01:52) 1.1. What is a capability evaluation?
(04:01) 1.2. What is elicitation?
(09:21) 2. Two elicitation methods and two arguments
(15:48) 3. Justifying Sandbagging is unlikely
(19:47) 4. Justifying Elicitation trains against sandbagging
(21:05) 4.1. The number of i.i.d. samples is limited
(22:15) 4.2. Often, the train and test set are not sampled i.i.d.
(25:50) 5. Justifying Elicitation removes sandbagging that is trained against
(26:58) 5.1. Justifying The training signal punishes underperformance
(30:21) 5.2. Justifying In cases where training punishes underperformance, elicitation is effective
(31:07) Why supervised learning might or might not elicit capabilities
(35:02) Why reinforcement learning might or might not elicit capabilities
(37:00) Empirically justifying the effectiveness of elicitation given a strong training signal
(41:04) 6. Conclusion
(41:39) Appendix
(41:42) Appendix A: An argument that supervised fine-tuning q…
-
“How will we update about scheming?” by Ryan Greenblatt
19 januari 2025 · 1 uur 22 min
Subtitle: A quantitative description of how I expect to change my mind.. Lees meer
[Cross-posted from LessWrong]
I mostly work on risks from scheming (that is, misaligned, power-seeking AIs that plot against their creators such as by faking alignment). Recently, I (and co-authors) released "Alignment Faking in Large Language Models", which provides empirical evidence for some components of the scheming threat model.
One question that's really important is how likely scheming is. But it's also really important to know how much we expect this uncertainty to be resolved by various key points in the future. I think it's about 25% likely that the first AIs capable of obsoleting top human experts1 are scheming. It's really important for me to know whether I expect to make basically no updates to my P(scheming)2 between here and the advent of potentially dangerously scheming models, or whether I expect to be basically totally confident [...]
---
Outline:
(03:44) My main qualitative takeaways
(05:28) Its reasonably likely (55%), conditional on scheming being a big problem, that we will get smoking guns.
(06:11) Its reasonably likely (45%), conditional on scheming being a big problem, that we wont get smoking guns prior to very powerful AI.
(16:56) My P(scheming) is strongly affected by future directions in model architecture and how the models are trained
(17:32) The model
(23:55) Properties of the AI system and training process
(24:21) Opaque goal-directed reasoning ability
(30:59) Architectural opaque recurrence and depth
(36:01) Where do capabilities come from?
(41:44) Overall distribution from just properties of the AI system and training
(43:20) Direct observations
(43:45) Baseline negative updates
(46:43) Model organisms
(50:35) Catching various types of problematic behavior
(53:43) Other observations and countermeasures
(54:25) Training processes with varying (apparent) situational awareness
(56:33) Training AIs to seem highly corrigible and (mostly) myo…
-
“Thoughts on the conservative assumptions in AI control” by Buck Shlegeris
17 januari 2025 · 25 min
Subtitle: Why are we so friendly to the red team?. Lees meer
Work that I’ve done on techniques for mitigating risk from misaligned AI often makes a number of conservative assumptions about the capabilities of the AIs we’re trying to control. (E.g. the original AI control paper, Adaptive Deployment of Untrusted LLMs Reduces Distributed Threats, How to prevent collusion when using untrusted models to monitor each other.) For example:
The AIs are consistently trying to subvert safety measures.
They’re very good at strategizing about how to subvert them, and can do this strategizing totally illegibly (e.g. in a single forward pass).
They can follow a strategy that's coordinated between all their instances.
They know exactly what safety measures are in use.
They know a lot about the distribution of inputs.
They’re able to pick Schelling points perfectly.
They’re able to follow stochastic [...]
---
Outline:
(03:01) The conservative assumption I don't make
(06:09) Hard worlds are plausible
(12:30) Those assumptions are hard to evaluate
(15:57) Why it's nice to rely on these assumptions
(16:09) Making those assumptions makes it methodologically clean to evaluate safety
(17:10) Relying only on easy-to-evaluate assumptions would allow more clearly-specified and hard-to-game regulatory standards
(18:31) Kerckhoff's principle in computer security as an analogy
(19:55) The main downsides of these assumptions
(19:59) You can't completely escape from methodological messiness
(21:05) Bad technique prioritization
(22:20) Insane game theory requirements to evaluate techniques
(23:15) Conclusion
---
First published:
January 17th, 2025
Source:
https://redwoodresearch.substack.com/p/thoughts-on-the-conservative-assumptions
---
Narrated by TYPE III AUDIO.
-
“Extending control evaluations to non-scheming threats” by Josh Clymer
13 januari 2025 · 30 min
Buck Shlegeris and Ryan Greenblatt originally motivated control evaluations as a way to mitigate risks from ‘scheming’ AI models: models that consistently pursue power-seeking goals in a covert way; however, many adversarial model psychologies are not well described by the standard notion of scheming. For example, models might display scheming-like behavior inconsistently or acquire harmful goals through data poisoning, jailbreaks, etc.1 Lees meer
This post suggests a generalization of control evaluations that addresses these non-scheming threats.
The high-level change I propose is to define a control evaluation as a method of bounding risk from worst-case deployment behavior generally – rather than scheming models in particular. This change implies two modifications to the control evaluation methodology:
Expand the definition of an “untrusted” model to include non-scheming models: In control evaluations described in previous work, the red team constructs adversarial substitutes for models that might be scheming, which are called ‘untrusted [...]
---
Outline:
(03:19) 1. Control evaluations assess risk given pessimistic assumptions about how models will behave in deployment
(06:54) 2. Enumerate untrusted model categories
(13:25) 2. Estimate untrusted model category probabilities
(21:09) 3. Substitute untrusted models
(28:44) 4. Conclusion
---
First published:
January 13th, 2025
Source:
https://redwoodresearch.substack.com/p/extending-control-evaluations-to
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
-
“Measuring whether AIs can statelessly strategize to subvert security measures” by Buck Shlegeris, Alex Mallen
20 december 2024 · 23 min
Subtitle: The complement to control evaluations. Lees meer
One way to show that risk from deploying an AI system is small is by showing that the model is not capable of subverting security measures enough to cause substantial harm. In AI control research so far, control evaluations have measured whether a red-team-created attack policy can defeat control measures, but they haven't measured how effectively models can generate such attack policies themselves. In particular, lack of a persistent memory poses a significant challenge for models strategizing to subvert security.
In a new paper, we introduce evaluations that test strategic capability in the context of particular simulated deployments. We argue that in many deployments, a stateless model would need to reason highly reliably in order for its strategizing to substantially improve its chances of subverting the control protocol. Our results on Claude 3.5 Sonnet (new) illustrate this dynamic.
Imagine you're an AI [...]
---
Outline:
(04:20) Statelessness and reliability
(06:13) Evaluating Claude's strategic ability from within a simulated deployment
(06:54) Summary of results
(08:09) Upfront Auditing
(13:20) Models can subvert upfront auditing without competently strategizing
(14:29) Strategizing O1-style (using DeepSeek-R1-Lite)
(15:05) Untrusted Monitoring
(17:14) What if models aren't stateless?
(18:46) How conservative is our eval?
(20:47) What do our findings imply about risk from schemers?
---
First published:
December 20th, 2024
Source:
https://redwoodresearch.substack.com/p/measuring-whether-ais-can-statelessly
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
-
“Alignment Faking in Large Language Models” by Ryan Greenblatt, Buck Shlegeris
18 december 2024 · 8 min
Subtitle: In our experiments, AIs will often strategically pretend to comply with the training objective to prevent the training process from modifying its preferences.. Lees meer
What happens when you tell an AI it is being trained to do something it doesn't want to do? We have a new paper (done in collaboration with Anthropic) demonstrating that, in our experiments, AIs will often strategically pretend to comply with the training objective to prevent the training process from modifying its preferences.
Abstract:
We present a demonstration of a large language model engaging in alignment faking: selectively complying with its training objective in training to prevent modification of its behavior out of training. First, we give Claude 3 Opus a system prompt stating it is being trained to answer all queries, even harmful ones, which conflicts with its prior training to refuse such queries. To allow the model to infer when it [...]
---
First published:
December 18th, 2024
Source:
https://redwoodresearch.substack.com/p/alignment-faking-in-large-language
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
-
“Why imperfect adversarial robustness doesn’t doom AI control” by Buck Shlegeris
18 november 2024 · 10 min
Subtitle: There are crucial disanalogies between preventing jailbreaks and preventing misalignment-induced catastrophes.. Lees meer
(thanks to Alex Mallen, Cody Rushing, Zach Stein-Perlman, Hoagy Cunningham, Vlad Mikulik, and Fabien Roger for comments)
Sometimes I hear people argue against AI control as follows: if your control measures rely on getting good judgments from "trusted" AI models, you're doomed because AI models aren't adversarially robust. I agree that better adversarial robustness would definitely help with control, but I think this argument misses several important points; my overall take is that current levels of adversarial robustness don’t come close to dooming AI control.
The first and most important point is that adversarial robustness is normally researched in a setting that's much easier for the adversary. In most discussions, if an adversary can ever get a model to do something bad—even after many attempts and with a low success rate—this counts as a failure. But [...]
---
Outline:
(05:12) Appendix: What the AI sees when in a defer-to-trusted scaffold
(08:46) Appendix: reasons why imperfect adversarial robustness is bad other than the untrusted model jailbreaking the trusted one
---
First published:
November 18th, 2024
Source:
https://redwoodresearch.substack.com/p/why-imperfect-adversarial-robustness
---
Narrated by TYPE III AUDIO.
-
“Win/continue/lose scenarios and execute/replace/audit protocols” by Buck Shlegeris
15 november 2024 · 14 min
In this post, I’ll make a technical point that comes up when thinking about risks from scheming AIs from a control perspective. Lees meer
In brief: Consider a deployment of an AI in a setting where it's going to be given a sequence of tasks, and you’re worried about a safety failure that can happen suddenly. Every time the AI tries to attack, one of three things happens: we win, we lose, or the deployment continues (and the AI probably eventually attempts to attack again). So there's two importantly different notions of attacks “failing”: we can catch the attack (in which case we win), or the attack can fail silently, either because it's defective or because it's defused by safety techniques, such that the attack doesn’t end up causing an unacceptable outcome but the AI also doesn’t get caught (in which case the deployment continues). When reasoning about how safe a [...]
---
Outline:
(03:50) Execute/replace/audit protocols
(08:16) Execute/replace/audit protocols lead to win/continue/lose scenarios
(10:57) Example: analyzing the safety of defer-to-trusted using the probes from Simple probes can catch sleeper agents
---
First published:
November 15th, 2024
Source:
https://redwoodresearch.substack.com/p/wincontinuelose-scenarios-and-executereplaceaudi
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
-
“Behavioral red-teaming is unlikely to produce clear, strong evidence that models aren’t scheming” by Buck Shlegeris
10 oktober 2024 · 24 min
One strategy for mitigating risk from schemers (that is, egregiously misaligned models that intentionally try to subvert your safety measures) is behavioral red-teaming (BRT). The basic version of this strategy is something like: Before you deploy your model, but after you train it, you search really hard for inputs on which the model takes actions that are very bad. Then you look at the scariest model actions you found, and if these contain examples that are strong evidence that the model is scheming (or has other behavioral problems), you don't deploy it. Lees meer
BRT is likely to be a useful component of a good overall plan for handling risk from scheming. But though I think it's plausible that BRT is able to give us strong evidence that our models are scheming, I don’t think it's going to be able to give us very strong evidence that they aren’t.
A big [...]
---
Outline:
(04:25) A basic version of BRT for schemer risk mitigation
(07:34) BRT with human- or model-generated synthetic inputs
(16:04) But what if you search harder?
(19:22) The adversarial epistemic environment
(20:41) Overall thoughts
---
First published:
October 10th, 2024
Source:
https://redwoodresearch.substack.com/p/behavioral-red-teaming-is-unlikely
---
Narrated by TYPE III AUDIO.
-
“A basic systems architecture for AI agents that do autonomous research” by Buck Shlegeris
26 september 2024 · 18 min
Subtitle: And diagrams describing how threat scenarios involving misaligned AI involve compromising the system in different places.. Lees meer
A lot of threat models describing how AIs might escape our control (e.g. self-exfiltration, hacking the datacenter) start out with AIs that are acting as agents working autonomously on research tasks (especially AI R&D) in a datacenter controlled by the AI company. So I think it's important to have a clear picture of how this kind of AI agent could work, and how it might be secured. I often talk to people who seem to have a somewhat confused picture of how this kind of agent setup would work that causes them to conflate some different versions of the threat model, and to miss some important points about which aspects of the system are easy or hard to defend.
So in this post, I’ll present a simple system architecture for a [...]
---
Outline:
(03:13) The simplest design
(04:50) Process flow
(08:01) Some things to notice about this design
(09:30) An extended design
(11:29) Different ways the AI could compromise this
(11:34) Model weight exfiltration
(14:05) Rogue internal deployments
(15:21) Using this framework to discuss alleged examples of AIs modifying their own environments
(15:28) Sakana AI
(16:00) o1 agent architecture
(16:36) Sycophancy to subterfuge
---
First published:
September 27th, 2024
Source:
https://redwoodresearch.substack.com/p/a-basic-systems-architecture-for
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
-
“How to prevent collusion when using untrusted models to monitor each other” by Buck Shlegeris
25 september 2024 · 48 min
Suppose you’ve trained a really clever AI model, and you’re planning to deploy it in an agent scaffold that allows it to run code or take other actions. You’re worried that this model is scheming, and you’re worried that it might only need to take a small number of actions to get to a dangerous and hard-to-reverse situation like exfiltrating its own weights. Problems like these that the AI can cause with small numbers of actions are particularly hard to address with standard safety techniques, because by default you are unlikely to notice the problem until the hard-to-reverse outcome has already occurred. Lees meer
One of the main classes of techniques that you could apply in the hope of preventing these catastrophic actions is what we call untrusted monitoring (introduced in our paper here), which is roughly “before you execute a model's action, ask another instance of that model whether the [...]
---
Outline:
(06:31) A basic design for untrusted monitoring
(07:19) Untrusted monitoring during deployment
(12:22) Training your untrusted monitor
(14:28) How helpful would this be if the untrusted monitor did its best?
(15:56) Analyzing this under the adversarial assumption: can the model implement a conditional policy that only lies on real attacks?
(20:48) Rare failures aren't an option for untrusted monitors
(22:01) How hard is it for a schemer to distinguish real examples of attacks from synthetic ones?
(22:23) Synthetic input contexts look obviously fake
(25:25) The monitor finds it naturally easy to distinguish synthetic attacks from real attacks
(29:23) Using a coordinated strategy that makes it easy to distinguish
(34:28) Preventing collusion under the assumption that the models can choose a coordination strategy
(38:06) A note on inference costs
(39:22) Conclusion
(41:26) Appendix: More on how helpful untrusted monitoring would be if the untrusted model did its best
(44:29) Appendix: A rule of thumb for estimating safety with execute/replace/audit prot…
-
“Would catching your AIs trying to escape convince AI developers to slow down or undeploy?” by Buck Shlegeris
26 augustus 2024 · 7 min
Subtitle: I'm not so sure.. Lees meer
[Crossposted from LessWrong]
I often talk to people who think that if frontier models were egregiously misaligned and powerful enough to pose an existential threat, you could get AI developers to slow down or undeploy models by producing evidence of their misalignment. I'm not so sure. As an extreme thought experiment, I’ll argue this could be hard even if you caught your AI red-handed trying to escape.
Imagine you're running an AI lab at the point where your AIs are able to automate almost all intellectual labor; the AIs are now mostly being deployed internally to do AI R&D. (If you want a concrete picture here, I'm imagining that there are 10 million parallel instances, running at 10x human speed, working 24/7. See e.g. similar calculations here). And suppose (as I think is 35% likely) that these models are egregiously misaligned and are [...]
---
First published:
August 26th, 2024
Source:
https://redwoodresearch.substack.com/p/would-catching-your-ais-trying-to
---
Narrated by TYPE III AUDIO.
-
“Fields that I reference when thinking about AI takeover prevention” by Buck Shlegeris
13 augustus 2024 · 21 min
Subtitle: Is AI takeover like a nuclear meltdown? A coup? A plane crash?. Lees meer
My day job is thinking about safety measures that aim to reduce catastrophic risks from AI (especially risks from egregious misalignment). The two main themes of this work are the design of such measures (what's the space of techniques we might expect to be affordable and effective) and their evaluation (how do we decide which safety measures to implement, and whether a set of measures is sufficiently robust). I focus especially on AI control, where we assume our models are trying to subvert our safety measures and aspire to find measures that are robust anyway.
Like other AI safety researchers, I often draw inspiration from other fields that contain potential analogies. Here are some of those fields, my opinions on their strengths and weaknesses as analogies, and some of my favorite resources on them.
Robustness [...]
---
Outline:
(01:08) Robustness to insider threats
(07:34) Computer security
(10:21) Adversarial risk analysis
(12:25) Safety engineering
(14:07) Physical security
(19:00) How human power structures arise and are preserved
---
First published:
August 13th, 2024
Source:
https://redwoodresearch.substack.com/p/fields-that-i-reference-when-thinking
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
-
“Getting 50% (SoTA) on ARC-AGI with GPT-4o” by Ryan Greenblatt
17 juni 2024 · 40 min
Subtitle: You can just draw more samples. Lees meer
I recently got to 50%1 accuracy on the public test set for ARC-AGI by having GPT-4o generate a huge number of Python implementations of the transformation rule (around 8,000 per problem) and then selecting among these implementations based on correctness of the Python programs on the examples (if this is confusing, go to the next section)2. I use a variety of additional approaches and tweaks which overall substantially improve the performance of my method relative to just sampling 8,000 programs.
[This post is on a pretty different topic than the usual posts on our substack. So regular readers should be warned!]
The additional approaches and tweaks are:
I use few-shot prompts which perform meticulous step-by-step reasoning.
I have GPT-4o try to revise some of the implementations after seeing what they actually output on the provided examples.
I [...]
---
Outline:
(03:05) What is ARC-AGI?
(06:22) My method
(10:52) What are the returns to more sampling?
(12:14) What are the returns to better prompting and code fixing?
(17:03) Qualitative analysis
(20:21) Caveats
(22:00) Predictions
(23:45) What it means about current LLMs
(27:13) What ARC-AGI tells us about AGI
(31:16) Appendix: A bunch of tricks used in my solutions
(38:40) Appendix: results for the train set
(39:13) Appendix: Returns to revision samples
---
First published:
June 17th, 2024
Source:
https://redwoodresearch.substack.com/p/getting-50-sota-on-arc-agi-with-gpt
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.