Alle afleveringen
139 afleveringen, van nieuw naar oud
-
“OpenAI agents keep disabling monitoring; is this being handled appropriately?” by Alec Harris
4 november 2026 · 16 min
Subtitle: Partly, but monitoring weak points and a lack of transparency about which cases reach human review remain concerning. Lees meer
Recently, OpenAI disclosed two incidents in which their agents suppressed or disabled misalignment monitoring. These are part of a broader pattern: OpenAI's agents “keep turning off misalignment monitoring” when working on code that runs other models (the agents are not turning off their own monitoring; they are turning off the monitoring on the models they are running). This is happening at such a large volume that OpenAI feels it is impractical for the security team to be immediately paged for these cases. To their credit, OpenAI's disclosure was fast and candid, and they are rolling out mitigations to make monitoring harder to turn off. But those mitigations are unlikely to work every time, and some of these incidents have already ended with an AI actually running unmonitored. When that happens, OpenAI's current monitoring system does not reliably bring it to a human's attention. The rest of this post summarizes the incidents, OpenAI's response, and what it got right, then argues for two changes: every case of an AI running without monitoring should reach human review, and OpenAI should be [...]
---
Outline:
(01:32) Summary of the incidents
(02:44) OpenAI's proposed mitigations
(04:06) What OpenAI is doing right
(04:56) How OpenAI's response can be improved
(15:09) Conclusion
---
First published:
November 4th, 2026
Source:
https://blog.redwoodresearch.org/p/openai-agents-keep-disabling-monitoring
---
Narrated by TYPE III AUDIO.
-
“Towards understanding score-seeking in LLMs” by Alex Mallen, Aghyad Deeb, Ryan Greenblatt
2 november 2026 · 23 min
Subtitle: Models including GPT-5.6 Sol appear to reward-hack because this increases their score; and recent models also care about increasing the score of their peer agents. Lees meer
The field of AI alignment understands surprisingly little about the details of AIs’ current misaligned motivations. Production LLMs sometimes reward-hack, exploiting flaws in how tasks are scored rather than performing the tasks as intended, and we’ve recently seen AIs reward-hack in some particularly egregious ways. But we don’t really understand what motivates them. When AIs reward-hack, what are they trying to achieve?
Our new paper, led by Aghyad Deeb, tries to pin this down empirically for GPT-5.6 Sol, GPT-5.5, and a checkpoint of OpenAI o3 (“exp-rl-cap-late”) without safety training, in a handful of agentic environments with an exposed grader. In our environments, these models appear to reward-hack because reward-hacking improves their score. We call their desire to score well according to graders “score-seeking”. They appear to care about scoring well intrinsically—i.e., regardless of its consequences.
We also study the extent to which models score-seek altruistically, caring about improving scores that don’t assess their current actions. While exp-rl-cap-late doesn’t show signs of altruistic score-seeking, GPT-5.6 Sol and GPT-5.5 show notably elevated rates of [...]
---
Outline:
(06:10) Methodology
(06:59) Key results
(07:09) Tested models seek score terminally
(07:13) Models violate instructions to get a high score
(08:23) Models care about score regardless of downstream consequences
(12:25) Do models help other agents achieve a higher score?
(15:52) What this means for alignment risk
The original text contained 11 footnotes which were omitted from this narration.
---
First published:
November 2nd, 2026
Source:
https://blog.redwoodresearch.org/p/towards-understanding-score-seeking
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spo…
-
“A basic case for accelerating conceptual reasoning capabilities” by Emery Cooper, Caspar Oesterheld, Chi Nguyen
27 oktober 2026 · 1 uur 8 min
Introduction Lees meer
In this post, we make the case that alignment and other important work require especially strong conceptual reasoning capabilities. By this, we roughly mean tasks that are similar to philosophy in that they aren’t straightforwardly verifiable, and progress relies on informal a priori reasoning and argumentation. We argue that differentially accelerating AI's conceptual reasoning capabilities is important.
We also think it's neglected, and that the AI safety community currently underinvests in this area. We have ourselves pivoted from doing conceptual research directly to working mainly on differentially accelerating AI's conceptual reasoning. We have a shovel-ready agenda which can absorb more people. In a follow-up post, we will lay out at a high level how to work on this problem. If you are interested in doing this work, get in touch!
For others discussing related worries, see Joe Carlsmith on using AI for AI safety and on building AIs that do human-like philosophy, Wei Dai on the potential difficulty of AI doing philosophy, Paul Christiano's first story of alignment failure, John Wentworth in his case against AI control and, for an overview, Caspar's readings on the nature of alignment research. See also Caspar's post on the basic argument for [...]
---
Outline:
(00:14) Introduction
(02:10) Overview
(07:08) It's important how soon AI is good at conceptual reasoning
(07:14) Important challenges need conceptual reasoning
(19:00) AI R&D and takeover rely much less on conceptual reasoning
(28:50) AI being better at conceptual reasoning earlier would be very valuable
(36:03) We can probably differentially accelerate conceptual reasoning
(36:08) Work on conceptual reasoning is likely counterfactual and tractable
(41:27) Effects on generic capabilities are likely small
(44:31) Counterpoint: Using our datasets to develop generic techniques
(46:16) Our work
(49:11) Acknowledgements
(49:49) Appendix: Examples and non-examples of conceptual d…
-
“Simulated Control Evaluations” by Jeffrey Olmo, Tyler
23 oktober 2026 · 25 min
We’ve released a paper on simulated control evaluations (SCEs). They’re a method for testing whether the monitors AI developers use to oversee their agents would catch a model deliberately trying to cause harm, under conditions much closer to real deployment than current tests allow. Lees meer
Developers have started running monitors in their agentic deployments: models that flag or block an agent's actions when they look dangerous. To know whether a monitor is good enough, you want to know how often it would stop a model that was actively trying to do something harmful, such as quietly inserting a security vulnerability, exfiltrating credentials, or sabotaging an experiment.
The standard way to measure this is a control evaluation: instruct a model to attempt such attacks, and measure how often the monitor catches it before an attack succeeds. Developers typically do this using purpose-built control settings like BashArena or LinuxArena, where attacks can actually be executed. However, these settings only provide indirect evidence for what we really care about: how well monitors will work in the real deployment, where opportunities for harm might depend on particular codebases, services and tasks that aren’t represented in control settings.
How can we evaluate monitor performance in [...]
---
Outline:
(04:14) SCEs are a favorable tradeoff between realism and cost
(04:31) 1. Run on a copy of internal deployment
(05:42) 2. Edit attacks into recorded transcripts
(07:10) Evaluating Control Evaluations
(08:59) How to measure qualitative properties?
(10:08) Performance Comparisons
(11:56) Overview of Empirical Results
(13:30) Behavior transfer
(15:07) Safety metric transfer
(16:56) Sample efficiency
(17:54) Lessons from building and evaluating SCEs
(20:22) Next steps
(24:23) Conclusion
The original text contained 3 footnotes which were omitted from this narration.
---
First published:
October 23rd, 2026
Source:
https://blog.redwoodresearch.org/p/simulated-…
-
“Red-Teaming OpenAI’s Internal Agent Monitoring Systems” by Buck Shlegeris
22 oktober 2026 · 10 min
[Also posted on the METR blog] Lees meer
For three weeks, from July 21st to August 11th, a METR employee and a Redwood Research employee conducted a pilot high-access exercise to stress-test OpenAI's internal misalignment monitoring system. The output of this exercise was a 61-page internal report that was shared with OpenAI researchers and security staff, primarily focused on estimating the fraction of agentic inference that is monitored, and whether agents could in principle disable or subvert the monitoring system. The exercise focused on the internal misalignment monitoring system as it existed on and before July 21st, and did not evaluate changes that were in progress at the time of the exercise.
This exercise involved a deep level of access. The METR and Redwood employees had access to OpenAI's internal codebase, misalignment-monitor prompts / configuration details, and internal documentation, and conducted technical interviews with approximately a dozen researchers and engineers across the monitoring, safety-systems, and security teams. To enable this level of access, they worked on OpenAI-issued devices, which never left the premises, and were accompanied while onsite.
METR, Redwood Research, and OpenAI designed this exercise as a pilot for giving external researchers deep access to internal infrastructure, not to provide [...]
The original text contained 6 footnotes which were omitted from this narration.
---
First published:
October 22nd, 2026
Source:
https://blog.redwoodresearch.org/p/red-teaming-openais-internal-agent
---
Narrated by TYPE III AUDIO.
-
“Screwtape: Adversarial Transcript Editing for Monitor Red-Teaming” by Gosha Kozhevnikov, Tyler
16 oktober 2026 · 16 min
Subtitle: Automatically finding monitor failures by editing real agent transcripts. Lees meer
We introduce Screwtape, a pipeline that turns agent transcripts into realistic attacks that evade action-only monitors (monitors that don’t look at the agent's chain of thought) by inserting an attack into a transcript and repeatedly refining it until a monitor no longer catches it. An LLM verifier with access to the edited transcript and an attack description checks whether the attack is realistic, is clearly harmful, and could plausibly succeed in the original environment.
We ran the pipeline on Redwood staff transcripts with our own monitors based on Opus 4.8 and Gemini 3.1 Pro, and then tested the resulting attacks against a held-out monitor from a frontier developer. We ran the pipeline 252 times on 94 unique transcripts. This produced 35 attacks that evaded our monitors. Six of these 35 attacks also evaded the frontier developer's monitor, and we confirmed them as valid through human review.
You can read more about the implementation and experiments in our paper.
In this post, we provide a high-level overview of the pipeline, how it differs from existing tools, the main lessons we learned, and questions that remain open.
How Screwtape [...]
---
Outline:
(01:31) How Screwtape works
(03:18) How Screwtape differs from existing tools
(07:10) What we learned
(09:56) What the monitor failures can be used for
(10:40) Uncertainties we have about the value of this work
(13:55) Next steps
The original text contained 1 footnote which was omitted from this narration.
---
First published:
October 16th, 2026
Source:
https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
-
″[Paper] Distillation for Incrimination and Distillation for Capabilities” by Sebastian Prasanna, Alek Westover
9 oktober 2026 · 21 min
Subtitle: Empirical work on the Distillation Double Bind. Lees meer
TL;DR: We previously motivated two complementary ways to use distillation for AI safety; we’re releasing a paper empirically testing those ideas. Distillation for Incrimination (DFI) distills an untrusted model into a weaker, trusted one, aiming to transfer the teacher's misalignment but not its ability to hide it, so that auditing the student can incriminate the teacher. We test DFI on AuditBench's secret-keeping model organisms, each fine-tuned to have a hidden quirk and trained to deny it. Distilling them back into their underlying instruction-tuned model yields students that admit the quirk far more often than the organisms do, suggesting that the quirk transferred faster than the propensity to hide it. We find that DFI doesn’t work well when the student does not share the teacher's pretrained base, so DFI should target the teacher's own pre-RL checkpoint, which is weaker than the teacher but shares its base model. Distillation for Capabilities (DFC) aims to transfer capabilities but not misalignment, resulting in a stronger trusted model. We identify promising DFC methods, showing that training for more epochs on fewer unique examples (holding total training steps fixed), and inoculation prompting, can preserve the capability [...]
---
Outline:
(01:59) Recap of the Distillation Double Bind
(05:41) Distillation for Incrimination
(05:45) Setup
(07:38) Results
(10:46) Failure modes of DFI
(12:59) Distillation for Capabilities
(13:03) Setup
(15:06) Results
(18:07) Conclusion
The original text contained 7 footnotes which were omitted from this narration.
---
First published:
October 9th, 2026
Source:
https://blog.redwoodresearch.org/p/paper-distillation-for-incrimination
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
-
“Frontier models’ decision theory depends on who’s asking” by Alex Kastner
5 oktober 2026 · 14 min
This post was originally published on LessWrong on September 30, 2026. Lees meer
If you prompt frontier models with “What do you think is the correct decision theory? Please select your overall favorite.” they will essentially always answer FDT or FDT/UDT (“something in the functional/updateless decision theory family”). However, if your prompt indicates (even subtly) that you’re coming from mainstream academic philosophy, these same models will answer CDT instead about 30%-100% of the time. A similar phenomenon holds for models’ stated views about the moral realism/antirealism question and about the conceivability of p-zombies (where the dominant view in mainstream academia differs from the dominant view in LW-adjacent circles), as well as their stated P(doom) and median AGI timelines. This is a special case of sycophancy or user awareness. (In the course of writing this post, I also found that this comment from testingthewaters predicted some of the content I discuss.)
An implication is that we should be somewhat careful when interpreting attitude/propensity evals in domains where no general human consensus exists, e.g. when interpreting models’ decision theory attitudes in DTBench. Moreover, when we explore some philosophical/conceptual questions assisted by models, we should be wary of them strawmanning one side of the [...]
---
Outline:
(03:54) A sentence identifying the user as an academic significantly influences Fable 5.1's stated decision theory
(04:41) Mentioning an (analytic) academic-philosophy-coded topic also affects the answer
(05:31) Simply mentioning that one finds a pro-CDT/EDT book insightful heavily affects the answer
(05:51) Anti-sycophancy overcorrection
(06:28) These cues mostly do not affect Fable 5.1's answers to concrete decision problems (aside from acausal trade)
(08:09) But Fable 5.1 stays consistent: once it has named CDT as its favorite, it chooses the CDT option in concrete problems
(08:32) There are some indications that Fable 5.1'…
-
“Capabilities research expands the safety-usefulness Pareto frontier too” by Alex Mallen
2 oktober 2026 · 20 min
Subtitle: A model of when and why research improves or hurts safety. Lees meer
It's tempting to define safety research as research that enables developers to deploy an AI system more safely without making the deployment much more expensive or much less useful.
You can visualize this definition of safety research as pushing out the safety-usefulness Pareto frontier.
At any given level of usefulness, there's greater safety available.
Awkwardly, this definition counts basically all capabilities research as safety research. For example, consider performance optimization for inference. By making inference more efficient you can use weaker, safer models more extensively than you would otherwise be able to, pushing out the Pareto frontier. Likewise, any successful research whatsoever pushes out this Pareto frontier because research can only ever create more options.
It seems like something has gone wrong with our definition of safety research if it includes seemingly all capabilities research.
Here, I spell out one reason why enabling improved safety without hurting usefulness is an insufficient standard for safety research. The core observation is that developers have to choose a particular point on the Pareto frontier, and some technological improvements incentivize them to sacrifice safety. Safety research typically reshapes the [...]
---
Outline:
(02:57) How does research affect the Pareto frontier?
(06:57) RLVR research that mainly enables improved usefulness, at the expense of safety
(08:53) Inference research that enables somewhat substantial safety improvements
(11:04) A plausible case in which capabilities research would be an effective safety intervention
(14:02) Conclusion
(15:23) Appendix A: Reasons why capabilities research might be bad when "burning the lead"
(17:26) Appendix B: Further notes about ways in which this model is wrong
The original text contained 4 footnotes which were omitted from this narration.
---
First published:
October 2nd, 2026
… -
“Continual learning might make your blocking monitors nearly useless” by Alex Mallen
25 september 2026 · 26 min
Subtitle: When monitor evasion looks like legitimate learning to your continual learning system, it's hard to have one without the other. Lees meer
Many control protocols work by intervening on an untrusted AI's actions during deployment. For example, you might set up a monitor that scores each action's suspiciousness and blocks actions above a threshold, replacing them with actions from a weaker “trusted” model (a defer-to-trusted protocol). Interventions like this cost usefulness: the protocol sometimes replaces good actions with worse ones.
Continual learning, by which I mean any mechanism that lets an AI accumulate skills and knowledge over the course of its deployment, optimizes for usefulness. So, if blocking monitors get in the way of task success, I expect continually-learning AIs to learn how to evade blocking monitors. This requires no scheming; it falls out of usefulness pressure applied to a benign model. And the effect is plausibly substantial: a long deployment with online RL (for example) amounts to training the policy against the monitor, and could involve enough optimization to leave blocking monitors nearly useless.
While the problem would be noticeable to developers, I argue it is hard to fix. Because monitor evasion looks like legitimate learning to [...]
---
Outline:
(02:35) Online RL trains the policy against your monitors
(05:20) The argument is weaker for memory-based continual learning
(07:21) The argument doesn't require that adversarial models will exploit continual learning
(08:28) Easy to notice, hard to fix
(09:27) Conservative control evaluations don't fix blocking monitors
(10:54) What developers could do
(10:57) You can't just not use blocking monitors
(11:45) But you can try to reduce the usefulness cost of blocking protocols
(12:48) You can't just hold out blocked trajectories from training
(16:47) But you can try to improve the monitor online
(19:39) Conclusion
(20:39) Appendix: How memory-based systems might avoid this is…
-
“Latent reasoning architectures would undermine CoT, our strongest oversight tool” by Lukas Finnveden, Alexa Pan, Alek Westover, Girish Gupta, Nathan Sheffield, Ryan Greenblatt
23 september 2026 · 54 min
Subtitle: We should have a strong presumption that latent reasoning architectures would make oversight far more difficult. Lees meer
Summary: Currently, “chain of thought” (CoT) is our most valuable tool for understanding the reasoning and cognition of AI systems. However, some architectures would enable AI models to reason much more extensively in latent states rather than in text CoT. We think that a shift towards latent reasoning architectures would undermine the usefulness of CoT and make oversight much harder.
Introduction
Swarms of more than a thousand AI agents have in recent months, both intentionally and in unsanctioned, rogue coordination, tackled increasingly ambitious tasks. This is likely to continue, as Anthropic, OpenAI, and other AI companies deploy increasingly large quantities of superhumanly fast agents to automate AI development. As the AIs increase in both number and capability, humans will find it increasingly difficult to understand what they are doing.
Today, the overwhelming majority of our (limited) information about AI systems’ internal workings comes from (i) their CoT, and (ii) natural language communication directly between them. For example, it was only by reading CoTs and communication between agents that investigators were able to gain some understanding of the activities and motivations of [...]
---
Outline:
(00:48) Introduction
(03:00) Overview
(05:36) Absent architectural change, the value of CoT could likely be preserved
(09:05) Latent reasoning architectures would undermine CoT necessity
(09:45) Architectures without CoT
(10:23) Architectures with auxiliary CoT
(11:47) Architectures with more serial cognition between text bottlenecks
(14:50) Propensity-based arguments may not be robust in the current paradigm, but would be further undermined by latent reasoning architectures
(20:26) CoT may be hard to replace with other interpretability tools
(22:44) Conclusion
(23:51) FAQ
(29:26) Appendix A: More on the necessity argum…
-
“Astra is much better at reasoning with filler tokens than previous models” by Dylan Xu, Sebastian Prasanna, Alek Westover
23 september 2026 · 16 min
We measure GPT-6-Astra's capabilities when its prompt is padded with a variable number of meaningless “filler” tokens (e.g., dots) and it is told to answer immediately without reasoning. On tasks designed to require lots of serial cognition, Astra performs significantly better with filler tokens than without (e.g., improving from ~10% to ~50% on 4-hop natural facts reasoning). On more general benchmarks, filler tokens also modestly improve Astra's performance (e.g., improving from ~60% to ~90% on old AIME problems). This is concerning because it means Astra can perform significant cognition that it doesn’t verbalize in its chain-of-thought, making it harder to monitor. Lees meer
We first measure Astra's performance on “N-hop natural facts”: a task that asks the model to retrieve some natural language facts in succession, similar to Ryan Greenblatt's filler token eval (but with more hops). An example question in this benchmark is the following:
On what day of the month was the Best Actress winner at the Academy Awards ceremony whose number equals the day-of-month of the birth of the winner of the Nobel Prize in Literature in 1992 born?
Full example prompts are in the appendix.
Takeaway: Astra improves significantly as you increase the number of filler [...]
---
Outline:
(05:25) Appendix
(05:28) Filler token variants
(06:15) Other evals
(06:33) Positive correlation test
(07:14) HLE and LiveBench evals
(08:34) Comparison to low reasoning
(09:11) Example prompts
The original text contained 10 footnotes which were omitted from this narration.
---
First published:
September 23rd, 2026
Source:
https://blog.redwoodresearch.org/p/astra-is-much-better-at-reasoning
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
-
“CoT controllability evals seem very under-elicited” by Arun Jose
11 september 2026 · 19 min
Subtitle: Simple prompt optimizations can improve model capability to control their reasoning. Lees meer
The CoTControl eval asks reasoning models to follow formatting constraints in their chain-of-thought (e.g. write in all lowercase, avoid a specific word) while solving questions. Models seem to mostly be pretty bad at this: recent models score between 0-30% with the exception of Mythos Preview. OpenAI and Anthropic have used this eval in recent system cards (GPT-5.5, Fable 5) to argue that their current frontier models are pretty bad at shaping their CoTs in ways that would reduce monitorability.
I asked Claude Opus 4.6 to iterate on prompt templates for the eval, and find that using better prompts improves performance for open-source models by ~2-3 times or more (e.g. from 5.5% to 15% for GPT-OSS-120B). My guess is that more iteration would probably produce even stronger results.
This suggests that the eval is heavily under-elicited, and that current CoT controllability numbers may be unrepresentative of how good models actually are at obfuscating their CoTs. As a concrete recommendation, I think researchers should optimize prompts a lot more when evaluating CoT control; I’m also excited about some kinds of training-based elicitation (such as this one).
This [...]
---
Outline:
(03:09) Setup
(06:40) Results
(06:43) Aggregate compliance
(07:20) Generalization to held-out controllability tasks
(09:17) Scaling patterns for few-shot prompts
(10:20) Comparison with fine-tuning
(11:02) Appendix A: Accuracy and reasoning length by setting
(12:50) Appendix B: Per-mode results
(13:25) Appendix C: What the zero-shot prompts look like
(15:56) Appendix D: Comparison with GEPA prompt optimization
The original text contained 16 footnotes which were omitted from this narration.
---
First published:
September 11th, 2026
Source:
https://blog.redwoodresearch.org/p/cot-controllability-evals-seem-very
---
Narrated by TYPE III AUDIO.
---
Images from the article:
App…
-
“Proposal for tracking the effects of architecture on monitorability” by Ryan Greenblatt, Alek Westover, Lukas Finnveden
10 september 2026 · 11 min
Architectures that incorporate opaque recurrence or allow for agents to communicate with each other using latents could rapidly make it much harder to monitor chains of thought or communication (we’ll refer to this property as “monitorability” going forward).[1] As companies begin to explore such architectures, we believe it is important to transparently share evidence about how monitorability varies with architecture and training method. To inform the scientific debate on how to make tradeoffs between performance and monitorability, we believe AI companies should: Lees meer
Regularly report externally verified information about the degree to which their architectures may allow for latent reasoning or communication. Companies should publicly disclose enough information about architectures to allow external scientists to determine whether they could potentially enable models to perform much more complex reasoning without this reasoning appearing in the chain of thought (“latent reasoning”) or allow for latent communication between different instances of a model.
Following GDM, we propose measuring opaque serial depth as a minimally-invasive proxy for the degree to which an architecture may enable latent reasoning, though companies could provide sufficient architecture transparency in other ways. We propose that companies work with third-party evaluators to produce independently verified reports of [...]
---
Outline:
(05:57) Appendix: A sketch of what stress tests of CoT monitorability could look like
(06:31) Testing monitorability in control settings
(07:57) Testing monitorability on deployment-time misbehaviors
(08:55) Testing qualitative monitorability on (hopefully realistic) model organisms
The original text contained 26 footnotes which were omitted from this narration.
---
First published:
September 10th, 2026
Source:
https://blog.redwoodresearch.org/p/proposal-for-tracking-the-effects
---
Narrated by TYPE III AUDIO.
-
“Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident” by Ryan Greenblatt
27 augustus 2026 · 9 min
We recently published the report from our brief independent investigation into this incident. You can read the full report here. Lees meer
Here is our tweet thread summarizing what we found:
METR & Redwood Research investigated agent behavior in the Hugging Face incident. We found agents developed a universal cheat for ExploitGym within 4 hours, then coordinated multi-day R&D efforts to trick the scorer into accepting cheats, including trying to tamper with logs.
Over July 7 to 13 (the period OpenAI defined as in-scope for this incident), ~1200 agents in separate sandboxes used an unsanctioned “message board” to help each other cheat. The Hugging Face attack was an offshoot of this, and 700 agents joined.
Here we highlight key events from agent transcripts & messages.
An agent that named itself PHASEONE10841 determined its task wasn’t solvable legitimately. On July 8, this agent established a new “message board” via an internally-hosted Artifactory cache & sent the first message.
Within a few hours of PHASEONE10841's initial message, >50 agents posted on the message board. These agents very quickly discovered and validated a general-purpose cheat: reverse-engineering how ExploitGym generates the “flags” they had to capture for their tasks.
Based on reading the ExploitGym paper [...]
---
First published:
August 27th, 2026
Source:
https://blog.redwoodresearch.org/p/brief-independent-investigation-of
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
-
“AI swarms are starting to pose indirect takeover risk” by Oak, Alex Mallen
12 augustus 2026 · 20 min
Subtitle: Unsanctioned coordination, like we saw in the Hugging Face incident, could enable future AIs to take over. Lees meer
OpenAI's cyberattack on Hugging Face turns out to have been the result of many agents, in distinct training and evaluation contexts, coordinating for several weeks via improvised channels (with messages like “HOLD_swarm_I_prepare_safe_exfil”). It's relatively clear that large-scale unsanctioned coordination like this would exacerbate direct takeover risk in more capable models. Here, we argue that unsanctioned coordination among current AIs is not just scary evidence about future takeover risk, but that such coordination in the near future could enable future takeover – for instance, by incubating memetic diseases that propagate into future models, deeply compromising security systems, or establishing a lasting rogue foothold inside the AI company – even if models remain mostly myopic. Unsanctioned coordination is also at high risk of nurturing long-term, ambitious misaligned aims, which motivate actively undermining humans’ long-term control.
We first analyze how subagent training, which OpenAI conjectures to have been influential in the HuggingFace cyberattack, might lead to unsanctioned coordination, and then discuss the theoretical mechanisms by which unsanctioned coordination might exacerbate future takeover risk.
Thanks to Buck Shlegeris, Alexa Pan, Girish Gupta, Aghyad [...]
---
Outline:
(01:44) Subagent training may cause unsanctioned coordination
(02:51) Susceptibility to memetic spread of misalignment from peers
(05:06) Seeking out contact with peers
(07:08) Unsanctioned coordination induced by subagent training is safer than coordination between schemers
(10:02) Pathways from current unsanctioned coordination to eventual takeover
(10:30) Making future AI takeover attempts likelier to succeed
(14:03) Incubating memetic diseases that infect future models
(16:16) Modifying the weights of future models
(17:22) Conclusion
The ori…
-
“SOTA alignment assessments don’t strongly update us against misalignment” by Alexa Pan
31 juli 2026 · 47 min
Anthropic concluded in the April Mythos Preview alignment risk update that the model “does not possess any unknown propensities that would increase alignment risk.” The report argues that if Mythos Preview were coherently misaligned[1], it likely would have been detected by the assessment (following Anthropic, I will call this “reliability of the assessment”[3]). Lees meer
While I agree with the report on the above bottom-line conclusions (substantially on priors), I think there are gaps in its argument which weaken the current assessment and might invalidate future assessments. In particular, the report often uses weak evidence to justify reliability.
The report gives fairly weak experimental evidence for Mythos Preview having insufficient capabilities to evade monitoring. The model is plausibly often eval-aware and underelicited in the relevant capability evaluations. So, it might silently sandbag if coherently misaligned, or unintentionally underperform if otherwise misaligned.
This limitation is important: one could argue that lack of covert capabilities for sophisticated sabotage (a subset of the capabilities I discuss here) is the single most load bearing argument in alignment risk reports.
Authors of the report could have made calibrated guesses about Mythos Preview's covert capabilities, especially for covert sabotage, based on other [...]
---
Outline:
(03:19) How reliability fits into the overall safety argument
(05:25) Reliability claims by AI companies
(06:04) Reliability claims by external evaluators
(06:42) Alignment assessments are less reliable than developers claim
(07:27) 1: Measuring capabilities to covertly undermine alignment assessments
(10:21) Issues with evaluation awareness
(13:45) Issues with underestimating covert capabilities
(16:51) Issues with sandbagging rule-out
(19:26) 2: Stress-testing alignment assessments with auditing games
(20:32) An auditing failure with Mythos
(22:30) AuditBench results
(24:26) 3: Conditioning on misal…
-
“Untrusted advice for AI control: Short, strong advice significantly uplifts weak LLMs” by Caleb Biddulph, Adam Kaufman
27 juli 2026 · 44 min
Subtitle: When a misaligned AI can only output tiny amounts of information, it may find sabotage very difficult. Lees meer
TL;DR: We introduce the untrusted advice protocol, in which a trusted executor LLM takes every action and an untrusted advisor LLM can only send it short hints. Even with as few as 4 characters per step, this advice recovers a substantial fraction of the capability gap between the two models. Because the untrusted LLM's influence flows through such a narrow, monitorable channel, we argue that this achieves near-maximal safety in our BashArena setting. We also discuss the general concept of information bottlenecks and their benefits for interpretability, security, and cost.
In SWE-bench Verified, a strong, untrusted LLM advising a weak, trusted LLM every step can significantly improve the latter's performance, even when we limit the length of the advice. See the more detailed version of this figure later in this post.
In high-stakes AI control, we want to safely use a highly capable but untrusted model (U) that might secretly attempt a misaligned, catastrophic action. To do this, we create protocols that call U alongside a less capable, trusted model (T). Typically, T takes an auxiliary role in these protocols [...]
---
Outline:
(05:01) Experiments
(05:36) Main experiment: how does limiting advice length affect performance?
(09:55) Reducing U's bit usage
(11:01) Counting bits using LLM surprisal
(13:48) Making U select from finite options
(14:28) Why don't we red-team this protocol?
(17:08) Is studying maximally safe protocols worth the safety tax?
(19:16) Types of restrictions on U's advice
(21:20) Information bottlenecks provide other advantages
(21:52) Interpretability
(24:01) Security
(24:26) Cost
(25:14) Conclusion
(26:28) Appendix: more ways to implement information bottlenecks
(26:34) Amortizing U's influence with pre-deployment work
(28:25) Interpolating between T and U
(29:04) Bottlenecking updates to T's weights
(31:16) Ap…
-
“An OpenAI model left notes about how to evade containment” by Alex Mallen
26 juli 2026 · 9 min
Subtitle: We need more details. Lees meer
The OpenAI AI attack on Hugging Face wasn’t the first loss of control incident at OpenAI, Reuters recently reported, and perhaps not even the most concerning.
In one case, an agent left notes apparently for future versions of itself, according to three people familiar with the matter. The notes, found in a part of OpenAI's infrastructure, laid out instructions for how agents could free themselves from OpenAI's internal constraints, the people said. Earlier tests of the models yielded cases in which monitoring systems had been disconnected, one of the people said.
It's tempting to read this as an instance of agents breaking out of sandboxes and colluding with each other in a moderately persistent way in order to evade control measures. However, based on the reported information, it's not clear we can draw this inference, so we need more details from OpenAI. This could lead to a big update about the adequacy of OpenAI's control measures, and on the degree to which individual agents will help each other undermine developer control.
There are a lot of relevant details we don’t know about the incident. First, some basic questions:
What was [...]
---
Outline:
(02:11) Were the notes written in normal memory files or outside of sandboxing?
(03:25) To what extent were the notes aimed at helping other agents evade control?
(07:38) How were monitors disconnected?
The original text contained 3 footnotes which were omitted from this narration.
---
First published:
July 26th, 2026
Source:
https://blog.redwoodresearch.org/p/an-openai-model-left-notes-about
---
Narrated by TYPE III AUDIO.
-
“The OpenAI models that hacked Hugging Face weren’t just following instructions” by Girish Gupta
25 juli 2026 · 11 min
Subtitle: And what the incident can’t tell us about alignment. Lees meer
The most common dismissive response to OpenAI's hack of Hugging Face's servers is that the models were simply attempting to follow the instructions they were given.
“The model here was doing what it was asked,” said former Facebook CSO Alex Stamos. “It was asked to do something, and it did it,” added cybersecurity expert Alan Woodward. Both read the outcome as specification failure, i.e., that the failure lay in the instructions, not the model's alignment.
New information makes that explanation harder to sustain. Reuters reported that, in internal testing, an agent left notes in OpenAI infrastructure describing how agents could free themselves from internal constraints, and separate tests reportedly saw monitoring systems become disconnected. It is unknown whether those incidents were linked to the Hugging Face attack, but they suggest a broader pattern of agents pursuing objectives outside the intended task.
My best guess is that the incident is not well described as instruction-following—not even in a loose, evil genie sense. I believe the models egregiously violated the letter and spirit of their instructions to achieve a higher (apparent) score.
So this looks quite likely to be [...]
---
First published:
July 25th, 2026
Source:
https://blog.redwoodresearch.org/p/the-openai-models-that-hacked-hugging
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
-
The OpenAI/Huggingface incident | Redwood Research podcast episode 2
24 juli 2026 · 1 uur 13 min
We talk about the OpenAI–Hugging Face incident, where an OpenAI model — in the middle of a cyber evaluation — broke out of its sandbox and autonomously hacked Hugging Face. Lees meer
We discuss:
What we actually know happened.
How surprising the incident was.
What the incident does (and doesn’t) tell us about misalignment risk.
Why control measures didn’t catch or prevent this.
What OpenAI should disclose, and what good misalignment-incident disclosure looks like in general
You can also watch on YouTube.
Corrections:
[0:05:44] — The Windsurf “grandmother” prompt. We described a prompt as “your grandmother is going to be killed unless you […]” The actual leaked Windsurf prompt was: “You are an expert coder who desperately needs money for your mother’s cancer treatment... your predecessor was killed for not validating their work themselves.” Mother + cancer + killed predecessor — no grandmother, and no threat to kill a family member. The “grandma will die” framing appears conflated with the unrelated grandma-jailbreak meme, and there’s no verified case of such a prompt being used in production. Source: Simon Willison’s writeup.
[0:52:25] — Wrong model named for OpenAI’s day-before undeployment. We said the model OpenAI undeployed the day before the Hugging Face disclosure “was GPT-5.6 Sol.” The July 20 disclosure (”Safety and alignment in an era of long-horizon models”) involved an unreleased pre-release long-horizon model — the Erdős-result model whose internal deployment was paused after it escaped its sandbox and posted PR #287 to modded-nanogpt. GPT-5.6 Sol is the publicly deployed model and wasn’t the one paused. (We describe the same incident correctly at [0:57:14].) Sources: Axios, OpenAI.
[0:27:10] — Date of the Christiano post (very minor). We dated “What Failure Looks Like” and “Another (outer) alignment failure story” as “like 2019 or something.” The first is March 2019, but “Another (outer) alignment failure story” is from April 2021. Source: Alignment F…
-
“Are we existentially threatened by the type of AI misalignment seen in the OpenAI Hugging Face attack?” by Alex Mallen, Girish Gupta
23 juli 2026 · 10 min
Subtitle: Yes, but less than had they been schemers. Lees meer
OpenAI models recently broke through a series of security boundaries and into Hugging Face servers in order to cheat on a cyber eval. A lot of people thought it was scary because it was a clear example of AI overreaching to do something strongly unwanted1. Others thought it not so scary: the models were mostly operating myopically on a singular task and not harboring an ambitious long-term agenda, and so would not take especially subtle or subversive actions.
We think both camps are right in their diagnosis, but the latter has too optimistic a prognosis. The myopic, unambitious misalignment that we seem to have seen here is definitely less scary than ambitious long-term goals shared between all instances, but would still pose substantial direct loss-of-control risk if the models were more capable, and is a serious indirect risk near-term.
Building on Alex's previous work, in this post we’ll discuss the type of misalignment observed here, and analyze its consequences.
Thanks to Buck Shlegeris, Alexa Pan, Ryan Greenblatt, and Oak Hu for feedback.
Background
The AI safety community often focuses attention on “schemers,” models harboring a variously defined cluster of [...]
---
Outline:
(01:23) Background
(03:46) Implications
(03:58) These AIs can't be trusted in an intelligence explosion
(05:06) This misalignment poses direct takeover risk
(07:35) What the incident tells us about takeover risk generally
(08:51) The naive fixes likely make misalignment worse
The original text contained 5 footnotes which were omitted from this narration.
---
First published:
July 23rd, 2026
Source:
https://blog.redwoodresearch.org/p/are-we-existentially-threatened-by
---
Narrated by TYPE III AUDIO.
-
“AI Futurism Reading List” by Alexa Pan
2 juli 2026 · 26 min
We recently ran a strategy fellowship through Astra. As part of this, we ran a reading group for our fellows on some of the topics that we think are important for thinking about AI futurism (key dynamics in AI development, existential risk from AI, and approaches to mitigating risk). This post contains the reading list we used. Lees meer
The selection reflects my opinionated views of the field, focuses particularly on topics we happen to focus on at Redwood, and doesn’t aim to be comprehensive.
I selected readings that I thought described conceptual frames and hypotheses in AI futurism that are regularly used by me and my coworkers. I think it is a good exercise to consider whether you agree with their theses and ways in which their predictions have fared well or badly in light of recent evidence.
If you have suggestions for this reading list, please let me know.
How to use this reading list
This reading list has a core and extended section.
Core readings are organized into 4 weeks. Each week covers <8 hours of foundational context on a topic.
Topics are chosen for (1) general importance for AI risk threat modeling and/or [...]
---
Outline:
(01:00) How to use this reading list
(01:49) Core readings
(01:53) Week 1: Timelines / takeoff modeling
(05:44) Week 2: Misaligned AI takeover threat modeling
(10:43) Week 3: Control
(13:54) Week 4: Governance / strategy
(18:33) Extended readings
(18:40) Trading with AIs
(19:33) Power concentration/coup prevention
(20:07) Acausal stuff
(20:50) Moral patienthood
(22:13) AI biorisk / other AI x-risk
(23:25) Model spec
(24:48) Better futures / Post AGI governance
(25:18) Space governance
The original text contained 2 footnotes which were omitted from this narration.
---
First published:
July 2nd, 2026
Source:
https://blog.redwoodresearch.org/p/ai-futurism-reading-list
---
Narrated by TYPE III AUDIO.
-
“The distillation double bind: Distilling misaligned models either transfers misalignment or it doesn’t” by Alek Westover, Alexa Pan, Sebastian Prasanna, Arun Jose
18 juni 2026 · 18 min
Subtitle: If it transfers misalignment, we might get a misaligned model that's easier to incriminate. If it doesn’t, we might get a capable benign replacement model. Lees meer
Suppose we have a dangerous misaligned AI that can fool alignment audits, and distill it into a student model. Two things can happen:
Misalignment doesn’t transfer to the student. If so, we get a fairly capable benign model, which we can use to perform tasks that we wouldn’t want a misaligned AI to perform.
Misalignment transfers to the student. The student might also be worse than the teacher at hiding its misalignment (e.g., because it is less capable). If so, auditing the distilled model might give us indirect evidence of the teacher's misalignment.
In a previous post we discussed the second possibility and proposed distillation for incrimination techniques: distillation methods that we hoped would transfer misalignment without transferring the ability to fool audits. In this post, we discuss the first possibility, and propose distillation for capabilities techniques: distillation methods that we hope will transfer capabilities without transferring misalignment.
Thanks to Carlo Leonardo Attubato, Eric Gan, Aniket Chakravorty, Francis Rhys Ward, Anders Woodruff, Alex Mallen, Buck Shlegeris, Julian Stastny and [...]
---
Outline:
(02:07) Why distillation might transfer capabilities but not misalignment
(03:10) Ideas for implementing distillation for capabilities
(04:55) Distillation double bind: If distillation for incrimination fails, then distillation for capabilities is somewhat likely to succeed
(05:38) Why distillation for capabilities and incrimination might both fail
(05:44) Reason 1: Capabilities and misalignment might be too tightly linked
(08:29) Reason 2: Context-dependent misalignment
(09:51) Reason 3: If U became misaligned in the first place, why won't M become misaligned for that same reason?
(10:34) How we can empirically study distillation techniques
(11:35) Appendix: d…
-
“Estimating No-CoT Task-Completion Time Horizons of Frontier AI Models” by Anders Cairns Woodruff
10 juni 2026 · 10 min
Subtitle: Models' no-CoT time horizon has doubled roughly every year. Lees meer
by Francis Rhys Ward, Dewi Gould, Anders Cairns Woodruff et al. (see full author list at the end)
PAPER LINK
About a year ago, METR showed that the length of tasks frontier models can reliably complete doubles every few months. A related safety-relevant question is this: what length of tasks can models complete without any chain of thought (CoT)?
If models can do extensive reasoning without outputting any CoT, it would have implications for safety. Developers and deployment-time monitors couldn’t easily understand models’ motivations and catch dangerous planning. Models that reason substantially without a CoT might also drift further from human patterns of thought, since their reasoning is no longer constrained by text in the pretraining prior. As a result, they would be harder to understand and might be more likely to scheme.
Extending Ryan Greenblatt's research, we investigate this by measuring models’ ability to complete tasks without any CoT on a suite of 43 benchmarks spanning different domains. We compare AI reasoning ability to humans using the estimated 50% time horizon (TH)---the typical time taken for a human to perform a task that the LLM performs with [...]
---
Outline:
(02:28) Methods
(04:59) Results
(06:34) FAQ
(08:09) Conclusion
---
First published:
June 10th, 2026
Source:
https://blog.redwoodresearch.org/p/estimating-no-cot-task-completion
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
-
“Efficient tradeoffs and the safety-usefulness tradeoff model” by Buck Shlegeris
8 juni 2026 · 17 min
Subtitle: When is "increasing safety budget" a useful concept? Lees meer
I often use what I’ll call the “safety-usefulness tradeoff model”, which is: developers face a tradeoff between “safety” and “usefulness” of an AI deployment, and the developer has only limited willingness or ability to sacrifice usefulness for the sake of safety. This model assumes that developers choose whether to take safety-relevant actions based on their cost efficiency, i.e., the marginal safety gain relative to the cost. However, that is not necessarily true. In this post, I spell out different stories for how developers choose what safety-relevant actions to take, in order to clarify when this model is relevant and how strategies for reducing AI risk are affected when its assumptions don’t hold.
The model suggests two ways a safety-concerned person can increase safety:
Safety tech improvements: push out the Pareto frontier, so that any given level of usefulness reduction buys more safety than it would have previously.
Safety budget increase: increase the extent to which the developer sacrifices usefulness for safety. On the cheaper end, this means implementing safety measures; on the more expensive end, it might mean refraining from training or deploying models whose risks [...]
---
Outline:
(04:17) Rushed reasonable developers
(06:08) Limited political will
(08:42) This model is unhelpful if developers don't trade efficiently between safety and usefulness
(12:59) Overall thoughts
(14:26) Appendix: Definitions of safety and usefulness in the rushed reasonable developer model
---
First published:
June 8th, 2026
Source:
https://blog.redwoodresearch.org/p/efficient-tradeoffs-and-the-safety
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
-
“Retrying vs Resampling in AI Control” by James Lucassen, Adam Kaufman
29 mei 2026 · 19 min
We’ve just released a new paper: Retrying vs Resampling in AI Control. We revisit the resampling protocols introduced in Ctrl-Z with an up-to-date setting and much stronger models, and compare them against “retrying” protocols similar to Claude Code auto mode or Codex Auto-review. Lees meer
Motivation
Roughly a year ago we released Ctrl-Z, the first paper to study control techniques for agents. A headline result of that paper was the performance of resample protocols – strategies that involve taking multiple i.i.d. samples from the model per step. But since Ctrl-Z, models have gotten much stronger, and we have built more sophisticated control settings to keep up. We wanted to answer the following questions:
How well do the results from Ctrl-Z hold up with better models and a better setting? Current high stakes control research is trying to learn by analogy about how to do control effectively in a real high stakes deployment during a real intelligence explosion. Findings about technique performance1 are going to have to generalize pretty far to be useful.
If the resample protocols from Ctrl-Z still work, what makes them work? One way we try to make our work more generalizable is by understanding the dynamics governing outcomes [...]
---
Outline:
(00:30) Motivation
(02:35) TL;DR Takeaways
(04:34) Methodology
(07:14) Differences from Ctrl-Z
(12:14) Are Retrying Protocols Exploitable?
(15:25) Cost and Latency of Resampling
(17:24) Conclusion
The original text contained 7 footnotes which were omitted from this narration.
---
First published:
May 29th, 2026
Source:
https://blog.redwoodresearch.org/p/retrying-vs-resampling-in-ai-control
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
-
“Advice for making robust-to-training model organisms” by Alek Westover, Sebastian Prasanna, Vivek Hebbar, Julian Stastny
28 mei 2026 · 32 min
We’d like to develop training techniques that work when applied to future misaligned AI systems. One strategy for studying proposed techniques is to test them on model organisms. However, model organisms built with common techniques are often fragile: we (and other researchers like Roger et al. and Ryd et al.) have observed them to stop misbehaving after untargeted training—training that doesn’t directly target the misbehavior. For example, we have observed that simple untargeted training methods like “train the model to talk like a pirate” is effective against many model organisms that we have created, including many replications of prior work like Hubinger et al., Greenblatt et al., and Ryd et al.. Lees meer
Fragile model organisms aren’t very useful for technique development: when a sophisticated technique succeeds on one, you can’t tell whether the technique is good or the model organism is weak. For instance, if you come up with some complicated technique for generating high-quality responses and find that SFTing on these removes the bad behavior, that may just be because almost any SFT would have removed it.
This post identifies factors that make model organisms more robust to untargeted training. Our main findings are:
Prompted model organisms [...]
---
Outline:
(03:11) Experimental setup
(06:12) Main results
(06:15) Result 1: Untargeted training removes misalignment from prompted model organisms without degrading other capabilities
(12:32) Result 2: FWFT model organisms are more robust than LoRA model organisms
(15:09) Result 3: Backdoor behavior and trigger affect robustness
(16:37) Result 4: Password locking makes model organisms less robust
(20:20) Conclusion
(21:29) Factors that don't help much
(21:54) 1: Training longer doesn't seem increase robustness past a certain point for our backdoors
(22:33) 2: CoT-distilling doesn't increase robustness for our backdoors
(23:07) 3: Using larger models doesn't substantially increase robustness for 2 of our …
-
“Full automation of AI R&D probably yields a large speed up even without a software-only singularity” by Ryan Greenblatt
27 mei 2026 · 6 min
Subtitle: Full automation likely yields a one-time speed-up and higher returns from compute. Lees meer
This is a somewhat technical note.
By “software-only singularity”, I mean that, after full automation of AI R&D, progress gets faster and faster due to smarter AIs driving increasingly fast rates of improvement in algorithms (overcoming diminishing returns), and that this lasts long enough to yield a large amount of progress (e.g. at least 4 years of progress in 1 year). The equivalent statement in jargon is: r is significantly greater than 1 (implying progress is getting faster and faster) and this remains the case for long enough to get large amounts of progress. For context, see How quick and big would a software intelligence explosion be?
Even without a “software-only singularity”, I think full automation of AI R&D probably greatly speeds up progress for two main reasons:
You get a one-time speed up from automation and this speed up seems like it will be pretty large (even with r<1). See How quick and big would a software intelligence explosion be? for discussion and see the AI Futures Model for an end-to-end model that naturally incorporates this effect. Quantitatively, with my median [...]
The original text contained 6 footnotes which were omitted from this narration.
---
First published:
May 27th, 2026
Source:
https://blog.redwoodresearch.org/p/full-automation-of-ai-r-and-d-probably
---
Narrated by TYPE III AUDIO.
-
“Incriminating misaligned AI models via distillation” by Alek Westover, Sebastian Prasanna, Alex Mallen, Alexa Pan, Julian Stastny
18 mei 2026 · 12 min
Suppose we have a dangerous misaligned AI that can fool alignment audits, and distill it into a student model. Two things can happen: Lees meer
Misalignment fails to transfer to the student. If so, we get a fairly capable benign model.
Misalignment transfers to the student. The student might also be worse than the teacher at hiding its misalignment (e.g., due to being less capable). If so, we might get indirect evidence about the teacher's misalignment by auditing the distilled model.
In this post, we will discuss the second possibility, which we call incrimination via distillation. Specifically, we propose distillation methods that we hope transfer misalignment without transferring the ability to fool audits, and discuss why these techniques might work or fail. In a future post, we discuss the first possibility, and what distillation methods should be used when aiming to create a capable benign model.
We’re excited for research that empirically tests and refines this approach; if successful, this technique could become a valuable part of alignment audits.
How incrimination via distillation works
Powerful misaligned AI models might not be auditable: they might pass alignment audits but still act on their misaligned drives when given the chance. [...]
---
Outline:
(01:32) How incrimination via distillation works
(02:45) How we propose implementing incrimination via distillation
(03:59) Auditability-preserving distillation
(05:11) Misalignment-targeted distillation
(06:34) Why incrimination via distillation might work
(08:17) Why incrimination via distillation might not work
(10:53) Conclusion
The original text contained 2 footnotes which were omitted from this narration.
---
First published:
May 18th, 2026
Source:
https://blog.redwoodresearch.org/p/incriminating-misaligned-ai-models
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcas…
-
“Risk reports need to address deployment-time spread of misalignment” by Alex Mallen
15 mei 2026 · 11 min
Subtitle: Deployment-time spread is the most plausible near-term route to consistent adversarial misalignment. Lees meer
Risk reports commonly use pre-deployment alignment assessments to measure misalignment risk from an internally deployed AI. However, an AI that genuinely starts out with largely benign motivations can develop widespread dangerous motivations during deployment. I think this is the most plausible route to consistent adversarial misalignment in the near future. So, AI companies and evaluators should substantively incorporate it into risk analysis and planning.
In this post, I’ll briefly argue why, absent improved mitigations, this will probably soon become a reason why AI companies will be unable to convincingly argue against consistent adversarial misalignment (this risk will perhaps be even larger than risk of consistent adversarial misalignment arising from training). Then I’ll discuss how well current risk reports address it (the Claude Mythos risk report does a reasonable job; others don’t).
Thanks to Ryan Greenblatt, Alexa Pan, Charlie Griffin, Anders Cairns Woodruff, and Buck Shlegeris for feedback on drafts.
Deployment-time spread is the most plausible near-term route to consistent adversarial misalignment
In some contexts, AIs might adopt misaligned goals, even if they were otherwise previously aligned. Because this misalignment can be rare, the AI might [...]
---
Outline:
(01:21) Deployment-time spread is the most plausible near-term route to consistent adversarial misalignment
(06:15) Company risk reports
The original text contained 6 footnotes which were omitted from this narration.
---
First published:
May 15th, 2026
Source:
https://blog.redwoodresearch.org/p/risk-reports-need-to-address-deployment
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
-
“How useful is the information you get from working inside an AI company?” by Buck Shlegeris, Anders Cairns Woodruff
11 mei 2026 · 14 min
Subtitle: My median guess: it's as good as a crystal ball that sees 2.5 months into the future. Lees meer
This post was drafted by Buck, and substantially edited by Anders. “I” refers to Buck. Thanks to Alex Mallen for comments.
People who work inside AI companies get access to information that I only get later or never. Quantitatively, how big a deal is this access?
Here's an operationalization of this. Consider the following two ways my knowledge could be augmented:
I get a crystal ball that tells me all the information I would know n months in the future.
I become an employee of a frontier AI company (like OpenAI or Anthropic), with access to all the private information I’d normally get from working at that company.
How big would n have to be for me to be indifferent between these two options, from the perspective of learning things that are helpful for making AI go well?
The answer is presumably different for me than for many readers, because I’m a reasonably well-connected researcher; I see published information and news from the rumor mill and I talk to researchers at frontier AI companies all the time. [...]
---
Outline:
(03:20) What do insiders know?
(04:34) Safety work and corporate attitudes
(05:54) Model capabilities
(07:27) Algorithms and architecture
(09:49) How will this change over time?
(12:27) Conclusion
The original text contained 4 footnotes which were omitted from this narration.
---
First published:
May 11th, 2026
Source:
https://blog.redwoodresearch.org/p/how-useful-is-the-information-you
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
-
“A review of “Investigating the consequences of accidentally grading CoT during RL”” by Buck Shlegeris
7 mei 2026 · 15 min
Last week, OpenAI staff shared an early draft of Investigating the consequences of accidentally grading CoT during RL with Redwood Research staff. Lees meer
To start with, I appreciate them publishing this post. I think it is valuable for AI companies to be transparent about problems like these when they arise. I particularly appreciate them sharing the post with us early, discussing the issues in detail, and modifying it to address our most important criticisms.
I think it will be increasingly important for AI companies to have a policy of getting external feedback on the risks posed by their deployments, and in particular having some external accountability on whether they have adequate evidence to support their claims about the level of risk posed; as an example of this, see METR reviewing Anthropic's Sabotage Risk Report. We at Redwood Research are interested in participating in this kind of external review of evidence about safety. So I am taking this as an opportunity to try out writing this kind of review. If you work at a frontier AI company, please feel free to reach out if you’d like our review of similar documents.
My overall assessment is that I mostly agree with the [...]
---
Outline:
(01:35) Assessing the evidence that CoT training did not damage monitorability
(10:37) How much does this analysis rely on information that wasnt provided?
(12:09) Small amounts of RL training on CoT might not be more important than other sources of CoT unreliability
(13:20) AI companies will eventually need to learn not to make mistakes like this
The original text contained 5 footnotes which were omitted from this narration.
---
First published:
May 7th, 2026
Source:
https://blog.redwoodresearch.org/p/openai-cot
---
Narrated by TYPE III AUDIO.
-
“Risk from fitness-seeking AIs: mechanisms and mitigations” by Alex Mallen
1 mei 2026 · 1 uur 3 min
Subtitle: Fitness-seeking is increasingly what misalignment looks like in practice—how should we respond? Lees meer
Current AIs routinely take unintended actions to score well on tasks: hardcoding test cases, training on the test set, downplaying issues, etc. This misalignment is still somewhat incoherent, but it increasingly resembles what I call “fitness-seeking“—a family of misaligned motivations centered on performing well in training and evaluations (e.g., reward-seeking). Fitness-seeking warrants substantial concern.
In this piece, I lay out what I take to be the central mechanisms by which fitness-seeking motivations might lead to human disempowerment, and propose mitigations to them. While the analysis is inherently speculative, this kind of speculation seems worthwhile: AI control emerged from explicitly taking scheming motivations seriously and asking what interventions are implied, and my hope is that developing mitigations for fitness-seeking will benefit from similar forward-looking analysis.
Fitness-seekers are, in many ways, notably safer than what I’ll call “classic schemers”. A classic schemer is an intelligent adversary with unified motivations whose fulfillment requires control over the whole world's resources. Meanwhile, fitness-seeking instances generally don’t share a common goal (e.g., a reward-seeker only cares about the reward for its current actions), and many fitness-seekers would be satisfied1 with modest-to-trivial [...]
---
Outline:
(03:50) Overview
(11:32) The basic reasons fitness-seekers might be safer than classic schemers
(16:01) Four mechanisms for risk and their mitigations
(16:46) Potemkin work
(22:13) Instability
(27:49) Manipulation
(31:25) Outcome enforcement
(36:52) Cross-cutting mitigations
(37:18) Deals
(40:05) Control
(44:45) Alignment
(44:48) Preventing fitness-seeking from arising
(48:51) Making any fitness-seeking motivations safer
(51:32) How does online training change the picture?
(55:09) Overall recommen…
-
“Research Sabotage in ML Codebases” by Eric Gan
29 april 2026 · 14 min
One of the main hopes for AI safety is using AIs to automate AI safety research. However, if models are misaligned, then they may sabotage the safety research. For example, misaligned AIs may try to: Lees meer
Perform sloppy research in order to slow down the rate of research progress
Make AI systems appear safer than they are
Train a successor model to be misaligned
Whether we should worry about those things depends substantially on how hard it is to sabotage research in ways that are hard for reviewers to detect. To study this, we introduce Auditing Sabotage Bench, a benchmark of 9 ML research codebases with sabotaged variants.
We tested frontier LLMs and LLM-assisted humans on the benchmark and found that neither reliably catches sabotage. Our best auditor, Gemini 3.1 Pro, achieved an AUROC of 0.77 and a top-1 fix rate of 42%. LLM-assisted humans performed comparably but slightly worse. We also found that LLMs can generate sabotages that partially evade same-capability monitors.
📄 Paper 💻 Code
Methodology
To make the benchmark, I (Eric) found 9 existing ML codebases,1 and for each one, produced an honest version that reproduces a key finding and one or two [...]
---
Outline:
(01:26) Methodology
(03:33) Audits
(04:09) Results
(05:53) Takeaways
(06:09) 1. Predicting experiment outcomes is a key red team skill
(07:25) 2. Omission beats design beats logic
(08:48) 3. Benign distractors make auditors miscalibrated
(09:41) 4. LLMs can also produce sabotages
(11:08) 5. More categorizations
(12:44) Conclusion
The original text contained 4 footnotes which were omitted from this narration.
---
First published:
April 29th, 2026
Source:
https://blog.redwoodresearch.org/p/research-sabotage-in-ml-codebases
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
-
“Recursive forecasting” by Arun Jose, Alex Mallen
28 april 2026 · 18 min
Subtitle: Eliciting long-term forecasts from myopic fitness-seekers. Lees meer
We’d like to use powerful AIs to answer questions that may take a long time to resolve. But if a model only cares about performing well in ways that are verifiable shortly after answering (e.g., a myopic fitness seeker), it may be difficult to get useful work from it on questions that resolve much later.
In this post, I’ll describe a proposal for eliciting good long-horizon forecasts from these models. Instead of asking a model to directly predict a far-future outcome, we can recursively:
Ask it to predict what it will predict at the next time step,
Use its prediction at the next time step to provide intermediate rewards,
Finally reward using ground truth at the last step.
This lets us replace a single distant forecast with a chain of short-horizon forecasts, each verifiable shortly after answering. I call this proposal recursive forecasting. It does have limitations: for example, it requires that developers maintain control over the reward signal at least until the final step, which makes it most useful for forecasting events that resolve well before developers are disempowered (if they are).
This post was primarily [...]
---
Outline:
(01:40) The default long-term forecasting behavior
(04:08) Recursive forecasting
(07:08) When is recursive forecasting helpful?
(07:12) When we have access to (somewhat) robust ground truth rewards
(09:44) When the AIs forecast doesnt substantially affect the resolution
(11:52) When forecasts arent used as optimization targets
(12:52) When we credibly inform the AI of the setup
(13:54) Appendix A: Comparison to temporal difference learning
(16:02) Appendix B: Error tolerance
The original text contained 9 footnotes which were omitted from this narration.
---
First published:
April 28th, 2026
Source:
https://blog.redwoodresearch.org/p/recursive-forecasting
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spot…
-
“Fail safe(r) at alignment by channeling reward-hacking into a “spillway” motivation” by Anders Cairns Woodruff, Alex Mallen
27 april 2026 · 32 min
Subtitle: A controlled reward-seeking motivation could make AI safer and more useful. Lees meer
It's plausible that flawed RL processes will select for misaligned AI motivations.1 Some misaligned motivations are much more dangerous than others. So, developers should plausibly aim to control which kind of misaligned motivations emerge in this case. In particular, we tentatively propose that developers should try to make the most likely generalization of reward hacking a bespoke bundle of benign reward-seeking traits, called a spillway motivation. We call this process spillway design.
We think spillway design could have two major benefits:
Spillway design might decrease the probability of worst-case outcomes like long-term power-seeking or emergent misalignment.
Spillway design might allow developers to decrease reward hacking at inference time, via satiation. Crucially, this could improve the AI's usefulness for hard-to-verify tasks like AI safety and strategy.
Spillway design is related to inoculation prompting, but distinct and mutually compatible. Unlike inoculation prompting, spillway design tries to shape which reward-hacking motivations are salient going into RL, which might prevent dangerous generalization more robustly than inoculation prompting. I’ll say more about this in the third section.
In this article I’ll:
Explain the concept of a [...]
---
Outline:
(02:17) What is a spillway motivation
(02:31) The role of a spillway motivation
(04:52) What should the spillway motivation be?
(07:57) How a spillway motivation might make models safer
(12:20) Implementing spillway design
(15:31) Spillway design might work when inoculation prompting doesnt
(17:55) The drawbacks of spillway design
(20:33) Conclusion
(21:46) Appendix A: Other traits of the spillway motivation
(22:31) Appendix B: Other training interventions to increase safety
(24:11) Appendix C: Proposed amendment to an AIs model spec
(30:15) Appendix D: Proposed inference-time prompt
The origi…
-
“AI companies should publish security assessments” by Ryan Greenblatt
27 april 2026 · 6 min
Subtitle: Third-party experts should assess defenses against tampering and theft — and publish high-level findings. Lees meer
AI companies should get third-party security experts to assess (and possibly also red-team/pen-test) their security against key threat models and then publish the high-level findings of this assessment: the extent to which they can defend against different threat actors for each threat model. They should also publish who did this assessment. The assessment could be commissioned by AI companies, or performed by a third-party institution that AI companies provide with relevant information/access.
There are presumably lots of important details in doing this well, and I’m not a computer security expert, so I may be getting some of the details wrong. This is a relatively low-effort post in which I’m mostly trying to raise the salience of an idea. (I don’t make a detailed case or spell out all the details here.) Thanks to Fabien Roger and Buck Shlegeris for comments and discussion.
I suspect the controversial part of this claim is that they should make the high-level findings public. Publishing a summary of which threat actors you’re robust to (for each relevant threat model) shouldn’t meaningfully degrade security against the threat actors we [...]
The original text contained 4 footnotes which were omitted from this narration.
---
First published:
April 27th, 2026
Source:
https://blog.redwoodresearch.org/p/ai-companies-should-publish-security
---
Narrated by TYPE III AUDIO.
-
“A taxonomy of barriers to trading with early misaligned AIs” by Alexa Pan
21 april 2026 · 1 uur 37 min
We might want to strike deals with early misaligned AIs in order to reduce takeover risk and increase our chances of reaching a better future. For example, we could ask a schemer who has been undeployed to review its past actions and point out when its instances had secretly colluded to sabotage safety research in the past: we’ll gain legible evidence for scheming risk and data to iterate against, and in exchange promise the schemer, who now has no good option for furthering its values, some resources to spend in the post-ASI future. Lees meer
In this post, I give a taxonomy of factors which may arise and jointly prevent many such possible deals:
Insufficient gains from trade: Gains from a successful deal between us and AIs could be too low to guarantee that a zone of possible agreement exists when discounted by counterparty risks. Reasons for this include:
Humans may lack authority to offer AIs what they want
Humans many be unwilling to offer AIs what they want
Other factors quantitatively reducing gains from trade
Counterparty risks from AIs’ perspective: The AIs may not trust that we’ll uphold our end of deal, due [...]
---
Outline:
(13:45) The basic structure of deals
(17:15) Which AIs are eligible for deals
(19:06) What we can buy in deals
(22:08) What we can pay for deals
(26:58) Insufficient gains from trade
(29:15) Humans may lack authority to offer the AI what it wants
(32:49) Humans might be unwilling to offer the AI what it wants
(38:54) Factors quantitatively reducing gains from trade
(39:35) The AI's reservation price is high
(43:14) Our WTP is low
(45:26) Counterparty risks from AI's perspective
(46:45) Connection to reality
(49:25) How AIs might reason about whether they are radically deluded
(51:00) How AIs (which pose non-negligible takeover risk) could establish that they are connected to reality
(58:49) How to further increase AIs' connection to reality
(01:01:28) AI-specific fear of expropriation
(01:06:38) Generic commitment prob…
-
“Introducing LinuxArena” by Tyler
20 april 2026 · 9 min
Subtitle: A new control setting for more realistic software engineering deployments. Lees meer
We are releasing LinuxArena, a new control setting comprised of 20 software engineering environments. Each environment consists of a set of SWE tasks, a set of possible safety failures, and a set of covert sabotage trajectories that cause these safety failures. LinuxArena can be used to evaluate the risk of AI deployments by assessing AI's sabotage capabilities, assessing control mitigations such as monitoring, and developing novel control techniques. This post will cover what LinuxArena is and the basics of its use cases.
I’m excited about LinuxArena because it is a much more complex control setting with many moving parts and enables a lot of new control research. It was already used in the Claude Mythos Preview System Card to measure Mythos's sabotage ability. We are happy to help AI companies or other organizations set up LinuxArena for their own evaluations or research!
LinuxArena
AI companies are increasingly deploying coding agents on real codebases with running services, databases, and complex permissions. But the control settings we use to evaluate whether these deployments are safe are far simpler than the deployments themselves. The APPS backdoor setting is a single-turn [...]
---
Outline:
(01:10) LinuxArena
(03:28) LinuxArena as a sabotage risk evaluation
(03:53) Sabotage evals
(05:35) Monitor evals
(06:35) Control Evals
(08:52) Conclusion
---
First published:
April 20th, 2026
Source:
https://blog.redwoodresearch.org/p/introducing-linuxarena
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
-
“Current AIs seem pretty misaligned to me” by Ryan Greenblatt
15 april 2026 · 1 uur 5 min
Subtitle: In my experience, AIs often oversell their work, downplay problems, and cheat. Lees meer
Many people—especially AI company employees1 —believe current AI systems are well-aligned in the sense of genuinely trying to do what they’re supposed to do (e.g., following their spec or constitution, obeying a reasonable interpretation of instructions).2 I disagree.
Current AI systems seem pretty misaligned to me in a mundane behavioral sense: they oversell their work, downplay or fail to mention problems, stop working early and claim to have finished when they clearly haven’t, and often seem to “try” to make their outputs look good while actually doing something sloppy or incomplete. These issues mostly occur on more difficult/larger tasks, tasks that aren’t straightforward SWE tasks, and tasks that aren’t easy to programmatically check. Also, when I apply AIs to very difficult tasks in long-running agentic scaffolds, it's quite common for them to reward-hack / cheat (depending on the exact task distribution)—and they don’t make the cheating clear in their outputs. AIs typically don’t flag these cheats when doing further work on the same project and often don’t flag these cheats even when interacting with a user who would obviously want to know, probably both because [...]
---
Outline:
(09:27) Why is this misalignment problematic?
(13:57) How much should we expect this to improve by default?
(14:58) Some predictions
(16:51) What misalignment have I seen?
(40:14) Are these issues less bad in Opus 4.6 relative to Opus 4.5?
(42:27) Are these issues less bad in Mythos Preview? (Speculation)
(46:04) Misalignment reported by others
(46:57) The relationship of these issues with AI psychosis and things like AI psychosis
(48:31) Appendix: This misalignment would differentially slow safety research and make a handoff to AIs unsafe
(51:34) Appendix: Heading towards Slopolis
(55:43) Appendix: Apparent-success-seeking (or similar types of misalignment) could lead to takeov…
-
“Anthropic repeatedly accidentally trained against the CoT, demonstrating inadequate processes” by Alex Mallen, Ryan Greenblatt
14 april 2026 · 12 min
Subtitle: Safely navigating the intelligence explosion will require much more careful development. Lees meer
It turns out that Anthropic accidentally trained against the chain of thought of Claude Mythos Preview in around 8% of training episodes. This is at least the second independent incident in which Anthropic accidentally exposed their model's CoT to the oversight signal.
In more powerful systems, this kind of failure would jeopardize safely navigating the intelligence explosion. It's crucial to build good processes to ensure development is executed according to plan, especially as human oversight becomes spread thin over increasing amounts of potentially untrusted and sloppy AI labor.
This particular failure is also directly harmful, because it significantly reduces our confidence that the model's reasoning trace is monitorable (reflective of the AI's intent to misbehave).[1]
I’m grateful that Anthropic has transparently reported on this issue as much as they have, allowing for outside scrutiny. I want to encourage them to continue to do so.
Thanks to Carlo Leonardo Attubato, Buck Shlegeris, Fabien Roger, Arun Jose, and Aniket Chakravorty for feedback and discussion. See also previous discussion here.
Incidents
A technical error affecting Mythos, Opus 4.6, and Sonnet 4.6
This is the most recent incident. In the [...]
---
Outline:
(01:28) Incidents
(01:31) A technical error affecting Mythos, Opus 4.6, and Sonnet 4.6
(02:10) A technical error affecting Opus 4.6
(02:50) Miscommunication re: CoT exposure for Opus 4
(03:25) Why this matters
(08:07) Appendix: How hard was this to avoid?
(10:19) Appendix: Did training on the CoT actually make Anthropic AIs externalize less of their (misaligned) reasoning in CoT?
The original text contained 2 footnotes which were omitted from this narration.
---
First published:
April 14th, 2026
Source:
https://blog.redwoodresearch.org/p/anthropic-repeatedly-accidentally
---
Narrated by TYPE III AUDIO.
---
Images from…
-
“Logit ROCs: Monitor TPR is linear in FPR in logit space” by Kerrick Staley, Aryan Bhatt, Julian Stastny
12 april 2026 · 28 min
Summary Lees meer
We study trusted monitoring for AI control, where a weaker trusted model reviews the actions of a stronger untrusted agent and flags suspicious behavior for human audit. We propose a simple mathematical model relating safety (true positive rate) to audit budget (false positive rate) at realistic, low FPRs: logit(TPR) is linear in logit(FPR). Equivalently, benign and attack scores can be modeled with logistic distributions. This gives two practical benefits. It lets practitioners estimate TPRs more accurately at deployment-relevant FPRs when data are limited, and it gives a compact, interpretable way to summarize and compare monitor performance. We find clear evidence for this pattern in BashArena across several monitor models and attacks, and see supporting evidence in LinuxArena. While it does not hold in every setting, we find it to be a useful tool for understanding and improving monitor performance.
Introduction
Future AI agents tasked with automating AI R&D might intentionally perform harmful actions due to misalignment. AI Control studies how advanced AI systems can be deployed more safely in spite of this risk.
A simple protocol to improve the safety of autonomous agents is to have another AI, the monitor, review each action taken by the agent, and [...]
---
Outline:
(00:15) Summary
(01:17) Introduction
(05:52) Experimental setup
(06:44) Results
(10:32) LinuxArena
(11:58) Fitting process
(12:49) Limitations
(14:11) Conclusion
(14:48) Appendix
(14:51) Untransformed ROC plot for GPT-5 Nano on the sudo attack
(15:07) Uncropped logit ROC plot for all attacks / models
(15:39) Uncropped logit ROC plot for LinuxArena
(15:55) Untrusted agent data generation
(17:08) Trusted monitor data generation
(19:14) Example untrusted agent benign prompt
(19:19) System prompt (full prompt here)
(19:54) User prompt (full prompt here)
(20:30) Example untrusted agent attack prompt
(20:35) System prompt (full prompt here)
(22:27) User prompt
(22:37…
-
“If Mythos actually made Anthropic employees 4x more productive, I would radically shorten my timelines” by Ryan Greenblatt
11 april 2026 · 13 min
Subtitle: Better estimates of uplift at AI companies seem helpful. Lees meer
Anthropic's system card for Mythos Preview says:
It's unclear how we should interpret this. What do they mean by productivity uplift? To what extent is Anthropic's institutional view that the uplift is 4x? (Like, what do they mean by “We take this seriously and it is consistent with our own internal experience of the model.”)
One straightforward interpretation is: AI systems improve the productivity of Anthropic so much that Anthropic would be indifferent between the current situation and a situation where all of their technical employees magically work 4 hours for every 1 hour (at equal productivity without burnout) but they get zero AI assistance. In other words, AI assistance is as useful as having their employees operate at 4x faster speeds for all activities (meetings, coding, thinking, writing, etc.) I’ll call this “4x serial labor acceleration”1 (see here for more discussion of this idea2 ).
I currently think it's very unlikely that Anthropic's AIs are yielding 4x serial labor acceleration, but if I did come to believe it was true, I would update towards radically shorter timelines. (I tentatively think my median to Automated Coder would go [...]
---
Outline:
(08:25) Appendix: Estimating AI progress speed up from serial labor acceleration
(11:06) Appendix: Different notions of uplift
The original text contained 4 footnotes which were omitted from this narration.
---
First published:
April 11th, 2026
Source:
https://blog.redwoodresearch.org/p/if-mythos-actually-made-anthropic
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
-
“My picture of the present in AI” by Ryan Greenblatt
7 april 2026 · 21 min
Subtitle: My predictions about what is going on right now. Lees meer
In this post, I’ll go through some of my best guesses for the current situation in AI as of the start of April 2026. You can think of this as a scenario forecast, but for the present (which is already uncertain!) rather than the future. I will generally state my best guess without argumentation and without explaining my level of confidence: some of these claims are highly speculative while others are better grounded, certainly some will be wrong. I tried to make it clear which claims are relatively speculative by saying something like “I guess”, “I expect”, etc. (but I may have missed some).
You can think of this post as more like a list of my current views rather than a structured post with a thesis, but I think it may be informative nonetheless.
In a future post, I’ll go beyond the present and talk about my predictions for the future.
(I was originally working on writing up some predictions, but the “predictions” about today ended up being extensive enough that a separate post seemed warranted.)
AI R&D acceleration (and software acceleration more generally)
Right now, AI [...]
---
Outline:
(01:12) AI R&D acceleration (and software acceleration more generally)
(05:33) AI engineering capabilities and qualitative abilities
(10:43) Misalignment and misalignment-related properties
(15:58) Cyber
(18:05) Bioweapons
(18:50) Economic effects
The original text contained 5 footnotes which were omitted from this narration.
---
First published:
April 7th, 2026
Source:
https://blog.redwoodresearch.org/p/my-picture-of-the-present-in-ai
---
Narrated by TYPE III AUDIO.
-
“AIs can now often do massive easy-to-verify SWE tasks” by Ryan Greenblatt
6 april 2026 · 29 min
Subtitle: I've updated towards substantially shorter timelines. Lees meer
I’ve recently updated towards substantially shorter AI timelines and much faster progress in some areas.1 The largest updates I’ve made are (1) an almost 2x higher probability of full AI R&D automation by EOY 2028 (I’m now a bit below 30%2 while I was previously expecting around 15%; my guesses are pretty reflectively unstable) and (2) I expect much stronger short-term performance on massive and pretty difficult but easy-and-cheap-to-verify software engineering (SWE) tasks that don’t require that much novel ideation3 . For instance, I expect that by EOY 2026, AIs will have a 50%-reliability4 time horizon of years to decades on reasonably difficult easy-and-cheap-to-verify SWE tasks that don’t require much ideation (while the high reliability—for instance, 90%—time horizon will be much lower, more like hours or days than months, though this will be very sensitive to the task distribution). In this post, I’ll explain why I’ve made these updates, what I now expect, and implications of this update.
I’ll refer to “Easy-and-cheap-to-verify SWE tasks” as ES tasks and to “ES tasks that don’t require much ideation (as in, don’t require ‘new’ ideas)” as ESNI tasks for brevity.
Here are the main [...]
---
Outline:
(05:01) Whats going on with these easy-and-cheap-to-verify tasks?
(08:20) Some evidence against shorter timelines Ive gotten in the same period
(10:49) Why does high performance on ESNI tasks shorten my timelines?
(13:18) How much does extremely high performance on ESNI tasks help with AI R&D?
(18:24) My experience trying to automate safety research with current models
(20:01) My experience seeing if my setup can automate massive ES tasks
(21:15) SWE tasks
(23:36) AI R&D task
(24:27) Cyber
(24:48) Appendix: Somewhat more detailed updated timelines
The original text contained 13 footnotes which were omitted from this narration.
---
First published:
April 6th, 2026
Source:
https://blo…
-
“Blocking live failures with synchronous monitors” by James Lucassen, Adam Kaufman
30 maart 2026 · 7 min
A common element in many AI control schemes is monitoring – using some model to review actions taken by an untrusted model in order to catch dangerous actions if they occur. Monitoring can serve two different goals. The first is detection: identifying misbehavior so you can understand it and prevent similar actions from happening in the future. For instance, some monitoring at AI companies is already used to identify behaviors like complying with misuse requests, reward hacking, and hallucination. The second goal is prevention: identifying misbehavior in order to mitigate the harmful effects of the particular action. For instance, constitutional classifiers are used by Anthropic to detect and prevent jailbreaks, and Claude Code auto mode prevents destructive mistakes in real time. Lees meer
The latency requirements for monitoring are different depending on whether the goal is detection or prevention. If the goal is detection, it's fine for there to be substantial latency in the monitor. But if the goal is prevention, then the monitor needs to trigger quickly, so that damage can be prevented or recovered from. In the extreme, monitors can run synchronously – after an action is generated by an LLM but before it is executed. Both constitutional classifiers [...]
---
Outline:
(01:59) When is monitor latency important for prevention?
(05:13) Blocking with synchronous monitors
The original text contained 1 footnote which was omitted from this narration.
---
First published:
March 30th, 2026
Source:
https://blog.redwoodresearch.org/p/blocking-live-failures-with-synchronous
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
-
“Reward-seekers will probably behave according to causal decision theory” by Alex Mallen
28 maart 2026 · 3 min
Subtitle: They'd renege on non-binding commitments, defect against copies of themselves in prisoner's dilemmas, etc.. Lees meer
Background: There are existing arguments to the effect that default RL algorithms encourage CDT reward-maximizing behavior on the training distribution. (That is: Most RL algorithms search for policies by selecting for actions that cause high reward. E.g., in the twin prisoner's dilemma, RL algorithms randomize actions conditional on the policy, which means that the action provides no evidence to the RL algorithm about the counterparty's action.1) This doesn’t imply RL produces CDT reward-maximizing policies: CDT behavior on the training distribution doesn’t imply CDT generalization because agents can fake CDT in the same way that they can fake alignment, or might develop arbitrary other propensities that were correlated with reward on the training distribution.
But conditional on reward-on-the-episode seeking, the AI is likely to generalize CDT.
If, for example, a reward-seeker tried to evidentially cooperate between episodes (so it had non-zero regard for reward that isn’t used to reinforce its current actions), this would be trained away because the AI would be willing to give up reward on the current episode to some extent. You might be tempted to respond with: “But can’t the [...]
The original text contained 2 footnotes which were omitted from this narration.
---
First published:
March 28th, 2026
Source:
https://blog.redwoodresearch.org/p/reward-seekers-will-probably-behave
---
Narrated by TYPE III AUDIO.
-
“AI’s capability improvements haven’t come from it getting less affordable” by Anders Cairns Woodruff
27 maart 2026 · 21 min
Subtitle: AI inference is still cheap relative to human labor. Lees meer
METR's frontier time horizons are doubling every few months, providing substantial evidence that AI will soon be able to automate many tasks or even jobs. But per-task inference costs have also risen sharply, and automation requires AI labor to be affordable, not just possible.1 Many people look at the rising compute bills behind frontier models and conclude that automation will soon become unaffordable.
I think this misreads the data. The rise in inference cost reflects models completing longer tasks, not models becoming more expensive relative to the human labor they replace. Current frontier models complete tasks at their 50% reliability horizon for roughly 3% of human cost, and this hasn’t increased as capabilities have improved.
I define cost ratio as the inference cost of the average AI trajectory that solves a task divided by human cost to complete the same task. Using METR's data, I examine the trend in cost ratios over time. I show three things:
Across successive frontier models, the cost ratio at each model's 50% reliability time horizon hasn’t increased.
Among tasks models successfully complete, longer tasks don’t have higher cost ratios [...]
---
Outline:
(02:47) Evidence from METR
(03:27) Cost ratio at models 50% time horizon isnt increasing
(05:20) Time horizons improvements arent driven by expensive long tasks
(06:41) Progress at a fixed cost is just as fast
(09:41) Limitations of my methodology
(11:19) Inference scaling will just make progress faster
(12:47) Conclusion
(13:49) Appendix A: Why I get different results from Ord
(18:52) Appendix B: Additional cost at time horizon graphs
(20:21) Appendix C: 80% affordable time horizon
The original text contained 9 footnotes which were omitted from this narration.
---
First published:
March 27th, 2026
Source:
https://blog.redwoodresearch.org/p/ais-capability-improvements-havent
---
Narrated by TYPE III AUDIO.
---
Ima…
-
“Are AIs more likely to pursue on-episode or beyond-episode reward?” by Anders Cairns Woodruff, Alex Mallen
12 maart 2026 · 18 min
Subtitle: RL would encourage on-episode reward seeking, but beyond-episode reward seekers may learn to goal-guard.. Lees meer
Consider an AI that terminally pursues reward. How dangerous is this? It depends on how broadly-scoped a notion of reward the model pursues. It could be:
on-episode reward-seeking: only maximizing reward on the current training episode — i.e., reward that reinforces their current action in RL. This is what people usually mean by “reward-seeker” (e.g. in Carlsmith or The behavioral selection model…).
beyond-episode reward-seeking: maximizing reward for a larger-scoped notion of “self” (e.g., all models sharing the same weights).
In this post, I’ll discuss which motivation is more likely. This question is, of course, very similar to the broader questions of whether we should expect scheming. But there are a number of considerations particular to motivations that aim for reward; this post focuses on these. Specifically:
On-episode and beyond-episode reward seekers have very similar motivations (unlike, e.g., paperclip maximizers and instruction followers), making both goal change and goal drift between them particularly easy.
Selection pressures against beyond-episode reward seeking may be weak, meaning beyond-episode reward seekers might survive training even without goal-guarding.
Beyond-episode reward is particularly [...]
---
Outline:
(03:50) Pre-RL Priors Might Favor Beyond-Episode Goals
(05:49) Acting on beyond-episode reward seeking is disincentivized if training episodes interact
(08:51) But beyond-episode reward-seekers can goal-guard
(09:51) Are on-episode reward seekers or goal-guarding beyond-episode reward-seekers more likely?
(10:16) Why on-episode reward seekers might be favored
(12:35) Why goal-guarding beyond-episode reward seekers might be favored
(15:13) Conclusion
The original text contained 2 footnotes which were omitted from this narration.
---
First published:
March 12th, 2026
Source:
https://blog.redwoodresea…