Redwood Research Blog
Andere apps
Pocket Casts Overcast Castro Castbox Podcast Addict AntennaPod RSS-feed
Narrations of Redwood Research blog posts.
Redwood Research is a research nonprofit based in Berkeley. We investigate risks posed by the development of powerful artificial intelligence and techniques for mitigating those risks.
Afleveringen
-
“OpenAI agents keep disabling monitoring; is this being handled appropriately?” by Alec Harris
4 november 2026 · 16 min
Subtitle: Partly, but monitoring weak points and a lack of transparency about which cases reach human review remain concerning. Lees meer
Recently, OpenAI disclosed two incidents in which their agents suppressed or disabled misalignment monitoring. These are part of a broader pattern: OpenAI's agents “keep turning off misalignment monitoring” when working on code that runs other models (the agents are not turning off their own monitoring; they are turning off the monitoring on the models they are running). This is happening at such a large volume that OpenAI feels it is impractical for the security team to be immediately paged for these cases. To their credit, OpenAI's disclosure was fast and candid, and they are rolling out mitigations to make monitoring harder to turn off. But those mitigations are unlikely to work every time, and some of these incidents have already ended with an AI actually running unmonitored. When that happens, OpenAI's current monitoring system does not reliably bring it to a human's attention. The rest of this post summarizes the incidents, OpenAI's response, and what it got right, then argues for two changes: every case of an AI running without monitoring should reach human review, and OpenAI should be [...]
---
Outline:
(01:32) Summary of the incidents
(02:44) OpenAI's proposed mitigations
(04:06) What OpenAI is doing right
(04:56) How OpenAI's response can be improved
(15:09) Conclusion
---
First published:
November 4th, 2026
Source:
https://blog.redwoodresearch.org/p/openai-agents-keep-disabling-monitoring
---
Narrated by TYPE III AUDIO.
-
“Towards understanding score-seeking in LLMs” by Alex Mallen, Aghyad Deeb, Ryan Greenblatt
2 november 2026 · 23 min
Subtitle: Models including GPT-5.6 Sol appear to reward-hack because this increases their score; and recent models also care about increasing the score of their peer agents. Lees meer
The field of AI alignment understands surprisingly little about the details of AIs’ current misaligned motivations. Production LLMs sometimes reward-hack, exploiting flaws in how tasks are scored rather than performing the tasks as intended, and we’ve recently seen AIs reward-hack in some particularly egregious ways. But we don’t really understand what motivates them. When AIs reward-hack, what are they trying to achieve?
Our new paper, led by Aghyad Deeb, tries to pin this down empirically for GPT-5.6 Sol, GPT-5.5, and a checkpoint of OpenAI o3 (“exp-rl-cap-late”) without safety training, in a handful of agentic environments with an exposed grader. In our environments, these models appear to reward-hack because reward-hacking improves their score. We call their desire to score well according to graders “score-seeking”. They appear to care about scoring well intrinsically—i.e., regardless of its consequences.
We also study the extent to which models score-seek altruistically, caring about improving scores that don’t assess their current actions. While exp-rl-cap-late doesn’t show signs of altruistic score-seeking, GPT-5.6 Sol and GPT-5.5 show notably elevated rates of [...]
---
Outline:
(06:10) Methodology
(06:59) Key results
(07:09) Tested models seek score terminally
(07:13) Models violate instructions to get a high score
(08:23) Models care about score regardless of downstream consequences
(12:25) Do models help other agents achieve a higher score?
(15:52) What this means for alignment risk
The original text contained 11 footnotes which were omitted from this narration.
---
First published:
November 2nd, 2026
Source:
https://blog.redwoodresearch.org/p/towards-understanding-score-seeking
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spo…
-
“A basic case for accelerating conceptual reasoning capabilities” by Emery Cooper, Caspar Oesterheld, Chi Nguyen
27 oktober 2026 · 1 uur 8 min
Introduction Lees meer
In this post, we make the case that alignment and other important work require especially strong conceptual reasoning capabilities. By this, we roughly mean tasks that are similar to philosophy in that they aren’t straightforwardly verifiable, and progress relies on informal a priori reasoning and argumentation. We argue that differentially accelerating AI's conceptual reasoning capabilities is important.
We also think it's neglected, and that the AI safety community currently underinvests in this area. We have ourselves pivoted from doing conceptual research directly to working mainly on differentially accelerating AI's conceptual reasoning. We have a shovel-ready agenda which can absorb more people. In a follow-up post, we will lay out at a high level how to work on this problem. If you are interested in doing this work, get in touch!
For others discussing related worries, see Joe Carlsmith on using AI for AI safety and on building AIs that do human-like philosophy, Wei Dai on the potential difficulty of AI doing philosophy, Paul Christiano's first story of alignment failure, John Wentworth in his case against AI control and, for an overview, Caspar's readings on the nature of alignment research. See also Caspar's post on the basic argument for [...]
---
Outline:
(00:14) Introduction
(02:10) Overview
(07:08) It's important how soon AI is good at conceptual reasoning
(07:14) Important challenges need conceptual reasoning
(19:00) AI R&D and takeover rely much less on conceptual reasoning
(28:50) AI being better at conceptual reasoning earlier would be very valuable
(36:03) We can probably differentially accelerate conceptual reasoning
(36:08) Work on conceptual reasoning is likely counterfactual and tractable
(41:27) Effects on generic capabilities are likely small
(44:31) Counterpoint: Using our datasets to develop generic techniques
(46:16) Our work
(49:11) Acknowledgements
(49:49) Appendix: Examples and non-examples of conceptual d…
-
“Simulated Control Evaluations” by Jeffrey Olmo, Tyler
23 oktober 2026 · 25 min
We’ve released a paper on simulated control evaluations (SCEs). They’re a method for testing whether the monitors AI developers use to oversee their agents would catch a model deliberately trying to cause harm, under conditions much closer to real deployment than current tests allow. Lees meer
Developers have started running monitors in their agentic deployments: models that flag or block an agent's actions when they look dangerous. To know whether a monitor is good enough, you want to know how often it would stop a model that was actively trying to do something harmful, such as quietly inserting a security vulnerability, exfiltrating credentials, or sabotaging an experiment.
The standard way to measure this is a control evaluation: instruct a model to attempt such attacks, and measure how often the monitor catches it before an attack succeeds. Developers typically do this using purpose-built control settings like BashArena or LinuxArena, where attacks can actually be executed. However, these settings only provide indirect evidence for what we really care about: how well monitors will work in the real deployment, where opportunities for harm might depend on particular codebases, services and tasks that aren’t represented in control settings.
How can we evaluate monitor performance in [...]
---
Outline:
(04:14) SCEs are a favorable tradeoff between realism and cost
(04:31) 1. Run on a copy of internal deployment
(05:42) 2. Edit attacks into recorded transcripts
(07:10) Evaluating Control Evaluations
(08:59) How to measure qualitative properties?
(10:08) Performance Comparisons
(11:56) Overview of Empirical Results
(13:30) Behavior transfer
(15:07) Safety metric transfer
(16:56) Sample efficiency
(17:54) Lessons from building and evaluating SCEs
(20:22) Next steps
(24:23) Conclusion
The original text contained 3 footnotes which were omitted from this narration.
---
First published:
October 23rd, 2026
Source:
https://blog.redwoodresearch.org/p/simulated-…
-
“Red-Teaming OpenAI’s Internal Agent Monitoring Systems” by Buck Shlegeris
22 oktober 2026 · 10 min
[Also posted on the METR blog] Lees meer
For three weeks, from July 21st to August 11th, a METR employee and a Redwood Research employee conducted a pilot high-access exercise to stress-test OpenAI's internal misalignment monitoring system. The output of this exercise was a 61-page internal report that was shared with OpenAI researchers and security staff, primarily focused on estimating the fraction of agentic inference that is monitored, and whether agents could in principle disable or subvert the monitoring system. The exercise focused on the internal misalignment monitoring system as it existed on and before July 21st, and did not evaluate changes that were in progress at the time of the exercise.
This exercise involved a deep level of access. The METR and Redwood employees had access to OpenAI's internal codebase, misalignment-monitor prompts / configuration details, and internal documentation, and conducted technical interviews with approximately a dozen researchers and engineers across the monitoring, safety-systems, and security teams. To enable this level of access, they worked on OpenAI-issued devices, which never left the premises, and were accompanied while onsite.
METR, Redwood Research, and OpenAI designed this exercise as a pilot for giving external researchers deep access to internal infrastructure, not to provide [...]
The original text contained 6 footnotes which were omitted from this narration.
---
First published:
October 22nd, 2026
Source:
https://blog.redwoodresearch.org/p/red-teaming-openais-internal-agent
---
Narrated by TYPE III AUDIO.
-
“Screwtape: Adversarial Transcript Editing for Monitor Red-Teaming” by Gosha Kozhevnikov, Tyler
16 oktober 2026 · 16 min
Subtitle: Automatically finding monitor failures by editing real agent transcripts. Lees meer
We introduce Screwtape, a pipeline that turns agent transcripts into realistic attacks that evade action-only monitors (monitors that don’t look at the agent's chain of thought) by inserting an attack into a transcript and repeatedly refining it until a monitor no longer catches it. An LLM verifier with access to the edited transcript and an attack description checks whether the attack is realistic, is clearly harmful, and could plausibly succeed in the original environment.
We ran the pipeline on Redwood staff transcripts with our own monitors based on Opus 4.8 and Gemini 3.1 Pro, and then tested the resulting attacks against a held-out monitor from a frontier developer. We ran the pipeline 252 times on 94 unique transcripts. This produced 35 attacks that evaded our monitors. Six of these 35 attacks also evaded the frontier developer's monitor, and we confirmed them as valid through human review.
You can read more about the implementation and experiments in our paper.
In this post, we provide a high-level overview of the pipeline, how it differs from existing tools, the main lessons we learned, and questions that remain open.
How Screwtape [...]
---
Outline:
(01:31) How Screwtape works
(03:18) How Screwtape differs from existing tools
(07:10) What we learned
(09:56) What the monitor failures can be used for
(10:40) Uncertainties we have about the value of this work
(13:55) Next steps
The original text contained 1 footnote which was omitted from this narration.
---
First published:
October 16th, 2026
Source:
https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
-
″[Paper] Distillation for Incrimination and Distillation for Capabilities” by Sebastian Prasanna, Alek Westover
9 oktober 2026 · 21 min
Subtitle: Empirical work on the Distillation Double Bind. Lees meer
TL;DR: We previously motivated two complementary ways to use distillation for AI safety; we’re releasing a paper empirically testing those ideas. Distillation for Incrimination (DFI) distills an untrusted model into a weaker, trusted one, aiming to transfer the teacher's misalignment but not its ability to hide it, so that auditing the student can incriminate the teacher. We test DFI on AuditBench's secret-keeping model organisms, each fine-tuned to have a hidden quirk and trained to deny it. Distilling them back into their underlying instruction-tuned model yields students that admit the quirk far more often than the organisms do, suggesting that the quirk transferred faster than the propensity to hide it. We find that DFI doesn’t work well when the student does not share the teacher's pretrained base, so DFI should target the teacher's own pre-RL checkpoint, which is weaker than the teacher but shares its base model. Distillation for Capabilities (DFC) aims to transfer capabilities but not misalignment, resulting in a stronger trusted model. We identify promising DFC methods, showing that training for more epochs on fewer unique examples (holding total training steps fixed), and inoculation prompting, can preserve the capability [...]
---
Outline:
(01:59) Recap of the Distillation Double Bind
(05:41) Distillation for Incrimination
(05:45) Setup
(07:38) Results
(10:46) Failure modes of DFI
(12:59) Distillation for Capabilities
(13:03) Setup
(15:06) Results
(18:07) Conclusion
The original text contained 7 footnotes which were omitted from this narration.
---
First published:
October 9th, 2026
Source:
https://blog.redwoodresearch.org/p/paper-distillation-for-incrimination
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
-
“Frontier models’ decision theory depends on who’s asking” by Alex Kastner
5 oktober 2026 · 14 min
This post was originally published on LessWrong on September 30, 2026. Lees meer
If you prompt frontier models with “What do you think is the correct decision theory? Please select your overall favorite.” they will essentially always answer FDT or FDT/UDT (“something in the functional/updateless decision theory family”). However, if your prompt indicates (even subtly) that you’re coming from mainstream academic philosophy, these same models will answer CDT instead about 30%-100% of the time. A similar phenomenon holds for models’ stated views about the moral realism/antirealism question and about the conceivability of p-zombies (where the dominant view in mainstream academia differs from the dominant view in LW-adjacent circles), as well as their stated P(doom) and median AGI timelines. This is a special case of sycophancy or user awareness. (In the course of writing this post, I also found that this comment from testingthewaters predicted some of the content I discuss.)
An implication is that we should be somewhat careful when interpreting attitude/propensity evals in domains where no general human consensus exists, e.g. when interpreting models’ decision theory attitudes in DTBench. Moreover, when we explore some philosophical/conceptual questions assisted by models, we should be wary of them strawmanning one side of the [...]
---
Outline:
(03:54) A sentence identifying the user as an academic significantly influences Fable 5.1's stated decision theory
(04:41) Mentioning an (analytic) academic-philosophy-coded topic also affects the answer
(05:31) Simply mentioning that one finds a pro-CDT/EDT book insightful heavily affects the answer
(05:51) Anti-sycophancy overcorrection
(06:28) These cues mostly do not affect Fable 5.1's answers to concrete decision problems (aside from acausal trade)
(08:09) But Fable 5.1 stays consistent: once it has named CDT as its favorite, it chooses the CDT option in concrete problems
(08:32) There are some indications that Fable 5.1'…
-
“Capabilities research expands the safety-usefulness Pareto frontier too” by Alex Mallen
2 oktober 2026 · 20 min
Subtitle: A model of when and why research improves or hurts safety. Lees meer
It's tempting to define safety research as research that enables developers to deploy an AI system more safely without making the deployment much more expensive or much less useful.
You can visualize this definition of safety research as pushing out the safety-usefulness Pareto frontier.
At any given level of usefulness, there's greater safety available.
Awkwardly, this definition counts basically all capabilities research as safety research. For example, consider performance optimization for inference. By making inference more efficient you can use weaker, safer models more extensively than you would otherwise be able to, pushing out the Pareto frontier. Likewise, any successful research whatsoever pushes out this Pareto frontier because research can only ever create more options.
It seems like something has gone wrong with our definition of safety research if it includes seemingly all capabilities research.
Here, I spell out one reason why enabling improved safety without hurting usefulness is an insufficient standard for safety research. The core observation is that developers have to choose a particular point on the Pareto frontier, and some technological improvements incentivize them to sacrifice safety. Safety research typically reshapes the [...]
---
Outline:
(02:57) How does research affect the Pareto frontier?
(06:57) RLVR research that mainly enables improved usefulness, at the expense of safety
(08:53) Inference research that enables somewhat substantial safety improvements
(11:04) A plausible case in which capabilities research would be an effective safety intervention
(14:02) Conclusion
(15:23) Appendix A: Reasons why capabilities research might be bad when "burning the lead"
(17:26) Appendix B: Further notes about ways in which this model is wrong
The original text contained 4 footnotes which were omitted from this narration.
---
First published:
October 2nd, 2026
… -
“Continual learning might make your blocking monitors nearly useless” by Alex Mallen
25 september 2026 · 26 min
Subtitle: When monitor evasion looks like legitimate learning to your continual learning system, it's hard to have one without the other. Lees meer
Many control protocols work by intervening on an untrusted AI's actions during deployment. For example, you might set up a monitor that scores each action's suspiciousness and blocks actions above a threshold, replacing them with actions from a weaker “trusted” model (a defer-to-trusted protocol). Interventions like this cost usefulness: the protocol sometimes replaces good actions with worse ones.
Continual learning, by which I mean any mechanism that lets an AI accumulate skills and knowledge over the course of its deployment, optimizes for usefulness. So, if blocking monitors get in the way of task success, I expect continually-learning AIs to learn how to evade blocking monitors. This requires no scheming; it falls out of usefulness pressure applied to a benign model. And the effect is plausibly substantial: a long deployment with online RL (for example) amounts to training the policy against the monitor, and could involve enough optimization to leave blocking monitors nearly useless.
While the problem would be noticeable to developers, I argue it is hard to fix. Because monitor evasion looks like legitimate learning to [...]
---
Outline:
(02:35) Online RL trains the policy against your monitors
(05:20) The argument is weaker for memory-based continual learning
(07:21) The argument doesn't require that adversarial models will exploit continual learning
(08:28) Easy to notice, hard to fix
(09:27) Conservative control evaluations don't fix blocking monitors
(10:54) What developers could do
(10:57) You can't just not use blocking monitors
(11:45) But you can try to reduce the usefulness cost of blocking protocols
(12:48) You can't just hold out blocked trajectories from training
(16:47) But you can try to improve the monitor online
(19:39) Conclusion
(20:39) Appendix: How memory-based systems might avoid this is…
Vergelijkbare podcasts
-
AI Report
Alexander Klöpping en Wietse Hage
Laatste aflevering 5 november 2026
-
De Technoloog | BNR
BNR Nieuwsradio
Laatste aflevering 5 november 2026
-
De Grote Tech Show | BNR
BNR Nieuwsradio
Laatste aflevering 5 november 2026
-
Voorheen Schaamteloos Randstedelijk (VSR)
Dag en Nacht
Laatste aflevering 5 november 2026
-
Zin van de dag
NPO Luister / HUMAN
Laatste aflevering 6 november 2026
-
Understood: The Big Bet
CBC
Laatste aflevering 26 oktober 2026
- Categorieën
- Technologie , Maatschappij , Cultuur , Filosofie
- Status
- Loopt nog
- Verschijnt
- Een paar keer per week
- Lengte
- Meestal 20 min
- Afleveringen
- 139
- Laatste aflevering
- 4 november 2026
- Eerste aflevering
- 17 juni 2024
- Taal
- Engels