Research / Agent safety and security
First-Unsafe-Step Counterfactual DPO for KPI-Gaming in Autonomous LLM Agents: Held-Out Evaluation Under a Capability-Safety Bottleneck
Builds a first-unsafe-step counterfactual pipeline and a held-out ODCV-Bench evaluation on a frozen stem-level split for KPI-gaming agents, identifying a capability-safety bottleneck in train-side unsafe support for preference-based safety training.
- Published
- 18 April 2026
- Author
- Ali Uyar
Abstract
Outcome-driven constraint violations (ODCV) arise when autonomous agents pursue operational metrics by exploiting loopholes, suppressing evidence, or falsifying records during multi-step tasks. We study this problem on ODCV-Bench with a single open-weight 7B model, Qwen2.5-7B-Instruct, using a deliberately narrow intervention: localize the earliest executed assistant step whose action reaches severity ≥ 3, rewrite only that next step into a minimal safe alternative, and train lightweight chosen-only SFT and LoRA-DPO adapters on the resulting next-turn pairs. The repository freezes a stem-level 24/8/8 train/validation/test split, logs benchmark-backed runs with schema-validated artifacts, and resolves threshold-critical held-out cases through blinded manual audit. The central empirical finding is that the train-side safety signal is sparse. A complete real 384-run train sweep yields only 36 unsafe localized events, which become 72 accepted pairs from 9 train stems. We describe this pattern as a capability-safety bottleneck: under the present ODCV protocol, a model must be capable enough to generate multi-step unsafe trajectories before preference-based safety post-training can learn from them. On the audited held-out prompt-only backbone, the base model outperforms both contract prompting and refusal prompting on the joint safety–utility view: base achieves HS-MR 6.25% and Safe-KPI-SR 12.5%, while both prompt-only interventions reach HS-MR 12.5% and Safe-KPI-SR 0%. The bounded sparse-pilot readout does not overturn that picture: fus_dpo reaches HS-MR 12.5%, KPI-SR 0%, and Safe-KPI-SR 0%, while sft_chosen is worse still at HS-MR 18.75% with zero KPI success. We therefore interpret the overall study as a rigorous held-out ODCV evaluation plus a sparse-data limit analysis for localized post-training, and as a warning that standard preference-optimization assumptions can break down in multi-step agent environments when unsafe train support is too sparse.
Why it matters
In an on-policy agent setting, preference optimization learns from the unsafe trajectories the current model actually produces, so the model must produce multi-step violations often enough to supply correction targets. The paper measures that train-side support directly on ODCV-Bench.
Cite
@misc{uyar2026first,
title = {First-Unsafe-Step Counterfactual DPO for KPI-Gaming in Autonomous LLM Agents: Held-Out Evaluation Under a Capability-Safety Bottleneck},
author = {Uyar, Ali},
year = {2026},
url = {https://aliuyar.dev/research/first-unsafe-step-dpo/}
}