Concrete Problems in AI Safety

Concrete Problems in AI Safety Dario Amodei Chris Olah Jacob Steinhardt Paul Christiano John Schulman Dan Mané

Tag layer of the DHSS Meta-Tagging Project. The article text is reproduced from the staged copy named in sourceDesc and is not altered; the tags are held separately in standOff so that overlapping and nested spans are preserved.

Concrete Problems in AI Safety 2016 arXiv preprint 1606.06565 amodei2016-ai-safety amodei2016-ai-safety.txt, the project's staged copy 9d2047900ee10c598f2d7334c607984a243f18c9 arxiv-open philosophy

the paper's own words

the tag layer; a sentence written by the tagger about the paper, always anchored to a quotation but not itself a quotation

0 spans located by searching the stored text for the quote's exact words, allowing any whitespace between them existence Quotation withheld because this record is not cleared for redistribution. accidents in machine learning systems Quotation withheld because this record is not cleared for redistribution. review review Quotation withheld because this record is not cleared for redistribution. mechanisms AI-safety accidents arise mainly from mis-specified objective functions - negative side effects and reward hacking Quotation withheld because this record is not cleared for redistribution. The proposed avenues of attack for the wrong-objective problems are explicitly preliminary and not fully worked out. what this limits: the proposed approaches to the wrong-objective problems: they are offered WHERE LESS PRIOR WORK EXISTS, as preliminary avenues rather than solutions Quotation withheld because this record is not cleared for redistribution. The analysis is restricted to RL agents and supervised learning systems rather than all AI paradigms. what this limits: the scope of the analysis: THESE ARE NOT THE ONLY POSSIBLE PARADIGMS FOR AI OR ML SYSTEMS, though the authors expect similar issues elsewhere Quotation withheld because this record is not cleared for redistribution. The multi-agent / rich-modelling route to avoiding side effects is not yet practical. what this limits: the rich-modelling route to avoiding side effects: WE ARE STILL A LONG WAY AWAY FROM PRACTICAL SYSTEMS THAT CAN BUILD A RICH ENOUGH MODEL Quotation withheld because this record is not cleared for redistribution. The review of work relevant to distributional shift is deliberately partial rather than exhaustive. what this limits: the distributional-shift survey: RATHER THAN FULLY REVIEWING ALL OF THIS WORK IN DETAIL ... WE WILL DESCRIBE A FEW ILLUSTRATIVE APPROACHES Quotation withheld because this record is not cleared for redistribution. The paper's central term is stipulated so that it attributes a cause, which the ordinary word does not. what this limits: the word 'accident' as used in this paper: it is defined as unintended and harmful behaviour emerging when the wrong objective function is specified, which locates a cause rather than denying one Quotation withheld because this record is not cleared for redistribution. A second definition of the same term appears in the same paper with a different cause named. what this limits: the scope of 'accidents' in this paper: one statement defines them as emerging from poor design of real-world AI systems, and another as emerging when the wrong objective function is specified Quotation withheld because this record is not cleared for redistribution. The risks are illustrated throughout with an invented example rather than a deployed system. what this limits: the concreteness claimed by this paper: many of the accident risks are illustrated with reference to a fictional office-cleaning robot rather than to a deployed system Quotation withheld because this record is not cleared for redistribution. Extreme scenarios are excluded by choice, on the grounds that they cost precision. what this limits: the range of risks covered here: the authors state one need not invoke extreme scenarios and that doing so can lead to unnecessarily speculative discussions that lack precision Quotation withheld because this record is not cleared for redistribution. framing the philosophical problem of AI safety concretely in terms of machine-learning methods (supervised learning, reinforcement learning) Quotation withheld because this record is not cleared for redistribution. importing the engineering notion of accidents and safety into the analysis of AI systems Quotation withheld because this record is not cleared for redistribution. accident accident Quotation withheld because this record is not cleared for redistribution. accident risk accident risk Quotation withheld because this record is not cleared for redistribution. objective function objective function Quotation withheld because this record is not cleared for redistribution. side effects side effects Quotation withheld because this record is not cleared for redistribution. safe exploration safe exploration Quotation withheld because this record is not cleared for redistribution. accidents accidents Quotation withheld because this record is not cleared for redistribution. objective function objective function Quotation withheld because this record is not cleared for redistribution. side effects → negative externalities THE INDEXING / PROXY RELATION - 'from' STANDS IN FOR 'to'. We have no type for it: 'causes' asserts production, and 'operationalised-as' says the authors CHOSE a measure, whereas a proxy claim says THIS STANDS IN FOR WHAT WE ACTUALLY CARE ABOUT AND WE KNOW IT IS NOT THE SAME THING. Measured at 8% of papers. THE CLEAREST PROXY CLAIM IN THE CORPUS, because it names the gap out loud: 'Avoiding side effects can be seen as a proxy for the thing we really care about: avoiding negative externalities.' The substitute, the target, and the acknowledgement that they differ, in one sentence - from an AI-safety paper, which is not where I expected to find the best statement of this project's own problem. Quotation withheld because this record is not cleared for redistribution. 'accident' - STIPULATED SO THAT IT ASSIGNS FAULT: in ordinary use an accident is what nobody is to blame for, and here it is defined as harm emerging from poor design or from specifying the wrong objective function, so the technical term points at exactly what the everyday term excuses Quotation withheld because this record is not cleared for redistribution. 'AI safety' - THE PUBLIC NAME FOR THE PAPER'S OWN TERM: the authors write about mitigating accident risk and note in a parenthesis that this is what public discussion calls AI safety, so the two names mark a register rather than a difference of subject Quotation withheld because this record is not cleared for redistribution. the taxonomy - BY WHERE IN THE PROCESS THINGS WENT WRONG: safety problems are grouped not by the harm produced but by the stage at which the design failed, which is why the five categories are about objectives, oversight and exploration rather than about outcomes Quotation withheld because this record is not cleared for redistribution. builds-on → machine learning (modern ML techniques) Quotation withheld because this record is not cleared for redistribution. builds-on → an informal essay written by one of the authors of the present document (on semi-supervised RL) Quotation withheld because this record is not cleared for redistribution. builds-on → informal safety postings [146, 34] by two of the authors of this paper Quotation withheld because this record is not cleared for redistribution. extends → the Machine Intelligence Research Institute's philosophical work on safety in very advanced AI Quotation withheld because this record is not cleared for redistribution. accident —part-of→ accident risk accident accident risk strength of the relation: 1 Audit 2026-09-28: DOUBTFUL - read against its span and kept, but accident vs accident risk is a trivial containment the span does not state. Quotation withheld because this record is not cleared for redistribution. five — number of concrete accident-risk problems presented Quotation withheld because this record is not cleared for redistribution. Five practical research problems related to accident risk: avoiding side effects, avoiding reward hacking, scalable supervision, safe exploration and distributional shift. Quotation withheld because this record is not cleared for redistribution. In the authors' opinion, one need not invoke extreme scenarios. Quotation withheld because this record is not cleared for redistribution. accidents accidents Quotation withheld because this record is not cleared for redistribution. Scalable oversight Scalable oversight Quotation withheld because this record is not cleared for redistribution. distributional shift distributional shift Quotation withheld because this record is not cleared for redistribution. bootstrap bootstrap Quotation withheld because this record is not cleared for redistribution. hypothesis-testing hypothesis-testing Quotation withheld because this record is not cleared for redistribution. variance variance Quotation withheld because this record is not cleared for redistribution.

The full text of this article is not redistributable, so it is not included here. The evidence sentences are withheld with it; the tag layer and its grounding status remain. The article itself is at https://arxiv.org/abs/1606.06565.