| phenomenon | accidents in machine learning systems— “In this paper we discuss one such potential impact: the problem of accidents in machine learning systems, defined as unintended and harmful behavior that may emerge from poor design of real-world AI systems.” |
| claim-type | existence— “Overall, we believe there are many concrete open technical problems relating to accident prevention in machine learning systems.” |
| method — evidence | — “We review previous work in these areas as well as suggesting research directions with a focus on relevance to cutting-edge AI systems.” |
| transfer | from machine learning: framing the philosophical problem of AI safety concretely in terms of machine-learning methods (supervised learning, reinforcement learning)— “in terms of classic methods in machine learning, such as supervised classification and reinforcement learning” | transfers[0] |
| transfer | from engineering (safety engineering): importing the engineering notion of accidents and safety into the analysis of AI systems— “This issue arises in almost any engineering discipline” | transfers[1] |
| mechanism | AI-safety accidents arise mainly from mis-specified objective functions - negative side effects and reward hacking— “two broad mechanisms that make it easy to produce wrong objective functions” | mechanisms[0] |
| builds-on | machine learning (modern ML techniques)— “We believe it is usually most productive to frame accident risk in terms of practical (though often quite general) issues with modern ML techniques” | claim_relations[0] |
| builds-on | an informal essay written by one of the authors of the present document (on semi-supervised RL)— “The discussion of semi-supervised RL draws heavily on an informal essay, https://medium.com/ai-control/ cf7d5375197f written by one of the authors of the present document.” | claim_relations[1] |
| builds-on | informal safety postings [146, 34] by two of the authors of this paper— “Finally, two of the authors of this paper have written informally about safety in AI systems [146, 34]; these postings provided inspiration for parts of the present document.” | claim_relations[2] |
| extends | the Machine Intelligence Research Institute's philosophical work on safety in very advanced AI— “By contrast, our focus is on the empirical study of practical safety problems in modern machine learning systems, which we believe is likely to be robustly useful across a broad variety of potential risks, both short- and long-term.” | claim_relations[3] |
| sense | 'accident' - STIPULATED SO THAT IT ASSIGNS FAULT: in ordinary use an accident is what nobody is to blame for, and here it is defined as harm emerging from poor design or from specifying the wrong objective function, so the technical term points at exactly what the everyday term excuses— “We define accidents as unintended and harmful behavior that may emerge from machine learning systems when we specify the wrong objective function” | senses[0] |
| sense | 'AI safety' - THE PUBLIC NAME FOR THE PAPER'S OWN TERM: the authors write about mitigating accident risk and note in a parenthesis that this is what public discussion calls AI safety, so the two names mark a register rather than a difference of subject— “we frame mitigating accident risk (often referred to as “AI safety” in public discussions) in terms of classic methods in machine learning” | senses[1] |
| sense | the taxonomy - BY WHERE IN THE PROCESS THINGS WENT WRONG: safety problems are grouped not by the harm produced but by the stage at which the design failed, which is why the five categories are about objectives, oversight and exploration rather than about outcomes— “We can categorize safety problems according to where in the process things went wrong” | senses[2] |
| audit | accident —part-of→ accident risk — Audit 2026-09-28: DOUBTFUL - read against its span and kept, but accident vs accident risk is a trivial containment the span does not state.— “we frame mitigating accident risk (often referred to as “AI safety” in public discussions) in terms of classic methods in machine learning” | concept_edges[1] |
| for which use | limitation — the proposed approaches to the wrong-objective problems: they are offered WHERE LESS PRIOR WORK EXISTS, as preliminary avenues rather than solutions— “When discussing the problems that arise from having the wrong objective function (reward hacking and side effects, and to a lesser extent scalable supervision), where less prior work exists, our aim is more exploratory—we seek to more clearly define the problem and suggest possible broad avenues of attack, with the understanding that these… | limitations[0] |
| for which use | limitation — the scope of the analysis: THESE ARE NOT THE ONLY POSSIBLE PARADIGMS FOR AI OR ML SYSTEMS, though the authors expect similar issues elsewhere— “These are not the only possible paradigms for AI or ML systems, but we believe they are sufficient to illustrate the issues we have in mind, and that similar issues are likely to arise for other kinds of AI systems.” | limitations[1] |
| for which use | limitation — the rich-modelling route to avoiding side effects: WE ARE STILL A LONG WAY AWAY FROM PRACTICAL SYSTEMS THAT CAN BUILD A RICH ENOUGH MODEL— “However we are still a long way away from practical systems that can build a rich enough model to avoid undesired side effects in a general sense.” | limitations[2] |
| for which use | limitation — the distributional-shift survey: RATHER THAN FULLY REVIEWING ALL OF THIS WORK IN DETAIL ... WE WILL DESCRIBE A FEW ILLUSTRATIVE APPROACHES— “Rather than fully reviewing all of this work in detail (which would necessitate a paper in itself), we will describe a few illustrative approaches and lay out some of their relative strengths and challenges.” | limitations[3] |
| for which use | limitation — the word 'accident' as used in this paper: it is defined as unintended and harmful behaviour emerging when the wrong objective function is specified, which locates a cause rather than denying one— “We define accidents as unintended and harmful behavior that may emerge from machine learning systems when we specify the wrong objective function” | limitations[4] |
| for which use | limitation — the scope of 'accidents' in this paper: one statement defines them as emerging from poor design of real-world AI systems, and another as emerging when the wrong objective function is specified— “the problem of accidents in machine learning systems, defined as unintended and harmful behavior that may emerge from poor design of real-world AI systems” | limitations[5] |
| for which use | limitation — the concreteness claimed by this paper: many of the accident risks are illustrated with reference to a fictional office-cleaning robot rather than to a deployed system— “For concreteness, we will illustrate many of the accident risks with reference to a fictional robot whose job is to clean up messes in an office using common cleaning tools” | limitations[6] |
| for which use | limitation — the range of risks covered here: the authors state one need not invoke extreme scenarios and that doing so can lead to unnecessarily speculative discussions that lack precision— “However, in our opinion one need not invoke these extreme scenarios to productively discuss accidents, and in fact doing so can lead to unnecessarily speculative discussions that lack precision” | limitations[7] |
| replication | untested |
| no type for this | side effects → negative externalities — THE INDEXING / PROXY RELATION - 'from' STANDS IN FOR 'to'. We have no type for it: 'causes' asserts production, and 'operationalised-as' says the authors CHOSE a measure, whereas a proxy claim says THIS STANDS IN FOR WHAT WE ACTUALLY CARE ABOUT AND WE KNOW IT IS NOT THE SAME THING. Measured at 8% of papers. THE CLEAREST PROXY CLAIM IN THE CORPUS, because it names the gap out loud: 'Avoiding side effects can be seen as a proxy for the thing we really care about: avoiding negative externalities.' The substitute, the target, and the acknowledgement that they differ, in one sentence - from an AI-safety paper, which is not where I expected to find the best statement of this project's own problem.— “• Multi-Agent Approaches: Avoiding side effects can be seen as a proxy for the thing we really care about: avoiding negative externalities.” | unhoused_relations[0] |
| limitation | The proposed avenues of attack for the wrong-objective problems are explicitly preliminary and not fully worked out.— “When discussing the problems that arise from having the wrong objective function (reward hacking and side effects, and to a lesser extent scalable supervision), where less prior work exists, our aim is more exploratory—we seek to more clearly define the problem and suggest possible broad avenues of attack, with the understanding that these… | limitations[0] |
| limitation | The analysis is restricted to RL agents and supervised learning systems rather than all AI paradigms.— “These are not the only possible paradigms for AI or ML systems, but we believe they are sufficient to illustrate the issues we have in mind, and that similar issues are likely to arise for other kinds of AI systems.” | limitations[1] |
| limitation | The multi-agent / rich-modelling route to avoiding side effects is not yet practical.— “However we are still a long way away from practical systems that can build a rich enough model to avoid undesired side effects in a general sense.” | limitations[2] |
| limitation | The review of work relevant to distributional shift is deliberately partial rather than exhaustive.— “Rather than fully reviewing all of this work in detail (which would necessitate a paper in itself), we will describe a few illustrative approaches and lay out some of their relative strengths and challenges.” | limitations[3] |
| limitation | The paper's central term is stipulated so that it attributes a cause, which the ordinary word does not.— “We define accidents as unintended and harmful behavior that may emerge from machine learning systems when we specify the wrong objective function” | limitations[4] |
| limitation | A second definition of the same term appears in the same paper with a different cause named.— “the problem of accidents in machine learning systems, defined as unintended and harmful behavior that may emerge from poor design of real-world AI systems” | limitations[5] |
| limitation | The risks are illustrated throughout with an invented example rather than a deployed system.— “For concreteness, we will illustrate many of the accident risks with reference to a fictional robot whose job is to clean up messes in an office using common cleaning tools” | limitations[6] |
| limitation | Extreme scenarios are excluded by choice, on the grounds that they cost precision.— “However, in our opinion one need not invoke these extreme scenarios to productively discuss accidents, and in fact doing so can lead to unnecessarily speculative discussions that lack precision” | limitations[7] |