Concrete Problems in AI Safety

Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, Dan Mané · arXiv preprint · 2016 · philosophy · record amodei2016-ai-safety
tag layer · index · concepts · authors · עברית

Where this record's text came from. faceted 2026-08 by reading the locally held text (pdftext); every evidence string verified as an exact substring of that text before merging. Claim type was hard to pin: the paper is an agenda-setting review whose central assertion is that concrete accident-risk problems exist and are tractable now, so 'existence' fits better than 'method' (no single technique is offered) or 'prediction'. method is 'review' because the paper reviews prior work and proposes experiments rather than running any. The MIRI relation is a scope/approach differentiation rather than a factual disagreement; 'extends' is the nearest available type and the verbatim sentence is the paper's own contrast clause. Many citations in the paper are bare bracketed numbers with no author names, so most citation contexts did not qualify as explicit relations to named prior work and were left out. 2026-08-14 and 2026-08-15 schema-0.3 passes: every quote verified as an exact substring of the staged text and anchored at merge time; concept terms verified present; every edge endpoint an existing node. Update, 15 September 2026: EXCEPTION, measured with the merge gate's own predicate (Paper.locate): 1 quote(s) on this record are not found as one exact string in the current staged text - content_tags.statistics[4].evidence. The sentence above does not hold for them as written. Where a row records why - the quote is grounded in another copy of the paper, joins two passages with an ellipsis, or differs from the staged text by extraction noise - that note stays on the row.

The quality of the text these quotes were cut from. symbol-font-not-extractable: 2 '(cid:NNN)' glyph-index markers remain in a staged text of 87,125 characters, where the PDF's embedded font had no ToUnicode map. THE QUOTES ON THIS RECORD ARE THE PAPER'S AND THE STAGED COPY IS THE DAMAGED ONE - which is the opposite of the usual case and is why it went unnoticed: one of its 51 live quotes is not an exact substring of this text, and it was located in it by aligning the quote's WORD SEQUENCE under exactly this damage - the words are there, the characters inside them are not. Not repaired in place: editing the staged text moves every offset after the edit and re-hashes every anchor on the record. Measured at 113aae60fefc3eb1, re-measured unchanged at 54b63482778ddf56.

Some words in these quotes contain the wrong character. The PDF this text was extracted from substituted 1 character inside a word — a currency sign or a stray accent where a letter pair belongs, as in signi¢cantly for significantly, or a single ligature glyph, as in conﬁdence. They are valid characters in the wrong place, so they survive every encoding check. The quotes keep them for the same reason as above, and a search of this page will not find the affected words.

This record’s own method is review, and the tags below follow from that. A review’s findings, quantities and statistics are things stated IN this paper, and some of them report results obtained elsewhere. No tag on this page names a different author. Measured across every review in this corpus, 81% of a review’s tags carry no citation at all — most of them are the review speaking. The corpus does not adjudicate the rest: read an unmarked tag as a claim this paper makes, not necessarily one it made first.

1 quote on this page was cut from a different copy of this article. This project holds two extractions of this paper: the plain text every other quote here is anchored in, and the publisher’s own HTML edition, which we are not licensed to republish. It is verbatim in the second and cannot be found in the first — not because the tagging is wrong, but because the extraction damaged the passage: a page number dropped into the middle of a number, a formula moved out of its sentence, a ligature where two letters belong. The record names the copy for each one. The quote is the paper’s; the extraction is not.

How to read this page. The article itself is not reproduced here — this paper is not cleared for redistribution, so what follows is the tag layer alone. The evidence sentences are withheld with the article. The table at the foot lists every tag the record carries; each row names the facet it belongs to, and the whose words column says whether the tag records the paper's own wording or the tagger's sentence about the paper. Every tag links to each other paper in the corpus carrying it. The machine-readable edition of the same thing is the TEI file beside this one, where the tags live in standOff, and the article is at its publisher via the link below. When we have changed our mind. Some entries in the table begin with a word in capitals and the reason for it. EVIDENCE CORRECTED — the finding still stands, but we had quoted the wrong sentence from the paper, and the quotation has been replaced with the right one. RETRACTED — the entry itself was wrong; it stays on the page, marked, with the reason, rather than being deleted. WITHDRAWN AND REPLACED — the entry was wrong, and a corrected version of it appears in the entries below. Each mark carries its own reason, in brackets, because the reasons differ from one entry to the next. An entry we have WITHDRAWN is removed from this page and kept in the record, with its reason. A line reading Audit and a date is a later re-reading of the entry against its whole sentence: DOUBTFUL means it was kept, and the reason says what is loose about it. Quotation marks inside the tagger's sentence. Where a tag is the tagger's own sentence, a phrase in quotation marks inside it is borrowed from the paper — the tagger's wording around it is not. The paper's own sentence is the evidence quote for that row; on this page it is withheld with the article, and the article is at its publisher. Nothing inside quotation marks anywhere on this page is ours: if it is quoted, it is the paper's. Whose words a concept carries. A concept entry records a term the paper itself uses, so the term, the sense given for it, and its evidence are the paper's own words, quoted where the paper fixes them. This is the opposite of a sense entry in the tag table, which is the tagger's sentence about how the paper uses a term. Where a concept entry could not be grounded in the paper's words, the entry says so on its own row.

Other fields define these differently the same word, defined another way in another discipline
reward 16 fields · learning 7 fields · robustness 3 fields · bootstrap 2 fields · linear regression 2 fields · variance 2 fields

Tags on this paper

Every tag the record carries, each linking to every other paper in the corpus that carries it. A tag appears here whether or not its words occur in the article — the spans below are only the ones a sentence in this paper licenses. Of the 11 concept tags on this paper, 7 are grounded in a quoted sentence.

AI safety value alignment reward hacking scalable oversight accident accident risk objective function side effects safe exploration accidents AI-safety accidents arise mainly from mis-specified objective functions - negative side effects and reward hacking safety objective learning reward exploration systems robustness supervision distributional shift bootstrap hypothesis-testing hypothesis testing regression linear regression variance review

Legend

relation to other work	`claim_relations`	how this paper stands to another: agrees, contradicts, extends, qualifies
claim type	`claim_type`	the shape of the claim: existence, mechanism, method, critique, prediction, null
concept edge	`concept_edges`	an asserted relation between two concepts, with the sentence asserting it
concept	`concepts`	an idea the paper works with, quoted where the paper uses it
content tag	`content_tags`	a finding, definition, quantity, statistic or variable named in the paper
limitation	`limitations`	what the paper says its evidence cannot support
mechanism	`mechanisms`	how the reported effect is said to come about
method	`method`	how the paper produced its evidence
phenomenon	`phenomenon`	the effect or regularity under study
sense	`senses`	a distinct use of a term, quoted where the paper fixes it
transfer	`transfers`	a method or concept carried in from another field
unrecordable relation	`unhoused_relations`	a relation the paper asserts that this vocabulary cannot express, with the reason

The paper

The full text is not reproduced here. This article is not cleared for redistribution, so the page carries the tag layer only. Its evidence sentences are withheld; the grounding status remains with each tag, and the article itself is at https://arxiv.org/abs/1606.06565. The reading view exists for the papers whose licence permits it.

Every tag on this page

claim_relations 4 · claim_type 1 · concept_edges 1 · concepts 7 · content_tags 9 · limitations 8 · mechanisms 1 · method 1 · phenomenon 1 · senses 3 · transfers 2 · unhoused_relations 1

Relations on this page carry a quotation for the claim. Where a relation was separately evidenced — and so tested against the vocabulary for its type — the row says so; none of 1 relation here carry that mark. The rest are the tagger’s reading of the sentence quoted beside them.

This record holds no controls. A layer is empty when the tagger recorded nothing in it — which is not the same as the paper having nothing to record.

facet	tag	field	conf.	whose words
claim_type	existence Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_type`		the paper's words
phenomenon	accidents in machine learning systems Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`phenomenon`		the paper's words
method	review Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: review	`method`		the paper's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: AI-safety accidents arise mainly from mis-specified objective functions - negative side effects and reward hacking	`mechanisms[0]`		the tagger's words
limitations	The proposed avenues of attack for the wrong-objective problems are explicitly preliminary and not fully worked out. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the proposed approaches to the wrong-objective problems: they are offered WHERE LESS PRIOR WORK EXISTS, as preliminary avenues rather than solutions	`limitations[0]`		the tagger's words
limitations	The analysis is restricted to RL agents and supervised learning systems rather than all AI paradigms. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the scope of the analysis: THESE ARE NOT THE ONLY POSSIBLE PARADIGMS FOR AI OR ML SYSTEMS, though the authors expect similar issues elsewhere	`limitations[1]`		the tagger's words
limitations	The multi-agent / rich-modelling route to avoiding side effects is not yet practical. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the rich-modelling route to avoiding side effects: WE ARE STILL A LONG WAY AWAY FROM PRACTICAL SYSTEMS THAT CAN BUILD A RICH ENOUGH MODEL	`limitations[2]`		the tagger's words
limitations	The review of work relevant to distributional shift is deliberately partial rather than exhaustive. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the distributional-shift survey: RATHER THAN FULLY REVIEWING ALL OF THIS WORK IN DETAIL ... WE WILL DESCRIBE A FEW ILLUSTRATIVE APPROACHES	`limitations[3]`		the tagger's words
limitations	The paper's central term is stipulated so that it attributes a cause, which the ordinary word does not. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the word 'accident' as used in this paper: it is defined as unintended and harmful behaviour emerging when the wrong objective function is specified, which locates a cause rather than denying one	`limitations[4]`	1.0	the tagger's words
limitations	A second definition of the same term appears in the same paper with a different cause named. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the scope of 'accidents' in this paper: one statement defines them as emerging from poor design of real-world AI systems, and another as emerging when the wrong objective function is specified	`limitations[5]`	1.0	the tagger's words
limitations	The risks are illustrated throughout with an invented example rather than a deployed system. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the concreteness claimed by this paper: many of the accident risks are illustrated with reference to a fictional office-cleaning robot rather than to a deployed system	`limitations[6]`	1.0	the tagger's words
limitations	Extreme scenarios are excluded by choice, on the grounds that they cost precision. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the range of risks covered here: the authors state one need not invoke extreme scenarios and that doing so can lead to unnecessarily speculative discussions that lack precision	`limitations[7]`	1.0	the tagger's words
transfers	framing the philosophical problem of AI safety concretely in terms of machine-learning methods (supervised learning, reinforcement learning) Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`transfers[0]`		the tagger's words
transfers	importing the engineering notion of accidents and safety into the analysis of AI systems Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`transfers[1]`		the tagger's words
concepts	accident Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: accident	`concepts[4]`		the paper's words
concepts	accident risk Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: accident risk	`concepts[5]`		the paper's words
concepts	objective function Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: objective function	`concepts[6]`		the paper's words
concepts	side effects Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: side effects	`concepts[7]`		the paper's words
concepts	safe exploration Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: safe exploration	`concepts[8]`		the paper's words
concepts	accidents Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: accidents	`concepts[9]`		the paper's words
concepts	objective function Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: objective function	`concepts[10]`		the paper's words
unhoused_relations	side effects → negative externalities Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. THE INDEXING / PROXY RELATION - 'from' STANDS IN FOR 'to'. We have no type for it: 'causes' asserts production, and 'operationalised-as' says the authors CHOSE a measure, whereas a proxy claim says THIS STANDS IN FOR WHAT WE ACTUALLY CARE ABOUT AND WE KNOW IT IS NOT THE SAME THING. Measured at 8% of papers. THE CLEAREST PROXY CLAIM IN THE CORPUS, because it names the gap out loud: 'Avoiding side effects can be seen as a proxy for the thing we really care about: avoiding negative externalities.' The substitute, the target, and the acknowledgement that they differ, in one sentence - from an AI-safety paper, which is not where I expected to find the best statement of this project's own problem.	`unhoused_relations[0]`		the tagger's words
senses	'accident' - STIPULATED SO THAT IT ASSIGNS FAULT: in ordinary use an accident is what nobody is to blame for, and here it is defined as harm emerging from poor design or from specifying the wrong objective function, so the technical term points at exactly what the everyday term excuses Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[0]`	1.0	the tagger's words
senses	'AI safety' - THE PUBLIC NAME FOR THE PAPER'S OWN TERM: the authors write about mitigating accident risk and note in a parenthesis that this is what public discussion calls AI safety, so the two names mark a register rather than a difference of subject Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[1]`	1.0	the tagger's words
senses	the taxonomy - BY WHERE IN THE PROCESS THINGS WENT WRONG: safety problems are grouped not by the harm produced but by the stage at which the design failed, which is why the five categories are about objectives, oversight and exploration rather than about outcomes Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[2]`	1.0	the tagger's words
claim_relations	builds-on → machine learning (modern ML techniques) Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[0]`		the tagger's words
claim_relations	builds-on → an informal essay written by one of the authors of the present document (on semi-supervised RL) Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[1]`		the tagger's words
claim_relations	builds-on → informal safety postings [146, 34] by two of the authors of this paper Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[2]`		the tagger's words
claim_relations	extends → the Machine Intelligence Research Institute's philosophical work on safety in very advanced AI Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[3]`		the tagger's words
concept_edges	accident —part-of→ accident risk Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. strength of the relation: 1 Audit 2026-09-28: DOUBTFUL - read against its span and kept, but accident vs accident risk is a trivial containment the span does not state. also in: accident · accident risk	`concept_edges[1]`		the tagger's words
content_tags	five — number of concrete accident-risk problems presented Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.quantities[0]`		the tagger's words
content_tags	Five practical research problems related to accident risk: avoiding side effects, avoiding reward hacking, scalable supervision, safe exploration and distributional shift. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.findings[0]`		the tagger's words
content_tags	In the authors' opinion, one need not invoke extreme scenarios. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.findings[1]`		the tagger's words
content_tags	accidents Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: accidents	`content_tags.definitions[0]`		the tagger's words
content_tags	Scalable oversight Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: Scalable oversight	`content_tags.definitions[1]`		the tagger's words
content_tags	distributional shift Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: distributional shift	`content_tags.definitions[2]`		the tagger's words
content_tags	bootstrap Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: bootstrap	`content_tags.statistics[1]`		the tagger's words
content_tags	hypothesis-testing Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: hypothesis-testing	`content_tags.statistics[3]`		the tagger's words
content_tags	variance Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: variance	`content_tags.statistics[5]`		the tagger's words

Interpretive choices

Tags that could not be placed in the text:

content_tags.statistics[4] — quotation withheld — the page has already reproduced as much of this paper as the rights budget allows
NOT MACHINE-VOUCHED: not-in-our-copy - The quote is in the original web page we fetched for this paper, but not in the text we tag against, which was extracted differently. Checked with ligatures, hyphens, spacing and punctuation set aside.

What was deliberately not tagged, and why, is recorded per paper by the tagger. Where a paper has no such record, this page says so rather than implying that nothing was declined.