Inherent Trade-Offs in the Fair Determination of Risk Scores

Jon Kleinberg, Sendhil Mullainathan, Manish Raghavan · ITCS / arXiv · 2016 · law · record kleinberg2016-fairness
tag layer · index · concepts · authors · עברית

Where this record's text came from. faceted 2026-08 by reading the locally held text (pdftext); every evidence string verified as an exact substring of that text before merging. Two facets had to bend the suggested vocabularies. method is 'formal-proof': the paper is pure theory (two characterisation theorems plus an NP-completeness reduction) and none of the listed values — behavioural-experiment, fmri, electrophysiology, simulation, review, meta-analysis, system-or-tool-building, systematic-review, practice-report — applies; leaving it empty would have lost real information. Claim type is 'existence' for an impossibility theorem: the paper establishes that an inherent trade-off exists (equivalently, that no satisfying method exists); 'null-result' was rejected because that shape denotes an empirical effect that failed to appear, and 'critique' was rejected because the authors explicitly disclaim arguing for or against any definition. The Hardt et al. relation is labelled 'extends' for want of a 'parallel/concurrent-work' type — the paper describes it as concurrent work covering the binary case, not as something it builds on. A further stated assumption, 'Our model will assume that the value pσ for each σ is known to the procedure.', is captured through the footnote quoted in limitations. 2026-08-14 schema-0.3 pass: every quote verified as an exact substring of the staged text and anchored at merge time; concept terms verified present; every edge endpoint an existing node.

How to read this page. The article itself is not reproduced here — this paper is not cleared for redistribution, so what follows is the tag layer alone. The evidence sentences are withheld with the article. The table at the foot lists every tag the record carries; each row names the facet it belongs to, and the whose words column says whether the tag records the paper's own wording or the tagger's sentence about the paper. Every tag links to each other paper in the corpus carrying it. The machine-readable edition of the same thing is the TEI file beside this one, where the tags live in standOff, and the article is at its publisher via the link below. When we have changed our mind. Some entries in the table begin with a word in capitals and the reason for it. EVIDENCE CORRECTED — the finding still stands, but we had quoted the wrong sentence from the paper, and the quotation has been replaced with the right one. RETRACTED — the entry itself was wrong; it stays on the page, marked, with the reason, rather than being deleted. WITHDRAWN AND REPLACED — the entry was wrong, and a corrected version of it appears in the entries below. Each mark carries its own reason, in brackets, because the reasons differ from one entry to the next. An entry we have WITHDRAWN is removed from this page and kept in the record, with its reason. A line reading Audit and a date is a later re-reading of the entry against its whole sentence: DOUBTFUL means it was kept, and the reason says what is loose about it. Quotation marks inside the tagger's sentence. Where a tag is the tagger's own sentence, a phrase in quotation marks inside it is borrowed from the paper — the tagger's wording around it is not. The paper's own sentence is the evidence quote for that row; on this page it is withheld with the article, and the article is at its publisher. Nothing inside quotation marks anywhere on this page is ours: if it is quoted, it is the paper's. Whose words a concept carries. A concept entry records a term the paper itself uses, so the term, the sense given for it, and its evidence are the paper's own words, quoted where the paper fixes them. This is the opposite of a sense entry in the tag table, which is the tagger's sentence about how the paper uses a term. Where a concept entry could not be grounded in the paper's words, the entry says so on its own row.

Other fields define these differently the same word, defined another way in another discipline
algorithmic fairness 2 fields · classification 2 fields · fairness 2 fields

Tags on this paper

Every tag the record carries, each linking to every other paper in the corpus that carries it. A tag appears here whether or not its words occur in the article — the spans below are only the ones a sentence in this paper licenses. Of the 10 concept tags on this paper, 8 are grounded in a quoted sentence.

algorithmic fairness calibration risk score impossibility result balance for the positive class balance for the negative class statistical parity base rates Balance for the positive and negative classes an impossibility theorem: calibration and balanced error rates across groups cannot all hold simultaneously except when base rates are equal or prediction is perfect fairness groups calibrated risk conditions classification recidivism defendants mean average value mathematical-proof

Method, as described: formal-proof

Legend

relation to other work	`claim_relations`	how this paper stands to another: agrees, contradicts, extends, qualifies
claim type	`claim_type`	the shape of the claim: existence, mechanism, method, critique, prediction, null
concept edge	`concept_edges`	an asserted relation between two concepts, with the sentence asserting it
concept	`concepts`	an idea the paper works with, quoted where the paper uses it
content tag	`content_tags`	a finding, definition, quantity, statistic or variable named in the paper
limitation	`limitations`	what the paper says its evidence cannot support
mechanism	`mechanisms`	how the reported effect is said to come about
method	`method`	how the paper produced its evidence
phenomenon	`phenomenon`	the effect or regularity under study
sense	`senses`	a distinct use of a term, quoted where the paper fixes it
transfer	`transfers`	a method or concept carried in from another field

The paper

The full text is not reproduced here. This article is not cleared for redistribution, so the page carries the tag layer only. Its evidence sentences are withheld; the grounding status remains with each tag, and the article itself is at https://arxiv.org/abs/1609.05807. The reading view exists for the papers whose licence permits it.

Every tag on this page

claim_relations 4 · claim_type 1 · concept_edges 3 · concepts 8 · content_tags 6 · limitations 9 · mechanisms 1 · method 1 · phenomenon 1 · senses 4 · transfers 1

Relations on this page carry a quotation for the claim. Where a relation was separately evidenced — and so tested against the vocabulary for its type — the row says so; 1 of 3 relations here carries that mark. The rest are the tagger’s reading of the sentence quoted beside them.

This record holds no controls. A layer is empty when the tagger recorded nothing in it — which is not the same as the paper having nothing to record.

facet	tag	field	conf.	whose words
claim_type	existence Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_type`		the paper's words
phenomenon	fairness conditions for probabilistic risk scores Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`phenomenon`		the paper's words
method	mathematical-proof Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: mathematical-proof	`method`		the paper's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: an impossibility theorem: calibration and balanced error rates across groups cannot all hold simultaneously except when base rates are equal or prediction is perfect	`mechanisms[0]`		the tagger's words
limitations	the incompatibility holds except in highly constrained special cases Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the incompatibility result itself: it holds EXCEPT IN HIGHLY CONSTRAINED SPECIAL CASES	`limitations[0]`		the tagger's words
limitations	The computational tractability of finding a minimum-loss fair assignment (or even deciding whether a non-trivial one exists) is left open. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: any algorithmic use of the fairness conditions: IT IS AN OPEN QUESTION WHETHER THERE IS A POLYNOMIAL-TIME ALGORITHM TO FIND A FAIR ASSIGNMENT OF MINIMUM LOSS, OR EVEN TO DETERMINE WHETHER A NON-TRIVIAL FAIR SOLUTION EXISTS	`limitations[1]`		the tagger's words
limitations	The NP-completeness result covers only the integral case, not the general model where a feature vector may be split across bins. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the NP-completeness result only: IT APPLIES ONLY TO THE CASE IN WHICH EVERYONE WITH A GIVEN FEATURE VECTOR MUST BE ASSIGNED TO THE SAME BIN	`limitations[2]`		the tagger's words
limitations	The paper deliberately does not analyse which fairness definitions are violated in real applications or datasets. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: any applied reading of the impossibility result: NOR IS OUR INTENTION TO ANALYZE WHICH DEFINITIONS OF FAIRNESS ARE VIOLATED IN PARTICULAR APPLICATIONS OR DATASETS	`limitations[3]`		the tagger's words
limitations	The exposition is restricted to two groups. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the two-group presentation: WE FOCUS ON THE CASE OF TWO GROUPS FOR SIMPLICITY OF EXPOSITION, with extension described as straightforward	`limitations[4]`		the tagger's words
limitations	The model assumes the true positive-class rate for each feature vector is known to the procedure; the unknown case is not analysed. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the known-p-sigma assumption, and the direction it cuts: THE CASE IN WHICH THE VALUE OF p-sigma IS UNKNOWN IS AN IMPORTANT VERSION OF THE PROBLEM AS WELL; HOWEVER, SINCE OUR MAIN RESULTS ESTABLISH STRONG LIMITATIONS ON WHAT IS ACHIEVABLE, THESE LIMI[tations carry over]	`limitations[5]`		the tagger's words
limitations	Withholding the group attribute from the procedure does not stop it exhibiting bias between groups, and the paper says so in a parenthesis as it defines the model. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: a claim that a procedure is unbiased because it never sees group membership: this model's rule has access only to the feature vector, and the authors state that this does not mean the rule is incapable of exhibiting bias between the two groups	`limitations[6]`	1.0	the tagger's words
limitations	The modelling assumption that group membership adds nothing beyond the feature vector is defended by a construction in which the feature vector conveys group perfectly. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the generality claimed for this model: the assumption that group adds no information beyond the feature vector is justified by allowing instances where people of different groups never share a feature vector, so that the feature vector implicitly conveys the group	`limitations[7]`	1.0	the tagger's words
limitations	A fourth and legally mandated notion of fairness is set aside as often neither feasible nor desirable, so the impossibility result does not speak to it. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: this theorem as covering fairness generally: statistical parity is excluded from the three conditions, on the stated ground that classification and risk assessment are broader activities where it is often neither feasible nor desirable	`limitations[8]`	1.0	the tagger's words
transfers	probabilistic fairness criteria (calibration, error-rate balance) applied to risk scores used in consequential legal / policy decisions Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`transfers[0]`		the tagger's words
concepts	calibration Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: calibration	`concepts[1]`		the paper's words
concepts	risk score Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: risk score	`concepts[2]`		the paper's words
concepts	balance for the positive class Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: balance for the positive class	`concepts[4]`		the paper's words
concepts	balance for the negative class Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: balance for the negative class	`concepts[5]`		the paper's words
concepts	statistical parity Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: statistical parity	`concepts[6]`		the paper's words
concepts	base rates Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: base rates	`concepts[7]`		the paper's words
concepts	Balance for the positive and negative classes Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: Balance for the positive and negative classes	`concepts[8]`		the paper's words
concepts	statistical parity Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: statistical parity	`concepts[9]`		the paper's words
senses	fairness - AS CALIBRATION WITHIN GROUPS (condition A): within each group, the people in a bin scored v really do belong to the positive class at rate v, i.e. the scores mean what they claim to mean when read separately in each group Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[0]`	1.0	the tagger's words
senses	fairness - AS BALANCE FOR THE NEGATIVE CLASS (condition B): people who do NOT have the property get the same average score whichever group they are in, so the errors are not systematically worse for one group's negative instances Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[1]`	1.0	the tagger's words
senses	fairness - AS BALANCE FOR THE POSITIVE CLASS (condition C): people who DO have the property get the same average score whichever group they are in - the condition the COMPAS critique turned on Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[2]`	1.0	the tagger's words
senses	the impossibility - AS A FACT ABOUT BASE RATES, NOT ABOUT A DOMAIN: the authors state that the trade-off is not a fact about medicine but about risk estimates whenever base rates differ, which is the paper stating its own transfer condition Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[3]`	1.0	the tagger's words
claim_relations	contradicts → the assumption that fairness criteria can be jointly satisfied Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[0]`		the tagger's words
claim_relations	extends → Dietrich et al.'s empirical observation that error-rate symmetry across groups is hard to achieve when base rates differ Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[1]`		the tagger's words
claim_relations	builds-on → the Angwin et al. critique of the COMPAS risk tool and its counter-arguments Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[2]`		the tagger's words
claim_relations	extends → the concurrent binary-classifier analysis of Hardt et al. [19] Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[3]`		the tagger's words
concept_edges	balance for the positive class —distinct-from→ statistical parity Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. strength of the relation: 1 also in: balance for the positive class · statistical parity	`concept_edges[0]`		the tagger's words
concept_edges	balance for the negative class —distinct-from→ statistical parity Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. strength of the relation: 1 also in: balance for the negative class · statistical parity	`concept_edges[1]`		the tagger's words
concept_edges	Balance for the positive and negative classes —distinct-from→ statistical parity Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. the relation itself is separately evidenced · strength of the relation: 1 · note: RE-GROUNDS another row of this record: span stated the relation, endpoint names did not match its words. · the relation stated: the balance conditions are a goal that can be pursued INDEPENDENTLY of statistical parity also in: Balance for the positive and negative classes · statistical parity	`concept_edges[2]`		the tagger's words
content_tags	three — the number of fairness conditions proven jointly unsatisfiable Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.quantities[0]`		the tagger's words
content_tags	Except in highly constrained special cases, no method can satisfy the three fairness conditions simultaneously. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.findings[0]`		the tagger's words
content_tags	Even satisfying all three conditions approximately requires the data to lie in an approximate version of one of the constrained special cases. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.findings[1]`		the tagger's words
content_tags	base rates Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: base rates	`content_tags.variables[0]`		the tagger's words
content_tags	balance for the positive class Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: balance for the positive class	`content_tags.definitions[1]`		the tagger's words
content_tags	mean Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. about this quote: QUOTE RECOVERED FROM THE PAPER'S OWN CHARACTERS. The recorded quote was LETTERS-ONLY IDENTICAL to this stretch of the staged text but not an exact substring of it - spaces, hyphens, ligature glyphs or mathematical operators differed, so it could never anchor. This is the file's own text between the same two letters: not produced by a rule about the damage, copied out of the file. Round-trip checked (the recovered span is letters-only equal to the original) and length-checked. The text as originally recorded is kept in evidence as extracted. also in: mean	`content_tags.statistics[1]`		the tagger's words

Interpretive choices

Tags that could not be placed in the text:

none — every tag was located in the text

What was deliberately not tagged, and why, is recorded per paper by the tagger. Where a paper has no such record, this page says so rather than implying that nothing was declined.