Inherent Trade-Offs in the Fair Determination of Risk Scores

Inherent Trade-Offs in the Fair Determination of Risk Scores Jon Kleinberg Sendhil Mullainathan Manish Raghavan

Tag layer of the DHSS Meta-Tagging Project. The article text is reproduced from the staged copy named in sourceDesc and is not altered; the tags are held separately in standOff so that overlapping and nested spans are preserved.

Inherent Trade-Offs in the Fair Determination of Risk Scores 2016 ITCS / arXiv 1609.05807 kleinberg2016-fairness kleinberg2016-fairness.txt, the project's staged copy 4de38059f62cdb6e1407e16ae061c9e630f2853d arxiv-open law

the paper's own words

the tag layer; a sentence written by the tagger about the paper, always anchored to a quotation but not itself a quotation

0 spans located by searching the stored text for the quote's exact words, allowing any whitespace between them existence Quotation withheld because this record is not cleared for redistribution. fairness conditions for probabilistic risk scores Quotation withheld because this record is not cleared for redistribution. mathematical-proof mathematical-proof Quotation withheld because this record is not cleared for redistribution. mechanisms an impossibility theorem: calibration and balanced error rates across groups cannot all hold simultaneously except when base rates are equal or prediction is perfect Quotation withheld because this record is not cleared for redistribution. the incompatibility holds except in highly constrained special cases what this limits: the incompatibility result itself: it holds EXCEPT IN HIGHLY CONSTRAINED SPECIAL CASES Quotation withheld because this record is not cleared for redistribution. The computational tractability of finding a minimum-loss fair assignment (or even deciding whether a non-trivial one exists) is left open. what this limits: any algorithmic use of the fairness conditions: IT IS AN OPEN QUESTION WHETHER THERE IS A POLYNOMIAL-TIME ALGORITHM TO FIND A FAIR ASSIGNMENT OF MINIMUM LOSS, OR EVEN TO DETERMINE WHETHER A NON-TRIVIAL FAIR SOLUTION EXISTS Quotation withheld because this record is not cleared for redistribution. The NP-completeness result covers only the integral case, not the general model where a feature vector may be split across bins. what this limits: the NP-completeness result only: IT APPLIES ONLY TO THE CASE IN WHICH EVERYONE WITH A GIVEN FEATURE VECTOR MUST BE ASSIGNED TO THE SAME BIN Quotation withheld because this record is not cleared for redistribution. The paper deliberately does not analyse which fairness definitions are violated in real applications or datasets. what this limits: any applied reading of the impossibility result: NOR IS OUR INTENTION TO ANALYZE WHICH DEFINITIONS OF FAIRNESS ARE VIOLATED IN PARTICULAR APPLICATIONS OR DATASETS Quotation withheld because this record is not cleared for redistribution. The exposition is restricted to two groups. what this limits: the two-group presentation: WE FOCUS ON THE CASE OF TWO GROUPS FOR SIMPLICITY OF EXPOSITION, with extension described as straightforward Quotation withheld because this record is not cleared for redistribution. The model assumes the true positive-class rate for each feature vector is known to the procedure; the unknown case is not analysed. what this limits: the known-p-sigma assumption, and the direction it cuts: THE CASE IN WHICH THE VALUE OF p-sigma IS UNKNOWN IS AN IMPORTANT VERSION OF THE PROBLEM AS WELL; HOWEVER, SINCE OUR MAIN RESULTS ESTABLISH STRONG LIMITATIONS ON WHAT IS ACHIEVABLE, THESE LIMI[tations carry over] Quotation withheld because this record is not cleared for redistribution. Withholding the group attribute from the procedure does not stop it exhibiting bias between groups, and the paper says so in a parenthesis as it defines the model. what this limits: a claim that a procedure is unbiased because it never sees group membership: this model's rule has access only to the feature vector, and the authors state that this does not mean the rule is incapable of exhibiting bias between the two groups Quotation withheld because this record is not cleared for redistribution. The modelling assumption that group membership adds nothing beyond the feature vector is defended by a construction in which the feature vector conveys group perfectly. what this limits: the generality claimed for this model: the assumption that group adds no information beyond the feature vector is justified by allowing instances where people of different groups never share a feature vector, so that the feature vector implicitly conveys the group Quotation withheld because this record is not cleared for redistribution. A fourth and legally mandated notion of fairness is set aside as often neither feasible nor desirable, so the impossibility result does not speak to it. what this limits: this theorem as covering fairness generally: statistical parity is excluded from the three conditions, on the stated ground that classification and risk assessment are broader activities where it is often neither feasible nor desirable Quotation withheld because this record is not cleared for redistribution. probabilistic fairness criteria (calibration, error-rate balance) applied to risk scores used in consequential legal / policy decisions Quotation withheld because this record is not cleared for redistribution. calibration calibration Quotation withheld because this record is not cleared for redistribution. risk score risk score Quotation withheld because this record is not cleared for redistribution. balance for the positive class balance for the positive class Quotation withheld because this record is not cleared for redistribution. balance for the negative class balance for the negative class Quotation withheld because this record is not cleared for redistribution. statistical parity statistical parity Quotation withheld because this record is not cleared for redistribution. base rates base rates Quotation withheld because this record is not cleared for redistribution. Balance for the positive and negative classes Balance for the positive and negative classes Quotation withheld because this record is not cleared for redistribution. statistical parity statistical parity Quotation withheld because this record is not cleared for redistribution. fairness - AS CALIBRATION WITHIN GROUPS (condition A): within each group, the people in a bin scored v really do belong to the positive class at rate v, i.e. the scores mean what they claim to mean when read separately in each group Quotation withheld because this record is not cleared for redistribution. fairness - AS BALANCE FOR THE NEGATIVE CLASS (condition B): people who do NOT have the property get the same average score whichever group they are in, so the errors are not systematically worse for one group's negative instances Quotation withheld because this record is not cleared for redistribution. fairness - AS BALANCE FOR THE POSITIVE CLASS (condition C): people who DO have the property get the same average score whichever group they are in - the condition the COMPAS critique turned on Quotation withheld because this record is not cleared for redistribution. the impossibility - AS A FACT ABOUT BASE RATES, NOT ABOUT A DOMAIN: the authors state that the trade-off is not a fact about medicine but about risk estimates whenever base rates differ, which is the paper stating its own transfer condition Quotation withheld because this record is not cleared for redistribution. contradicts → the assumption that fairness criteria can be jointly satisfied Quotation withheld because this record is not cleared for redistribution. extends → Dietrich et al.'s empirical observation that error-rate symmetry across groups is hard to achieve when base rates differ Quotation withheld because this record is not cleared for redistribution. builds-on → the Angwin et al. critique of the COMPAS risk tool and its counter-arguments Quotation withheld because this record is not cleared for redistribution. extends → the concurrent binary-classifier analysis of Hardt et al. [19] Quotation withheld because this record is not cleared for redistribution. balance for the positive class —distinct-from→ statistical parity balance for the positive class statistical parity strength of the relation: 1 Quotation withheld because this record is not cleared for redistribution. balance for the negative class —distinct-from→ statistical parity balance for the negative class statistical parity strength of the relation: 1 Quotation withheld because this record is not cleared for redistribution. Balance for the positive and negative classes —distinct-from→ statistical parity Balance for the positive and negative classes statistical parity the relation itself is separately evidenced · strength of the relation: 1 · note: RE-GROUNDS another row of this record: span stated the relation, endpoint names did not match its words. · the relation stated: the balance conditions are a goal that can be pursued INDEPENDENTLY of statistical parity Quotation withheld because this record is not cleared for redistribution. three — the number of fairness conditions proven jointly unsatisfiable Quotation withheld because this record is not cleared for redistribution. Except in highly constrained special cases, no method can satisfy the three fairness conditions simultaneously. Quotation withheld because this record is not cleared for redistribution. Even satisfying all three conditions approximately requires the data to lie in an approximate version of one of the constrained special cases. Quotation withheld because this record is not cleared for redistribution. base rates base rates Quotation withheld because this record is not cleared for redistribution. balance for the positive class balance for the positive class Quotation withheld because this record is not cleared for redistribution. mean mean about this quote: QUOTE RECOVERED FROM THE PAPER'S OWN CHARACTERS. The recorded quote was LETTERS-ONLY IDENTICAL to this stretch of the staged text but not an exact substring of it - spaces, hyphens, ligature glyphs or mathematical operators differed, so it could never anchor. This is the file's own text between the same two letters: not produced by a rule about the damage, copied out of the file. Round-trip checked (the recovered span is letters-only equal to the original) and length-checked. The text as originally recorded is kept in evidence as extracted. Quotation withheld because this record is not cleared for redistribution.

The full text of this article is not redistributable, so it is not included here. The evidence sentences are withheld with it; the tag layer and its grounding status remain. The article itself is at https://arxiv.org/abs/1609.05807.