Why Should I Trust You? Explaining the Predictions of Any Classifier

Why Should I Trust You? Explaining the Predictions of Any Classifier Marco Tulio Ribeiro Sameer Singh Carlos Guestrin

Tag layer of the DHSS Meta-Tagging Project. The article text is reproduced from the staged copy named in sourceDesc and is not altered; the tags are held separately in standOff so that overlapping and nested spans are preserved.

Why Should I Trust You? Explaining the Predictions of Any Classifier 2016 KDD / arXiv 1602.04938 ribeiro2016-lime ribeiro2016-lime.txt, the project's staged copy a7d1281b5d00af5e68d9a3c7ca17907abce14a20 arxiv-open statistics

the paper's own words

the tag layer; a sentence written by the tagger about the paper, always anchored to a quotation but not itself a quotation

0 spans located by searching the stored text for the quote's exact words, allowing any whitespace between them method Quotation withheld because this record is not cleared for redistribution. human trust in individual predictions of black-box classifiers Quotation withheld because this record is not cleared for redistribution. system-or-tool-building system-or-tool-building Quotation withheld because this record is not cleared for redistribution. mechanisms THE EXPLANATION'S VOCABULARY IS NOT THE MODEL'S: an interpretable representation is defined separately from the features the classifier actually uses - presence or absence of a word, or of a contiguous patch of similar pixels - while the model may work over embeddings or a three-channel tensor Quotation withheld because this record is not cleared for redistribution. mechanisms INTERPRETABILITY IS DEFINED AS AUDIENCE-RELATIVE, not as a property of the model class: the authors state that a linear model, a gradient vector or an additive model MAY OR MAY NOT be interpretable, because the criterion has to take the user's limitations into account Quotation withheld because this record is not cleared for redistribution. mechanisms and the audience-relativity is made concrete rather than left as a caveat: practitioners may be able to read a small Bayesian network where laymen are better served by a few weighted features Quotation withheld because this record is not cleared for redistribution. mechanisms THE IMPOSSIBILITY IS STATED BEFORE THE CRITERION IS WEAKENED: an explanation cannot be completely faithful unless it IS the complete description of the model, so the requirement imposed instead is that it be faithful in the vicinity of the instance being explained Quotation withheld because this record is not cleared for redistribution. mechanisms AND THE IMPLICATION IS STATED TO RUN ONE WAY ONLY: local fidelity does not imply global fidelity - a feature that matters globally may not matter locally, and the reverse - while global fidelity would imply local, and globally faithful interpretable explanations are said to remain a challenge Quotation withheld because this record is not cleared for redistribution. mechanisms the explainer is required to treat the model as a BLACK BOX, and the reason given is not only that current classifiers are uninterpretable but that model-agnosticism buys the ability to explain classifiers that do not exist yet Quotation withheld because this record is not cleared for redistribution. mechanisms THE TRADE-OFF IS PUT IN THE OBJECTIVE RATHER THAN LEFT TO JUDGEMENT: unfaithfulness in the neighbourhood is minimised while a separate measure of the explanation's own complexity is held low enough for a human to read, and the authors note the same formulation admits different explanation families, fidelity functions and complexity measures Quotation withheld because this record is not cleared for redistribution. mechanisms the neighbourhood is explored BY SAMPLING rather than by assumption, precisely to keep the method model-agnostic: no assumption is made about the model, and the local loss is approximated by drawing samples weighted by proximity to the instance Quotation withheld because this record is not cleared for redistribution. mechanisms samples are drawn both near the instance and FAR from it, weighted by proximity - which is what lets a globally complex model still be approximated locally, and what the authors say makes the method fairly robust to sampling noise Quotation withheld because this record is not cleared for redistribution. mechanisms A CLASSIFIER AT 94% HELD-OUT ACCURACY IS SHOWN TO BE UNTRUSTWORTHY BY ITS EXPLANATIONS: the words it relies on are said to have no connection to either class, and one of them appears in 22% of training examples with 99% of those in a single class - and removing the headers only substitutes the names of prolific posters, which would not generalise either Quotation withheld because this record is not cleared for redistribution. explanations are only locally faithful (learned locally around each prediction) what this limits: the reach of any explanation the method produces: it works Quotation withheld, so faithfulness is local and not global Quotation withheld because this record is not cleared for redistribution. Some model behaviours cannot be captured at all by the chosen interpretable representation. what this limits: the choice of interpretable representation: CERTAIN INTERPRETABLE REPRESENTATIONS WILL NOT BE POWERFUL ENOUGH TO EXPLAIN CERTAIN BEHAVIORS Quotation withheld because this record is not cleared for redistribution. Sparse linear explanations may be unfaithful where the underlying model is highly non-linear even locally. what this limits: any guarantee that an explanation exists at all: IF THE UNDERLYING MODEL IS HIGHLY NON-LINEAR EVEN IN THE LOCALITY OF THE PREDICTION, THERE MAY NOT BE A FAITHFUL EXPLANATION Quotation withheld because this record is not cleared for redistribution. The submodular pick step is not worked out for images. what this limits: image applications of the global pick step: HOW TO PERFORM THE PICK STEP FOR IMAGES ... WE WOULD LIKE TO ADDRESS THIS LIMITATION IN THE FUTURE Quotation withheld because this record is not cleared for redistribution. The husky-versus-wolf human-subject experiment had a small sample. what this limits: the husky-versus-wolf human-subject result: ALTHOUGH OUR SAMPLE SIZE IS SMALL Quotation withheld because this record is not cleared for redistribution. Interpretability is a property of the reader, not of the explanation: the same model may or may not be interpretable depending on who is looking at it. what this limits: any claim that a method produces interpretable explanations: interpretability is stated to depend on the user's limitations, so a linear model, a gradient vector or an additive model may or may not be interpretable depending on the audience Quotation withheld because this record is not cleared for redistribution. The vocabulary of the explanation is deliberately not the vocabulary of the model. what this limits: an explanation read as a description of what the model computes: the input variables in the explanation may need to differ from the model's features, so the two are stated in different terms by design Quotation withheld because this record is not cleared for redistribution. Complete faithfulness is stated to be impossible short of reproducing the model itself. what this limits: an explanation offered as an account of the model: the paper states complete faithfulness is often impossible unless the explanation is the complete description of the model, so what is offered instead is local fidelity Quotation withheld because this record is not cleared for redistribution. the explanation-based trust assessment checked against human experts eliminates: an explanation that satisfies a metric without helping a person - the human check is the only thing that can test a claim about trust · outcome (stated by the tagger, no quotation): held Quotation withheld because this record is not cleared for redistribution. interpretability interpretability Quotation withheld because this record is not cleared for redistribution. local fidelity local fidelity Quotation withheld because this record is not cleared for redistribution. global fidelity global fidelity Quotation withheld because this record is not cleared for redistribution. LIME LIME Quotation withheld because this record is not cleared for redistribution. model-agnostic model-agnostic Quotation withheld because this record is not cleared for redistribution. trust - SENSE 1, IN A PREDICTION: whether a user is willing to act on one individual output, which is the question an explanation of a single prediction addresses Quotation withheld because this record is not cleared for redistribution. interpretability - AUDIENCE-RELATIVE: it depends on the target audience, so practitioners may read a small Bayesian network while laymen need a few weighted features, and the same artefact is interpretable for one and not the other Quotation withheld because this record is not cleared for redistribution. fidelity - LOCAL AND GLOBAL, AND THE IMPLICATION RUNS ONE WAY ONLY: local fidelity does not give global fidelity, and features that matter globally may not matter locally Quotation withheld because this record is not cleared for redistribution. claim_type: method a second reading reaching the same value. Recorded: method — note: A BLIND second reading by the project on 2026-08-25: the record's own claim type/method/phenomenon were not looked at until every value here was written and substring-verified. Not applied - a person decides. Sampling frame: the 504 staged records carrying no alternatives block, so this never re-measures the 202. All 11 records with 130+ tags were taken as a census and 19 more spread one per discipline across 22 disciplines - the dense non-neuroscience cell is the one the 202 do not contain (inside them, 94-100% of the densest bands are neuroscience; among these 11, one is)., Quotation withheld because this record is not cleared for redistribution. method: system-or-tool-building a second reading reaching the same value. Recorded: system-or-tool-building — note: A BLIND second reading by the project on 2026-08-25: the record's own claim type/method/phenomenon were not looked at until every value here was written and substring-verified. Not applied - a person decides. Sampling frame: the 504 staged records carrying no alternatives block, so this never re-measures the 202. All 11 records with 130+ tags were taken as a census and 19 more spread one per discipline across 22 disciplines - the dense non-neuroscience cell is the one the 202 do not contain (inside them, 94-100% of the densest bands are neuroscience; among these 11, one is)., Quotation withheld because this record is not cleared for redistribution. phenomenon: trust in the predictions of black-box classifiers, and whether a prediction can be understood well enough to act on a second reading DIFFERING from the recorded value. Recorded: human trust in individual predictions of black-box classifiers — note: A BLIND second reading by the project on 2026-08-25: the record's own claim type/method/phenomenon were not looked at until every value here was written and substring-verified. Not applied - a person decides. Sampling frame: the 504 staged records carrying no alternatives block, so this never re-measures the 202. All 11 records with 130+ tags were taken as a census and 19 more spread one per discipline across 22 disciplines - the dense non-neuroscience cell is the one the 202 do not contain (inside them, 94-100% of the densest bands are neuroscience; among these 11, one is)., Quotation withheld because this record is not cleared for redistribution. builds-on → model-agnostic explanation by learning an interpretable model on the original model's predictions [2, 7, 22] Quotation withheld because this record is not cleared for redistribution. extends → prior perturbation-based explanation work [24] Quotation withheld because this record is not cleared for redistribution. builds-on → the 1-1/e greedy approximation guarantee for submodular coverage maximisation [15] Quotation withheld because this record is not cleared for redistribution. local fidelity —distinct-from→ global fidelity local fidelity global fidelity strength of the relation: 1 Quotation withheld because this record is not cleared for redistribution. LIME explains the predictions of any classifier in an interpretable and faithful way by learning an interpretable model locally around each prediction. about this quote: QUOTE REPAIRED: the extractor split a hyphen-space and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted. Quotation withheld because this record is not cleared for redistribution. SP-LIME selects a set of representative instances with explanations to address the problem of trusting the model. Quotation withheld because this record is not cleared for redistribution. LIME LIME about this quote: QUOTE REPAIRED: the extractor split a hyphen-space and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted. Quotation withheld because this record is not cleared for redistribution. model-agnostic explanation model-agnostic explanation Quotation withheld because this record is not cleared for redistribution. Bayesian Bayesian Quotation withheld because this record is not cleared for redistribution. mean mean Quotation withheld because this record is not cleared for redistribution. regression regression Quotation withheld because this record is not cleared for redistribution. sample-size sample-size about this quote: QUOTE REPAIRED: the extractor split a hyphen-space and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted. Quotation withheld because this record is not cleared for redistribution. standard-error standard-error Quotation withheld because this record is not cleared for redistribution. statistical-significance statistical-significance Quotation withheld because this record is not cleared for redistribution.

The full text of this article is not redistributable, so it is not included here. The evidence sentences are withheld with it; the tag layer and its grounding status remain. The article itself is at https://arxiv.org/abs/1602.04938.