How the machine ‘thinks’: Understanding opacity in machine learning algorithms

How the machine ‘thinks’: Understanding opacity in machine learning algorithms Jenna Burrell

Tag layer of the DHSS Meta-Tagging Project. The article text is reproduced from the staged copy named in sourceDesc and is not altered; the tags are held separately in standOff so that overlapping and nested spans are preserved.

How the machine ‘thinks’: Understanding opacity in machine learning algorithms 2016 Big Data & Society 10.1177/2053951715622512 burrell2016-how-the-machine-thinks-understandi burrell2016-how-the-machine-thinks-understandi.txt, the project's staged copy 96cbbc866c7a0409cad50380d86b7bfbc8df2367 cc-by-nc ai-ethics

the paper's own words

the tag layer; a sentence written by the tagger about the paper, always anchored to a quotation but not itself a quotation

0 spans located by searching the stored text for the quote's exact words, allowing any whitespace between them opacity in socially consequential algorithmic classification - spam filters, credit scoring, fraud detection, search ranking - and specifically what makes machine learning classifiers hard to understand Quotation withheld because this record is not cleared for redistribution. what the reasons are for not knowing how a classification was arrived at - proprietary secrecy, technical difficulty, or something else Quotation withheld because this record is not cleared for redistribution. a three-way typology of opacity - corporate secrecy, technical illiteracy, and the mismatch between high-dimensional optimisation and human interpretation - with the argument that each form needs a different remedy Quotation withheld because this record is not cleared for redistribution. method Quotation withheld because this record is not cleared for redistribution. the recipient of an algorithmic classification rarely has any concrete sense of how or why it was arrived at Quotation withheld because this record is not cleared for redistribution. conceptual-analysis conceptual-analysis Quotation withheld because this record is not cleared for redistribution. mechanisms THE MACHINE BUILDS ITS OWN REPRESENTATION WITHOUT REGARD FOR HUMAN COMPREHENSION - optimisation over training data does not naturally accord with human semantic explanation, so the opacity is a property of the method and not of anyone's choice or skill Quotation withheld because this record is not cleared for redistribution. mechanisms USEFULNESS REQUIRES COMPLEXITY: a machine learning algorithm CAN be implemented so its logic is fully comprehensible, but such an instance is unlikely to be useful - accuracy carries a degree of unavoidable complexity Quotation withheld because this record is not cleared for redistribution. mechanisms THE OPACITY IS IN THE INTERPLAY, NOT IN EITHER PART: the dataset may be large but comprehensible and the code may be written with clarity, and the complexity arises between them, in the mechanism of the algorithm operating on data Quotation withheld because this record is not cleared for redistribution. mechanisms THE DECISION LOGIC CHANGES AS THE ALGORITHM LEARNS, so reading the code as written does not tell you what it does after training Quotation withheld because this record is not cleared for redistribution. mechanisms DIMENSIONALITY MANIPULATION BREAKS THE FEATURE-TO-DIMENSION CORRESPONDENCE: with PCA or the kernel trick the relation is not one-to-one, so a 'weighted input' can no longer be traced to a human-readable property Quotation withheld because this record is not cleared for redistribution. mechanisms INTUITION FAILS AT HIGH DIMENSIONS - each of thousands of features subtly and imperceptibly shifts the classification, so debugging and improvement get harder as inputs are added status: Domingos (2012) notes that ‘intuition fails at high-dimensions.’ Quotation withheld because this record is not cleared for redistribution. mechanisms SOME ALGORITHMS ARE OPAQUE IN ORDER TO WORK: spam, scam and fraud detection face an adversary who adapts, so disclosure would defeat the system - opacity here is functional rather than merely protective Quotation withheld because this record is not cleared for redistribution. mechanisms SECRECY MAY BE COVER RATHER THAN COMPETITIVE NECESSITY: Pasquale's reading is that the adversary is regulation itself, which makes the same observable opacity a different problem with a different remedy status: he argues that a kind of adversarial situation is indeed in play Quotation withheld because this record is not cleared for redistribution. the audit's explanation was produced by imposing human interpretive reasoning on a process of statistical optimisation - 'machine thinking was resolved to human interpretation' - which the author flags as a limit of his own method Quotation withheld because this record is not cleared for redistribution. ambiguities remained after the audit and the author doubts the explanation was necessarily a correct one Quotation withheld because this record is not cleared for redistribution. end-user explanations that list a few key criteria are at best incomplete and at worst false reassurance - which applies to the remedy the paper is closest to endorsing Quotation withheld because this record is not cleared for redistribution. the paper's own examples are handwriting recognition and spam filtering, where the feature-to-dimension relation IS one-to-one, and it states that this is often not the case Quotation withheld because this record is not cleared for redistribution. no single tool or process will alleviate black-boxed classification, and the right combination is said to depend on the application space rather than being specified Quotation withheld because this record is not cleared for redistribution. the difficulty is stated to persist even for those with specialised training and even for computer scientists, which bounds what any literacy remedy can achieve Quotation withheld because this record is not cleared for redistribution. the code audit found the word 'Nigeria' neutrally weighted in the spam classifier - a check that produced reassuring evidence rather than the expected discriminatory finding, and the paper reports it that way eliminates: discriminatory weighting of the word 'Nigeria' as the explanation of the spam classifier's behaviour - the audit found it neutrally weighted · outcome: partial Quotation withheld because this record is not cleared for redistribution. the author states plainly that the explanation his own audit produced may not be correct: innocuous words like 'visit' and 'want' were weighted as spam indicators and remained unexplained eliminates: the audit's own explanation being taken as a description of what the classifier does, when words the author calls innocuous carry spam weight and remain unaccounted for · outcome: failed Quotation withheld because this record is not cleared for redistribution. opacity opacity Quotation withheld because this record is not cleared for redistribution. machine learning machine learning Quotation withheld because this record is not cleared for redistribution. classification classification Quotation withheld because this record is not cleared for redistribution. algorithm algorithm Quotation withheld because this record is not cleared for redistribution. interpretability interpretability Quotation withheld because this record is not cleared for redistribution. remediable incomprehensibility remediable incomprehensibility Quotation withheld because this record is not cleared for redistribution. curse of dimensionality curse of dimensionality Quotation withheld because this record is not cleared for redistribution. training data training data Quotation withheld because this record is not cleared for redistribution. features features Quotation withheld because this record is not cleared for redistribution. code audit code audit Quotation withheld because this record is not cleared for redistribution. auditors auditors Quotation withheld because this record is not cleared for redistribution. discrimination discrimination Quotation withheld because this record is not cleared for redistribution. principal component analysis principal component analysis Quotation withheld because this record is not cleared for redistribution. kernel trick kernel trick Quotation withheld because this record is not cleared for redistribution. trade secrets trade secrets Quotation withheld because this record is not cleared for redistribution. open source open source Quotation withheld because this record is not cleared for redistribution. semantic interpretation semantic interpretation Quotation withheld because this record is not cleared for redistribution. spam spam Quotation withheld because this record is not cleared for redistribution. opacity - THREE DIFFERENT THINGS UNDER ONE WORD, and the paper's whole point is that they are routinely conflated: (1) intentional corporate or state secrecy, (2) the fact that reading code is a specialist skill, (3) the mismatch between high-dimensional mathematical optimisation and human-scale semantic reasoning. Only the third is intrinsic to machine learning Quotation withheld because this record is not cleared for redistribution. opacity - AND THE SECOND AND THIRD SENSES ARE THE PAIR THAT GET CONFUSED, both reading as 'this is technical and complex'. The paper separates them because they have opposite remedies: education fixes illiteracy and cannot touch dimensionality Quotation withheld because this record is not cleared for redistribution. opacity (Pasquale's sense) - 'REMEDIABLE INCOMPREHENSIBILITY', which builds the remedy into the definition. Burrell's third form is precisely the case that definition cannot cover, since no disclosure makes it comprehensible Quotation withheld because this record is not cleared for redistribution. code audit - IN THIS PAPER, TWO INCOMPATIBLE THINGS: reading the source, which is what the legal literature proposes, and observing the algorithm in action on data, which is what the third form of opacity actually requires. The paper's critique of audits turns entirely on which one is meant Quotation withheld because this record is not cleared for redistribution. critiques → proposals for code or algorithm audits as a way to evaluate for discriminatory classification, which the paper argues underestimate what reading the code of a complicated system entails and are non-specific about classes of algorithm Quotation withheld because this record is not cleared for redistribution. qualifies → Pasquale's definition of opacity as 'remediable incomprehensibility' and the regulatory-disclosure remedy that follows from it Quotation withheld because this record is not cleared for redistribution. qualifies → the emerging literature on the politics of algorithms, which treats algorithmic opacity as largely intentional corporate self-protection Quotation withheld because this record is not cleared for redistribution. extends → scholarship on digital inequality, which has focused on the distribution of computational resources and skills rather than on subjection to computational classification Quotation withheld because this record is not cleared for redistribution. curse of dimensionality —causes→ opacity curse of dimensionality opacity strength of the relation: 0.9 Audit 2026-09-28: DOUBTFUL - read against its span and kept, but the span shown does not name dimensionality in its opening; causes is loose. Quotation withheld because this record is not cleared for redistribution. machine learning —causes→ opacity machine learning opacity strength of the relation: 0.9 Quotation withheld because this record is not cleared for redistribution. opacity —renamed-across-fields→ interpretability opacity interpretability strength of the relation: 0.95 Quotation withheld because this record is not cleared for redistribution. trade secrets —causes→ opacity trade secrets opacity strength of the relation: 0.9 Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but trade-secret opacity is 'one argument in the emerging literature'; attributed. Quotation withheld because this record is not cleared for redistribution. Three forms of opacity are distinguished: corporate or institutional concealment, code literacy as a specialist skill, and the mismatch between high-dimensional optimisation and human-scale interpretation. Quotation withheld because this record is not cleared for redistribution. The form of opacity in an application must be identified before choosing a course of action to mitigate it. Quotation withheld because this record is not cleared for redistribution. The challenge is not only reading code but understanding the algorithm in action on data. Quotation withheld because this record is not cleared for redistribution. machine learning opacity persists even for those with specialised training, even for computer scientists Quotation withheld because this record is not cleared for redistribution. opacity opacity Quotation withheld because this record is not cleared for redistribution. remediable incomprehensibility remediable incomprehensibility Quotation withheld because this record is not cleared for redistribution. interpretability interpretability Quotation withheld because this record is not cleared for redistribution. curse of dimensionality curse of dimensionality Quotation withheld because this record is not cleared for redistribution.

The full text of this article is not redistributable, so it is not included here. The evidence sentences are withheld with it; the tag layer and its grounding status remain. The article itself is at https://doi.org/10.1177/2053951715622512.