How the machine ‘thinks’: Understanding opacity in machine learning algorithms

Jenna Burrell · Big Data & Society · 2016 · ai-ethics · 10.1177/2053951715622512 · record burrell2016-how-the-machine-thinks-understandi
tag layer · index · concepts · authors · עברית

Where this record's text came from. 2026-08-20 schema-0.3 pass: every quote verified as an exact substring of the staged text and anchored at merge time; concept terms verified present; every edge endpoint an existing node.

How to read this page. The article itself is not reproduced here — this paper is not cleared for redistribution, so what follows is the tag layer alone. The evidence sentences are withheld with the article. The table at the foot lists every tag the record carries; each row names the facet it belongs to, and the whose words column says whether the tag records the paper's own wording or the tagger's sentence about the paper. Every tag links to each other paper in the corpus carrying it. The machine-readable edition of the same thing is the TEI file beside this one, where the tags live in standOff, and the article is at its publisher via the link below. When we have changed our mind. Some entries in the table begin with a word in capitals and the reason for it. EVIDENCE CORRECTED — the finding still stands, but we had quoted the wrong sentence from the paper, and the quotation has been replaced with the right one. RETRACTED — the entry itself was wrong; it stays on the page, marked, with the reason, rather than being deleted. WITHDRAWN AND REPLACED — the entry was wrong, and a corrected version of it appears in the entries below. Each mark carries its own reason, in brackets, because the reasons differ from one entry to the next. An entry we have WITHDRAWN is removed from this page and kept in the record, with its reason. A line reading Audit and a date is a later re-reading of the entry against its whole sentence: DOUBTFUL means it was kept, and the reason says what is loose about it. Quotation marks inside the tagger's sentence. Where a tag is the tagger's own sentence, a phrase in quotation marks inside it is borrowed from the paper — the tagger's wording around it is not. The paper's own sentence is the evidence quote for that row; on this page it is withheld with the article, and the article is at its publisher. Nothing inside quotation marks anywhere on this page is ours: if it is quoted, it is the paper's. Whose words a concept carries. A concept entry records a term the paper itself uses, so the term, the sense given for it, and its evidence are the paper's own words, quoted where the paper fixes them. This is the opposite of a sense entry in the tag table, which is the tagger's sentence about how the paper uses a term. Where a concept entry could not be grounded in the paper's words, the entry says so on its own row.

Other fields define these differently the same word, defined another way in another discipline
machine learning 14 fields · principal component analysis 5 fields · algorithm 5 fields · interpretability 3 fields · discrimination 2 fields · training data 2 fields · and 2 more →

Tags on this paper

Every tag the record carries, each linking to every other paper in the corpus that carries it. A tag appears here whether or not its words occur in the article — the spans below are only the ones a sentence in this paper licenses. Of the 18 concept tags on this paper, all 18 are grounded in a quoted sentence.

opacity machine learning classification algorithm interpretability remediable incomprehensibility curse of dimensionality training data features code audit auditors discrimination principal component analysis kernel trick trade secrets open source semantic interpretation spam THE MACHINE BUILDS ITS OWN REPRESENTATION WITHOUT REGARD FOR HUMAN COMPREHENSION - optimisation over training data does not naturally accord with human semantic explanation, so the opacity is a property of the method and not of anyone's choice or skill USEFULNESS REQUIRES COMPLEXITY: a machine learning algorithm CAN be implemented so its logic is fully comprehensible, but such an instance is unlikely to be useful - accuracy carries a degree of unavoidable complexity THE OPACITY IS IN THE INTERPLAY, NOT IN EITHER PART: the dataset may be large but comprehensible and the code may be written with clarity, and the complexity arises between them, in the mechanism of the algorithm operating on data THE DECISION LOGIC CHANGES AS THE ALGORITHM LEARNS, so reading the code as written does not tell you what it does after training DIMENSIONALITY MANIPULATION BREAKS THE FEATURE-TO-DIMENSION CORRESPONDENCE: with PCA or the kernel trick the relation is not one-to-one, so a 'weighted input' can no longer be traced to a human-readable property INTUITION FAILS AT HIGH DIMENSIONS - each of thousands of features subtly and imperceptibly shifts the classification, so debugging and improvement get harder as inputs are added SOME ALGORITHMS ARE OPAQUE IN ORDER TO WORK: spam, scam and fraud detection face an adversary who adapts, so disclosure would defeat the system - opacity here is functional rather than merely protective SECRECY MAY BE COVER RATHER THAN COMPETITIVE NECESSITY: Pasquale's reading is that the adversary is regulation itself, which makes the same observable opacity a different problem with a different remedy conceptual-analysis

Method, as described: citation of computer science literature and known industry practice, plus testing and manipulation of code as a lightweight code audit, worked through handwriting recognition and spam filtering examples

Legend

relation to other work	`claim_relations`	how this paper stands to another: agrees, contradicts, extends, qualifies
claim type	`claim_type`	the shape of the claim: existence, mechanism, method, critique, prediction, null
concept edge	`concept_edges`	an asserted relation between two concepts, with the sentence asserting it
concept	`concepts`	an idea the paper works with, quoted where the paper uses it
content tag	`content_tags`	a finding, definition, quantity, statistic or variable named in the paper
contribution	`contribution`	what the paper claims to add
control	`controls`	an experiment run to eliminate a rival explanation
limitation	`limitations`	what the paper says its evidence cannot support
mechanism	`mechanisms`	how the reported effect is said to come about
method	`method`	how the paper produced its evidence
object of study	`object_of_study`	what the paper is about — the thing it investigates
phenomenon	`phenomenon`	the effect or regularity under study
research question	`research_question`	the question the paper sets itself
sense	`senses`	a distinct use of a term, quoted where the paper fixes it

The paper

The full text is not reproduced here. This article is not cleared for redistribution, so the page carries the tag layer only. Its evidence sentences are withheld; the grounding status remains with each tag, and the article itself is at https://doi.org/10.1177/2053951715622512. The reading view exists for the papers whose licence permits it.

Every tag on this page

claim_relations 4 · claim_type 1 · concept_edges 4 · concepts 18 · content_tags 8 · contribution 1 · controls 2 · limitations 6 · mechanisms 8 · method 1 · object_of_study 1 · phenomenon 1 · research_question 1 · senses 4

Relations on this page carry a quotation for the claim. Where a relation was separately evidenced — and so tested against the vocabulary for its type — the row says so; none of 4 relations here carry that mark. The rest are the tagger’s reading of the sentence quoted beside them.

facet	tag	field	conf.	whose words
object_of_study	opacity in socially consequential algorithmic classification - spam filters, credit scoring, fraud detection, search ranking - and specifically what makes machine learning classifiers hard to understand Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`object_of_study`		the paper's words
research_question	what the reasons are for not knowing how a classification was arrived at - proprietary secrecy, technical difficulty, or something else Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`research_question`		the paper's words
contribution	a three-way typology of opacity - corporate secrecy, technical illiteracy, and the mismatch between high-dimensional optimisation and human interpretation - with the argument that each form needs a different remedy Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`contribution`		the paper's words
claim_type	method Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_type`		the paper's words
phenomenon	the recipient of an algorithmic classification rarely has any concrete sense of how or why it was arrived at Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`phenomenon`		the paper's words
method	conceptual-analysis Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: conceptual-analysis	`method`		the paper's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: THE MACHINE BUILDS ITS OWN REPRESENTATION WITHOUT REGARD FOR HUMAN COMPREHENSION - optimisation over training data does not naturally accord with human semantic explanation, so the opacity is a property of the method and not of anyone's choice or skill	`mechanisms[0]`	0.95	the tagger's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: USEFULNESS REQUIRES COMPLEXITY: a machine learning algorithm CAN be implemented so its logic is fully comprehensible, but such an instance is unlikely to be useful - accuracy carries a degree of unavoidable complexity	`mechanisms[1]`	0.95	the tagger's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: THE OPACITY IS IN THE INTERPLAY, NOT IN EITHER PART: the dataset may be large but comprehensible and the code may be written with clarity, and the complexity arises between them, in the mechanism of the algorithm operating on data	`mechanisms[2]`	0.95	the tagger's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: THE DECISION LOGIC CHANGES AS THE ALGORITHM LEARNS, so reading the code as written does not tell you what it does after training	`mechanisms[3]`	0.95	the tagger's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: DIMENSIONALITY MANIPULATION BREAKS THE FEATURE-TO-DIMENSION CORRESPONDENCE: with PCA or the kernel trick the relation is not one-to-one, so a 'weighted input' can no longer be traced to a human-readable property	`mechanisms[4]`	0.95	the tagger's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. status: Domingos (2012) notes that ‘intuition fails at high-dimensions.’ also in: INTUITION FAILS AT HIGH DIMENSIONS - each of thousands of features subtly and imperceptibly shifts the classification, so debugging and improvement get harder as inputs are added	`mechanisms[5]`	0.95	the tagger's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: SOME ALGORITHMS ARE OPAQUE IN ORDER TO WORK: spam, scam and fraud detection face an adversary who adapts, so disclosure would defeat the system - opacity here is functional rather than merely protective	`mechanisms[6]`	0.95	the tagger's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. status: he argues that a kind of adversarial situation is indeed in play also in: SECRECY MAY BE COVER RATHER THAN COMPETITIVE NECESSITY: Pasquale's reading is that the adversary is regulation itself, which makes the same observable opacity a different problem with a different remedy	`mechanisms[7]`	0.9	the tagger's words
limitations	the audit's explanation was produced by imposing human interpretive reasoning on a process of statistical optimisation - 'machine thinking was resolved to human interpretation' - which the author flags as a limit of his own method Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`limitations[0]`	0.95	the tagger's words
limitations	ambiguities remained after the audit and the author doubts the explanation was necessarily a correct one Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`limitations[1]`	0.95	the tagger's words
limitations	end-user explanations that list a few key criteria are at best incomplete and at worst false reassurance - which applies to the remedy the paper is closest to endorsing Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`limitations[2]`	0.95	the tagger's words
limitations	the paper's own examples are handwriting recognition and spam filtering, where the feature-to-dimension relation IS one-to-one, and it states that this is often not the case Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`limitations[3]`	0.9	the tagger's words
limitations	no single tool or process will alleviate black-boxed classification, and the right combination is said to depend on the application space rather than being specified Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`limitations[4]`	0.95	the tagger's words
limitations	the difficulty is stated to persist even for those with specialised training and even for computer scientists, which bounds what any literacy remedy can achieve Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`limitations[5]`	0.95	the tagger's words
controls	the code audit found the word 'Nigeria' neutrally weighted in the spam classifier - a check that produced reassuring evidence rather than the expected discriminatory finding, and the paper reports it that way Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. eliminates: discriminatory weighting of the word 'Nigeria' as the explanation of the spam classifier's behaviour - the audit found it neutrally weighted · outcome: partial	`controls[0]`	0.9	the tagger's words
controls	the author states plainly that the explanation his own audit produced may not be correct: innocuous words like 'visit' and 'want' were weighted as spam indicators and remained unexplained Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. eliminates: the audit's own explanation being taken as a description of what the classifier does, when words the author calls innocuous carry spam weight and remain unaccounted for · outcome: failed	`controls[1]`	0.95	the tagger's words
concepts	opacity Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: opacity	`concepts[0]`	0.95	the paper's words
concepts	machine learning Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: machine learning	`concepts[1]`	0.95	the paper's words
concepts	classification Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: classification	`concepts[2]`	0.95	the paper's words
concepts	algorithm Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: algorithm	`concepts[3]`	0.95	the paper's words
concepts	interpretability Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: interpretability	`concepts[4]`	0.95	the paper's words
concepts	remediable incomprehensibility Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: remediable incomprehensibility	`concepts[5]`	0.95	the paper's words
concepts	curse of dimensionality Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: curse of dimensionality	`concepts[6]`	0.95	the paper's words
concepts	training data Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: training data	`concepts[7]`	0.95	the paper's words
concepts	features Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: features	`concepts[8]`	0.95	the paper's words
concepts	code audit Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: code audit	`concepts[9]`	0.95	the paper's words
concepts	auditors Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: auditors	`concepts[10]`	0.9	the paper's words
concepts	discrimination Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: discrimination	`concepts[11]`	0.95	the paper's words
concepts	principal component analysis Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: principal component analysis	`concepts[12]`	0.95	the paper's words
concepts	kernel trick Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: kernel trick	`concepts[13]`	0.95	the paper's words
concepts	trade secrets Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: trade secrets	`concepts[14]`	0.95	the paper's words
concepts	open source Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: open source	`concepts[15]`	0.9	the paper's words
concepts	semantic interpretation Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: semantic interpretation	`concepts[16]`	0.95	the paper's words
concepts	spam Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: spam	`concepts[17]`	0.9	the paper's words
senses	opacity - THREE DIFFERENT THINGS UNDER ONE WORD, and the paper's whole point is that they are routinely conflated: (1) intentional corporate or state secrecy, (2) the fact that reading code is a specialist skill, (3) the mismatch between high-dimensional mathematical optimisation and human-scale semantic reasoning. Only the third is intrinsic to machine learning Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[0]`	0.95	the tagger's words
senses	opacity - AND THE SECOND AND THIRD SENSES ARE THE PAIR THAT GET CONFUSED, both reading as 'this is technical and complex'. The paper separates them because they have opposite remedies: education fixes illiteracy and cannot touch dimensionality Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[1]`	0.95	the tagger's words
senses	opacity (Pasquale's sense) - 'REMEDIABLE INCOMPREHENSIBILITY', which builds the remedy into the definition. Burrell's third form is precisely the case that definition cannot cover, since no disclosure makes it comprehensible Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[2]`	0.9	the tagger's words
senses	code audit - IN THIS PAPER, TWO INCOMPATIBLE THINGS: reading the source, which is what the legal literature proposes, and observing the algorithm in action on data, which is what the third form of opacity actually requires. The paper's critique of audits turns entirely on which one is meant Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[3]`	0.95	the tagger's words
claim_relations	critiques → proposals for code or algorithm audits as a way to evaluate for discriminatory classification, which the paper argues underestimate what reading the code of a complicated system entails and are non-specific about classes of algorithm Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[0]`	0.95	the tagger's words
claim_relations	qualifies → Pasquale's definition of opacity as 'remediable incomprehensibility' and the regulatory-disclosure remedy that follows from it Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[1]`	0.9	the tagger's words
claim_relations	qualifies → the emerging literature on the politics of algorithms, which treats algorithmic opacity as largely intentional corporate self-protection Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[2]`	0.95	the tagger's words
claim_relations	extends → scholarship on digital inequality, which has focused on the distribution of computational resources and skills rather than on subjection to computational classification Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[3]`	0.9	the tagger's words
concept_edges	curse of dimensionality —causes→ opacity Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. strength of the relation: 0.9 Audit 2026-09-28: DOUBTFUL - read against its span and kept, but the span shown does not name dimensionality in its opening; causes is loose. also in: curse of dimensionality · opacity	`concept_edges[0]`		the tagger's words
concept_edges	machine learning —causes→ opacity Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. strength of the relation: 0.9 also in: machine learning · opacity	`concept_edges[1]`		the tagger's words
concept_edges	opacity —renamed-across-fields→ interpretability Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. strength of the relation: 0.95 also in: opacity · interpretability	`concept_edges[2]`		the tagger's words
concept_edges	trade secrets —causes→ opacity Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. strength of the relation: 0.9 Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but trade-secret opacity is 'one argument in the emerging literature'; attributed. also in: trade secrets · opacity	`concept_edges[3]`		the tagger's words
content_tags	Three forms of opacity are distinguished: corporate or institutional concealment, code literacy as a specialist skill, and the mismatch between high-dimensional optimisation and human-scale interpretation. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.findings[0]`	0.95	the tagger's words
content_tags	The form of opacity in an application must be identified before choosing a course of action to mitigate it. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.findings[1]`	0.95	the tagger's words
content_tags	The challenge is not only reading code but understanding the algorithm in action on data. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.findings[2]`	0.95	the tagger's words
content_tags	machine learning opacity persists even for those with specialised training, even for computer scientists Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.findings[3]`	0.95	the tagger's words
content_tags	opacity Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: opacity	`content_tags.definitions[0]`	0.95	the tagger's words
content_tags	remediable incomprehensibility Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: remediable incomprehensibility	`content_tags.definitions[1]`	0.95	the tagger's words
content_tags	interpretability Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: interpretability	`content_tags.definitions[2]`	0.95	the tagger's words
content_tags	curse of dimensionality Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: curse of dimensionality	`content_tags.definitions[3]`	0.95	the tagger's words

Interpretive choices

Tags that could not be placed in the text:

none — every tag was located in the text

What was deliberately not tagged, and why, is recorded per paper by the tagger. Where a paper has no such record, this page says so rather than implying that nothing was declined.