How We Do Things with Words: Analyzing Text as Social and Cultural Data

Dong Nguyen, Maria Liakata, Simon DeDeo, Jacob Eisenstein · Frontiers in Artificial Intelligence / arXiv · 2020 · digital-humanities · 10.3389/frai.2020.00062 · record nguyen2020-text-as-data
tag layer · index · concepts · authors · עברית

Where this record's text came from. faceted 2026-08 by reading the locally held text (xml); every evidence string verified as an exact substring of that text before merging. Non-empirical guidance article; no data or results of its own, so replication status is untested. It repeatedly uses Chandrasekharan et al. (2017) on Reddit hate speech as a running worked example rather than as a claim it relates itself to, so that is not recorded as a claim_relation. Discipline is a compromise: the four authors write from NLP, digital humanities, and social science simultaneously ('given our diversity of disciplinary backgrounds'), and the facet cannot express that plurality. Method 'practice-report' was chosen over 'review' because the article reports the authors' own research practice rather than surveying a literature. 2026-08-12, 2026-08-26 and 2026-08-28 schema-0.3 passes: every quote verified as an exact substring of the staged text and anchored at merge time; concept terms verified present; every edge endpoint an existing node. Update, 15 September 2026: EXCEPTION, measured with the merge gate's own predicate (Paper.locate): 3 quote(s) on this record are not found as one exact string in the current staged text - content_tags.findings[1].evidence, content_tags.findings[2].evidence, content_tags.statistics[0].evidence. The sentence above does not hold for them as written. Where a row records why - the quote is grounded in another copy of the paper, joins two passages with an ellipsis, or differs from the staged text by extraction noise - that note stays on the row.

3 quotes on this page were cut from a different copy of this article. This project holds two extractions of this paper: the plain text every other quote here is anchored in, and the publisher’s own HTML edition, which we are not licensed to republish. They are verbatim in the second and cannot be found in the first — not because the tagging is wrong, but because the extraction damaged the passage: a page number dropped into the middle of a number, a formula moved out of its sentence, a ligature where two letters belong. The record names the copy for each one. The quote is the paper’s; the extraction is not.

How to read this page. The article itself is not reproduced here — this paper is not cleared for redistribution, so what follows is the tag layer alone. The evidence sentences are withheld with the article. The table at the foot lists every tag the record carries; each row names the facet it belongs to, and the whose words column says whether the tag records the paper's own wording or the tagger's sentence about the paper. Every tag links to each other paper in the corpus carrying it. The machine-readable edition of the same thing is the TEI file beside this one, where the tags live in standOff, and the article is at its publisher via the link below. When we have changed our mind. Some entries in the table begin with a word in capitals and the reason for it. EVIDENCE CORRECTED — the finding still stands, but we had quoted the wrong sentence from the paper, and the quotation has been replaced with the right one. RETRACTED — the entry itself was wrong; it stays on the page, marked, with the reason, rather than being deleted. WITHDRAWN AND REPLACED — the entry was wrong, and a corrected version of it appears in the entries below. Each mark carries its own reason, in brackets, because the reasons differ from one entry to the next. An entry we have WITHDRAWN is removed from this page and kept in the record, with its reason. A line reading Audit and a date is a later re-reading of the entry against its whole sentence: DOUBTFUL means it was kept, and the reason says what is loose about it. Quotation marks inside the tagger's sentence. Where a tag is the tagger's own sentence, a phrase in quotation marks inside it is borrowed from the paper — the tagger's wording around it is not. The paper's own sentence is the evidence quote for that row; on this page it is withheld with the article, and the article is at its publisher. Nothing inside quotation marks anywhere on this page is ours: if it is quoted, it is the paper's. Whose words a concept carries. A concept entry records a term the paper itself uses, so the term, the sense given for it, and its evidence are the paper's own words, quoted where the paper fixes them. This is the opposite of a sense entry in the tag table, which is the tagger's sentence about how the paper uses a term. Where a concept entry could not be grounded in the paper's words, the entry says so on its own row.

Other fields define these differently the same word, defined another way in another discipline
metadata 7 fields · reliability 5 fields · ground truth 5 fields · sample-size 5 fields · gold standard 4 fields · privacy 3 fields · and 7 more →

Tags on this paper

Every tag the record carries, each linking to every other paper in the corpus that carries it. A tag appears here whether or not its words occur in the article — the spans below are only the ones a sentence in this paper licenses. Of the 82 concept tags on this paper, 60 are grounded in a quoted sentence. 77 carry the tagger's gloss of what the term means in this paper, which is not something the article can license.

text as data natural language processing annotation measurement validity computational social science background concept systematized concept operationalization conceptual validity content validity ground truth gold standard reliability validity annotation scheme inter-annotator agreement Krippendorff's alpha close reading ambiguity thick hate speech deductive crowdsourcing topic modeling unobservable insight driven automating tasks Domain experts dual use Born-digital data unobtrusive measurements privacy provenance black box APIs data biases spurious results generalizability “comparison” set matching design control group counterfactual scenario metadata Discretization classification scheme Supervised learning unsupervised learning natural structure unit of text unit of analysis interpretability human coders annotation examples expert annotators intra -annotator agreement Disagreement between annotators racial bias pre-processing steps OCR errors Tokenization Multi-word terms normalization word pieces stoplist Dictionaries domain-specific dictionaries spurious features post-hoc explanations Topic models number of topics stratified random sample synthetic data explanatory power predictive performance opportunities for interpretation qualitative analyses working across disciplines qualitative insights reliable results reproducibility efforts needs of the user Computational text analysis models that are already somewhat interpretable text computational social humanities measurement interpretation corpus hypothesis-testing hypothesis testing sample-size sample size F-score practice-report

Legend

second reading	`alternatives`	an independent re-reading of a facet, recorded whether or not it agrees
relation to other work	`claim_relations`	how this paper stands to another: agrees, contradicts, extends, qualifies
claim type	`claim_type`	the shape of the claim: existence, mechanism, method, critique, prediction, null
concept edge	`concept_edges`	an asserted relation between two concepts, with the sentence asserting it
concept	`concepts`	an idea the paper works with, quoted where the paper uses it
content tag	`content_tags`	a finding, definition, quantity, statistic or variable named in the paper
limitation	`limitations`	what the paper says its evidence cannot support
mechanism	`mechanisms`	how the reported effect is said to come about
method	`method`	how the paper produced its evidence
object of study	`object_of_study`	what the paper is about — the thing it investigates
phenomenon	`phenomenon`	the effect or regularity under study
sense	`senses`	a distinct use of a term, quoted where the paper fixes it
transfer	`transfers`	a method or concept carried in from another field
unrecordable relation	`unhoused_relations`	a relation the paper asserts that this vocabulary cannot express, with the reason

The paper

The full text is not reproduced here. This article is not cleared for redistribution, so the page carries the tag layer only. Its evidence sentences are withheld; the grounding status remains with each tag, and the article itself is at https://doi.org/10.3389/frai.2020.00062. The reading view exists for the papers whose licence permits it.

Every tag on this page

alternatives 2 · claim_relations 6 · claim_type 1 · concept_edges 12 · concepts 60 · content_tags 21 · limitations 11 · mechanisms 8 · method 1 · object_of_study 1 · phenomenon 1 · senses 25 · transfers 2 · unhoused_relations 14

Relations on this page carry a quotation for the claim. Where a relation was separately evidenced — and so tested against the vocabulary for its type — the row says so; 3 of 12 relations here carry that mark. The rest are the tagger’s reading of the sentence quoted beside them.

This record holds no controls. A layer is empty when the tagger recorded nothing in it — which is not the same as the paper having nothing to record.

facet	tag	field	conf.	whose words
object_of_study	The research process itself - how a contested social or cultural concept travels from a question, through a definition and an operationalisation, to a measurement that can be validated. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`object_of_study`		the paper's words
claim_type	method Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_type`		the paper's words
phenomenon	computational text analysis of social and cultural concepts Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`phenomenon`		the paper's words
method	practice-report Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: practice-report	`method`		the paper's words
mechanisms	A concept must be narrowed from the full range of meanings it carries to the single formulation a study adopts, and those are two different objects with two different names Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`mechanisms[0]`		the tagger's words
mechanisms	Computation earns its place where patterns exist only in aggregate and are invisible in any single case Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`mechanisms[1]`		the tagger's words
mechanisms	Where there is no objective standard, accuracy against annotations measures consistency rather than correctness Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`mechanisms[2]`		the tagger's words
mechanisms	Inter-annotator reliability differs from accuracy in taking no coder as correct and discounting agreement expected by chance Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`mechanisms[3]`		the tagger's words
mechanisms	Validation without a gold standard proceeds by sampling across the full range of scores and asking whether each case makes sense under the adopted definition Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`mechanisms[4]`		the tagger's words
mechanisms	Reading the texts beside the model output informs the model as much as the model informs the reading Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`mechanisms[5]`		the tagger's words
mechanisms	Annotator disagreement can be a property of the concept rather than a defect of the annotators, and can be carried forward instead of resolved Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`mechanisms[6]`		the tagger's words
mechanisms	A METHODS GUIDE THAT MAKES `FIND THE COMPETING DEFINITIONS` A REQUIRED STEP, IN A CHECKLIST: `Conceptualization: What is this all about? What are the core concepts you are addressing? And are you being true to their core meaning? What are competing definitions? Which is best suited to the task and why? Does the systematized concept you've selected reflect an adequate understanding of the background concept? How do domain experts approach the topic?` This is Adcock and Collier's vocabulary turned into a procedure a researcher is told to follow before touching data - and it is the closest thing in the corpus to a specification of what this project does. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`mechanisms[7]`	0.95	the tagger's words
limitations	operationalizing rich concepts computationally can raise conceptual-validity concerns and yield shallow conclusions Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: any computational operationalisation of a rich concept: it raises Quotation withheld	`limitations[0]`		the tagger's words
limitations	The guidance is drawn from the authors' own experience and is explicitly acknowledged as imperfect. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the guidance this paper offers: OUR GUIDANCE IS BASED ON OUR OWN EXPERIENCES AND IS THEREFORE INHERENTLY IMPERFECT	`limitations[1]`		the tagger's words
limitations	The proposed set of guiding questions is acknowledged to be incomplete. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the set of guiding questions: WE PROVIDE A SET OF QUESTIONS, THOUGH UNAVOIDABLY INCOMPLETE, THAT CAN SERVE AS A GUIDE FOR THINKING THROUGH CHALLENGES IN EACH STEP OF THE RESEARCH PROCESS	`limitations[2]`		the tagger's words
limitations	The authors position computational text analysis as an addition to, not a substitute for, other analytic approaches — i.e. it cannot answer such questions alone. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: replacement of other methods, named as the use it does not support: COMPUTATIONAL TEXT ANALYSIS IS NOT A REPLACEMENT FOR BUT RATHER AN ADDITION TO THE APPROACHES ONE CAN TAKE TO ANALYZE SOCIAL AND CULTURAL PHENOMENA USING TEXTUAL DATA	`limitations[3]`		the tagger's words
limitations	The guidance is offered as the authors' own experience and explicitly imperfect Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. duplicate of: limitations[1] · what this limits: the same scope as limitations[1] - the guidance, from experience and INHERENTLY IMPERFECT	`limitations[4]`		the tagger's words
limitations	For social and cultural concepts there is no ground truth to appeal to, and definitions are contested and change Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: validation of any social or cultural concept measure: FOR SOCIAL AND CULTURAL CONCEPTS THERE IS NO ABSOLUTE GROUND TRUTH	`limitations[5]`		the tagger's words
limitations	An operationalisation is never a faithful rendering of the concept it stands for Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: every measured construct in the paper and the genre: OPERATIONALIZATIONS ARE NEVER PERFECT TRANSLATIONS, AND ARE OFTEN REFINED OVER THE COURSE OF AN INVESTIGATION	`limitations[6]`		the tagger's words
limitations	Many of the concepts at issue cannot be observed at all, which is what makes validation hard rather than merely laborious Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: validation in the humanities and social sciences specifically: MANY RELEVANT CONCEPTS ARE FUNDAMENTALLY UNOBSERVABLE, SUCH AS LATENT TRAITS OF POLITICAL ACTORS - unlike tasks such as SPAM FILTERING	`limitations[7]`		the tagger's words
limitations	Human labels are not neutral either - they carry the annotators' priors and perspectives Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: any human-labelled ground truth: WHEN THE GROUND TRUTH COMES FROM PEOPLE, IT MAY BE INFLUENCED BY IDEOLOGICAL PRIORS, PRIORS, PRIMING, SIMPLE DIFFERENCES OF OPINION OR PERSPECTIVE	`limitations[8]`		the tagger's words
limitations	The research process is presented linearly but is not, and later stages feed back into the question itself Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the paper's own linear presentation of the research process: THE RESEARCH PROCESS SOUNDS MORE OR LESS LINEAR THIS WAY, BUT EACH OF THESE PHASES OVERLAPS, AND IN SOME INSTANCES REQUIRES US TO RETURN TO PREVIOUS STEPS	`limitations[9]`		the tagger's words
limitations	A poor operationalisation does not merely add noise - it can produce conclusions that are shallow rather than wrong Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: conclusions drawn from an operationalisation: CHOICES REGARDING HOW TO OPERATIONALIZE AND ANALYZE THESE CONCEPTS CAN RAISE SERIOUS Quotation withheld	`limitations[10]`		the tagger's words
transfers	Adcock and Collier's background/systematized concept distinction and their notion of content validity, imported to discipline the conceptualization step of computational text analysis Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`transfers[0]`		the tagger's words
transfers	Close reading carried into computational work not as a rival but as a validation step that informs the model Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`transfers[1]`		the tagger's words
concepts	ground truth Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: ground truth	`concepts[10]`	0.9	the paper's words
concepts	gold standard Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: gold standard	`concepts[11]`	0.9	the paper's words
concepts	hate speech Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: hate speech	`concepts[20]`	0.9	the paper's words
concepts	insight driven Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: insight driven	`concepts[25]`	0.95	the paper's words
concepts	automating tasks Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: automating tasks	`concepts[26]`	0.95	the paper's words
concepts	Domain experts Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: Domain experts	`concepts[27]`	0.95	the paper's words
concepts	dual use Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: dual use	`concepts[28]`	0.95	the paper's words
concepts	Born-digital data Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: Born-digital data	`concepts[29]`	0.95	the paper's words
concepts	unobtrusive measurements Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: unobtrusive measurements	`concepts[30]`	0.95	the paper's words
concepts	privacy Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: privacy	`concepts[31]`	0.95	the paper's words
concepts	provenance Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: provenance	`concepts[32]`	0.95	the paper's words
concepts	black box APIs Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: black box APIs	`concepts[33]`	0.95	the paper's words
concepts	data biases Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: data biases	`concepts[34]`	0.95	the paper's words
concepts	spurious results Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: spurious results	`concepts[35]`	0.95	the paper's words
concepts	generalizability Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: generalizability	`concepts[36]`	0.95	the paper's words
concepts	“comparison” set Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: “comparison” set	`concepts[37]`	0.95	the paper's words
concepts	matching design Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: matching design	`concepts[38]`	0.95	the paper's words
concepts	control group Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: control group	`concepts[39]`	0.95	the paper's words
concepts	counterfactual scenario Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: counterfactual scenario	`concepts[40]`	0.95	the paper's words
concepts	metadata Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: metadata	`concepts[41]`	0.95	the paper's words
concepts	Discretization Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: Discretization	`concepts[42]`	0.95	the paper's words
concepts	classification scheme Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: classification scheme	`concepts[43]`	0.95	the paper's words
concepts	Supervised learning Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: Supervised learning	`concepts[44]`	0.95	the paper's words
concepts	unsupervised learning Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: unsupervised learning	`concepts[45]`	0.95	the paper's words
concepts	natural structure Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: natural structure	`concepts[46]`	0.95	the paper's words
concepts	unit of text Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: unit of text	`concepts[47]`	0.95	the paper's words
concepts	unit of analysis Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: unit of analysis	`concepts[48]`	0.95	the paper's words
concepts	interpretability Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: interpretability	`concepts[49]`	0.95	the paper's words
concepts	human coders Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: human coders	`concepts[50]`	0.95	the paper's words
concepts	annotation examples Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: annotation examples	`concepts[51]`	0.95	the paper's words
concepts	expert annotators Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: expert annotators	`concepts[52]`	0.95	the paper's words
concepts	intra -annotator agreement Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: intra -annotator agreement	`concepts[53]`	0.95	the paper's words
concepts	Disagreement between annotators Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: Disagreement between annotators	`concepts[54]`	0.95	the paper's words
concepts	racial bias Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: racial bias	`concepts[55]`	0.95	the paper's words
concepts	pre-processing steps Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: pre-processing steps	`concepts[56]`	0.95	the paper's words
concepts	OCR errors Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: OCR errors	`concepts[57]`	0.95	the paper's words
concepts	Tokenization Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: Tokenization	`concepts[58]`	0.95	the paper's words
concepts	Multi-word terms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: Multi-word terms	`concepts[59]`	0.95	the paper's words
concepts	normalization Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: normalization	`concepts[60]`	0.95	the paper's words
concepts	word pieces Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: word pieces	`concepts[61]`	0.95	the paper's words
concepts	stoplist Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: stoplist	`concepts[62]`	0.95	the paper's words
concepts	Dictionaries Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: Dictionaries	`concepts[63]`	0.95	the paper's words
concepts	domain-specific dictionaries Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: domain-specific dictionaries	`concepts[64]`	0.95	the paper's words
concepts	spurious features Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: spurious features	`concepts[65]`	0.95	the paper's words
concepts	post-hoc explanations Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: post-hoc explanations	`concepts[66]`	0.95	the paper's words
concepts	Topic models Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: Topic models	`concepts[67]`	0.95	the paper's words
concepts	number of topics Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: number of topics	`concepts[68]`	0.95	the paper's words
concepts	stratified random sample Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: stratified random sample	`concepts[69]`	0.95	the paper's words
concepts	synthetic data Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: synthetic data	`concepts[70]`	0.95	the paper's words
concepts	explanatory power Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: explanatory power	`concepts[71]`	0.95	the paper's words
concepts	predictive performance Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: predictive performance	`concepts[72]`	0.95	the paper's words
concepts	opportunities for interpretation Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: opportunities for interpretation	`concepts[73]`	0.95	the paper's words
concepts	qualitative analyses Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: qualitative analyses	`concepts[74]`	0.95	the paper's words
concepts	working across disciplines Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: working across disciplines	`concepts[75]`	0.95	the paper's words
concepts	qualitative insights Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: qualitative insights	`concepts[76]`	0.95	the paper's words
concepts	reliable results Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: reliable results	`concepts[77]`	0.95	the paper's words
concepts	reproducibility efforts Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: reproducibility efforts	`concepts[78]`	0.95	the paper's words
concepts	needs of the user Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: needs of the user	`concepts[79]`	0.95	the paper's words
concepts	Computational text analysis Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: Computational text analysis	`concepts[80]`	0.95	the paper's words
concepts	models that are already somewhat interpretable Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: models that are already somewhat interpretable	`concepts[81]`	0.95	the paper's words
unhoused_relations	Born-digital data → unobtrusive measurements Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. Because born-digital sources do not require direct elicitation, they enable unobtrusive measurement. Enables is absent from the governed types, and the gate correctly refused a causes edge without its governed effect cue.	`unhoused_relations[0]`	0.95	the tagger's words
unhoused_relations	Domain experts → qualitative insights Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. Domain experts identify qualitative insights that can strengthen computational analysis and enrich the answer. Advises/contributes is not a governed relation type.	`unhoused_relations[1]`	0.95	the tagger's words
unhoused_relations	black box APIs → data biases Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. Closed commercial APIs are likely to introduce biases whose form and severity researchers cannot inspect. Introduces-risk-of is not a governed type, and the wording is probabilistic.	`unhoused_relations[2]`	0.95	the tagger's words
unhoused_relations	matching design → control group Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. The matching design populates the control group with untreated forums similar to the treated ones. Constructs/selects is absent from the governed types.	`unhoused_relations[3]`	0.95	the tagger's words
unhoused_relations	control group → counterfactual scenario Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. The matched control group is used to estimate what would have happened without the closures. Estimates-counterfactual is absent and should not be forced into operationalised-as without its governed cue.	`unhoused_relations[4]`	0.95	the tagger's words
unhoused_relations	Discretization → validity Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. Overly broad discretized categories can threaten validity. Threatens is absent from the governed types, and deterministic causation would overstate 'can'.	`unhoused_relations[5]`	0.95	the tagger's words
unhoused_relations	unit of text → unit of analysis Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. The labeled unit of text can differ from the final unit of analysis. Differs-in-level-from is not captured by the governed identity or distinction types.	`unhoused_relations[6]`	0.95	the tagger's words
unhoused_relations	expert annotators → reliable results Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. Highly trained or expert annotators tend to produce more reliable results for complex concepts. Suitable-for/tends-to-improve is absent and explicitly hedged.	`unhoused_relations[7]`	0.95	the tagger's words
unhoused_relations	pre-processing steps → reproducibility efforts Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. Preprocessing choices have a large impact on reproducibility efforts. Affects is absent from the governed relation types.	`unhoused_relations[8]`	0.95	the tagger's words
unhoused_relations	post-hoc explanations → models that are already somewhat interpretable Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. Post-hoc explanations are an alternative to restricting analysis to models that are already somewhat interpretable, but may be unfaithful. Alternative-to-with-fidelity-risk is absent.	`unhoused_relations[9]`	0.95	the tagger's words
unhoused_relations	number of topics → needs of the user Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. The appropriate topic count is determined by the needs of the user rather than the collection. Determined-by-analytical-purpose is absent from the governed types.	`unhoused_relations[10]`	0.95	the tagger's words
unhoused_relations	explanatory power → predictive performance Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. For insight-driven studies, improving explanatory power can be more useful than small predictive-performance gains. Preferred-over-for-purpose is absent from the governed types.	`unhoused_relations[11]`	0.95	the tagger's words
unhoused_relations	qualitative analyses → Computational text analysis Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. Moving between large-scale computational text analysis and small-scale qualitative analysis combines their strengths. Complements is absent from the governed types.	`unhoused_relations[12]`	0.95	the tagger's words
unhoused_relations	Tokenization → Multi-word terms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. Deciding whether multi-word terms are tokenized as one unit can dramatically alter textual patterns. Treatment-of is absent and causation would misstate the relevant agent.	`unhoused_relations[13]`	0.95	the tagger's words
senses	concept - BACKGROUND: everything the term has meant across the literature, contested and changing Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[0]`	1.0	the tagger's words
senses	concept - SYSTEMATIZED: the one definition this study adopts, chosen for its question and framework Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[1]`	1.0	the tagger's words
senses	accuracy - as VALIDITY: what it is usually presented as, and what it cannot be without an objective standard Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[2]`	1.0	the tagger's words
senses	accuracy - as RELIABILITY: consistency against a human sample, which is what it actually measures when no ground truth exists Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[3]`	1.0	the tagger's words
senses	ambiguity - in computational linguistics: a problem to be solved, on the assumption that one correct interpretation exists Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[4]`	1.0	the tagger's words
senses	ambiguity - in literary criticism: prized, alongside interpretation and argumentation, above definitive conclusions Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[5]`	1.0	the tagger's words
senses	annotator disagreement - as NOISE: a failure of the scheme or the coders, to be reduced Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[6]`	1.0	the tagger's words
senses	annotator disagreement - as SIGNAL: a property of the concept, which can be carried into the analysis rather than resolved away Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[7]`	1.0	the tagger's words
senses	Insight-driven analysis - describes or explains a phenomenon rather than optimizing a prediction task Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[8]`	0.95	the tagger's words
senses	Dual use - a protective hate-speech method may also be adapted to suppress marginalized speech or dissent Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[9]`	0.95	the tagger's words
senses	Born-digital evidence - massive and unobtrusive, yet ethically constrained by contextual privacy and absent consent Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[10]`	0.95	the tagger's words
senses	Black-box collection - commercial APIs can introduce unknown biases into research inference Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[11]`	0.95	the tagger's words
senses	Restricted generalization - biased data can support within-population comparison when claims stay within that population Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[12]`	0.95	the tagger's words
senses	Counterfactual control - matched untreated forums estimate what would have happened without platform closure Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[13]`	0.95	the tagger's words
senses	Discretization tradeoff - categories facilitate nonlinear analysis but overly broad bins threaten validity Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[14]`	0.95	the tagger's words
senses	Supervised versus unsupervised learning - labels learned from annotated data versus structure learned from unlabeled data Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[15]`	0.95	the tagger's words
senses	Annotation disagreement - can reveal either scheme weaknesses or inherent ambiguity and may itself be retained as signal Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[16]`	0.95	the tagger's words
senses	Preprocessing provenance - operationally consequential choices should be documented like editorial or biomedical protocols Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[17]`	0.95	the tagger's words
senses	Tokenization - seemingly basic segmentation that can fail on creative orthography and transform multi-word patterns Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[18]`	0.95	the tagger's words
senses	Word-piece segmentation - effective for prediction but potentially unsuitable when linguistic-unit interpretation matters Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[19]`	0.95	the tagger's words
senses	Domain dictionary construction - automation proposes candidates and human theory filters spurious features Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[20]`	0.95	the tagger's words
senses	Post-hoc explanation - interprets individual predictions without guaranteeing fidelity to the underlying model Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[21]`	0.95	the tagger's words
senses	Topic count - a user-selected analytical scale like map scale or microscope magnification, not an intrinsic truth Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[22]`	0.95	the tagger's words
senses	Content revalidation - jointly interrogates concept, indicators, and scores and may force recoding or method replacement Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[23]`	0.95	the tagger's words
senses	Computational and qualitative analysis - complementary scales for trends and individual stories Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[24]`	0.95	the tagger's words
alternatives	claim_type: method Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. a second reading reaching the same value. Recorded: method — note: a second reading reaching the same value from a different sentence. Not applied - a person decides.	`alternatives.claim_type`		the tagger's words
alternatives	phenomenon: computational text analysis of social and cultural concepts Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. a second reading reaching the same value. Recorded: computational text analysis of social and cultural concepts — note: a second reading reaching the same value from a different sentence. Not applied - a person decides.	`alternatives.phenomenon`		the tagger's words
claim_relations	builds-on → natural language processing (computational text methods) Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[0]`		the tagger's words
claim_relations	builds-on → Bender and Friedman (2018), Gebru et al. (2018) on dataset documentation and Mitchell et al. (2019) on model documentation Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[1]`		the tagger's words
claim_relations	critiques → the gold-standard framing of evaluation when applied to contested social and cultural concepts Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[2]`		the tagger's words
claim_relations	extends → Adcock and Collier (2001) on measurement validity in political science Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[3]`		the tagger's words
claim_relations	supports → Piper's account of validation as further discovery in two directions Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[4]`		the tagger's words
claim_relations	extends → the datasheets and model-cards line of work on documenting datasets and models Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[5]`		the tagger's words
concept_edges	gold standard —same-as→ ground truth Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. strength of the relation: 1 also in: gold standard · ground truth	`concept_edges[2]`		the tagger's words
concept_edges	ground truth —distinct-from→ unobservable Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. strength of the relation: 1 Audit 2026-09-28: DOUBTFUL - read against its span and kept, but the span calls some concepts fundamentally unobservable; distinct-from ground truth is inferred. also in: ground truth · unobservable	`concept_edges[3]`		the tagger's words
concept_edges	reliability —distinct-from→ validity Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. strength of the relation: 1 Audit 2026-09-28: DOUBTFUL - read against its span and kept, but the span defines reliability only; the contrast with validity is not in it. also in: reliability · validity	`concept_edges[4]`		the tagger's words
concept_edges	inter-annotator agreement —operationalised-as→ reliability Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. strength of the relation: 1 Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but the span glosses inter-annotator agreement as 'intercoder reliability'; operationalised-as is loose. also in: inter-annotator agreement · reliability	`concept_edges[5]`		the tagger's words
concept_edges	Krippendorff's alpha —operationalised-as→ inter-annotator agreement Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. strength of the relation: 1 Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but endpoint by anaphora: the span answers 'how inter-annotator agreement will be measured' (previous sentence); operationalised-as is loose. also in: Krippendorff's alpha · inter-annotator agreement	`concept_edges[6]`		the tagger's words
concept_edges	content validity —operationalised-as→ annotation scheme Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. strength of the relation: 1 Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but the span writes 'the codebook'; 'annotation scheme' is a variant, and operationalised-as is loose. also in: content validity · annotation scheme	`concept_edges[7]`		the tagger's words
concept_edges	close reading —operationalised-as→ validity Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. strength of the relation: 0.9 Audit 2026-09-28: DOUBTFUL - read against its span and kept, but the span says validation takes the form of close reading for some; the direction of operationalised-as is reversed. also in: close reading · validity	`concept_edges[8]`		the tagger's words
concept_edges	ambiguity —contrasted-with→ ground truth Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. strength of the relation: 1 Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but attributed: the span quotes Kirschenbaum. also in: ambiguity · ground truth	`concept_edges[9]`		the tagger's words
concept_edges	deductive —part-of→ annotation scheme Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. strength of the relation: 1 Audit 2026-09-28: DOUBTFUL - read against its span and kept, but the span describes deductive coding rules; part-of the annotation scheme is inferred. also in: deductive · annotation scheme	`concept_edges[11]`		the tagger's words
concept_edges	insight driven —contrasted-with→ automating tasks Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. the relation itself is separately evidenced · strength of the relation: 1 · note: The article explicitly contrasts insight-driven social inquiry with task automation and prediction. also in: insight driven · automating tasks	`concept_edges[14]`	0.95	the tagger's words
concept_edges	data biases —causes→ spurious results Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. the relation itself is separately evidenced · strength of the relation: 0.5 — the record marks this weak · note: The paper says biases may lead to spurious results; weight reflects 'may'. Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but the span says data biases MAY lead to spurious results; hedged. also in: data biases · spurious results	`concept_edges[15]`	0.95	the tagger's words
concept_edges	Supervised learning —distinct-from→ unsupervised learning Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. the relation itself is separately evidenced · strength of the relation: 1 · note: The contrast is labeled data versus unlabeled data. also in: Supervised learning · unsupervised learning	`concept_edges[16]`	0.95	the tagger's words
content_tags	0.80 or above — the conventional threshold for acceptable intercoder agreement in communication research Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.quantities[0]`		the tagger's words
content_tags	450 18th-century novels — the corpus in which close reading beside model output found an unrecognised genre Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.quantities[1]`		the tagger's words
content_tags	A human reader is better equipped than a computer to make logical inferences, resolve ambiguities and apply cultural knowledge. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.findings[0]`		the tagger's words
content_tags	a concept often has several valid definitions, and definitions may be contested over time Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.findings[3]`		the tagger's words
content_tags	validity then has to be judged by other techniques Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.findings[4]`		the tagger's words
content_tags	this kind of validation is rare in NLP, yet matters most when it is hard to tell what drives a model Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.findings[5]`		the tagger's words
content_tags	disagreement between annotators can reveal weaknesses in the annotation scheme or the inherent ambiguity of what is being measured Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.findings[6]`		the tagger's words
content_tags	annotation carries bias: tweets in African American English, and by self-identified African Americans, were more often labelled offensive Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.findings[7]`		the tagger's words
content_tags	close reading of 450 eighteenth-century novels alongside model output revealed a genre absent from literary history's own categories Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.findings[8]`		the tagger's words
content_tags	conceptual validity Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: conceptual validity	`content_tags.definitions[0]`		the tagger's words
content_tags	ground truth Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: ground truth	`content_tags.definitions[1]`		the tagger's words
content_tags	background concept Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: background concept	`content_tags.definitions[2]`		the tagger's words
content_tags	systematized concept Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: systematized concept	`content_tags.definitions[3]`		the tagger's words
content_tags	reliability Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: reliability	`content_tags.definitions[4]`		the tagger's words
content_tags	validity Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: validity	`content_tags.definitions[5]`		the tagger's words
content_tags	annotation scheme Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: annotation scheme	`content_tags.definitions[6]`		the tagger's words
content_tags	content validity Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: content validity	`content_tags.definitions[7]`		the tagger's words
content_tags	sample-size Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: sample-size	`content_tags.statistics[1]`		the tagger's words
content_tags	Krippendorff's alpha Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: Krippendorff's alpha	`content_tags.statistics[2]`		the tagger's words
content_tags	inter-annotator agreement Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: inter-annotator agreement	`content_tags.statistics[3]`		the tagger's words
content_tags	F-score Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: F-score	`content_tags.statistics[4]`		the tagger's words

Interpretive choices

Tags that could not be placed in the text:

content_tags.findings[1] — quotation withheld — the page has already reproduced as much of this paper as the rights budget allows
NOT MACHINE-VOUCHED: declared-ellipsis - An elided quote: the quote marks its cuts with an ellipsis, and each of its 2 pieces is in our copy of the paper, some differing slightly from our copy in letters, notation or a small word ('the notion of ground truth is uncommon i...'; 'using them as a “ground truth” requires ...'). The joined string is in no copy.
content_tags.findings[2] — quotation withheld — the page has already reproduced as much of this paper as the rights budget allows
NOT MACHINE-VOUCHED: not-in-our-copy - The quote is in the original web page we fetched for this paper, but not in the text we tag against, which was extracted differently. Checked with ligatures, hyphens, spacing and punctuation set aside.
content_tags.statistics[0] — quotation withheld — the page has already reproduced as much of this paper as the rights budget allows
NOT MACHINE-VOUCHED: not-in-our-copy - The quote is in the original web page we fetched for this paper, but not in the text we tag against, which was extracted differently. Checked with ligatures, hyphens, spacing and punctuation set aside.

What was deliberately not tagged, and why, is recorded per paper by the tagger. Where a paper has no such record, this page says so rather than implying that nothing was declined.