How We Do Things with Words: Analyzing Text as Social and Cultural Data

How We Do Things with Words: Analyzing Text as Social and Cultural Data Dong Nguyen Maria Liakata Simon DeDeo Jacob Eisenstein

Tag layer of the DHSS Meta-Tagging Project. The article text is reproduced from the staged copy named in sourceDesc and is not altered; the tags are held separately in standOff so that overlapping and nested spans are preserved.

How We Do Things with Words: Analyzing Text as Social and Cultural Data 2020 Frontiers in Artificial Intelligence / arXiv 10.3389/frai.2020.00062 1907.01468 nguyen2020-text-as-data nguyen2020-text-as-data.txt, the project's staged copy acff3b3b1cb92251590ca039d0cd215e937f5421 cc-by digital-humanities

the paper's own words

the tag layer; a sentence written by the tagger about the paper, always anchored to a quotation but not itself a quotation

0 spans located by searching the stored text for the quote's exact words, allowing any whitespace between them The research process itself - how a contested social or cultural concept travels from a question, through a definition and an operationalisation, to a measurement that can be validated. Quotation withheld because this record is not cleared for redistribution. method Quotation withheld because this record is not cleared for redistribution. computational text analysis of social and cultural concepts Quotation withheld because this record is not cleared for redistribution. practice-report practice-report Quotation withheld because this record is not cleared for redistribution. A concept must be narrowed from the full range of meanings it carries to the single formulation a study adopts, and those are two different objects with two different names Quotation withheld because this record is not cleared for redistribution. Computation earns its place where patterns exist only in aggregate and are invisible in any single case Quotation withheld because this record is not cleared for redistribution. Where there is no objective standard, accuracy against annotations measures consistency rather than correctness Quotation withheld because this record is not cleared for redistribution. Inter-annotator reliability differs from accuracy in taking no coder as correct and discounting agreement expected by chance Quotation withheld because this record is not cleared for redistribution. Validation without a gold standard proceeds by sampling across the full range of scores and asking whether each case makes sense under the adopted definition Quotation withheld because this record is not cleared for redistribution. Reading the texts beside the model output informs the model as much as the model informs the reading Quotation withheld because this record is not cleared for redistribution. Annotator disagreement can be a property of the concept rather than a defect of the annotators, and can be carried forward instead of resolved Quotation withheld because this record is not cleared for redistribution. A METHODS GUIDE THAT MAKES `FIND THE COMPETING DEFINITIONS` A REQUIRED STEP, IN A CHECKLIST: `Conceptualization: What is this all about? What are the core concepts you are addressing? And are you being true to their core meaning? What are competing definitions? Which is best suited to the task and why? Does the systematized concept you've selected reflect an adequate understanding of the background concept? How do domain experts approach the topic?` This is Adcock and Collier's vocabulary turned into a procedure a researcher is told to follow before touching data - and it is the closest thing in the corpus to a specification of what this project does. Quotation withheld because this record is not cleared for redistribution. operationalizing rich concepts computationally can raise conceptual-validity concerns and yield shallow conclusions what this limits: any computational operationalisation of a rich concept: it raises Quotation withheld Quotation withheld because this record is not cleared for redistribution. The guidance is drawn from the authors' own experience and is explicitly acknowledged as imperfect. what this limits: the guidance this paper offers: OUR GUIDANCE IS BASED ON OUR OWN EXPERIENCES AND IS THEREFORE INHERENTLY IMPERFECT Quotation withheld because this record is not cleared for redistribution. The proposed set of guiding questions is acknowledged to be incomplete. what this limits: the set of guiding questions: WE PROVIDE A SET OF QUESTIONS, THOUGH UNAVOIDABLY INCOMPLETE, THAT CAN SERVE AS A GUIDE FOR THINKING THROUGH CHALLENGES IN EACH STEP OF THE RESEARCH PROCESS Quotation withheld because this record is not cleared for redistribution. The authors position computational text analysis as an addition to, not a substitute for, other analytic approaches — i.e. it cannot answer such questions alone. what this limits: replacement of other methods, named as the use it does not support: COMPUTATIONAL TEXT ANALYSIS IS NOT A REPLACEMENT FOR BUT RATHER AN ADDITION TO THE APPROACHES ONE CAN TAKE TO ANALYZE SOCIAL AND CULTURAL PHENOMENA USING TEXTUAL DATA Quotation withheld because this record is not cleared for redistribution. The guidance is offered as the authors' own experience and explicitly imperfect duplicate of: limitations[1] · what this limits: the same scope as limitations[1] - the guidance, from experience and INHERENTLY IMPERFECT Quotation withheld because this record is not cleared for redistribution. For social and cultural concepts there is no ground truth to appeal to, and definitions are contested and change what this limits: validation of any social or cultural concept measure: FOR SOCIAL AND CULTURAL CONCEPTS THERE IS NO ABSOLUTE GROUND TRUTH Quotation withheld because this record is not cleared for redistribution. An operationalisation is never a faithful rendering of the concept it stands for what this limits: every measured construct in the paper and the genre: OPERATIONALIZATIONS ARE NEVER PERFECT TRANSLATIONS, AND ARE OFTEN REFINED OVER THE COURSE OF AN INVESTIGATION Quotation withheld because this record is not cleared for redistribution. Many of the concepts at issue cannot be observed at all, which is what makes validation hard rather than merely laborious what this limits: validation in the humanities and social sciences specifically: MANY RELEVANT CONCEPTS ARE FUNDAMENTALLY UNOBSERVABLE, SUCH AS LATENT TRAITS OF POLITICAL ACTORS - unlike tasks such as SPAM FILTERING Quotation withheld because this record is not cleared for redistribution. Human labels are not neutral either - they carry the annotators' priors and perspectives what this limits: any human-labelled ground truth: WHEN THE GROUND TRUTH COMES FROM PEOPLE, IT MAY BE INFLUENCED BY IDEOLOGICAL PRIORS, PRIORS, PRIMING, SIMPLE DIFFERENCES OF OPINION OR PERSPECTIVE Quotation withheld because this record is not cleared for redistribution. The research process is presented linearly but is not, and later stages feed back into the question itself what this limits: the paper's own linear presentation of the research process: THE RESEARCH PROCESS SOUNDS MORE OR LESS LINEAR THIS WAY, BUT EACH OF THESE PHASES OVERLAPS, AND IN SOME INSTANCES REQUIRES US TO RETURN TO PREVIOUS STEPS Quotation withheld because this record is not cleared for redistribution. A poor operationalisation does not merely add noise - it can produce conclusions that are shallow rather than wrong what this limits: conclusions drawn from an operationalisation: CHOICES REGARDING HOW TO OPERATIONALIZE AND ANALYZE THESE CONCEPTS CAN RAISE SERIOUS Quotation withheld Quotation withheld because this record is not cleared for redistribution. Adcock and Collier's background/systematized concept distinction and their notion of content validity, imported to discipline the conceptualization step of computational text analysis Quotation withheld because this record is not cleared for redistribution. Close reading carried into computational work not as a rival but as a validation step that informs the model Quotation withheld because this record is not cleared for redistribution. ground truth ground truth Quotation withheld because this record is not cleared for redistribution. gold standard gold standard Quotation withheld because this record is not cleared for redistribution. hate speech hate speech Quotation withheld because this record is not cleared for redistribution. insight driven insight driven Quotation withheld because this record is not cleared for redistribution. automating tasks automating tasks Quotation withheld because this record is not cleared for redistribution. Domain experts Domain experts Quotation withheld because this record is not cleared for redistribution. dual use dual use Quotation withheld because this record is not cleared for redistribution. Born-digital data Born-digital data Quotation withheld because this record is not cleared for redistribution. unobtrusive measurements unobtrusive measurements Quotation withheld because this record is not cleared for redistribution. privacy privacy Quotation withheld because this record is not cleared for redistribution. provenance provenance Quotation withheld because this record is not cleared for redistribution. black box APIs black box APIs Quotation withheld because this record is not cleared for redistribution. data biases data biases Quotation withheld because this record is not cleared for redistribution. spurious results spurious results Quotation withheld because this record is not cleared for redistribution. generalizability generalizability Quotation withheld because this record is not cleared for redistribution. “comparison” set “comparison” set Quotation withheld because this record is not cleared for redistribution. matching design matching design Quotation withheld because this record is not cleared for redistribution. control group control group Quotation withheld because this record is not cleared for redistribution. counterfactual scenario counterfactual scenario Quotation withheld because this record is not cleared for redistribution. metadata metadata Quotation withheld because this record is not cleared for redistribution. Discretization Discretization Quotation withheld because this record is not cleared for redistribution. classification scheme classification scheme Quotation withheld because this record is not cleared for redistribution. Supervised learning Supervised learning Quotation withheld because this record is not cleared for redistribution. unsupervised learning unsupervised learning Quotation withheld because this record is not cleared for redistribution. natural structure natural structure Quotation withheld because this record is not cleared for redistribution. unit of text unit of text Quotation withheld because this record is not cleared for redistribution. unit of analysis unit of analysis Quotation withheld because this record is not cleared for redistribution. interpretability interpretability Quotation withheld because this record is not cleared for redistribution. human coders human coders Quotation withheld because this record is not cleared for redistribution. annotation examples annotation examples Quotation withheld because this record is not cleared for redistribution. expert annotators expert annotators Quotation withheld because this record is not cleared for redistribution. intra -annotator agreement intra -annotator agreement Quotation withheld because this record is not cleared for redistribution. Disagreement between annotators Disagreement between annotators Quotation withheld because this record is not cleared for redistribution. racial bias racial bias Quotation withheld because this record is not cleared for redistribution. pre-processing steps pre-processing steps Quotation withheld because this record is not cleared for redistribution. OCR errors OCR errors Quotation withheld because this record is not cleared for redistribution. Tokenization Tokenization Quotation withheld because this record is not cleared for redistribution. Multi-word terms Multi-word terms Quotation withheld because this record is not cleared for redistribution. normalization normalization Quotation withheld because this record is not cleared for redistribution. word pieces word pieces Quotation withheld because this record is not cleared for redistribution. stoplist stoplist Quotation withheld because this record is not cleared for redistribution. Dictionaries Dictionaries Quotation withheld because this record is not cleared for redistribution. domain-specific dictionaries domain-specific dictionaries Quotation withheld because this record is not cleared for redistribution. spurious features spurious features Quotation withheld because this record is not cleared for redistribution. post-hoc explanations post-hoc explanations Quotation withheld because this record is not cleared for redistribution. Topic models Topic models Quotation withheld because this record is not cleared for redistribution. number of topics number of topics Quotation withheld because this record is not cleared for redistribution. stratified random sample stratified random sample Quotation withheld because this record is not cleared for redistribution. synthetic data synthetic data Quotation withheld because this record is not cleared for redistribution. explanatory power explanatory power Quotation withheld because this record is not cleared for redistribution. predictive performance predictive performance Quotation withheld because this record is not cleared for redistribution. opportunities for interpretation opportunities for interpretation Quotation withheld because this record is not cleared for redistribution. qualitative analyses qualitative analyses Quotation withheld because this record is not cleared for redistribution. working across disciplines working across disciplines Quotation withheld because this record is not cleared for redistribution. qualitative insights qualitative insights Quotation withheld because this record is not cleared for redistribution. reliable results reliable results Quotation withheld because this record is not cleared for redistribution. reproducibility efforts reproducibility efforts Quotation withheld because this record is not cleared for redistribution. needs of the user needs of the user Quotation withheld because this record is not cleared for redistribution. Computational text analysis Computational text analysis Quotation withheld because this record is not cleared for redistribution. models that are already somewhat interpretable models that are already somewhat interpretable Quotation withheld because this record is not cleared for redistribution. Born-digital data → unobtrusive measurements Because born-digital sources do not require direct elicitation, they enable unobtrusive measurement. Enables is absent from the governed types, and the gate correctly refused a causes edge without its governed effect cue. Quotation withheld because this record is not cleared for redistribution. Domain experts → qualitative insights Domain experts identify qualitative insights that can strengthen computational analysis and enrich the answer. Advises/contributes is not a governed relation type. Quotation withheld because this record is not cleared for redistribution. black box APIs → data biases Closed commercial APIs are likely to introduce biases whose form and severity researchers cannot inspect. Introduces-risk-of is not a governed type, and the wording is probabilistic. Quotation withheld because this record is not cleared for redistribution. matching design → control group The matching design populates the control group with untreated forums similar to the treated ones. Constructs/selects is absent from the governed types. Quotation withheld because this record is not cleared for redistribution. control group → counterfactual scenario The matched control group is used to estimate what would have happened without the closures. Estimates-counterfactual is absent and should not be forced into operationalised-as without its governed cue. Quotation withheld because this record is not cleared for redistribution. Discretization → validity Overly broad discretized categories can threaten validity. Threatens is absent from the governed types, and deterministic causation would overstate 'can'. Quotation withheld because this record is not cleared for redistribution. unit of text → unit of analysis The labeled unit of text can differ from the final unit of analysis. Differs-in-level-from is not captured by the governed identity or distinction types. Quotation withheld because this record is not cleared for redistribution. expert annotators → reliable results Highly trained or expert annotators tend to produce more reliable results for complex concepts. Suitable-for/tends-to-improve is absent and explicitly hedged. Quotation withheld because this record is not cleared for redistribution. pre-processing steps → reproducibility efforts Preprocessing choices have a large impact on reproducibility efforts. Affects is absent from the governed relation types. Quotation withheld because this record is not cleared for redistribution. post-hoc explanations → models that are already somewhat interpretable Post-hoc explanations are an alternative to restricting analysis to models that are already somewhat interpretable, but may be unfaithful. Alternative-to-with-fidelity-risk is absent. Quotation withheld because this record is not cleared for redistribution. number of topics → needs of the user The appropriate topic count is determined by the needs of the user rather than the collection. Determined-by-analytical-purpose is absent from the governed types. Quotation withheld because this record is not cleared for redistribution. explanatory power → predictive performance For insight-driven studies, improving explanatory power can be more useful than small predictive-performance gains. Preferred-over-for-purpose is absent from the governed types. Quotation withheld because this record is not cleared for redistribution. qualitative analyses → Computational text analysis Moving between large-scale computational text analysis and small-scale qualitative analysis combines their strengths. Complements is absent from the governed types. Quotation withheld because this record is not cleared for redistribution. Tokenization → Multi-word terms Deciding whether multi-word terms are tokenized as one unit can dramatically alter textual patterns. Treatment-of is absent and causation would misstate the relevant agent. Quotation withheld because this record is not cleared for redistribution. concept - BACKGROUND: everything the term has meant across the literature, contested and changing Quotation withheld because this record is not cleared for redistribution. concept - SYSTEMATIZED: the one definition this study adopts, chosen for its question and framework Quotation withheld because this record is not cleared for redistribution. accuracy - as VALIDITY: what it is usually presented as, and what it cannot be without an objective standard Quotation withheld because this record is not cleared for redistribution. accuracy - as RELIABILITY: consistency against a human sample, which is what it actually measures when no ground truth exists Quotation withheld because this record is not cleared for redistribution. ambiguity - in computational linguistics: a problem to be solved, on the assumption that one correct interpretation exists Quotation withheld because this record is not cleared for redistribution. ambiguity - in literary criticism: prized, alongside interpretation and argumentation, above definitive conclusions Quotation withheld because this record is not cleared for redistribution. annotator disagreement - as NOISE: a failure of the scheme or the coders, to be reduced Quotation withheld because this record is not cleared for redistribution. annotator disagreement - as SIGNAL: a property of the concept, which can be carried into the analysis rather than resolved away Quotation withheld because this record is not cleared for redistribution. Insight-driven analysis - describes or explains a phenomenon rather than optimizing a prediction task Quotation withheld because this record is not cleared for redistribution. Dual use - a protective hate-speech method may also be adapted to suppress marginalized speech or dissent Quotation withheld because this record is not cleared for redistribution. Born-digital evidence - massive and unobtrusive, yet ethically constrained by contextual privacy and absent consent Quotation withheld because this record is not cleared for redistribution. Black-box collection - commercial APIs can introduce unknown biases into research inference Quotation withheld because this record is not cleared for redistribution. Restricted generalization - biased data can support within-population comparison when claims stay within that population Quotation withheld because this record is not cleared for redistribution. Counterfactual control - matched untreated forums estimate what would have happened without platform closure Quotation withheld because this record is not cleared for redistribution. Discretization tradeoff - categories facilitate nonlinear analysis but overly broad bins threaten validity Quotation withheld because this record is not cleared for redistribution. Supervised versus unsupervised learning - labels learned from annotated data versus structure learned from unlabeled data Quotation withheld because this record is not cleared for redistribution. Annotation disagreement - can reveal either scheme weaknesses or inherent ambiguity and may itself be retained as signal Quotation withheld because this record is not cleared for redistribution. Preprocessing provenance - operationally consequential choices should be documented like editorial or biomedical protocols Quotation withheld because this record is not cleared for redistribution. Tokenization - seemingly basic segmentation that can fail on creative orthography and transform multi-word patterns Quotation withheld because this record is not cleared for redistribution. Word-piece segmentation - effective for prediction but potentially unsuitable when linguistic-unit interpretation matters Quotation withheld because this record is not cleared for redistribution. Domain dictionary construction - automation proposes candidates and human theory filters spurious features Quotation withheld because this record is not cleared for redistribution. Post-hoc explanation - interprets individual predictions without guaranteeing fidelity to the underlying model Quotation withheld because this record is not cleared for redistribution. Topic count - a user-selected analytical scale like map scale or microscope magnification, not an intrinsic truth Quotation withheld because this record is not cleared for redistribution. Content revalidation - jointly interrogates concept, indicators, and scores and may force recoding or method replacement Quotation withheld because this record is not cleared for redistribution. Computational and qualitative analysis - complementary scales for trends and individual stories Quotation withheld because this record is not cleared for redistribution. claim_type: method a second reading reaching the same value. Recorded: method — note: a second reading reaching the same value from a different sentence. Not applied - a person decides. Quotation withheld because this record is not cleared for redistribution. phenomenon: computational text analysis of social and cultural concepts a second reading reaching the same value. Recorded: computational text analysis of social and cultural concepts — note: a second reading reaching the same value from a different sentence. Not applied - a person decides. Quotation withheld because this record is not cleared for redistribution. builds-on → natural language processing (computational text methods) Quotation withheld because this record is not cleared for redistribution. builds-on → Bender and Friedman (2018), Gebru et al. (2018) on dataset documentation and Mitchell et al. (2019) on model documentation Quotation withheld because this record is not cleared for redistribution. critiques → the gold-standard framing of evaluation when applied to contested social and cultural concepts Quotation withheld because this record is not cleared for redistribution. extends → Adcock and Collier (2001) on measurement validity in political science Quotation withheld because this record is not cleared for redistribution. supports → Piper's account of validation as further discovery in two directions Quotation withheld because this record is not cleared for redistribution. extends → the datasheets and model-cards line of work on documenting datasets and models Quotation withheld because this record is not cleared for redistribution. gold standard —same-as→ ground truth gold standard ground truth strength of the relation: 1 Quotation withheld because this record is not cleared for redistribution. ground truth —distinct-from→ unobservable ground truth unobservable strength of the relation: 1 Audit 2026-09-28: DOUBTFUL - read against its span and kept, but the span calls some concepts fundamentally unobservable; distinct-from ground truth is inferred. Quotation withheld because this record is not cleared for redistribution. reliability —distinct-from→ validity reliability validity strength of the relation: 1 Audit 2026-09-28: DOUBTFUL - read against its span and kept, but the span defines reliability only; the contrast with validity is not in it. Quotation withheld because this record is not cleared for redistribution. inter-annotator agreement —operationalised-as→ reliability inter-annotator agreement reliability strength of the relation: 1 Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but the span glosses inter-annotator agreement as 'intercoder reliability'; operationalised-as is loose. Quotation withheld because this record is not cleared for redistribution. Krippendorff's alpha —operationalised-as→ inter-annotator agreement Krippendorff's alpha inter-annotator agreement strength of the relation: 1 Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but endpoint by anaphora: the span answers 'how inter-annotator agreement will be measured' (previous sentence); operationalised-as is loose. Quotation withheld because this record is not cleared for redistribution. content validity —operationalised-as→ annotation scheme content validity annotation scheme strength of the relation: 1 Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but the span writes 'the codebook'; 'annotation scheme' is a variant, and operationalised-as is loose. Quotation withheld because this record is not cleared for redistribution. close reading —operationalised-as→ validity close reading validity strength of the relation: 0.9 Audit 2026-09-28: DOUBTFUL - read against its span and kept, but the span says validation takes the form of close reading for some; the direction of operationalised-as is reversed. Quotation withheld because this record is not cleared for redistribution. ambiguity —contrasted-with→ ground truth ambiguity ground truth strength of the relation: 1 Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but attributed: the span quotes Kirschenbaum. Quotation withheld because this record is not cleared for redistribution. deductive —part-of→ annotation scheme deductive annotation scheme strength of the relation: 1 Audit 2026-09-28: DOUBTFUL - read against its span and kept, but the span describes deductive coding rules; part-of the annotation scheme is inferred. Quotation withheld because this record is not cleared for redistribution. insight driven —contrasted-with→ automating tasks insight driven automating tasks the relation itself is separately evidenced · strength of the relation: 1 · note: The article explicitly contrasts insight-driven social inquiry with task automation and prediction. Quotation withheld because this record is not cleared for redistribution. data biases —causes→ spurious results data biases spurious results the relation itself is separately evidenced · strength of the relation: 0.5 — the record marks this weak · note: The paper says biases may lead to spurious results; weight reflects 'may'. Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but the span says data biases MAY lead to spurious results; hedged. Quotation withheld because this record is not cleared for redistribution. Supervised learning —distinct-from→ unsupervised learning Supervised learning unsupervised learning the relation itself is separately evidenced · strength of the relation: 1 · note: The contrast is labeled data versus unlabeled data. Quotation withheld because this record is not cleared for redistribution. 0.80 or above — the conventional threshold for acceptable intercoder agreement in communication research Quotation withheld because this record is not cleared for redistribution. 450 18th-century novels — the corpus in which close reading beside model output found an unrecognised genre Quotation withheld because this record is not cleared for redistribution. A human reader is better equipped than a computer to make logical inferences, resolve ambiguities and apply cultural knowledge. Quotation withheld because this record is not cleared for redistribution. a concept often has several valid definitions, and definitions may be contested over time Quotation withheld because this record is not cleared for redistribution. validity then has to be judged by other techniques Quotation withheld because this record is not cleared for redistribution. this kind of validation is rare in NLP, yet matters most when it is hard to tell what drives a model Quotation withheld because this record is not cleared for redistribution. disagreement between annotators can reveal weaknesses in the annotation scheme or the inherent ambiguity of what is being measured Quotation withheld because this record is not cleared for redistribution. annotation carries bias: tweets in African American English, and by self-identified African Americans, were more often labelled offensive Quotation withheld because this record is not cleared for redistribution. close reading of 450 eighteenth-century novels alongside model output revealed a genre absent from literary history's own categories Quotation withheld because this record is not cleared for redistribution. conceptual validity conceptual validity Quotation withheld because this record is not cleared for redistribution. ground truth ground truth Quotation withheld because this record is not cleared for redistribution. background concept background concept Quotation withheld because this record is not cleared for redistribution. systematized concept systematized concept Quotation withheld because this record is not cleared for redistribution. reliability reliability Quotation withheld because this record is not cleared for redistribution. validity validity Quotation withheld because this record is not cleared for redistribution. annotation scheme annotation scheme Quotation withheld because this record is not cleared for redistribution. content validity content validity Quotation withheld because this record is not cleared for redistribution. sample-size sample-size Quotation withheld because this record is not cleared for redistribution. Krippendorff's alpha Krippendorff's alpha Quotation withheld because this record is not cleared for redistribution. inter-annotator agreement inter-annotator agreement Quotation withheld because this record is not cleared for redistribution. F-score F-score Quotation withheld because this record is not cleared for redistribution.

The full text of this article is not redistributable, so it is not included here. The evidence sentences are withheld with it; the tag layer and its grounding status remain. The article itself is at https://doi.org/10.3389/frai.2020.00062.