phenomenon · claim-type · cross-domain-transfer · mechanism · claim-relation · replication-status · limitation — grounded in the full text| phenomenon | computational text analysis of social and cultural concepts— “In this article we describe our experiences with computational text analysis involving rich social and cultural concepts.” | |
| claim-type | method— “Second, we hope to provide a set of key questions that can guide work with thick social and cultural concepts.” | |
| method — evidence | — “In this article we have consolidated our experiences, as scholars from very different disciplines, in analyzing text as social and cultural data and described how the research process often unfolds.” | |
| object of study — evidence | — “In this article, we take the reader through a typical research process that involves measuring social or cultural concepts using computational methods, discussing both the opportunities and complications that often arise.” | |
| transfer | from political science: Adcock and Collier's background/systematized concept distinction and their notion of content validity, imported to discipline the conceptualization step of computational text analysis— “If we want to move beyond the use of ad hoc definitions, it can be useful to distinguish between what political scientists Adcock and Collier ( 2001 ) call the “background concept” and the “systematized concept.”” | transfers[0] |
| transfer | from literary studies: Close reading carried into computational work not as a rival but as a validation step that informs the model— “Model outputs tell us something about the texts, while a close reading of the texts alongside those outputs tells us something about the models that can be used for more effective model building.” | transfers[1] |
| mechanism | A concept must be narrowed from the full range of meanings it carries to the single formulation a study adopts, and those are two different objects with two different names— “The background concept comprises the full and diverse set of meanings that might be associated with a particular term.” | mechanisms[0] |
| mechanism | Computation earns its place where patterns exist only in aggregate and are invisible in any single case— “Moreover, many patterns are not obvious in any specific context, but only stand out in the aggregate.” | mechanisms[1] |
| mechanism | Where there is no objective standard, accuracy against annotations measures consistency rather than correctness— “Accuracy, as well as other measures such as precision, recall and F-score, are sometimes presented as a measure of validity, but if we do not have a genuinely objective determination of what something is supposed measure—as is often the case in text analysis—then accuracy is perhaps a better indication of reliability than of validity.” | mechanisms[2] |
| mechanism | Inter-annotator reliability differs from accuracy in taking no coder as correct and discounting agreement expected by chance— “In contrast, when the reliability is measured based on the output of different annotators, no coder is taken as the standard and the likelihood of coders reaching agreement by chance (rather than because they are “correct”) is factored into the resulting statistic.” | mechanisms[3] |
| mechanism | Validation without a gold standard proceeds by sampling across the full range of scores and asking whether each case makes sense under the adopted definition— “That is, take a stratified random sample, selecting observations from the full range of scores, and ask: Do these make sense in light of the systematized concept?” | mechanisms[4] |
| mechanism | Reading the texts beside the model output informs the model as much as the model informs the reading— “Model outputs tell us something about the texts, while a close reading of the texts alongside those outputs tells us something about the models that can be used for more effective model building.” | mechanisms[5] |
| mechanism | Annotator disagreement can be a property of the concept rather than a defect of the annotators, and can be carried forward instead of resolved— “Disagreement itself can be meaningful and can be integrated in subsequent analyses (Aroyo and Welty, 2013 ; Demeester et al., 2016 ).” | mechanisms[6] |
| mechanism | A METHODS GUIDE THAT MAKES `FIND THE COMPETING DEFINITIONS` A REQUIRED STEP, IN A CHECKLIST: `Conceptualization: What is this all about? What are the core concepts you are addressing? And are you being true to their core meaning? What are competing definitions? Which is best suited to the task and why? Does the systematized concept you've selected reflect an adequate understanding of the background concept? How do domain experts approach the topic?` This is Adcock and Collier's vocabulary turned into a procedure a researcher is told to follow before touching data - and it is the closest thing in the corpus to a specification of what this project does.— “Conceptualization: What is this all about? What are the core concepts you are addressing? And are you being true to their core meaning? What are competing definitions? Which is best suited to the task and why? Does the systematized concept you've selected reflect an adequate understanding of the background concept? How do domain experts… | mechanisms[7] |
| builds-on | natural language processing (computational text methods)— “the backbone of natural language processing” | claim_relations[0] |
| builds-on | Bender and Friedman (2018), Gebru et al. (2018) on dataset documentation and Mitchell et al. (2019) on model documentation— “These questions complement recent work providing guidance and suggestions about datasets (Bender and Friedman, 2018 ; Gebru et al., 2018 ) and models (Mitchell et al., 2019 ) more specifically.” | claim_relations[1] |
| critiques | the gold-standard framing of evaluation when applied to contested social and cultural concepts— “However, the notion of ground truth is uncommon in the humanities and social sciences and it is often taken too far in machine learning.” | claim_relations[2] |
| extends | Adcock and Collier (2001) on measurement validity in political science— “If we want to move beyond the use of ad hoc definitions, it can be useful to distinguish between what political scientists Adcock and Collier ( 2001 ) call the “background concept” and the “systematized concept.”” | claim_relations[3] |
| supports | Piper's account of validation as further discovery in two directions— “Applying this circular, iterative process to 450 18th-century novels written in three languages, Piper was able to uncover a new form of “ conversional novel ” that was not previously captured in “ literary history's received critical categories ” (Piper, 2015 , p. 92).” | claim_relations[4] |
| extends | the datasheets and model-cards line of work on documenting datasets and models— “These questions complement recent work providing guidance and suggestions about datasets (Bender and Friedman, 2018 ; Gebru et al., 2018 ) and models (Mitchell et al., 2019 ) more specifically.” | claim_relations[5] |
| sense | concept - BACKGROUND: everything the term has meant across the literature, contested and changing— “The background concept comprises the full and diverse set of meanings that might be associated with a particular term.” | senses[0] |
| sense | concept - SYSTEMATIZED: the one definition this study adopts, chosen for its question and framework— “That definition, in turn, represents the systematized concept: the formulation that is adopted for the study.” | senses[1] |
| sense | accuracy - as VALIDITY: what it is usually presented as, and what it cannot be without an objective standard— “Accuracy, as well as other measures such as precision, recall and F-score, are sometimes presented as a measure of validity, but if we do not have a genuinely objective determination of what something is supposed measure—as is often the case in text analysis—then accuracy is perhaps a better indication of reliability than of validity.” | senses[2] |
| sense | accuracy - as RELIABILITY: consistency against a human sample, which is what it actually measures when no ground truth exists— “In NLP and machine learning, most models are primarily evaluated by comparing the machine-generated labels against an annotated sample.” | senses[3] |
| sense | ambiguity - in computational linguistics: a problem to be solved, on the assumption that one correct interpretation exists— “In Computational Linguistics [.] ambiguity is almost uniformly treated as a problem to be solved; the focus is on disambiguation, with the assumption that one true, correct interpretation exists .” | senses[4] |
| sense | ambiguity - in literary criticism: prized, alongside interpretation and argumentation, above definitive conclusions— “Kirschenbaum ( 2007 , p. 1) notes that in literary criticism and the digital humanities more broadly “ interpretation, ambiguity, and argumentation are prized far above ground truth and definitive conclusions."” | senses[5] |
| sense | annotator disagreement - as NOISE: a failure of the scheme or the coders, to be reduced— “Disagreement between annotators can signal weaknesses of the annotation scheme, or highlight the inherent ambiguity in what we are trying to measure.” | senses[6] |
| sense | annotator disagreement - as SIGNAL: a property of the concept, which can be carried into the analysis rather than resolved away— “Disagreement itself can be meaningful and can be integrated in subsequent analyses (Aroyo and Welty, 2013 ; Demeester et al., 2016 ).” | senses[7] |
| sense | Insight-driven analysis - describes or explains a phenomenon rather than optimizing a prediction task— “Computational analysis of text motivated by these questions is insight driven: we aim to describe a phenomenon or explain how it came about. For example, what can we learn about how and why hate speech is used or how this changes over time? Is hate speech one thing, or does it comprise multiple forms of expression? Is there a clear boundary… | senses[8] |
| sense | Dual use - a protective hate-speech method may also be adapted to suppress marginalized speech or dissent— “Questions about potential “dual use” may also arise. Returning to our introductory example, Chandrasekharan et al. ( 2017 ) started with a deceptively simple question: if an internet platform eliminates forums for hate speech, does this impact hate speech in other forums? The research was motivated by the belief that a rising tide of online… | senses[9] |
| sense | Born-digital evidence - massive and unobtrusive, yet ethically constrained by contextual privacy and absent consent— “The use of born-digital data raises ethical concerns. Although early studies often treated privacy as a binary construct, many now acknowledge its complexity (danah boyd and Crawford, 2012 ). Conversations on private matters can be posted online, visible for all, but social norms regarding what should be considered public information may… | senses[10] |
| sense | Black-box collection - commercial APIs can introduce unknown biases into research inference— “Working with data that someone else has acquired presents additional problems related to provenance and contextualization. It may not always be possible to determine the criteria applied during the creation process. For example, why were certain newspapers digitized but not others, and what does this say about the collection? Similar… | senses[11] |
| sense | Restricted generalization - biased data can support within-population comparison when claims stay within that population | senses[12] |
| sense | Counterfactual control - matched untreated forums estimate what would have happened without platform closure | senses[13] |
| sense | Discretization tradeoff - categories facilitate nonlinear analysis but overly broad bins threaten validity Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | senses[14] |
| sense | Supervised versus unsupervised learning - labels learned from annotated data versus structure learned from unlabeled data Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | senses[15] |
| sense | Annotation disagreement - can reveal either scheme weaknesses or inherent ambiguity and may itself be retained as signal Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | senses[16] |
| sense | Preprocessing provenance - operationally consequential choices should be documented like editorial or biomedical protocols Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | senses[17] |
| sense | Tokenization - seemingly basic segmentation that can fail on creative orthography and transform multi-word patterns Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | senses[18] |
| sense | Word-piece segmentation - effective for prediction but potentially unsuitable when linguistic-unit interpretation matters Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | senses[19] |
| sense | Domain dictionary construction - automation proposes candidates and human theory filters spurious features Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | senses[20] |
| sense | Post-hoc explanation - interprets individual predictions without guaranteeing fidelity to the underlying model Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | senses[21] |
| sense | Topic count - a user-selected analytical scale like map scale or microscope magnification, not an intrinsic truth Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | senses[22] |
| sense | Content revalidation - jointly interrogates concept, indicators, and scores and may force recoding or method replacement Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | senses[23] |
| sense | Computational and qualitative analysis - complementary scales for trends and individual stories Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | senses[24] |
| edge | gold standard —same-as→ ground truth Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concept_edges[2] |
| audit | ground truth —distinct-from→ unobservable — Audit 2026-09-28: DOUBTFUL - read against its span and kept, but the span calls some concepts fundamentally unobservable; distinct-from ground truth is inferred. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concept_edges[3] |
| audit | reliability —distinct-from→ validity — Audit 2026-09-28: DOUBTFUL - read against its span and kept, but the span defines reliability only; the contrast with validity is not in it. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concept_edges[4] |
| audit | inter-annotator agreement —operationalised-as→ reliability — Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but the span glosses inter-annotator agreement as 'intercoder reliability'; operationalised-as is loose. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concept_edges[5] |
| audit | Krippendorff's alpha —operationalised-as→ inter-annotator agreement — Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but endpoint by anaphora: the span answers 'how inter-annotator agreement will be measured' (previous sentence); operationalised-as is loose. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concept_edges[6] |
| audit | content validity —operationalised-as→ annotation scheme — Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but the span writes 'the codebook'; 'annotation scheme' is a variant, and operationalised-as is loose. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concept_edges[7] |
| audit | close reading —operationalised-as→ validity — Audit 2026-09-28: DOUBTFUL - read against its span and kept, but the span says validation takes the form of close reading for some; the direction of operationalised-as is reversed. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concept_edges[8] |
| weight | close reading —operationalised-as→ validity — 0.9 Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concept_edges[8] |
| audit | ambiguity —contrasted-with→ ground truth — Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but attributed: the span quotes Kirschenbaum. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concept_edges[9] |
| audit | deductive —part-of→ annotation scheme — Audit 2026-09-28: DOUBTFUL - read against its span and kept, but the span describes deductive coding rules; part-of the annotation scheme is inferred. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concept_edges[11] |
| note | insight driven —contrasted-with→ automating tasks — The article explicitly contrasts insight-driven social inquiry with task automation and prediction. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concept_edges[14] |
| audit | data biases —causes→ spurious results — Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but the span says data biases MAY lead to spurious results; hedged. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concept_edges[15] |
| note | data biases —causes→ spurious results — The paper says biases may lead to spurious results; weight reflects 'may'. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concept_edges[15] |
| weight | data biases —causes→ spurious results — 0.5 Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concept_edges[15] |
| note | Supervised learning —distinct-from→ unsupervised learning — The contrast is labeled data versus unlabeled data. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concept_edges[16] |
| for which use | limitation — any computational operationalisation of a rich concept: it raises CONCERNS ABOUT CONCEPTUAL VALIDITY AND MAY LEAD TO SHALLOW OR OBVIOUS CONCLUSIONS Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[0] |
| for which use | limitation — the guidance this paper offers: OUR GUIDANCE IS BASED ON OUR OWN EXPERIENCES AND IS THEREFORE INHERENTLY IMPERFECT Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[1] |
| for which use | limitation — the set of guiding questions: WE PROVIDE A SET OF QUESTIONS, THOUGH UNAVOIDABLY INCOMPLETE, THAT CAN SERVE AS A GUIDE FOR THINKING THROUGH CHALLENGES IN EACH STEP OF THE RESEARCH PROCESS Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[2] |
| for which use | limitation — replacement of other methods, named as the use it does not support: COMPUTATIONAL TEXT ANALYSIS IS NOT A REPLACEMENT FOR BUT RATHER AN ADDITION TO THE APPROACHES ONE CAN TAKE TO ANALYZE SOCIAL AND CULTURAL PHENOMENA USING TEXTUAL DATA Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[3] |
| duplicate of | limitation — limitations[1] Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[4] |
| for which use | limitation — the same scope as limitations[1] - the guidance, from experience and INHERENTLY IMPERFECT Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[4] |
| for which use | limitation — validation of any social or cultural concept measure: FOR SOCIAL AND CULTURAL CONCEPTS THERE IS NO ABSOLUTE GROUND TRUTH Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[5] |
| for which use | limitation — every measured construct in the paper and the genre: OPERATIONALIZATIONS ARE NEVER PERFECT TRANSLATIONS, AND ARE OFTEN REFINED OVER THE COURSE OF AN INVESTIGATION Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[6] |
| for which use | limitation — validation in the humanities and social sciences specifically: MANY RELEVANT CONCEPTS ARE FUNDAMENTALLY UNOBSERVABLE, SUCH AS LATENT TRAITS OF POLITICAL ACTORS - unlike tasks such as SPAM FILTERING Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[7] |
| for which use | limitation — any human-labelled ground truth: WHEN THE GROUND TRUTH COMES FROM PEOPLE, IT MAY BE INFLUENCED BY IDEOLOGICAL PRIORS, PRIORS, PRIMING, SIMPLE DIFFERENCES OF OPINION OR PERSPECTIVE Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[8] |
| for which use | limitation — the paper's own linear presentation of the research process: THE RESEARCH PROCESS SOUNDS MORE OR LESS LINEAR THIS WAY, BUT EACH OF THESE PHASES OVERLAPS, AND IN SOME INSTANCES REQUIRES US TO RETURN TO PREVIOUS STEPS Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[9] |
| for which use | limitation — conclusions drawn from an operationalisation: CHOICES REGARDING HOW TO OPERATIONALIZE AND ANALYZE THESE CONCEPTS CAN RAISE SERIOUS CONCERNS ABOUT CONCEPTUAL VALIDITY AND MAY LEAD TO SHALLOW OR OBVIOUS CONCLUSIONS Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[10] |
| weight | Domain experts → qualitative insights — 0.7 Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | unhoused_relations[1] |
| weight | black box APIs → data biases — 0.7 Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | unhoused_relations[2] |
| weight | Discretization → validity — 0.7 Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | unhoused_relations[5] |
| weight | unit of text → unit of analysis — 0.7 Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | unhoused_relations[6] |
| weight | expert annotators → reliable results — 0.7 Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | unhoused_relations[7] |
| weight | post-hoc explanations → models that are already somewhat interpretable — 0.5 Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | unhoused_relations[9] |
| weight | explanatory power → predictive performance — 0.7 Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | unhoused_relations[11] |
| weight | qualitative analyses → Computational text analysis — 0.7 Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | unhoused_relations[12] |
| weight | Tokenization → Multi-word terms — 0.7 Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | unhoused_relations[13] |
| replication | untested | |
| no type for this | Born-digital data → unobtrusive measurements — Because born-digital sources do not require direct elicitation, they enable unobtrusive measurement. Enables is absent from the governed types, and the gate correctly refused a causes edge without its governed effect cue. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | unhoused_relations[0] |
| no type for this | Domain experts → qualitative insights — Domain experts identify qualitative insights that can strengthen computational analysis and enrich the answer. Advises/contributes is not a governed relation type. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | unhoused_relations[1] |
| no type for this | black box APIs → data biases — Closed commercial APIs are likely to introduce biases whose form and severity researchers cannot inspect. Introduces-risk-of is not a governed type, and the wording is probabilistic. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | unhoused_relations[2] |
| no type for this | matching design → control group — The matching design populates the control group with untreated forums similar to the treated ones. Constructs/selects is absent from the governed types. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | unhoused_relations[3] |
| no type for this | control group → counterfactual scenario — The matched control group is used to estimate what would have happened without the closures. Estimates-counterfactual is absent and should not be forced into operationalised-as without its governed cue. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | unhoused_relations[4] |
| no type for this | Discretization → validity — Overly broad discretized categories can threaten validity. Threatens is absent from the governed types, and deterministic causation would overstate 'can'. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | unhoused_relations[5] |
| no type for this | unit of text → unit of analysis — The labeled unit of text can differ from the final unit of analysis. Differs-in-level-from is not captured by the governed identity or distinction types. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | unhoused_relations[6] |
| no type for this | expert annotators → reliable results — Highly trained or expert annotators tend to produce more reliable results for complex concepts. Suitable-for/tends-to-improve is absent and explicitly hedged. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | unhoused_relations[7] |
| no type for this | pre-processing steps → reproducibility efforts — Preprocessing choices have a large impact on reproducibility efforts. Affects is absent from the governed relation types. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | unhoused_relations[8] |
| no type for this | post-hoc explanations → models that are already somewhat interpretable — Post-hoc explanations are an alternative to restricting analysis to models that are already somewhat interpretable, but may be unfaithful. Alternative-to-with-fidelity-risk is absent. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | unhoused_relations[9] |
| no type for this | number of topics → needs of the user — The appropriate topic count is determined by the needs of the user rather than the collection. Determined-by-analytical-purpose is absent from the governed types. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | unhoused_relations[10] |
| no type for this | explanatory power → predictive performance — For insight-driven studies, improving explanatory power can be more useful than small predictive-performance gains. Preferred-over-for-purpose is absent from the governed types. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | unhoused_relations[11] |
| no type for this | qualitative analyses → Computational text analysis — Moving between large-scale computational text analysis and small-scale qualitative analysis combines their strengths. Complements is absent from the governed types. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | unhoused_relations[12] |
| no type for this | Tokenization → Multi-word terms — Deciding whether multi-word terms are tokenized as one unit can dramatically alter textual patterns. Treatment-of is absent and causation would misstate the relevant agent. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | unhoused_relations[13] |
| limitation | operationalizing rich concepts computationally can raise conceptual-validity concerns and yield shallow conclusions Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[0] |
| limitation | The guidance is drawn from the authors' own experience and is explicitly acknowledged as imperfect. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[1] |
| limitation | The proposed set of guiding questions is acknowledged to be incomplete. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[2] |
| limitation | The authors position computational text analysis as an addition to, not a substitute for, other analytic approaches — i.e. it cannot answer such questions alone. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[3] |
| limitation | The guidance is offered as the authors' own experience and explicitly imperfect Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[4] |
| limitation | For social and cultural concepts there is no ground truth to appeal to, and definitions are contested and change Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[5] |
| limitation | An operationalisation is never a faithful rendering of the concept it stands for Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[6] |
| limitation | Many of the concepts at issue cannot be observed at all, which is what makes validation hard rather than merely laborious Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[7] |
| limitation | Human labels are not neutral either - they carry the annotators' priors and perspectives Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[8] |
| limitation | The research process is presented linearly but is not, and later stages feed back into the question itself Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[9] |
| limitation | A poor operationalisation does not merely add noise - it can produce conclusions that are shallow rather than wrong Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[10] |
finding · quantity · variable · definition · key-term — grounded in the full text| finding | A human reader is better equipped than a computer to make logical inferences, resolve ambiguities and apply cultural knowledge. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.findings[0] |
| finding | The notion of ground truth is uncommon and risky in the humanities/social sciences: labels often mean something other than expected, threatening conceptual validity. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.findings[1] |
| not machine-vouched | finding — declared-ellipsis - An elided quote: the quote marks its cuts with an ellipsis, and each of its 2 pieces is in our copy of the paper, some differing slightly from our copy in letters, notation or a small word ('the notion of ground truth is uncommon i...'; 'using them as a “ground truth” requires ...'). The joined string is in no copy. | content_tags.findings[1] |
| finding | Interdisciplinary collaboration is essential to realize the potential of text-as-data analysis. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.findings[2] |
| not machine-vouched | finding — not-in-our-copy - The quote is in the original web page we fetched for this paper, but not in the text we tag against, which was extracted differently. Checked with ligatures, hyphens, spacing and punctuation set aside. | content_tags.findings[2] |
| finding | a concept often has several valid definitions, and definitions may be contested over time Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.findings[3] |
| finding | validity then has to be judged by other techniques Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.findings[4] |
| finding | this kind of validation is rare in NLP, yet matters most when it is hard to tell what drives a model Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.findings[5] |
| finding | disagreement between annotators can reveal weaknesses in the annotation scheme or the inherent ambiguity of what is being measured Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.findings[6] |
| finding | annotation carries bias: tweets in African American English, and by self-identified African Americans, were more often labelled offensive Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.findings[7] |
| finding | close reading of 450 eighteenth-century novels alongside model output revealed a genre absent from literary history's own categories Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.findings[8] |
| quantity | 0.80 or above — the conventional threshold for acceptable intercoder agreement in communication research Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.quantities[0] |
| quantity | 450 18th-century novels — the corpus in which close reading beside model output found an unrecognised genre Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.quantities[1] |
| definition | conceptual validity Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.definitions[0] |
| definition | ground truth Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.definitions[1] |
| definition | background concept Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.definitions[2] |
| definition | systematized concept Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.definitions[3] |
| definition | reliability Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.definitions[4] |
| definition | validity Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.definitions[5] |
| definition | annotation scheme Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.definitions[6] |
| definition | content validity Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.definitions[7] |
| key-terms | text, computational, social, humanities, annotation, validity, measurement, interpretation, corpus, background concept, systematized concept, operationalization, conceptual validity, content validity, ground truth, gold standard, reliability, annotation scheme, inter-annotator agreement, Krippendorff's alpha, close reading, ambiguity, thick, unobservable |
| concept | ground truth — not defined in this paper's own words; the assumed correct answer, which the paper argues does not exist for contested concepts Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[10] |
| concept | gold standard — not defined in this paper's own words; its machine-learning name, and the frame this paper says is often taken too far Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[11] |
| concept | hate speech — not defined in this paper's own words; the running example of a contested concept with an external definition used as an anchor Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[20] |
| concept | insight driven — not defined in this paper's own words; analysis aimed at describing or explaining phenomena rather than merely predicting labels Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[25] |
| concept | automating tasks — not defined in this paper's own words; performance-oriented computational work contrasted with theory-led insight generation Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[26] |
| concept | Domain experts — not defined in this paper's own words; collaborators who revise questions and identify where qualitative knowledge strengthens analysis Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[27] |
| concept | dual use — not defined in this paper's own words; risk that a method developed for protective purposes can be repurposed for repression Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[28] |
| concept | Born-digital data — not defined in this paper's own words; naturally occurring digital sources enabling unobtrusive and large-scale study Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[29] |
| concept | unobtrusive measurements — not defined in this paper's own words; observation without direct elicitation that might alter participant responses Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[30] |
| concept | privacy — not defined in this paper's own words; non-binary ethical property not reducible to explicit visibility settings Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[31] |
| concept | provenance — not defined in this paper's own words; origin and creation context that may be inaccessible when reusing acquired data Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[32] |
| concept | black box APIs — not defined in this paper's own words; commercial interfaces whose closed collection process can introduce unknown research bias Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[33] |
| concept | data biases — not defined in this paper's own words; collection distortions that can create spurious results and constrain inference Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[34] |
| concept | spurious results — not defined in this paper's own words; findings induced by biased repurposed data rather than the target phenomenon Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[35] |
| concept | generalizability — not defined in this paper's own words; scope of extrapolation limited by language, target, and population biases Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[36] |
| concept | “comparison” set — not defined in this paper's own words; documents outside a core set needed to establish a distinctive property Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[37] |
| concept | matching design — not defined in this paper's own words; control-selection strategy matching forums to treated communities Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[38] |
| concept | control group — not defined in this paper's own words; untreated comparison forums selected to resemble the closed forums Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[39] |
| concept | counterfactual scenario — not defined in this paper's own words; estimated state in which the platform did not close the specific forums Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[40] |
| concept | metadata — not defined in this paper's own words; information about document creation or reception that can be valuable but error-prone Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[41] |
| concept | Discretization — not defined in this paper's own words; conversion of continuous variables into categories that can aid nonlinear analysis but threaten validity Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[42] |
| concept | classification scheme — not defined in this paper's own words; category system determining which variations are visible or hidden Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[43] |
| concept | Supervised learning — not defined in this paper's own words; learning labels from labeled data for inference on unlabeled text Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[44] |
| concept | unsupervised learning — not defined in this paper's own words; learning from unlabeled data, often for exploration Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[45] |
| concept | natural structure — not defined in this paper's own words; patterns uncovered from unlabeled data by unsupervised approaches Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[46] |
| concept | unit of text — not defined in this paper's own words; segment that receives an annotation or code Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[47] |
| concept | unit of analysis — not defined in this paper's own words; final inferential level that may differ from the annotated text segment Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[48] |
| concept | interpretability — not defined in this paper's own words; capacity to communicate patterns, detect spurious features, and support result interpretation Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[49] |
| concept | human coders — not defined in this paper's own words; people who label data for full coding or model training Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[50] |
| concept | annotation examples — not defined in this paper's own words; worked cases added as annotation guidelines are iteratively updated Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[51] |
| concept | expert annotators — not defined in this paper's own words; specialists favored for complex or domain-specific concepts Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[52] |
| concept | intra -annotator agreement — not defined in this paper's own words; repeatability check within individual coders Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[53] |
| concept | Disagreement between annotators — not defined in this paper's own words; signal of either scheme weakness or inherent ambiguity that may itself be analytically meaningful Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[54] |
| concept | racial bias — not defined in this paper's own words; annotation and detection disparity affecting African American English tweets Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[55] |
| concept | pre-processing steps — not defined in this paper's own words; underreported preparation choices that shape measures, analysis, and reproducibility Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[56] |
| concept | OCR errors — not defined in this paper's own words; recognition defects whose form varies with source, date, typography, and software Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[57] |
| concept | Tokenization — not defined in this paper's own words; segmentation of character sequences into word-like units with consequential assumptions Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[58] |
| concept | Multi-word terms — not defined in this paper's own words; multi-token expressions whose treatment as one unit can change discovered patterns Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[59] |
| concept | normalization — not defined in this paper's own words; loosely defined grouping of abbreviations, ordinals, and spelling variants Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[60] |
| concept | word pieces — not defined in this paper's own words; statistical sub-token segments optimized for prediction but not necessarily linguistically meaningful Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[61] |
| concept | stoplist — not defined in this paper's own words; task-specific vocabulary exclusion list Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[62] |
| concept | Dictionaries — not defined in this paper's own words; transparent category word lists used to code text Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[63] |
| concept | domain-specific dictionaries — not defined in this paper's own words; corpus-adapted lexicons that address limits of general-purpose dictionaries Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[64] |
| concept | spurious features — not defined in this paper's own words; non-target correlations learned from imbalanced or noisy datasets Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[65] |
| concept | post-hoc explanations — not defined in this paper's own words; prediction-level interpretations that may fail to faithfully describe the original model Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[66] |
| concept | Topic models — not defined in this paper's own words; usually unsupervised models constrained for interpretable exploration Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[67] |
| concept | number of topics — not defined in this paper's own words; analytical scale chosen according to user needs rather than an inherent collection value Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[68] |
| concept | stratified random sample — not defined in this paper's own words; sample across the score range used for qualitative content revalidation Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[69] |
| concept | synthetic data — not defined in this paper's own words; constructed observations permitting controlled validation comparisons Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[70] |
| concept | explanatory power — not defined in this paper's own words; insight-driven objective sometimes more useful than incremental prediction gains Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[71] |
| concept | predictive performance — not defined in this paper's own words; forecasting objective contrasted with explanatory gains Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[72] |
| concept | opportunities for interpretation — not defined in this paper's own words; disagreements between model output and human labels treated as analytically revealing cases Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[73] |
| concept | qualitative analyses — not defined in this paper's own words; small-scale inquiry combined iteratively with large-scale computation Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[74] |
| concept | working across disciplines — not defined in this paper's own words; interdisciplinary setting in which operational and analytic choices require negotiation Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[75] |
| concept | qualitative insights — not defined in this paper's own words; domain knowledge added to strengthen computational answers Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[76] |
| concept | reliable results — not defined in this paper's own words; annotation outcome more often produced by highly trained or expert coders for complex concepts Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[77] |
| concept | reproducibility efforts — not defined in this paper's own words; ability to reproduce analysis affected by preprocessing choices Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[78] |
| concept | needs of the user — not defined in this paper's own words; desired analytical scale that determines the appropriate topic count Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[79] |
| concept | Computational text analysis — not defined in this paper's own words; large-scale approach treated as a complement to qualitative inquiry Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[80] |
| concept | models that are already somewhat interpretable — not defined in this paper's own words; models restricted to a small number of human-readable features Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[81] |
| hypothesis-testing | Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.statistics[0] |
| not machine-vouched | hypothesis-testing — not-in-our-copy - The quote is in the original web page we fetched for this paper, but not in the text we tag against, which was extracted differently. Checked with ligatures, hyphens, spacing and punctuation set aside. | content_tags.statistics[0] |
| sample-size n/N | Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.statistics[1] |
| Krippendorff's alpha | Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.statistics[2] |
| inter-annotator agreement | Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.statistics[3] |
| F-score | Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.statistics[4] |
| claim type | method — a second reading reaching the same value; recorded: method Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | alternatives.claim_type |
| phenomenon | computational text analysis of social and cultural concepts — a second reading reaching the same value; recorded: computational text analysis of social and cultural concepts Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | alternatives.phenomenon |
When we have changed our mind. Some entries in the table begin with a word in capitals and the reason for it. EVIDENCE CORRECTED — the finding still stands, but we had quoted the wrong sentence from the paper, and the quotation has been replaced with the right one. RETRACTED — the entry itself was wrong; it stays on the page, marked, with the reason, rather than being deleted. WITHDRAWN AND REPLACED — the entry was wrong, and a corrected version of it appears in the entries below. Each mark carries its own reason, in brackets, because the reasons differ from one entry to the next. An entry we have WITHDRAWN is removed from this page and kept in the record, with its reason. A line reading Audit and a date is a later re-reading of the entry against its whole sentence: DOUBTFUL means it was kept, and the reason says what is loose about it. Quotation marks inside the tagger's sentence. Where a tag is the tagger's own sentence, a phrase in quotation marks inside it is borrowed from the paper — the tagger's wording around it is not. The paper's own sentence is the evidence quote for that row; on this page it is withheld with the article, and the article is at its publisher. Nothing inside quotation marks anywhere on this page is ours: if it is quoted, it is the paper's. Whose words a concept carries. A concept entry records a term the paper itself uses, so the term, the sense given for it, and its evidence are the paper's own words, quoted where the paper fixes them. This is the opposite of a sense entry in the tag table, which is the tagger's sentence about how the paper uses a term. Where a concept entry could not be grounded in the paper's words, the entry says so on its own row.
Tag layer © the Meta-Tagging Project, CC BY 4.0. The paper text is © its authors/publisher
and is linked, not reproduced. Excerpts shown are short quotations for identification and verification.
· back to index
· concepts · authors
· עברית
© 2026 Shir Sivroni · ORCID 0009-0000-2597-2824