The Meta-Tagging Project

Evaluating Definitions in a Verbatim-Grounded,
Field-Neutral Tag Layer
From provenance-preserving annotations to measurable concept membership

Shir Sivroni
Department of Mathematics and Computer Science, The Open University of Israel
ORCID: 0009-0000-2597-2824

Definition-evaluation corpus: 372 case–definition observations (289 literature-positive · 83 literature-negative) from 34 papers.
Tag-layer corpus: 697 published records spanning 103 disciplines.
Scope: This paper presents a field-neutral framework for representing concepts as provenance-grounded records and evaluating competing definitions against an explicitly specified corpus. Art serves as a worked example rather than as the object of the study.
Data and code: doi:10.5281/zenodo.21432188.
Objective. Cross-disciplinary scholarship lacks a shared terminology. Researchers therefore cannot determine from terminology alone whether a phenomenon has already been studied elsewhere under a different name, or whether identical labels denote equivalent constructs or merely homonyms. Both failures are documented beyond the present corpus. The preferential association of similar entities is studied as homophily in sociology and as assortative mixing in network physics (McPherson et al., 2001; Newman, 2003). Conversely, criticality denotes a phase-transition regime in statistical physics, whereas neuroscience often operationalises “brain criticality” through signatures such as power-law neuronal avalanches, which are not sufficient on their own to establish critical dynamics (Mariani et al., 2022). This paper introduces an evidence-bearing concept layer to make such questions empirically tractable. Each concept becomes a record linking attested uses and senses to verbatim source passages, enabling convergent senses to be joined, divergent senses retained, and competing definitions evaluated against explicit corpus judgements. Art provides the worked example. The framework thereby turns canonicalisation from an implicit editorial choice into an auditable research procedure.

Status and intended scale. What is reported here is a proof of concept, and its corpus is sized accordingly: 697 tagged records, of which 34 papers and 372 adjudicated observations carry the definition evaluation. The programme it belongs to is larger — an interactive tagging platform holding tens of thousands of papers, in which a reader assembles a corpus, submits a definition of their own, and has it judged against that corpus in a single run. Nothing in the method reported here depends on the corpus being small; what the small corpus establishes is that the procedure is auditable, and what it cannot yet establish is how the measurements behave at scale.

Seeing the idea. A prototype exists and can be looked at without an account: the definition scorer shows what the layer is meant to feel like — choose a concept, choose the corpus, and each candidate definition appears with its confusion matrix, its case table, and the licensing quotation under every score. It is a demonstration rather than a working tool: two concepts are loaded, the scored corpus is the 29 papers judged in one run, and a reader cannot yet submit a definition of their own for a fresh judging run. It is shown here so the shape of the thing is visible, not because it is ready to be used. The platform is at its home page and the tag layer, the TEI editions and the code are at the showcase.

Contents

Study overview

A concise orientation to the argument. Each panel opens to show the supporting rationale.

1

Research problem

Competing definitions are commonly assessed through argument and counterexample. This study asks whether the classifications they entail can also be evaluated systematically and reproducibly against source-grounded cases.

Rationale

Cross-disciplinary synthesis is hindered by terminological heterogeneity: related phenomena are described differently across fields, while shared labels may denote distinct constructs. Ontologies, citation graphs, and keyword indexes expose part of this structure, but often retain discipline-specific vocabularies or require intensive curation. They therefore do not reliably supply evidence-grounded cross-field join keys. The problem also extends to interpretive and social research: framing has been described as a “fractured paradigm” whose theoretical and operational uses span psychology, sociology, political science, and communication (Entman, 1993). Lexical overlap must therefore be treated as a question to be tested, not as evidence of conceptual identity.

↑ back to contents
3

Methodological contribution

Each candidate definition is operationalised as a binary decision rule and compared with cases that the source literature explicitly includes in or excludes from the target concept.

Method and implementation status

Earlier analyses found that free-text labels rarely joined across fields, whereas a controlled statistical vocabulary did. The methodological implication is that canonicalisation requires not merely a preferred label, but an explicit and testable membership rule.

Canonicalisation is a definitional operation. Assigning different surface forms to one canonical term entails a judgement about category membership. In many tagging workflows this judgement remains implicit; the framework proposed here makes it explicit and subjects its application to reliability analysis.

7.1 Concepts as records

A concept represented only as a string attached to a paper cannot directly encode distinct senses, contrast cases, relations among interpretations, or evidence provenance. The proposed representation therefore treats the concept as a record rather than as a label.

The concept_records.json build dated 9 August 2026, with material appended on 10 August, contains 2,103 ConceptNode records grouped into 2,464 paper-level mentions and comprising 2,599 Attestation objects. Of these attestations, 1,880 carry a quotation and 719 still require grounding. The schema also records role (defines / exemplifies / refutes / mentions) and assertion status (affirmed / denied / unresolved). These figures specify both the implemented data model and the remaining corpus-wide grounding work.

The reference model is a historical gazetteer: place → name variant → dated attestation, where the entity is the record and the document is the evidence. Here it is concept → wording → quotation. The structures are analogous; the substantive difference is that an abstract concept does not generally have a directly observable referent. Sense resolution must therefore remain evidence-bearing and open to reliability assessment.

7.2 A concept may hold more than one sense

The same lexical form may express distinct theoretical constructs. Collapsing those constructs into a single aggregate can therefore obscure rather than summarise the corpus. The schema can represent senses and evidence-bearing relations among them (contrasts, extends, narrows), although corpus-wide population of these fields awaits completion of the attestation-grounding stage.

A worked example concerns priming. In one literature, the prime and target are different stimuli linked by category, and the outcome is the interpretation assigned to an ambiguous description; in another, they are the same stimulus and the outcome is a latency reduction. The latter literature explicitly rejects facilitation of the stage implicated by the former. A canonical merge without sense disambiguation would therefore create a relation that the source literature does not support.

7.3 Two gates, not one

The verbatim rule proves that a sentence exists in a paper. That is a string comparison, and a string comparison establishes provenance and nothing about meaning. A second gate asks whether the sentence licenses the concept at all.

In an initial audit, 82 attestations passed the substring gate and four failed the meaning gate — graph theory tagged on a sentence that never names it, machine learning tagged on a clause arguing the opposite, deep learning tagged on a paper that trains no deep network, bias tagged inside a three-item list that did not establish its role. All four were exact quotations. This result shows that exact-string verification establishes provenance but is insufficient to establish that the cited sentence licenses the assigned concept.

7.4 Measuring the membership rule

The rest of this paper (§§8–13) sets out the measurement: a definition treated as a binary classifier over cases the literature itself adjudicated, ranked against rivals in a single run, after three logical gates that no statistical measure detects — circularity, recursion without a base case, and a free parameter the author never fixed. Inter-coder reliability is reported on the act of application itself (κ = 0.547, n = 140).

The contribution of this paper is methodological rather than a claim of new cross-disciplinary discovery. It specifies a concept-record schema, distinguishes provenance from semantic licensing, and evaluates definition application on a documented corpus. The public implementation remains incomplete where concept-level quotations and sense relations have not yet been populated.
↑ back to contents
4

Evaluation design

A case is positive only when a source explicitly includes it in the target concept and negative only when a source explicitly excludes it. Silence is treated as unresolved, not negative.

Validity gates and scoring

Candidate definitions are evaluated only on explicitly adjudicated cases. Each positive or negative label is linked to a verified quotation from the source paper. The labels therefore represent documented corpus judgements rather than claims of context-independent truth.

What the gates disqualify, and why before scoring rather than after

Some candidate definitions achieve high classification scores despite structural defects that the confusion matrix cannot identify. The framework therefore applies logical validity checks before comparative ranking and reports gated definitions as unranked.

Gate 1 — circularity

The definiens contains the definiendum. Our circular control ("whatever people call art") achieves an almost perfect MCC because it reproduces the corpus label. Predictive scores alone cannot identify this form of target leakage.

Gate 2 — recursion without a base case

Levinson's historical definition: "a thing intended for regard as prior artworks were correctly regarded." It defines art by reference to prior art and never states what the first art was. The mathematical defence (0 is a number; n+1 is a number) works only when the base is given. Here there is n+1 and no 0.

Gate 3 — failure of point identification

Gaut's cluster account: an object is art if it meets enough of ten criteria, none of which is necessary — and he never says how many "enough" is. Dutton, who defends the same approach, concedes the point explicitly: "A recognition-criteria approach ... does not tell us in advance how many of the criteria need be present."
We decomposed the ten criteria and coded 120 cases against each. Moving the threshold from k = 1 to k = 10 moves the score across a range of 0.6 MCC, and if k may be chosen per case the account attains MCC 1.000. Without a fixed threshold, the definition is not point-identified for quantitative evaluation: its reported score is contingent on an analyst-selected parameter.

Why the gates precede scoring

All three defects are external to the confusion matrix. A circular definition may classify well because it reproduces the target label, whereas a definition with a free threshold receives the score associated with the selected value. Applying these gates after model selection would permit structurally invalid candidates to influence the ranking. The gates are therefore evaluated before scoring and before automated definition induction.

Measurement versus refinement — a distinction that must be recorded

Iterative revision requires versioned definitions and a clear distinction between prospective evaluation and post hoc refinement:

Measurement — the wording was fixed before the cases were seen. The score is a test.
Refinement — the wording was fixed after it became clear which cases failed. The score is not a test, because the cases are already known.

Both activities are legitimate, but their scores have different inferential meanings. The data model therefore retains each version and records whether its wording preceded or followed inspection of the evaluation cases.

Sub-concepts must themselves be defined

A definition that depends on an undefined subsidiary term transfers part of the decision to tacit judgement. In one analysis, operationalising such a subsidiary term without changing the outer wording increased MCC from +0.380 to +0.664. The framework therefore decomposes subsidiary terms until further operationalisation no longer changes classification outcomes.

↑ back to contents
5

Interpretive scope

The framework does not determine what art is. It quantifies how closely a stated definition reproduces explicit judgements in a specified corpus.

Scope and intended use

The layer is intended as a hypothesis-generating instrument for far reading. It identifies candidate regions in which close reading may be informative; it does not replace interpretive analysis or establish context-independent conceptual truth.

6.1 Why a grounded layer, rather than reading on demand

A language model can generate an immediate cross-disciplinary answer, but the basis of that answer may be difficult to audit. A grounded layer imposes a slower evidential workflow in which each supported claim resolves to a source passage. Its intended advantage is traceability rather than speed.

↑ back to contents

Interactive graphical summary

Interactive Figure 1. Concept-record data model

This diagram specifies the record on which the proposed measurement workflow depends. It distinguishes implemented structure from corpus fields that still require population. Every field in the diagram is a link; following it reaches that field's entry in the glossary in Appendix B, which states what the field does, what it holds, and its data structure (string · number · array · dictionary · record · tree · graph).

ConceptNodenode in the concept tree — the main record■ id : str■ labels : dict[lang,str]■ external_ids : dict[src,str]■ parent : -> ConceptNode?■ children : -> ConceptNode[]■ sources : PaperSource[]■ senses : Sense[]■ sense_relations : SenseRelation[]■ criteria_added : Criterion[]■ contrast_cases : ContrastCase[]■ theorists : Theorist[]PaperSourceone paper7■ paper_id : str■ field : str■ year : int■ lang : str■ attest : Attestation[]Sensea reading of the concept58■ id : str■ label : str■ holder : str■ defining : -> Attest[]■ endorsed_by : -> paper_id[]Criteriona condition in the definition3■ id : str■ label : str■ kind : necessary|suff■ evidence : -> Attest[]■ counter_ev : -> Attest[]■ excludes : -> ContrastCase[]ContrastCasewhat is not the concept45■ id : str■ thing : str■ evidence : -> Attestation■ excluded_by : -> Criterion?■ judged : boolTheoristholder of a position71■ id : str■ name : str■ external_id : str■ senses : -> Sense[]■ evidence : -> Attest[]SenseRelationrelation between readings■ from_sense : -> Sense■ to_sense : -> Sense■ kind : contrasts|ext■ evidence : -> Attestation■ paper_id : strAttestationa single attestation — the smallest unit171■ id : str■ node_id : -> ConceptNode■ quote : str■ locator : str■ role : str■ asserted : P|N|U■ object_level : bool■ kind : str◆ ownership — the record contains themparent / childrenthe tree — same typeeach paper holds its own attestations
Rectangle = record · solid arrow = ownership · open arrow = reference. Every row links to the glossary.

Interactive Figure 2. Classification framework and confusion matrix

Each cell is interactive: selecting it opens the corresponding definition and formula using the same colour code.

The papers assert: YES, this is the concept The papers assert: NO, it is not The definition INCLUDES The definition EXCLUDES TP the definition included a case the papers call art HIT FP the definition included a case the papers say is not art FALSE ALARM FN the definition excluded a case the papers call art MISS TN the definition excluded a case the papers say is not art CORRECT REJECTION
This colour code is used consistently throughout the document.
The papers assert: this IS art 15 cases (role: example) The papers assert: this is NOT art 45 cases (40 of them judged) The INCLUDES The EXCLUDES TP not yet measured 15 cases recorded; not yet evaluated against the definition FP 17 Brillo · wallpaper · pruning · colour field · AI FN not yet measured = 15 − TP once TP is determined TN 23 14 excluded by the “object” condition Developmental snapshot: the two missing cells require evaluation of the 15 positive cases.
Historical intermediate state from ART_CASE/concept_art.json; retained to document the transition from partial to complete evaluation.
TPTP — true positive · hit?
HIT. A case that the definition includes and the source literature explicitly assigns to the concept.
TP = |{ x : definition(x) = 1 AND corpus(x) = art }|
↑ back
FPFP — false positive · false alarm?
FALSE ALARM. A case that the definition includes although the source literature explicitly excludes it from the concept.
FP = |{ x : definition(x) = 1 AND corpus(x) = not art }|
↑ back
FNFN — false negative · miss?
MISS. A case that the definition excludes although the source literature explicitly assigns it to the concept.
FN = |{ x : definition(x) = 0 AND corpus(x) = art }|
↑ back
TNTN — true negative · correct rejection?
CORRECT REJECTION. A case that both the definition and the source literature exclude from the concept.
TN = |{ x : definition(x) = 0 AND corpus(x) = not art }|
↑ back

Here the classifier is the candidate definition, whereas the reference label records what the source papers explicitly state. These labels are corpus-relative observations, not claims of metaphysical or context-independent truth.

The four cells

Under the stated corpus labels, each FN provides counterevidence to necessity, and each FP provides counterevidence to sufficiency. This mapping connects the logical analysis to the classification framework.

The table is generated from 372 adjudicated case observations (289 positive; 83 negative) drawn from 34 papers. Because 78% of observations are positive, the primary summary measure is MCC rather than F1: an all-positive classifier can obtain a high F1 while its MCC is zero.

DefinitionTPFPFNTN MCC
condition-based definition (object/action, intention, values)260132970+0.700
the aesthetic definition202118772+0.477
art as expression139215081+0.392
broad245504433+0.252
narrow73121682+0.251
mimesis94619577+0.238
circular2870283—
the cluster account266162367—
the historical definition24244779—
the institutional theory22026981—
open28224759—
The circular control demonstrates why validity gates precede ranking. Its near-perfect classification follows from reproducing the target label; the confusion matrix alone cannot diagnose this leakage.

What the matrix covers, and what it does not

Three of the six criteria are computed from these four cells: coverage (recall = TP/(TP+FN)), discrimination (precision = TP/(TP+FP)), and prediction — the same four cells on a held-out sample not used in constructing the definition. The remaining three lie outside it. Typicality ranking requires an order rather than labels, and is measured by rank correlation between the ranking a definition predicts and frequency of attestation in the corpus. Applicability is measured by κ. Minimality is a property of the definition rather than of the cases, and is measured by ablation: remove one condition, re-score; a condition whose removal changes nothing is ornament.

Three states, not two

A case is positive when a paper explicitly includes it in the concept and negative when a paper explicitly excludes it. The third state is unresolved: absence of an assertion is not evidence of negation. Treating silence as negative would systematically convert non-mention into exclusion and bias the evaluation.

The unit of analysis is accordingly neither the paper nor the passage but the (case, paper) pair. The same case may be positive in one paper and negative in another, and that disagreement is the most valuable datum in the corpus rather than noise: collapsing to the paper loses it, collapsing to the case loses who said it. Variance is decomposed with the paper as a random effect (ICC = between / (between + within)), so that twenty-five attestations from a single paper inflate nothing.

Full manuscript

Sections are collapsed for readability. Select a heading to open the complete text.

Abstract
Background, methods, results, and conclusions.

Background. Cross-disciplinary synthesis is constrained by terminological heterogeneity: related phenomena may be named differently, whereas identical labels may denote distinct constructs. Earlier work introduced a field-neutral tag layer in which each non-trivial assignment is linked to an exact licensing passage. The structure proved auditable, but free-text phenomenon labels rarely produced cross-field joins.

Methods. We distinguish two datasets. The tag-layer corpus contains 697 published records across 103 disciplines; its provenance audit re-checked 18,416 evidence strings. The definition-evaluation corpus contains 372 explicitly adjudicated case observations from 34 papers (289 positive; 83 negative). Candidate definitions are treated as binary decision rules, screened for circularity, unsupported recursion, and unidentified parameters, and then compared using Matthews correlation coefficient (MCC). Application reliability was estimated on a blinded sample of 140 case–definition pairs independently labelled by a second language-model system with the first labels withheld.

Results. The provenance audit identified three failed strings. Among 511 papers, 500 distinct free-text phenomenon labels yielded only two cross-disciplinary matches. By contrast, a controlled statistical vocabulary produced join keys spanning as many as 31 disciplines; re-audit of 519 paper–term assignments retained 420 and rejected 99 (19.1%). In the definition-evaluation corpus, the highest-ranked eligible definition achieved MCC = 0.700. Reliability of application was κ = 0.547 with 78.6% observed agreement (n = 140). The prespecified expectation of distinctly lower agreement for the cluster account was not supported in this sample.

Conclusions. Canonicalisation is not merely a normalisation step: it entails an explicit rule for concept membership. A defensible tag layer must therefore preserve provenance, represent alternative senses, and measure the reproducibility of its membership decisions. The concept_records.json build dated 9 August 2026, with material appended on 10 August, contains 2,103 concept records, 2,464 paper-level mentions, and 2,599 attestation objects; 1,880 attestations carry a quotation and 719 still require grounding. Sense relations also remain incompletely populated. The work should thus be read as a validated measurement framework with a quantified implementation gap, not as a completed corpus-wide semantic layer.

Scope. This study evaluates a field-neutral method rather than a substantive definition of art. In principle, any concept can be represented as an evidence-bearing record and competing definitions assessed against an explicitly delimited corpus. Art is used because its literature states definitional disagreements unusually clearly; game provides a secondary test case. The present demonstration remains bounded in two respects. First, the scored corpus includes 29 of 60 papers so that every definition is evaluated within a single, comparable run. Second, newly submitted definitions can currently be assessed only against previously adjudicated cases. User-defined corpora and fresh adjudication runs are implementation targets, not results reported here.

↑ back to contents
0. Scope and contribution
What the paper establishes, and what it does not.

A shared, verbatim-grounded annotation format can be audited at scale, but free-text phenomenon labels rarely support cross-disciplinary joining. This paper addresses the methodological consequence of that null finding.

The annotation layer

Each paper is annotated by the shape of its contribution rather than by topic alone: claim type, phenomenon, claim relations, replication status, and limitations. The shared tag registry (SCHEMA/tags.json, version 0.5) contains 45 tags in seven layers.

Verbatim grounding is a provenance constraint. Each non-trivial tag must be accompanied by an exact licensing passage. In the standing audit, 18,416 evidence strings were re-checked against source text and three failed. This procedure verifies that the cited text exists; it does not by itself verify that the passage semantically licenses the assigned tag.

Why canonicalisation forces a definition

Free-text phenomenon labels produced almost no cross-field joins, whereas the controlled statistical vocabulary produced many. This contrast identifies canonical vocabulary as a necessary joining layer. Canonicalisation, however, requires a rule for deciding which surface forms and cases belong under the same term. That rule is a definition in operational form.

Contribution

The paper links canonicalisation to explicit definition evaluation: concepts are represented as records capable of holding senses, evidence, and contrast cases, while candidate membership rules are evaluated on a documented corpus.

The conceptual and operational components are interdependent. A concept record provides a place to represent competing senses; a measured membership rule makes those senses empirically inspectable rather than merely asserted.

Implementation status

  • Implemented: the concept_records.json build dated 9 August 2026, with material appended on 10 August, contains 2,103 concept records grouped into 2,464 paper-level mentions and comprising 2,599 attestations; the schema also provides sense relations, explicit positive/negative/unresolved case states, definition scoring, and the three pre-evaluation validity gates.
  • Measured: definition performance on 372 adjudicated observations and blinded application reliability on 140 case–definition pairs.
  • Not yet complete: according to the same concept_records.json build, 1,880 of the 2,599 attestations carry a quotation and 719 still require grounding; sense and sense-relation fields are not yet populated corpus-wide.
The claim is deliberately limited. This paper contributes a traceable data model and an evaluated membership procedure. It does not overturn the earlier null finding, and it does not claim that the public corpus has already become a complete semantic knowledge layer.
↑ back to contents
1. Background and motivation
Terminological heterogeneity, auditable annotation, and the definition problem.

Scholarly findings are primarily communicated in natural-language prose, while cross-disciplinary retrieval depends on identifying relations among differently worded claims. Keyword search, citation networks, and discipline-specific ontologies each expose part of this structure, but none guarantees that semantically related observations will share a searchable label.

The Meta-Tagging Project addresses this problem by describing the form of a contribution and grounding each non-trivial assignment in a source passage. The earlier empirical result was negative but informative: source-faithful phenomenon strings seldom matched across disciplines. A controlled statistical vocabulary joined more effectively, suggesting that cross-field retrieval requires canonical categories.

Research question. If canonicalisation requires a membership rule, how can competing rules be represented, applied, and compared without discarding provenance or converting interpretive disagreement into unrecorded coder discretion?

The study answers this question in three steps. It distinguishes provenance verification from semantic licensing, represents concepts as evidence-bearing records capable of holding multiple senses, and evaluates candidate definitions as decision rules over explicitly adjudicated cases. The worked example concerns definitions of art, but the measurement problem is intended to generalise to other contested scholarly categories.

↑ back to contents
2. Conceptual framework
A small, field-neutral facet set with explicit provenance.

Each paper is annotated with a deliberately compact set of facets intended to describe the form of a contribution rather than its topic: claim-type (existence / mechanism / method / prediction / critique / null-result), phenomenon, claim-relations (builds-on / extends / contradicts / replicates, each linked to a named target), replication-status, and limitations.

The defining constraint is verbatim grounding. Because a tag is an interpretation of natural language, each non-trivial assignment must include a licensing excerpt and a resolvable provenance state. This design makes the annotation inspectable in a way that an ungrounded assertion or dense embedding is not. Human-readable HTML views and machine-readable JSON records are generated from the same canonical data.

2.1 Data structure

Each paper is represented by one canonical record. Tag assignments point to their licensing passages. The shared registry (SCHEMA/tags.json, version 0.5) currently contains 45 tags in seven layers. The registry centralises labels and descriptors, but formal scope notes, inclusion rules, and grounded examples are not yet complete for every tag; this is treated as an implementation requirement rather than as a finished property.

↑ back to contents
3. Corpus, annotation, and validation
Two corpora, provenance auditing, and deterministic derived outputs.

3.1 Tag-layer corpus

The tag-layer corpus is curated for disciplinary breadth and serves as a proof of format and procedure; it is not a probability sample and does not support claims of disciplinary representativeness. The current build contains 697 published records across 103 disciplines. The build pipeline validates annotation invariants, recalculates reported quantities, derives postings and record shards for search, exports an adjacency structure for implemented relations, and regenerates audit and reading views.

Of the 126 papers in the proof-of-format corpus, 10 had unusable text layers. Among the remaining 116 papers with usable text, faceting was completed for 116/116 papers (100%). These status values can be checked against DATA/corpus.json; the ten unusable records are retained with an explicit untaggable status rather than silently excluded.

A standing provenance audit re-checked 18,416 evidence strings against their associated source text and identified three failures. A further 623 strings matched only an older stored copy of the corresponding paper and are reported separately as a weaker provenance state.

3.2 Definition-evaluation corpus

The definition analysis uses a distinct corpus of 372 adjudicated case observations drawn from 34 papers: 289 observations in which a source explicitly includes the case in the concept and 83 in which a source explicitly excludes it. Unresolved or silent cases are not treated as negative. All candidate definitions are evaluated within the same run and under the same coding instruction.

↑ back to contents
4. Empirical results and the null finding
Free-text labels rarely join across disciplines.

In the audited proof-of-format, 116 papers across 38 disciplines were fully faceted. The resulting layer contained 127 claim-relation assignments and 87 limitation assignments, each verified by exact-string matching to the available source text.

Free-text phenomenon labels did not provide an effective cross-field join layer. In the audited corpus, only one pair of disciplines shared a phenomenon string verbatim. In the expanded corpus, 511 papers yielded 500 distinct phenomenon strings, only two of which crossed a disciplinary boundary. A separate novelty scan did not identify a candidate cross-disciplinary relation that had not already been described in the literature.

The result distinguishes conceptual overlap from string identity. Motifs such as priming or power-law structure may recur across fields, yet source-faithful free-text descriptions remain too specific to match reliably. These observations motivate a canonical vocabulary layer above, rather than in place of, the grounded source wording.

↑ back to contents
5. Controlled statistical vocabulary as a cross-field join layer
A controlled methodological vocabulary joins where free text does not.

A controlled vocabulary of statistical terms provides a direct test of the proposed canonical layer. The broadest audited term spans 31 disciplines. Re-audit of all 519 legacy paper–term assignments rejected 99 (19.1%) and retained 420 as semantically valid. Because source availability and exact local-body grounding are incomplete, both the primary counts and a stricter exact-local-basis sensitivity analysis are reported.

This result supports a bounded conclusion: controlled terms can function as cross-field join keys, but only because their labels and intended statistical meanings have already been normalised. The statistical layer therefore demonstrates the value of canonicalisation; it does not eliminate the need to specify and test membership rules for less standardised concepts.

5.2 Query implementation

The public interface implements static, client-side search using an inverted index of 1,943 keys mapped to document postings. Queries are served without a backend; their cost depends on postings-list lengths and output size rather than being independent of corpus size.

5.3 Unlabelled regions as hypotheses

Regions that contain multiple grounded observations but lack a shared label may indicate a terminological gap. This pattern is hypothesis-generating rather than decisive: absence of a label in the sampled literature does not establish that no term exists. In the memory taxonomy example, one such region corresponds approximately to what might be termed short-term implicit memory and warrants targeted literature review.

↑ back to contents
6. Interpretive scope
Corpus-relative measurement rather than an ontological verdict.

The framework estimates how well a stated definition reproduces explicit judgements in a specified corpus. It does not determine what art—or any other contested concept—ultimately is. Positive and negative labels record what a cited source explicitly includes or excludes; silence, ambiguity, and unexamined cases remain unresolved.

This distinction is central to the intended use. The tag layer supports far-reading and hypothesis generation by identifying patterns that warrant close reading. It does not replace historical context, interpretive argument, or expert adjudication. A high score therefore indicates fit to documented corpus judgements under a particular instruction, not metaphysical truth or universal disciplinary consensus.

↑ back to contents
7. Concept records and measurable membership
From labels to evidence-bearing records with explicit decision rules.

A concept is represented as a record rather than as a string attached to a paper. The normative model can store labels, external identifiers, source-grouped attestations, distinct senses, relations among senses, criteria, and contrast cases. This structure allows disagreement to remain visible instead of being collapsed into one preferred wording.

Canonicalisation is treated as a definitional operation: assigning several expressions to one key entails a rule about membership. That rule is made inspectable by linking decisions to quotations and measurable by applying candidate definitions to explicitly adjudicated cases. The interactive model is shown in Interactive Figure 1.

Current implementation boundary. The concept_records.json build dated 9 August 2026, with material appended on 10 August, contains 2,103 concept records grouped into 2,464 paper-level mentions and comprising 2,599 attestation objects. Of these, 1,880 carry a quotation and 719 still require grounding; the sense-relation layer also remains incomplete. The schema therefore identifies both the realised representation and the work needed for corpus-wide evidential completion.
↑ back to contents
8. Pre-evaluation validity gates
Circularity, unsupported recursion, and unidentified parameters.

Classification performance is interpreted only after three logical checks. First, a definition is circular when its definiens reproduces the definiendum; a high score in that case reflects target leakage. Second, a recursive definition requires an explicit base case. Third, a quantitative application must fix any threshold or parameter needed to determine membership.

The gates are applied before ranking because none can be diagnosed from a confusion matrix alone. A circular definition may classify nearly perfectly, while a free threshold can be chosen to maximise performance after the outcomes are known. Gated candidates remain visible for diagnostic comparison but are not assigned a rank.

The cluster-account sensitivity analysis illustrates point identification: changing the required number of satisfied criteria from one to ten spans 0.6 MCC, and allowing the threshold to vary by case produces perfect apparent fit. This is evidence of flexibility in the operationalisation, not evidence of perfect definition performance.

↑ back to contents
9. Evaluation criteria for candidate definitions
Nine criteria: three matrix measures, three external measures, and three prior gates.

The evaluation framework distinguishes properties measured by the confusion matrix from properties that require separate evidence. Three formal gates are applied before either class of measure is interpreted.

#CriterionWhat it asks Where measured
G1Non-circular Does the definiens contain the definiendum?gate
G2Recursive base If recursive — is the base case stated?gate
G3Point-identified Does it contain a parameter the author never fixed?gate
1Coverage How much of what is agreed to be the concept does it admit?recall
2Discrimination How much of what is agreed not to be the concept does it exclude?precision
3Prediction Does it hold on cases not used in building it?held-out sample
4Typicality ranking Does it explain why one instance is more central than another?rank correlation
5Applicability Will independent coders reach the same judgement?κ between coders
6Minimality Can a condition be removed with nothing changing?ablation
Applicability is a precondition for interpreting the matrix. If independent coders do not apply a definition consistently, variation in the four cells may reflect the coding operation rather than the definition alone. Reliability on the controlled statistical-term layer was κ = 0.767; reliability for the candidate definitions is reported in §12.
↑ back to contents
10. Classification framework and results
Corpus-relative confusion matrices and MCC-based comparison.

Each eligible definition is applied as a binary decision rule, D(x), to cases for which the source literature supplies an explicit inclusion or exclusion. This produces the four conventional cells: true positives and false negatives among literature-positive observations, and false positives and true negatives among literature-negative observations. The interactive explanation and complete result table are provided in Interactive Figure 2.

The primary summary measure is the Matthews correlation coefficient because 289 of the 372 observations (77.7%) are positive. MCC incorporates all four cells and assigns a value of zero to an all-positive classifier in this dataset, whereas F1 can remain high when true negatives are ignored.

Primary result. Among candidates that passed the logical gates, the condition-based definition achieved the highest observed MCC (0.700; TP = 260, FP = 13, FN = 29, TN = 70). The aesthetic definition ranked second descriptively (MCC = 0.477). These are within-corpus estimates; close comparisons require uncertainty intervals and held-out evaluation.

Gated candidates are displayed but unranked. In particular, the circular control produced near-perfect classification because it reproduced the target label. This is a diagnostic control demonstrating that predictive fit alone is insufficient evidence of a valid definition.

The reference variable has three states: positive, negative, and unresolved. Only the first two enter the confusion matrix. The observational unit is the case–paper pair so that different sources may explicitly classify the same case differently without that disagreement being erased.

↑ back to contents
11. Concept-record data model
Provenance, source grouping, senses, criteria, and contrast cases.

The proposed model centres on ConceptNode, with paper-level provenance stored through PaperSource and mention-level evidence represented by Attestation. Optional structures represent Sense, SenseRelation, Criterion, ContrastCase, and Theorist. Grouping attestations by source prevents many mentions from one paper from being treated as independent positions.

Each evidential relation is designed to resolve to a quotation and locator. Positive, negative, and unresolved assertion states are retained explicitly, and the object_level field distinguishes a case of the concept from a proposition about the concept. The linked, field-level glossary is available through Interactive Figure 1 and Appendix B.

The model is partly prospective. Record types and identifiers are implemented, but public attestation quotations, senses, and sense relations remain unpopulated. Claims in this article therefore distinguish schema capability from populated empirical content.

↑ back to contents
12. Reliability of definition application
Agreement between two independently prompted language-model coding systems.

Applicability is evaluated separately from predictive performance. If independent coders apply the same wording differently to the same case, the confusion matrix combines properties of the definition, the instruction, and the coding system.

A blinded sample of 140 case–definition pairs (20 per definition) was coded by a second language-model system that had not produced the first set of judgements. Both systems received the same operational question: “Does this definition, read literally, apply to this case?” The task was therefore definition application rather than an unconstrained classification of whether a case is art.

DefinitionκAgreementn
the historical definition+0.80090%20
condition-based definition (object/action, intention, values)+0.50075%20
the institutional theory+0.50075%20
the aesthetic definition+0.47975%20
the cluster account+0.41770%20
mimesis+0.41280%20
art as expression+0.34885%20
Overall+0.547 78.6% 140
The prespecified expectation was not supported in this sample. The coding protocol anticipated markedly lower agreement for the cluster account because the number of required criteria is unspecified. Its observed κ was 0.417, 0.090 below the mean of the remaining definitions. With only 20 observations per definition and no interval estimate for this contrast, the result does not establish equivalence or demonstrate that an unstated threshold converges across applications.
Reliability constrains interpretation. Overall κ was 0.547 and raw agreement was 78.6%. Under the conventional, and only heuristic, Landis–Koch labels, this falls in the “moderate” range. Reliability was descriptively higher for the controlled statistical-term layer (κ = 0.767), but the present design does not isolate whether that difference arises from task complexity, prompt wording, model behaviour, category structure, or sampling variation.

The historical definition produced the highest agreement (κ = 0.800) despite failing the recursive-base gate. This illustrates that reliability and substantive adequacy are distinct: a consistently applied definition is not necessarily an acceptable one. Per-definition estimates should be treated as descriptive because each rests on only 20 judgements.

↑ back to contents
13. Related work and interoperability
Positioning the framework within digital-humanities annotation research.

A set of projects presented in a 2026 summer school at the Open University of Israel illustrates a shared methodological problem: how to convert textual evidence into structured data without erasing its textual and interpretive character. The projects differ in their units of annotation, vocabularies, infrastructures, and reliability procedures.

ResearcherUnit of annotation VocabularyPlatformReliability
Gutman
figurative language as a network		 a figurative instance — a manually extracted sub-sentential span closed: 103 topicsDistilBERT + Gephinot measured
Dekel
public distant reading		 the whole novel — a work-level recordmixed: questionnaire Google Formsnot measured, by stated policy
Melnik
idioms and their flexibility		 an occurrence of an idiom in a corpuscorpus queries Hebrew corpus—
Marienberg-Milikowsky
interpretive tagging		 a span in the texthierarchical tagsetCATMA → ViS-À-ViS —
Sagiv
Hasidic stories		 an entity in the text: person, place, subjectthe TEI-XML standard TEI-Publisher—
Rusinek
MEHDIE · Kima		 a pair of records, not a span in textclosed: gazetteers WHG · Recogito · Wikidatanot measured
Rivlin-Angert
political delegitimisation		 the sentenceclosed: two-stage schemeDictaLM 2.0 κ = 0.82

Common ground and methodological distinction

These projects annotate phenomena in texts, including figurative expressions, place names, delegitimisation, and idioms. The present study uses annotation as an evidential substrate for a different question: which operational definition best fits a corpus of explicitly adjudicated cases? This shift makes category specification, provenance, and application reliability direct objects of evaluation.

The contribution addresses two recurring, but not universal, methodological gaps. Some category systems provide labels without operational scope conditions, and several projects do not report intercoder reliability. Rivlin-Angert and Mor-Lan are an important counterexample, reporting κ = 0.82 for three coders on a 170-sentence sample. The present framework combines verbatim evidence for each non-trivial assignment with explicit membership rules whose application can be tested.

Structural analogy and its limit

The Kima data model provides a useful structural analogy: place → name variant → dated attestation can be compared with concept → wording → quotation. The models are not identical, however. Geographic variants can often be checked against an external referent or phonetic normalisation, whereas abstract concept membership generally lacks an equivalent independent test. The latter therefore requires explicit adjudication and uncertainty states.

Interoperability — what must be emitted

Technical connection need not require a shared interpretive theory, but it does require compatible representation choices:

  • An opaque identifier instead of a label. Where the tag is the word, any change of wording silently breaks every external link. A stable identifier, with the words hanging off it as skos:prefLabel and skos:altLabel.
  • A textual anchor that is not an offset. A TextQuoteSelector (quotation plus preceding and following context) survives re-editions; character offsets do not. Offsets for tools that need them are derived from it at export time.
  • Resolution to an external authority — Wikidata, GeoNames. Once two layers point at the same identifier they join without any bilateral arrangement.

Two export formats are especially compatible with the practices reviewed here: two-tier TEI (metadata in teiHeader, span annotations in standOff), and a pair of Gephi CSV files (nodes and edges).

External validation opportunity. Applying this framework to an independently developed category system would test whether explicit scope rules improve reliability outside the corpus used to develop the method.
↑ back to contents
14. Limitations and threats to validity
Sampling, model dependence, implementation status, and reproducibility constraints.

The results should be interpreted within the following constraints.

  • Reliability is model-based and imprecisely estimated. κ = 0.547 over 140 blinded pairs reflects agreement between two language-model coding systems, not human coders. The per-definition sample is 20 and confidence intervals are not reported.
  • Scores are instruction-sensitive. The same definition scored +0.714 and +0.591 on identical cases under differently worded instructions. Cross-run comparisons therefore require an identical protocol; MCC here is a property of the definition–instruction–coder combination, not of a definition in isolation.
  • The negatives are not a representative sample. They are contrast cases someone troubled to write down, not a random sample of what is not the concept. Accuracy and specificity should therefore not be interpreted as population estimates. Researcher-labelled hard negatives are reported separately from literature-grounded cases.
  • There is no access to authorial intent. What was not written is not measured.
  • Recall has a ceiling. What was missed cannot be known without knowing the whole set; coverage is therefore coverage relative to a documented pool, and must be named as such.
  • The corpus is still small. 34 papers. The bootstrap unit is the paper, so the effective n is 34 rather than 372 — which is why the lower places in a ranking are not separable.
  • The public concept layer is not yet evidentially complete. The concept_records.json build dated 9 August 2026, with material appended on 10 August, contains 2,103 concept records grouped into 2,464 paper-level mentions and comprising 2,599 attestation objects. Of these, 1,880 carry a quotation and 719 still require grounding; sense and relation fields are likewise incomplete. The diagrams describe both the implemented data model and work not yet completed corpus-wide.
  • The registry remains underspecified for evaluation. Its 45 tags have labels and brief descriptors, but comprehensive scope notes, inclusion rules, and grounded examples are not yet available for every tag.
  • Independent recomputation of the κ table is currently incomplete. The public repository provides the coding manual, task instructions, and blank coding sheet, but not the completed response file required by the scoring script.
  • External validity has not been established. Cross-corpus evaluation and the proposed scoring workflow for independently developed category systems remain future work.
  • Open-textured positions require a different target. A view that rejects necessary and sufficient conditions cannot be evaluated as though it supplied such a classifier; its operational consequences would require a separately specified test.
↑ back to contents
15. Data, code, and materials availability
Public repository, archived releases, and current reproducibility boundary.

Project code, schemas, derived views, documentation, and the materials that can be redistributed are available in the public GitHub repository. Versioned releases are archived under the Zenodo concept DOI 10.5281/zenodo.21432188. The repository records source rights and links to restricted sources rather than redistributing content for which reuse permission has not been established.

The public reliability package includes the coding manual, instructions for the second coder, and a blank coding sheet. At the time of this paper (12 August 2026), the completed response file used to calculate the κ values was not present in the public repository; consequently, those values cannot yet be independently recomputed from the archived public materials alone. The reported results should therefore be read together with this explicit reproducibility qualification.

The HTML article is designed as a layered research object: the main argument remains linear, while methodological detail, model fields, and worked examples can be opened in place. Hyperlinks and colour coding are explanatory aids; the same semantic distinctions are also stated in text and retained in print output.

↑ back to contents
Appendix A — formulae
The formulae in full.

All measures are computed from the four cells. TP and FN are counted over cases the corpus adjudicated positive, FP and TN over those it adjudicated negative.

precision = TP / (TP + FP) — discrimination
recall = TP / (TP + FN) — coverage
F1 = 2 · precision · recall / (precision + recall)
MCC = (TP·TN − FP·FN) / sqrt((TP+FP)(TP+FN)(TN+FP)(TN+FN))

Why MCC and not F1. F1 ignores TN, so on a positively skewed corpus a classifier that says yes to everything obtains a high F1. MCC gives that same classifier exactly 0.000 (Matthews, 1975; Chicco & Jurman, 2020). F1 is reported beside it, never alone.

Reliability

κ = (p_o − p_e) / (1 − p_e)
p_o = observed agreement · p_e = agreement expected by chance from the two coders' marginals (Cohen, 1960). Interpretation of values is reported with the commonly used Landis and Koch (1977) descriptors, which are treated as heuristic rather than as evidential thresholds. For designs with more than two coders or missing observations, Krippendorff's alpha may be preferable (Hayes & Krippendorff, 2007).

Variance between papers and within them

The same case may be judged in several papers. Variance is decomposed by analysis of variance:

ICC = σ²_between / (σ²_between + σ²_within) (Shrout & Fleiss, 1979)
Higher ICC is consistent with observations clustering by paper and may motivate investigation of disciplinary or author-level differences.
Lower ICC indicates greater within-paper variation and may signal locally contested or context-dependent usage.

A hierarchical model with paper as a random effect can represent both levels and reduces, but does not automatically eliminate, the influence of many correlated attestations from a single source.

Confidence intervals and comparison

Definitions are ranked with a paired paper-level bootstrap of 2,000 resamples: two definitions share a place unless one achieves the higher score in at least 95% of resamples, so that ties are ties on the evidence rather than artefacts of overlapping intervals. The paired bootstrap resamples the same papers for both definitions and asks how often one actually beats the other, which is the question a ranking answers; papers rather than cases are resampled, because cases drawn from the same paper are not independent. Each definition's score is additionally reported with a 95% confidence interval from 3,000 paper-level bootstrap resamples, and paired case-level differences are assessed with McNemar's test (McNemar, 1947). Scoring is performed by TOOLS/art_score_v3.py with results in ART_CASE/scores_v3.json; the ranking and its tie rule are TOOLS/final_ranking.py, with output in ART_CASE/final_ranking.json.

↑ back to contents
Appendix B — field glossary
The field glossary — every field in the record diagram links here.
Implementation note. This glossary specifies the normative record model. Several fields—most importantly attestation quotations, senses, and sense relations—are not yet populated in the current public concept-layer build.

ConceptNode the concept record

The record that holds one concept. Everything we know about "art" hangs from it. It is also a node in a tree, so it can be split into sub-types without duplicating information.

↩ back
id string str
What it does: The concept's only key in the system.
What it holds: A short, fixed string, e.g. art (text)
↩ back
labels dictionary dict[lang,str]
What it does: The concept's name in each language.
What it holds: A dictionary: language code → name. he→omanut, en→art (key→value pairs)
↩ back
external_ids dictionary dict[src,str]
What it does: Links to authority records outside the project, so the layer can join to Wikidata and to MEHDIE.
What it holds: A dictionary: authority name → its identifier. wikidata→Q735 (key→value pairs)
↩ back
parent tree -> ConceptNode?
What it does: The broader concept this one derives from. For "painting" the parent is "art".
What it holds: A pointer to a record of the same type, or empty if this is the root (reference to a record of the same type: parent or children)
↩ back
children tree -> ConceptNode[]
What it does: The sub-types. This is what allows a concept to be split into ten sub-types quickly, with the children inheriting the parent's criteria by reference rather than by copy.
What it holds: A list of pointers to records of the same type (a list of items of one type)
↩ back
sources array PaperSource[]
What it does: The papers that mention the concept, grouped by paper -- because twenty-five mentions in one paper are one opinion, not twenty-five.
What it holds: A list of PaperSource records (a list of items of one type)
↩ back
senses array Sense[]
What it does: The different readings the concept receives. One record can hold contradictory senses without averaging between them.
What it holds: A list of Sense records (a list of items of one type)
↩ back
sense_relations graph SenseRelation[]
What it does: Who contradicts whom, who extends whom -- the concept's internal graph.
What it holds: A list of SenseRelation records (a list of items of one type)
↩ back
criteria_added array Criterion[]
What it does: The conditions this concept adds beyond its parent's. Inheritance is resolved by walking upward, so a change at the parent reaches every child immediately.
What it holds: A list of Criterion records (a list of items of one type)
↩ back
contrast_cases array ContrastCase[]
What it does: What is not the concept. Without these, discrimination cannot be measured -- only coverage.
What it holds: A list of ContrastCase records (a list of items of one type)
↩ back
theorists array Theorist[]
What it does: Who holds which reading.
What it holds: A list of Theorist records (a list of items of one type)

PaperSource one paper

One paper that mentions the concept, with all of its attestations inside it. This is what stops twenty-five mentions in a single paper from counting as twenty-five independent opinions: they are one author's position.

↩ back
paper_id string str
What it does: Which paper this is.
What it holds: A stable identifier for the paper (text)
↩ back
field string str
What it does: Its discipline -- so that crossing of fields can be checked.
What it holds: A discipline label (text)
↩ back
year number int
What it does: When it was published -- so that change over time is visible.
What it holds: A year (number)
↩ back
lang string str
What it does: The language of the source.
What it holds: A language code (text)
↩ back
attest array Attestation[]
What it does: Every attestation from this paper.
What it holds: A list of Attestation records (a list of items of one type)

Sense a reading of the concept

One reading the concept receives. A single record can hold contradictory senses without averaging between them, which is the point: disagreement is data.

↩ back
id string str
What it does: The sense's key.
What it holds: A short string (text)
↩ back
label string str
What it does: What this sense is called.
What it holds: A short phrase (text)
↩ back
holder string str
What it does: Who holds it.
What it holds: A theorist's name or identifier (text)
↩ back
defining array -> Attest[]
What it does: The quotations the sense is derived from -- not our paraphrase of them.
What it holds: A list of pointers to attestations (reference to another record, not a copy of it)
↩ back
endorsed_by array -> paper_id[]
What it does: Which papers hold this sense.
What it holds: A list of paper identifiers (a list of items of one type)

Criterion a condition in the definition

One condition a definition asserts, carrying both the evidence for it and the evidence against it.

↩ back
id string str
What it does: The condition's key.
What it holds: A short string (text)
↩ back
label string str
What it does: The condition in words.
What it holds: A phrase (text)
↩ back
kind closed list necessary|sufficient
What it does: Whether it is asserted as necessary or as sufficient -- two things that are measured differently.
What it holds: One of a closed list of values
↩ back
evidence array -> Attest[]
What it does: Quotations that support the condition.
What it holds: A list of pointers to attestations (reference to another record, not a copy of it)
↩ back
counter_ev array -> Attest[]
What it does: Quotations that refute it. Every item here is a refutation of necessity.
What it holds: A list of pointers to attestations (reference to another record, not a copy of it)
↩ back
excludes array -> ContrastCase[]
What it does: Which contrast cases this condition filters out.
What it holds: A list of pointers to contrast cases (reference to another record, not a copy of it)

ContrastCase what is not the concept

A case the literature states is NOT the concept. Without these, discrimination cannot be measured at all -- only coverage.

↩ back
id string str
What it does: The case's key.
What it holds: A short string (text)
↩ back
thing string str
What it does: What the case is, briefly.
What it holds: A short description (text)
↩ back
evidence record -> Attestation
What it does: The quotation it was taken from.
What it holds: A pointer to one attestation (reference to another record, not a copy of it)
↩ back
excluded_by record -> Criterion?
What it does: Which condition filtered it out -- this is how one learns which condition is doing work.
What it holds: A pointer to one criterion, or empty (reference to another record, not a copy of it)
↩ back
judged boolean bool
What it does: Whether the paper adjudicated it at all. An unadjudicated case is not counted as a negative; it goes into the unresolved rate.
What it holds: Yes or no

Theorist holder of a position

Who holds which reading. A node in the graph, not metadata.

↩ back
id string str
What it does: Internal key.
What it holds: A short string (text)
↩ back
name string str
What it does: The name as it appears.
What it holds: A name (text)
↩ back
external_id string str
What it does: Wikidata -- so that "Arthur Danto" and "Arthur C. Danto" are not counted twice.
What it holds: An authority identifier (text)
↩ back
senses array -> Sense[]
What it does: Which readings this person holds.
What it holds: A list of pointers to senses (reference to another record, not a copy of it)
↩ back
evidence array -> Attest[]
What it does: Where we know it from.
What it holds: A list of pointers to attestations (reference to another record, not a copy of it)

SenseRelation relation between readings

A weighted edge between two senses: who contradicts whom, who extends whom.

↩ back
from_sense record -> Sense
What it does: One end of the edge.
What it holds: A pointer to a sense (reference to another record, not a copy of it)
↩ back
to_sense record -> Sense
What it does: The other end.
What it holds: A pointer to a sense (reference to another record, not a copy of it)
↩ back
kind closed list contrasts|extends|narrows
What it does: Contradicts, extends, narrows -- and in future a weight on each edge.
What it holds: One of a closed list of values
↩ back
evidence record -> Attestation
What it does: The quotation that grounds the relation.
What it holds: A pointer to one attestation (reference to another record, not a copy of it)
↩ back
paper_id string str
What it does: Who asserts this relation.
What it holds: A paper identifier (text)

Attestation a single attestation

One mention in one paper. The schema is designed to hold the verbatim sentence and its locator; these fields remain unpopulated in the current public build. An attestation becomes evidentially auditable only when that grounding is present.

↩ back
id string str
What it does: The attestation's key.
What it holds: A short string (text)
↩ back
node_id record -> ConceptNode
What it does: Which concept or sub-type it belongs to. Always the most specific node that fits, otherwise information smears upward.
What it holds: A pointer to a concept record (reference to another record, not a copy of it)
↩ back
quote string str
What it does: The verbatim sentence from the paper. Must be an exact substring.
What it holds: The sentence itself (text)
↩ back
locator string str
What it does: Where in the text -- page or offset.
What it holds: A locator (text)
↩ back
role closed list str
What it does: What the mention does: defines, exemplifies, refutes, merely mentions.
What it holds: One of a closed list of values
↩ back
asserted closed list P|N|U
What it does: Yes / no / not adjudicated. The third value is what allows us to be honest about what the literature did not decide.
What it holds: One of three values
↩ back
object_level boolean bool
What it does: Whether this is an instance of the concept or a claim about it. Conflating the two destroyed an entire run.
What it holds: Yes or no
↩ back
kind closed list str
What it does: Artefact, performance, text, natural item, event, abstract. This is what makes it possible to define sub-concepts such as "an object" from the corpus rather than by assumption.
What it holds: One of a closed list of values
↑ back to contents
Appendix C — automated construction of candidate definitions
Generation from atomic conditions and the need for prior validity gates.

The exploratory procedure can assemble a candidate from atomic conditions extracted from definitions in the corpus and evaluate it under the same scoring framework.

Validity gates must precede optimisation. In an initial run, the candidate inventory included the circular condition “produced with the intention of making a work of art.” The search selected this condition because it maximised the observed score. Circular conditions are therefore marked in the data and excluded before candidate search; post hoc rejection would otherwise permit target leakage to determine the selected model.
Discrimination is constrained by corpus coverage. A later candidate used the property “exhibits an individual point of view” and scored highly until seven researcher-labelled negative controls were added. It admitted all seven. This sensitivity analysis shows that a corpus cannot test exclusions it does not contain. The controls are therefore stored separately from literature-grounded evidence, identified as researcher judgements, and reported both with and without inclusion. They probe failure modes but do not substitute for source-grounded negative cases.
↑ back to contents
Appendix D — reproducibility
Scripts, required inputs, and the boundary of the public package.

The analyses are scripted, but successful reproduction depends on access to the corresponding input files and on using the same instructions and software environment. The local research tree uses the following commands:

  • python TOOLS/final_ranking.py ART_CASE/cases_v3.json ART_CASE/art_v7_verdicts.json
  • python TOOLS/cluster_criteria.py then python TOOLS/cluster_k_sweep.py
  • python TOOLS/make_defs_kappa.py score KAPPA_STUDY_DEFS/responses.csv — requires the completed response file, which is not included in the public repository at this paper
  • python TOOLS/build_defs_preprint_en.py — this document

The threshold sweep for the cluster account is reported below as a sensitivity analysis:

k12345678910
MCC+0.282+0.620+0.770+0.826+0.882+0.847+0.781+0.738+0.660+0.576

Best fixed threshold k = 5 · worst k = 1 · range 0.6 MCC · with k chosen per case, MCC 1.0.

↑ back to contents
References
References.

Theory of definition

  • Belnap, N. D. (1993). On rigorous definitions. Philosophical Studies 72(2–3), 115–146. doi:10.1007/BF00989671
  • Gupta, A. & Mackereth, S. (2023). Definitions. The Stanford Encyclopedia of Philosophy (Winter 2023 ed.).
  • Suppes, P. (1957). Introduction to Logic. Van Nostrand.
  • Wittgenstein, L. (1953). Philosophical Investigations (G. E. M. Anscombe, trans.). Blackwell.

Terminological heterogeneity across disciplines

  • McPherson, M., Smith-Lovin, L. & Cook, J. M. (2001). Birds of a feather: homophily in social networks. Annual Review of Sociology 27, 415–444. doi:10.1146/annurev.soc.27.1.415
  • Newman, M. E. J. (2003). Mixing patterns in networks. Physical Review E 67, 026126. doi:10.1103/PhysRevE.67.026126
  • Mariani, B., Nicoletti, G., Bisio, M., Maschietto, M., Vassanelli, S. & Suweis, S. (2022). Disentangling the critical signatures of neural activity. Scientific Reports 12, 10770. doi:10.1038/s41598-022-13686-0
  • Entman, R. M. (1993). Framing: toward clarification of a fractured paradigm. Journal of Communication 43(4), 51–58. doi:10.1111/j.1460-2466.1993.tb01304.x

The definitional debate — the worked example

  • Ziff, P. (1953). The task of defining a work of art. The Philosophical Review 62(1), 58–78. doi:10.2307/2182722
  • Weitz, M. (1956). The role of theory in aesthetics. JAAC 15(1), 27–35. doi:10.2307/427491
  • Kennick, W. E. (1958). Does traditional aesthetics rest on a mistake? Mind 67(267), 317–334. doi:10.1093/mind/lxvii.267.317
  • Danto, A. (1964). The artworld. The Journal of Philosophy 61(19), 571–584. doi:10.2307/2022937
  • Dickie, G. (1974). Art and the Aesthetic: An Institutional Analysis. Cornell University Press.
  • Dickie, G. (1984). The Art Circle: A Theory of Art. Haven Publications.
  • Levinson, J. (1979). Defining art historically. The British Journal of Aesthetics 19(3), 232–250. doi:10.1093/bjaesthetics/19.3.232
  • Carroll, N. (1993). Historical narratives and the philosophy of art. JAAC 51(3). doi:10.2307/431506
  • Gaut, B. (2000). "Art" as a cluster concept. In N. Carroll (ed.), Theories of Art Today. University of Wisconsin Press.
  • Gaut, B. (2005). The cluster account of art defended. The British Journal of Aesthetics 45(3), 273–288. doi:10.1093/aesthj/ayi032
  • Dutton, D. (2006). A naturalist definition of art. JAAC 64(3), 367–377. doi:10.1111/j.1540-594X.2006.00217.x

Measurement

  • Matthews, B. W. (1975). Comparison of the predicted and observed secondary structure of T4 phage lysozyme. Biochimica et Biophysica Acta 405(2), 442–451. doi:10.1016/0005-2795(75)90109-9
  • Chicco, D. & Jurman, G. (2020). The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genomics 21, 6. doi:10.1186/s12864-019-6413-7
  • Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20(1), 37–46. doi:10.1177/001316446002000104
  • Landis, J. R. & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics 33(1), 159–174. doi:10.2307/2529310
  • Hayes, A. F. & Krippendorff, K. (2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures 1(1), 77–89. doi:10.1080/19312450709336664
  • Krippendorff, K. (2018). Content Analysis: An Introduction to Its Methodology (4th ed.). SAGE.
  • Shrout, P. E. & Fleiss, J. L. (1979). Intraclass correlations: uses in assessing rater reliability. Psychological Bulletin 86(2), 420–428. doi:10.1037/0033-2909.86.2.420
  • Efron, B. (1979). Bootstrap methods: another look at the jackknife. The Annals of Statistics 7(1), 1–26. doi:10.1214/aos/1176344552
  • McNemar, Q. (1947). Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika 12(2), 153–157. doi:10.1007/BF02295996

The community

  • Münz-Manor, O. & Marienberg-Milikowsky, I. (2023). Visualization of categorization: how to see the wood and the trees. Digital Humanities Quarterly 17(3).
  • Gutman, G. (2024). A network analysis of figurative topic classification. Digital Humanities Quarterly 18(3).
  • Dekel, Y. & Marienberg-Milikowsky, I. (2021). From distant to public reading. magazén 2(2), 225–252. doi:10.30687/mag/2724-3923/2021/04/003
  • Rivlin-Angert, N. & Mor-Lan, G. (2025). The enemy from within. EMNLP 2025, 16634–16647. doi:10.18653/v1/2025.emnlp-main.841
  • Sagi, T., Zaga, M., Rusinek, S., Lev, E., Lavee, M. & Hose, K. (2026). MEHDIE: the Middle East Heritage Data Integration Endeavor. Digital Scholarship in the Humanities 41(Suppl. 1), i238–i246. doi:10.1093/llc/fqaf111
↑ back to contents