Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change

Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change William L. Hamilton Jure Leskovec Dan Jurafsky

Tag layer of the DHSS Meta-Tagging Project. The article text is reproduced from the staged copy named in sourceDesc and is not altered; the tags are held separately in standOff so that overlapping and nested spans are preserved.

Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change 2016 ACL / arXiv 10.18653/v1/P16-1141 1605.09096 hamilton2016-diachronic-embeddings hamilton2016-diachronic-embeddings.txt, the project's staged copy 158f1a98d9a8eae1267766dc1c40e0d3be7db47e arxiv-open digital-humanities

the paper's own words

the tag layer; a sentence written by the tagger about the paper, always anchored to a quotation but not itself a quotation

0 spans located by searching the stored text for the quote's exact words, allowing any whitespace between them existence Quotation withheld because this record is not cleared for redistribution. semantic change Quotation withheld because this record is not cleared for redistribution. corpus-analysis corpus-analysis Quotation withheld because this record is not cleared for redistribution. mechanisms statistical laws of semantic change: the rate of change scales as an inverse power-law of word frequency (law of conformity) and rises with polysemy (law of innovation) Quotation withheld because this record is not cleared for redistribution. The paper's results are correlational; it explicitly declines to claim the causal mechanisms it speculates about. what this limits: the mechanistic story: WHILE OUR RESULTS CANNOT CONFIRM SUCH CAUSAL LINKS, THEY NONETHELESS HIGHLIGHT A NEW ROLE FOR FREQUENCY AND POLYSEMY - correlational by the authors' own statement Quotation withheld because this record is not cleared for redistribution. The polysemy proxy (local clustering coefficient in a PPMI co-occurrence network) is biased towards rating contextually diverse function words as highly polysemous. what this limits: every polysemy score: the measure IS SLIGHTLY BIASED TOWARDS RATING CONTEXTUALLY DIVERSE DISCOURSE FUNCTION WORDS (E.G., ALSO) AS HIGHLY POLYSEMOUS Quotation withheld because this record is not cleared for redistribution. Polysemy has no agreed ground truth to validate the measure against. what this limits: any validation of a polysemy measure at all: MEASURING WORD POLYSEMY IS A DIFFICULT AND FRAUGHT TASK, AS EVEN 'GROUND TRUTH' DICTIONARIES DIFFER IN THE NUMBER OF SENSES THEY ASSIGN TO WORDS Quotation withheld because this record is not cleared for redistribution. The main corpora contain artifacts from shifting sampling biases over time. what this limits: every result computed on that corpus: THE GOOGLE N-GRAM DATASETS ... CONTAIN MANY CORPUS ARTIFACTS DUE, E.G., TO SHIFTING SAMPLING BIASES OVER TIME Quotation withheld because this record is not cleared for redistribution. Part of the polysemy result (the negative curvature at high polysemy) is not robust across datasets or methods. what this limits: the polysemy curvature result specifically: IT VARIES ACROSS DATASETS AND WAS NOT PRESENT WITH SGNS, SO IT IS NOT AS ROBUST AS THE CLEAR LINEAR TREND Quotation withheld because this record is not cleared for redistribution. The paper's second law is stated in two logically different forms, two pages apart. what this limits: any citation of the law of innovation: the abstract states it as holding independent of frequency, while the introduction states it as holding after controlling for frequency and adds 'significantly', and the two are not the same claim Quotation withheld because this record is not cleared for redistribution. One of the three methods is stated to be prone to false discoveries. what this limits: any result computed with PPMI embeddings: the authors state Quotation withheld because this record is not cleared for redistribution. The choice between the two remaining methods is left unsettled by the paper's own benchmarks. what this limits: any rate of semantic change quoted from this work: between SVD and SGNS the results are described as somewhat equivocal, with each performing best on two of the four tasks Quotation withheld because this record is not cleared for redistribution. word embeddings (word2vec) applied across time to quantify historical semantic change Quotation withheld because this record is not cleared for redistribution. word frequency, in the law of innovation eliminates: the reading that polysemous words change faster only because they are rarer · outcome (stated by the tagger, no quotation): held Quotation withheld because this record is not cleared for redistribution. semantic change semantic change Quotation withheld because this record is not cleared for redistribution. law of innovation law of innovation Quotation withheld because this record is not cleared for redistribution. law of conformity law of conformity Quotation withheld because this record is not cleared for redistribution. polysemy polysemy Quotation withheld because this record is not cleared for redistribution. word frequency word frequency Quotation withheld because this record is not cleared for redistribution. PPMI PPMI Quotation withheld because this record is not cleared for redistribution. distributional methods distributional methods Quotation withheld because this record is not cleared for redistribution. co-occurrence relationships co-occurrence relationships Quotation withheld because this record is not cleared for redistribution. distributional hypothesis distributional hypothesis Quotation withheld because this record is not cleared for redistribution. semantic similarity/distance semantic similarity/distance Quotation withheld because this record is not cleared for redistribution. cosine similarity/distance cosine similarity/distance Quotation withheld because this record is not cleared for redistribution. SVD embeddings SVD embeddings Quotation withheld because this record is not cleared for redistribution. low-dimensional approximations of the PPMI embeddings low-dimensional approximations of the PPMI embeddings Quotation withheld because this record is not cleared for redistribution. dimensionality reduction dimensionality reduction Quotation withheld because this record is not cleared for redistribution. regularization regularization Quotation withheld because this record is not cleared for redistribution. SGNS ‘neural’ embeddings SGNS ‘neural’ embeddings Quotation withheld because this record is not cleared for redistribution. skip-gram with negative sampling skip-gram with negative sampling Quotation withheld because this record is not cleared for redistribution. incremental initialization incremental initialization Quotation withheld because this record is not cleared for redistribution. Google N-Gram datasets Google N-Gram datasets Quotation withheld because this record is not cleared for redistribution. corpus artifacts corpus artifacts Quotation withheld because this record is not cleared for redistribution. COHA corpus COHA corpus Quotation withheld because this record is not cleared for redistribution. orthogonal Procrustes orthogonal Procrustes Quotation withheld because this record is not cleared for redistribution. rotational alignment rotational alignment Quotation withheld because this record is not cleared for redistribution. Pair-wise similarity time-series Pair-wise similarity time-series Quotation withheld because this record is not cleared for redistribution. pairwise similarity shifts pairwise similarity shifts Quotation withheld because this record is not cleared for redistribution. Spearman correlation Spearman correlation Quotation withheld because this record is not cleared for redistribution. semantic displacement semantic displacement Quotation withheld because this record is not cleared for redistribution. cos-dist cos-dist Quotation withheld because this record is not cleared for redistribution. synchronic accuracy synchronic accuracy Quotation withheld because this record is not cleared for redistribution. diachronic validity diachronic validity Quotation withheld because this record is not cleared for redistribution. detecting known shifts detecting known shifts Quotation withheld because this record is not cleared for redistribution. discovering shifts from data discovering shifts from data Quotation withheld because this record is not cleared for redistribution. linear mixed model linear mixed model Quotation withheld because this record is not cleared for redistribution. random intercepts per word random intercepts per word Quotation withheld because this record is not cleared for redistribution. fixed effects per decade fixed effects per decade Quotation withheld because this record is not cleared for redistribution. contextual diversity contextual diversity Quotation withheld because this record is not cleared for redistribution. polysemousness polysemousness Quotation withheld because this record is not cleared for redistribution. empirical co-occurrence network empirical co-occurrence network Quotation withheld because this record is not cleared for redistribution. clustering coefficient clustering coefficient Quotation withheld because this record is not cleared for redistribution. discourse function words discourse function words Quotation withheld because this record is not cleared for redistribution. controlling for frequency controlling for frequency Quotation withheld because this record is not cleared for redistribution. null model null model Quotation withheld because this record is not cleared for redistribution. learning learning Quotation withheld because this record is not cleared for redistribution. sociocultural conformity bias sociocultural conformity bias Quotation withheld because this record is not cleared for redistribution. SGNS → skip-gram with negative sampling SGNS abbreviates skip-gram with negative sampling; acronym-expansion is absent from the edge registry. Quotation withheld because this record is not cleared for redistribution. SVD → singular value decomposition SVD abbreviates singular value decomposition; acronym-expansion is absent from the edge registry. Quotation withheld because this record is not cleared for redistribution. clustering coefficient → polysemousness A lower clustering coefficient indicates higher polysemousness; inverse-association is absent from the edge registry. Quotation withheld because this record is not cleared for redistribution. learning → law of conformity Learning errors are offered as a possible mechanism for the conformity law; hypothesized-mechanism-for is absent from the edge registry. Quotation withheld because this record is not cleared for redistribution. sociocultural conformity bias → law of conformity Sociocultural conformity bias is offered as another possible mechanism; hypothesized-mechanism-for is absent from the edge registry. Quotation withheld because this record is not cleared for redistribution. rotational alignment → orthogonal Procrustes Orthogonal Procrustes is used to obtain rotational alignment; implemented-by is absent from the edge registry. Quotation withheld because this record is not cleared for redistribution. polysemy → contextual diversity Contextual diversity is used as a proxy for polysemy; proxy-for is absent from the edge registry. Quotation withheld because this record is not cleared for redistribution. the law of innovation - TWO STATEMENTS, ONE NAME: the abstract has it holding 'independent of frequency' and the introduction 'after controlling for frequency', with 'significantly' added in the second; independence and a residual surviving adjustment are different claims, and the name that travels does not say which one is meant Quotation withheld because this record is not cleared for redistribution. a rate of semantic change - A NUMBER PRODUCED BY A CHOICE OF EMBEDDING: PPMI, SVD and SGNS are ranked against each other by the paper itself, PPMI is called clearly worse and prone to false discoveries from genre shifts, and between the other two the result is equivocal - so the construct's value depends on an instrument the paper declines to settle Quotation withheld because this record is not cleared for redistribution. a law - PROPOSED, NOT DERIVED: the authors say they PROPOSE two statistical laws relating frequency and polysemy to semantic change, fitted over six corpora, and the record's existing limitation already records that the analysis is correlational and the causal mechanisms are declined Quotation withheld because this record is not cleared for redistribution. Distributional hypothesis - word meaning represented implicitly by co-occurrence relationships Quotation withheld because this record is not cleared for redistribution. SVD embedding - low-dimensional regularized approximation to a PPMI representation Quotation withheld because this record is not cleared for redistribution. SGNS embedding - word/context vectors trained with skip-gram negative sampling Quotation withheld because this record is not cleared for redistribution. Google N-Grams / COHA - very large artifact-prone sample versus smaller genre-balanced corpus Quotation withheld because this record is not cleared for redistribution. Orthogonal Procrustes - cosine-preserving rotational alignment of embeddings across periods Quotation withheld because this record is not cleared for redistribution. Pair-wise similarity series - changing cosine similarity summarized against time by Spearman correlation Quotation withheld because this record is not cleared for redistribution. Semantic displacement - cosine distance between one word’s aligned representations at different times Quotation withheld because this record is not cleared for redistribution. Synchronic accuracy / diachronic validity - within-period similarity versus across-time change performance Quotation withheld because this record is not cleared for redistribution. Detection / discovery benchmarks - recovery of known shifts versus expert judgment of inferred shifts Quotation withheld because this record is not cleared for redistribution. SVD / SGNS trade-off - sensitivity to subtle shifts versus robustness to corpus artifacts Quotation withheld because this record is not cleared for redistribution. Linear mixed model - word random intercepts and decade fixed effects for displacement regression Quotation withheld because this record is not cleared for redistribution. Contextual diversity - network-neighborhood proxy for polysemousness Quotation withheld because this record is not cleared for redistribution. Local clustering coefficient - inverse polysemy score from co-occurrence-neighbor connectivity Quotation withheld because this record is not cleared for redistribution. Frequency control - necessary because polysemy proxy and word frequency are intrinsically correlated Quotation withheld because this record is not cleared for redistribution. Law of conformity null model - frequency-driven baseline for other semantic-change interactions Quotation withheld because this record is not cleared for redistribution. builds-on → prior distributional work on semantic change (Sagi et al., 2011; Wijaya and Yeniterzi, 2011; Gulordava and Baroni, 2011; Jatowt and Duh, 2014; Kulkarni et al., 2014; Xu and Kemp, 2015) and novel-sense detection (Lau et al., 2012; Mitra et al., 2014; Cook et al., 2014; Mitra et al., 2015; Frermann and Lapata, 2016) Quotation withheld because this record is not cleared for redistribution. extends → the same lines of quantitative / distributional work on semantic change Quotation withheld because this record is not cleared for redistribution. contradicts → Wilkins (1993); Hopper and Traugott (2003), who argued that semantic change leads to polysemy Quotation withheld because this record is not cleared for redistribution. replicates → Levy et al. (2015) Quotation withheld because this record is not cleared for redistribution. word frequency —causes→ semantic change word frequency semantic change direction on the measure: reduces · strength of the relation: 0.8 Quotation withheld because this record is not cleared for redistribution. polysemy —causes→ semantic change polysemy semantic change holds when: after controlling for word frequency · direction on the measure: increases · strength of the relation: 0.8 Quotation withheld because this record is not cleared for redistribution. distributional hypothesis —operationalised-as→ co-occurrence relationships distributional hypothesis co-occurrence relationships the relation itself is separately evidenced · strength of the relation: 1 · note: The paper explicitly says the methods operationalize the hypothesis through co-occurrence. Quotation withheld because this record is not cleared for redistribution. semantic similarity/distance —operationalised-as→ cosine similarity/distance semantic similarity/distance cosine similarity/distance the relation itself is separately evidenced · strength of the relation: 1 · note: Semantic similarity/distance is approximated by cosine similarity/distance. Quotation withheld because this record is not cleared for redistribution. SVD embeddings —is-a→ low-dimensional approximations of the PPMI embeddings SVD embeddings low-dimensional approximations of the PPMI embeddings the relation itself is separately evidenced · strength of the relation: 1 · note: The paper states SVD embeddings correspond to low-dimensional PPMI approximations. Quotation withheld because this record is not cleared for redistribution. pairwise similarity shifts —operationalised-as→ Spearman correlation pairwise similarity shifts Spearman correlation the relation itself is separately evidenced · strength of the relation: 1 · note: Pairwise shifts are assessed with Spearman correlation against time. Quotation withheld because this record is not cleared for redistribution. semantic change —operationalised-as→ cos-dist semantic change cos-dist the relation itself is separately evidenced · strength of the relation: 1 · note: Cosine distance between aligned representations is explicitly used as a semantic-change measure. Quotation withheld because this record is not cleared for redistribution. SVD embeddings —contrasted-with→ SGNS ‘neural’ embeddings SVD embeddings SGNS ‘neural’ embeddings the relation itself is separately evidenced · strength of the relation: 1 · note: The paper contrasts SVD sensitivity with SGNS robustness and their task strengths. Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but the span compares the methods 'SVD' and 'SGNS', not 'SVD embeddings' / 'SGNS ‘neural’ embeddings'. Quotation withheld because this record is not cleared for redistribution. 6 corpora / 4 languages / 2 centuries — empirical scope of the study Quotation withheld because this record is not cleared for redistribution. inverse power-law — how semantic-change rate scales with word frequency (law of conformity) Quotation withheld because this record is not cleared for redistribution. Two quantitative laws of semantic change: conformity (change rate falls as an inverse power of word frequency) and innovation (more polysemous words change faster, independent of frequency). about this quote: QUOTE REPAIRED: the extractor split a hyphen-space and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted. Quotation withheld because this record is not cleared for redistribution. rate of semantic change rate of semantic change Quotation withheld because this record is not cleared for redistribution. law of conformity law of conformity Quotation withheld because this record is not cleared for redistribution. law of innovation law of innovation Quotation withheld because this record is not cleared for redistribution. Bayesian Bayesian about this quote: QUOTE REPAIRED: the extractor split a hyphen-space and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted. Quotation withheld because this record is not cleared for redistribution. correlation correlation about this quote: QUOTE REPAIRED: the extractor split a hyphen-space and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted. Quotation withheld because this record is not cleared for redistribution. hypothesis-testing hypothesis-testing Quotation withheld because this record is not cleared for redistribution. statistical-significance statistical-significance Quotation withheld because this record is not cleared for redistribution. statistical-significance statistical-significance Quotation withheld because this record is not cleared for redistribution.

The full text of this article is not redistributable, so it is not included here. The evidence sentences are withheld with it; the tag layer and its grounding status remain. The article itself is at https://doi.org/10.18653/v1/P16-1141.