GloVe: Global Vectors for Word Representation

GloVe: Global Vectors for Word Representation Jeffrey Pennington Richard Socher Christopher D. Manning

Tag layer of the DHSS Meta-Tagging Project. The article text is reproduced from the staged copy named in sourceDesc and is not altered; the tags are held separately in standOff so that overlapping and nested spans are preserved.

GloVe: Global Vectors for Word Representation 2014 EMNLP 10.3115/v1/D14-1162 pennington2014-glove pennington2014-glove.txt, the project's staged copy 7e04499f0f4da137a1324b38d54fa808b5fcf7f8 open-access linguistics

the paper's own words

the tag layer; a sentence written by the tagger about the paper, always anchored to a quotation but not itself a quotation

0 spans located by searching the stored text for the quote's exact words, allowing any whitespace between them method Quotation withheld because this record is not cleared for redistribution. word vector representations from co-occurrence statistics Quotation withheld because this record is not cleared for redistribution. system-or-tool-building system-or-tool-building Quotation withheld because this record is not cleared for redistribution. mechanisms learn word vectors from the global co-occurrence matrix via a log-bilinear model on ratios of co-occurrence probabilities, so vector differences encode meaning Quotation withheld because this record is not cleared for redistribution. the origin of the linguistic regularities captured by word-vector arithmetic was not understood what this limits: the explanatory claim about vector arithmetic: Quotation withheld Quotation withheld because this record is not cleared for redistribution. The chosen exponent in the weighting function is justified only empirically, not derived. what this limits: every result produced with the weighting function: WE OFFER ONLY EMPIRICAL MOTIVATION FOR CHOOSING THE VALUE 3/4 - the exponent is fitted, not derived Quotation withheld because this record is not cleared for redistribution. The head-to-head comparison with word2vec is confounded by many performance-affecting parameters. what this limits: the head-to-head comparison with word2vec: A RIGOROUS QUANTITATIVE COMPARISON ... IS COMPLICATED BY THE EXISTENCE OF MANY PARAMETERS THAT HAVE A STRONG EFFECT ON PERFORMANCE Quotation withheld because this record is not cleared for redistribution. Unspecified parameters were left at defaults, a simplification the authors say a more thorough analysis should relax. what this limits: every reported comparison, and it is the confound item 2 names: WE SET ANY UNSPECIFIED PARAMETERS TO THEIR DEFAULT VALUES, ASSUMING THAT THEY ARE CLOSE TO OPTIMAL, THOUGH WE ACKNOWLEDGE THAT THIS SIMPLIFICATION SHO[uld be noted] Quotation withheld because this record is not cleared for redistribution. The largest-corpus (840B token) run used a different preprocessing, so its results are not comparable with the reported ones. what this limits: the scalability demonstration specifically: TO DEMONSTRATE THE SCALABILITY OF THE MODEL, WE ALSO TRAINED IT ON A MUCH LARGER SIXTH CORPUS, CONTAINING 840 BILLION TOKENS OF WEB DATA, BUT IN THIS [case the conditions differ] Quotation withheld because this record is not cleared for redistribution. The comparison that establishes the headline result runs the competing method at default settings, assumed near-optimal, and the paper says the assumption should be relaxed. what this limits: the claim that this model outperforms word2vec: unspecified parameters of the compared systems were left at default values on the assumption they are close to optimal, which the authors call a simplification that a more thorough analysis should relax Quotation withheld because this record is not cleared for redistribution. The exponent that shapes the weighting function has empirical motivation only. what this limits: results that depend on the weighting function: the authors state they offer only empirical motivation for the value 3/4, noting a similar fractional power scaling was found elsewhere Quotation withheld because this record is not cleared for redistribution. The distinction the model is built on is declared arbitrary by the authors. what this limits: any interpretation of the two vector sets as different kinds of thing: the paper states Quotation withheld and that the two roles may be exchanged Quotation withheld because this record is not cleared for redistribution. The largest corpus in the paper was processed differently, so its numbers do not sit on the same scale as the rest. what this limits: a comparison across the corpora reported here: for the 840-billion-token web corpus the vocabulary was not lowercased, and the authors state the results are therefore not directly comparable Quotation withheld because this record is not cleared for redistribution. The complexity result rests on an assumed distribution rather than a measured one. what this limits: the claim about how the model's cost scales: it requires assuming the number of co-occurrences of two words can be modelled as a power-law function of that pair's frequency rank Quotation withheld because this record is not cleared for redistribution. global matrix factorization (from latent semantic analysis) combined with local context windows to learn word vectors Quotation withheld because this record is not cleared for redistribution. holding corpus, vocabulary size, vector dimension and context window fixed across GloVe and word2vec, and plotting accuracy against training time eliminates: that the accuracy difference between GloVe and word2vec comes from differing hyperparameters or training budget rather than from the model itself · outcome: partial Quotation withheld because this record is not cleared for redistribution. word vector word vector Quotation withheld because this record is not cleared for redistribution. context word context word Quotation withheld because this record is not cleared for redistribution. word analogy word analogy Quotation withheld because this record is not cleared for redistribution. context window context window Quotation withheld because this record is not cleared for redistribution. syntactic subtask syntactic subtask Quotation withheld because this record is not cleared for redistribution. what the paper calls its own object - WORD VECTORS AND WORD REPRESENTATIONS, NEVER EMBEDDINGS: it uses those two terms 24 and 11 times; 'word embeddings' appears exactly once and describes another group's method rather than this paper's output, and 'distributional semantics' does not appear at all Quotation withheld because this record is not cleared for redistribution. 'word embeddings' IN THIS PAPER - SOMEONE ELSE'S OBJECT: the single occurrence attributes the term to Levy et al., alongside vLBL and ivLBL, as a rival PPMI-based approach; the authors never apply the word to what they themselves produce Quotation withheld because this record is not cleared for redistribution. the analogy evaluation - AS DIMENSIONS OF DIFFERENCE, NOT DISTANCE: it probes the vector space by examining the various dimensions of difference between vectors rather than the scalar distance between them, which is why a model can win on similarity and lose on analogy Quotation withheld because this record is not cleared for redistribution. the context window - SYMMETRIC OR ASYMMETRIC, AND THE CHOICE MOVES ONE SUBTASK ONLY: a window extending both ways is called symmetric and one extending only left asymmetric, and the syntactic subtask specifically improves with small asymmetric windows Quotation withheld because this record is not cleared for redistribution. claim_type: method a second reading reaching the same value. Recorded: method — note: A BLIND second reading by the project on 2026-08-25: the record's own claim type/method/phenomenon were not looked at until every value here was written and substring-verified. Not applied - a person decides. Sampling frame: the 504 staged records carrying no alternatives block, so this never re-measures the 202. All 11 records with 130+ tags were taken as a census and 19 more spread one per discipline across 22 disciplines - the dense non-neuroscience cell is the one the 202 do not contain (inside them, 94-100% of the densest bands are neuroscience; among these 11, one is)., Quotation withheld because this record is not cleared for redistribution. method: system-or-tool-building a second reading reaching the same value. Recorded: system-or-tool-building — note: A BLIND second reading by the project on 2026-08-25: the record's own claim type/method/phenomenon were not looked at until every value here was written and substring-verified. Not applied - a person decides. Sampling frame: the 504 staged records carrying no alternatives block, so this never re-measures the 202. All 11 records with 130+ tags were taken as a census and 19 more spread one per discipline across 22 disciplines - the dense non-neuroscience cell is the one the 202 do not contain (inside them, 94-100% of the densest bands are neuroscience; among these 11, one is)., Quotation withheld because this record is not cleared for redistribution. phenomenon: why linear regularities emerge in word vector spaces learned from co-occurrence statistics a second reading DIFFERING from the recorded value. Recorded: word vector representations from co-occurrence statistics — note: A BLIND second reading by the project on 2026-08-25: the record's own claim type/method/phenomenon were not looked at until every value here was written and substring-verified. Not applied - a person decides. Sampling frame: the 504 staged records carrying no alternatives block, so this never re-measures the 202. All 11 records with 130+ tags were taken as a census and 19 more spread one per discipline across 22 disciplines - the dense non-neuroscience cell is the one the 202 do not contain (inside them, 94-100% of the densest bands are neuroscience; among these 11, one is)., Quotation withheld because this record is not cleared for redistribution. builds-on → global matrix factorization (LSA) and local context-window methods Quotation withheld because this record is not cleared for redistribution. contradicts → Baroni et al. (2014) Quotation withheld because this record is not cleared for redistribution. extends → skip-gram and ivLBL window-based models Quotation withheld because this record is not cleared for redistribution. context window —causes→ syntactic subtask context window syntactic subtask strength of the relation: 0.8 Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but the span says performance on the syntactic subtask is better for small context windows; causes is loose. Quotation withheld because this record is not cleared for redistribution. 75% — accuracy on the word analogy task (GloVe headline result) Quotation withheld because this record is not cleared for redistribution. 6 billion — corpus size used to build the co-occurrence statistics Quotation withheld because this record is not cleared for redistribution. GloVe is a new global log-bilinear regression model combining the advantages of global matrix factorization and local context window methods. about this quote: QUOTE REPAIRED: the extractor split a hyphen-space and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted. Quotation withheld because this record is not cleared for redistribution. Only in the ratio of co-occurrence probabilities does noise from non-discriminative words cancel out. Quotation withheld because this record is not cleared for redistribution. word-word co-occurrence counts word-word co-occurrence counts about this quote: QUOTE REPAIRED: the extractor split a hyphen-space and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted. Quotation withheld because this record is not cleared for redistribution. GloVe GloVe about this quote: QUOTE REPAIRED: the extractor split a hyphen-space and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted. Quotation withheld because this record is not cleared for redistribution. word analogy task word analogy task Quotation withheld because this record is not cleared for redistribution. correlation correlation Quotation withheld because this record is not cleared for redistribution. regression regression about this quote: QUOTE REPAIRED: the extractor split a hyphen-space and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted. Quotation withheld because this record is not cleared for redistribution.

The full text of this article is not redistributable, so it is not included here. The evidence sentences are withheld with it; the tag layer and its grounding status remain. The article itself is at https://doi.org/10.3115/v1/D14-1162.