Efficient Estimation of Word Representations in Vector Space

Efficient Estimation of Word Representations in Vector Space Tomas Mikolov Kai Chen Greg Corrado Jeffrey Dean

Tag layer of the DHSS Meta-Tagging Project. The article text is reproduced from the staged copy named in sourceDesc and is not altered; the tags are held separately in standOff so that overlapping and nested spans are preserved.

Efficient Estimation of Word Representations in Vector Space 2013 ICLR Workshop 1301.3781 mikolov2013-word2vec mikolov2013-word2vec.txt, the project's staged copy 1bc408302c036b05aa2d5c99878a5f30d44ecb64 arxiv-open linguistics

the paper's own words

the tag layer; a sentence written by the tagger about the paper, always anchored to a quotation but not itself a quotation

0 spans located by searching the stored text for the quote's exact words, allowing any whitespace between them method Quotation withheld because this record is not cleared for redistribution. distributed vector representations of words (word embeddings) Quotation withheld because this record is not cleared for redistribution. system-or-tool-building system-or-tool-building Quotation withheld because this record is not cleared for redistribution. mechanisms learn distributed word vectors with a shallow model - CBOW predicts a word from its context, skip-gram predicts the context from the word - so words in similar contexts get similar vectors Quotation withheld because this record is not cleared for redistribution. for many languages the available training corpora are only a few billion words or less (a data constraint) what this limits: the results for lower-resource languages: Quotation withheld Quotation withheld because this record is not cleared for redistribution. The exact-match accuracy metric makes 100% unreachable because the models have no morphological information and synonyms count as errors. what this limits: the accuracy figures against their own ceiling: REACHING 100% ACCURACY IS LIKELY TO BE IMPOSSIBLE, AS THE CURRENT MODELS DO NOT HAVE ANY INPUT INFORMATION ABOUT WORD MORPHOLOGY Quotation withheld because this record is not cleared for redistribution. The test set excludes multi-word entities. what this limits: everything the evaluation can speak about: WE HAVE INCLUDED IN OUR TEST SET ONLY SINGLE TOKEN WORDS, THUS MULTI-WORD ENTITIES ARE NOT PRESENT (SUCH AS NEW YORK) - what falls outside is invisible, not wrong Quotation withheld because this record is not cleared for redistribution. Relationship accuracy in the qualitative examples still leaves much room for improvement. what this limits: the qualitative relationship examples: ACCURACY IS QUITE GOOD, ALTHOUGH THERE IS CLEARLY A LOT OF ROOM FOR FURTHER IMPROVEMENTS Quotation withheld because this record is not cleared for redistribution. On the Microsoft Sentence Completion Challenge the Skip-gram model alone is worse than LSA similarity; it only wins in combination with RNNLMs. what this limits: the Sentence Completion result specifically: THE SKIP-GRAM MODEL ITSELF DOES NOT PERFORM ON THIS TASK BETTER THAN LSA SIMILARITY - it contributes only in combination Quotation withheld because this record is not cleared for redistribution. The benchmark's question count is combinatorial: questions are formed by pairing items from short hand-made lists, so the number of questions is far larger than the number of independent facts being tested. what this limits: the size of this test set as a measure of evidence: a list of 68 hand-listed American cities and their states was expanded into about 2,500 questions by pairing entries at random, so questions within a category are not independent of one another Quotation withheld because this record is not cleared for redistribution. In the semantic half of a semantic benchmark, a correct-meaning answer is scored wrong unless it is the exact expected token. what this limits: any accuracy figure from this test set: the answer must be the exact expected token, and synonyms are counted as mistakes Quotation withheld because this record is not cleared for redistribution. The connection between the benchmark and any practical use is asserted as a belief, not shown. what this limits: this accuracy as evidence that the vectors are useful: the authors state a belief that usefulness for certain applications should be positively correlated with the metric Quotation withheld because this record is not cleared for redistribution. distributed vector representations of words, taken from neural language models into large-scale computational linguistics and simplified until they could be trained on billions of words Quotation withheld because this record is not cleared for redistribution. the same data used to train a previously published recurrent neural network language model, kept as the comparison point eliminates: a speed or quality gain that comes from a different or larger corpus rather than from the model - the RNNLM baseline saw the same data · outcome (stated by the tagger, no quotation): held Quotation withheld because this record is not cleared for redistribution. skip-gram skip-gram Quotation withheld because this record is not cleared for redistribution. word vectors word vectors Quotation withheld because this record is not cleared for redistribution. cosine distance cosine distance Quotation withheld because this record is not cleared for redistribution. CBOW CBOW Quotation withheld because this record is not cleared for redistribution. semantic - AS FIVE HAND-CHOSEN RELATION TYPES: the semantic half of the test set is five relations (common capital city, all capital cities, currency, city-in-state, Man-Woman), so 'captures semantics' means 'answers analogies of those five kinds' Quotation withheld because this record is not cleared for redistribution. similarity - AS COSINE DISTANCE AFTER A VECTOR OFFSET, with the three input words removed from the search, so the answer is the nearest remaining word rather than the nearest word Quotation withheld because this record is not cleared for redistribution. the analogy relation - AS ARITHMETIC ON VECTORS: a relation between two words is the difference of their vectors, and it is applied to a third word by addition Quotation withheld because this record is not cleared for redistribution. builds-on → neural-network language models Quotation withheld because this record is not cleared for redistribution. extends → the two-step NNLM architecture of [13, 14] Quotation withheld because this record is not cleared for redistribution. builds-on → the authors' earlier two-step word-vector training work [13, 14] Quotation withheld because this record is not cleared for redistribution. builds-on → the model-complexity comparison framework of [18] Quotation withheld because this record is not cleared for redistribution. CBOW —contrasted-with→ Skip-gram CBOW Skip-gram strength of the relation: 1 Quotation withheld because this record is not cleared for redistribution. 1.6 billion — training corpus size; high-quality vectors learned in under a day Quotation withheld because this record is not cleared for redistribution. < 1 day — training time for high-quality word vectors Quotation withheld because this record is not cleared for redistribution. The learned vectors capture linear regularities: vector(King) - vector(Man) + vector(Woman) is closest to vector(Queen). about this quote: QUOTE REPAIRED: the extractor split a hyphen-space and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted. Quotation withheld because this record is not cleared for redistribution. model architecture model architecture about this quote: QUOTE REPAIRED: the extractor split a hyphen-space and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted. Quotation withheld because this record is not cleared for redistribution. Continuous Bag-of-Words Continuous Bag-of-Words Quotation withheld because this record is not cleared for redistribution. Continuous Skip-gram Model Continuous Skip-gram Model Quotation withheld because this record is not cleared for redistribution. correlation correlation Quotation withheld because this record is not cleared for redistribution.

The full text of this article is not redistributable, so it is not included here. The evidence sentences are withheld with it; the tag layer and its grounding status remain. The article itself is at https://arxiv.org/abs/1301.3781.