Natural Language Processing (Almost) from Scratch

Natural Language Processing (Almost) from Scratch Ronan Collobert Jason Weston Léon Bottou Michael Karlen

Tag layer of the DHSS Meta-Tagging Project. The article text is reproduced from the staged copy named in sourceDesc and is not altered; the tags are held separately in standOff so that overlapping and nested spans are preserved.

Natural Language Processing (Almost) from Scratch 2011 Journal of Machine Learning Research collobert2011-nlp-scratch collobert2011-nlp-scratch.txt, the project's staged copy 1c9daca357f18ffa807edc132afe7903ce1b7afb jmlr-open linguistics

the paper's own words

the tag layer; a sentence written by the tagger about the paper, always anchored to a quotation but not itself a quotation

0 spans located by searching the stored text for the quote's exact words, allowing any whitespace between them method Quotation withheld because this record is not cleared for redistribution. neural sequence labelling across NLP benchmark tasks Quotation withheld because this record is not cleared for redistribution. system-or-tool-building system-or-tool-building Quotation withheld because this record is not cleared for redistribution. mechanisms THE FIELD'S GOAL IS POSED AS A QUESTION AND THEN REPORTED TO HAVE NO AGREED FORM: will a program ever convert English text into a data structure that UNAMBIGUOUSLY AND COMPLETELY DESCRIBES THE MEANING - and AMONG NUMEROUS PROBLEMS, NO CONSENSUS HAS EMERGED ABOUT THE FORM OF SUCH A DATA STRUCTURE. The obstacle named first is not difficulty but the absence of agreement on what the answer would look like Quotation withheld because this record is not cleared for redistribution. mechanisms THE FIELD'S TASKS ARE NAMED AS REDUCED OBJECTIVES ADOPTED BECAUSE THE REAL QUESTION IS UNRESOLVED: UNTIL SUCH FUNDAMENTAL PROBLEMS ARE RESOLVED, computer scientists MUST SETTLE FOR REDUCED OBJECTIVES - extracting SIMPLER REPRESENTATIONS describing RESTRICTED ASPECTS of the textual information. What the field measures itself on is described as a substitute, in the paper's second sentence Quotation withheld because this record is not cleared for redistribution. mechanisms THE SAME REPRESENTATIONS ARE GIVEN TWO DIFFERENT WARRANTS, AND ONE OF THEM IS A BELIEF: they are OFTEN MOTIVATED BY SPECIFIC APPLICATIONS, for instance bag-of-words for information retrieval - and they CAN ALSO BE MOTIVATED BY OUR BELIEF THAT THEY CAPTURE SOMETHING MORE GENERAL about natural language. An instrumental justification and an epistemic one, offered for the same objects Quotation withheld because this record is not cleared for redistribution. mechanisms THE ANNOTATIONS EXIST IN ORDER TO MAKE SYSTEMS COMPARABLE, AND THE BENCHMARKS ARE CREDITED WITH DRIVING THE RESEARCH: text corpora HAVE BEEN MANUALLY ANNOTATED with such data structures IN ORDER TO COMPARE THE PERFORMANCE of various systems, and THE AVAILABILITY OF STANDARD BENCHMARKS HAS STIMULATED RESEARCH. The ground truth's purpose is comparison, and the comparison is what made the field move about this quote: RE-SLICED, AND THE GUARD THAT CAUGHT IT WAS TWENTY MINUTES OLD. My end key was "...Natural Language Processing (NLP) and" - ending on "and" - because the page furniture ("Keywords: ... arXiv 1. Introduction") is spliced into that sentence and its real continuation is on the far side of the header. The dangling-word check refused it at drafting time. Quotation withheld because this record is not cleared for redistribution. mechanisms THE METHOD IS DEFINED BY WHAT IT REFUSES TO USE, AND THE COST IS STATED IN THE SAME SENTENCE: the versatility is achieved BY TRYING TO AVOID TASK-SPECIFIC ENGINEERING AND THEREFORE DISREGARDING A LOT OF PRIOR KNOWLEDGE - Quotation withheld. What is given up is named as the price of what is gained Quotation withheld because this record is not cleared for redistribution. mechanisms THE GROUND TRUTH ITSELF COMES IN TWO VOCABULARIES, AND THE FIX IS TO PICK ONE: the ground truth for three of the tasks IS PROVIDED USING TWO DIFFERENT TAGGING SCHEMES, and IN ORDER TO ELIMINATE THIS ADDITIONAL SOURCE OF VARIATIONS the authors DECIDED TO USE THE MOST EXPRESSIVE scheme FOR ALL TASKS. Two naming systems for the same annotations, normalised by choosing the more expressive - not the more common Quotation withheld because this record is not cleared for redistribution. mechanisms THE CHOICE OF HYPER-PARAMETERS IS CONCEDED TO BE ONE OF MANY EQUALLY GOOD ONES, WITH THE UNEXPLORED SPACE NAMED: although they ended up with A PARTICULAR CHOICE, MANY OTHER CHOICES ARE ALMOST EQUALLY AS GOOD - ALTHOUGH PERHAPS THERE ARE OTHERS THAT ARE BETTER, AS WE COULD NOT DO A FULL GRID SEARCH. The reported configuration is presented as a sample rather than an optimum Quotation withheld because this record is not cleared for redistribution. mechanisms THE SCALE OF THE EXPERIMENT IS ATTRIBUTED TO THE HARDWARE RATHER THAN TO THE SCIENCE, WRYLY: very long training times MAKE SUCH STRATEGIES NECESSARY FOR THE FORESEEABLE FUTURE - and IF WE HAD BEEN GIVEN COMPUTERS TEN TIMES FASTER, WE PROBABLY WOULD HAVE FOUND USES FOR DATASETS TEN TIMES BIGGER. The design is described as a function of the machines available Quotation withheld because this record is not cleared for redistribution. mechanisms TWO NEGATIVE RESULTS FROM OTHER GROUPS ARE CITED SIDE BY SIDE: one group obtained results SOMEWHAT BELOW THE STATE-OF-THE-ART using joint decoding for two tasks, and another DESCRIBED A NEGATIVE RESULT AT THE SAME JOINT TASK. Failures are gathered as evidence about a practice rather than about the groups that attempted it Quotation withheld because this record is not cleared for redistribution. avoiding task-specific features meant forgoing a large body of linguistic knowledge Quotation withheld because this record is not cleared for redistribution. The system is only 'almost' from scratch: words are lowercased and capitalization encoded, and learning from letter sequences was left out of scope. what this limits: the from-scratch claim: IDEALLY, AN EVEN MORE RAW INPUT WOULD BE TO LEARN FROM LETTER SEQUENCES RATHER THAN WORDS - words are lowercased and capitalisation encoded separately Quotation withheld because this record is not cleared for redistribution. Week-long training times made a proper hyperparameter search infeasible. what this limits: every hyperparameter-dependent result: TRAINING TIMES FOR SUCH LARGE SCALE SYSTEMS ARE COUNTED IN WEEKS, [SO] IT IS NOT FEASIBLE TO TRY MANY COMBINATIONS Quotation withheld because this record is not cleared for redistribution. Multi-task learning gave no or only marginal gains over separately trained per-task architectures. what this limits: the multi-task claim: NO (OR ONLY MARGINAL) IMPROVEMENTS WERE OBTAINED with the unified architecture over separately trained ones Quotation withheld because this record is not cleared for redistribution. The language model structure is probably too restrictive to capture hierarchical syntax; only the word embeddings are exploited downstream. what this limits: any syntactic reading of the language model: ITS STRUCTURE IS PROBABLY TOO RESTRICTIVE FOR SUCH GOALS, and only the word embeddings are exploited Quotation withheld because this record is not cleared for redistribution. The benchmark scores are treated as indirect measurements of something the paper does not measure directly, and the claim about it is a posit. what this limits: the benchmark tables reported here: the authors view the benchmarks as indirect measurements of the relevance of the internal representations, and posit that those representations are more general than any of the benchmarks Quotation withheld because this record is not cleared for redistribution. The paper names the failure mode its own design is exposed to. what this limits: the comparison of these results against other systems: the authors note there is a great temptation to optimize a system's performance for a specific benchmark Quotation withheld because this record is not cleared for redistribution. The pre-processing the architecture does keep is named, with a presumably attached to removing it. what this limits: the claim that this system works from raw input: the authors state they did some pre-processing, namely lowercasing and encoding capitalization as a separate feature, and say that with enough unlabeled data they could presumably learn without it Quotation withheld because this record is not cleared for redistribution. Task-specific engineering is added back in a later section, so the headline architecture and the best numbers are not the same system. what this limits: the best figures quoted from this paper: a later section evaluates how much further improvement comes from incorporating standard NLP task-specific engineering, which the main architecture avoids Quotation withheld because this record is not cleared for redistribution. the word embedding dimension held constant at 50 across the tasks, so the language models of the earlier section could be reused unchanged eliminates: a task-to-task difference that is really a difference in embedding capacity · outcome (stated by the tagger, no quotation): held Quotation withheld because this record is not cleared for redistribution. performance also reported with the embeddings FROZEN during task-specific training, beside the figures where they are allowed to move eliminates: attributing to the pre-trained embeddings a gain that actually comes from fine-tuning them on the task · outcome (stated by the tagger, no quotation): held Quotation withheld because this record is not cleared for redistribution. benchmark benchmark Quotation withheld because this record is not cleared for redistribution. internal representations internal representations Quotation withheld because this record is not cleared for redistribution. linguistic knowledge linguistic knowledge Quotation withheld because this record is not cleared for redistribution. lookup table lookup table Quotation withheld because this record is not cleared for redistribution. a priori NLP knowledge a priori NLP knowledge Quotation withheld because this record is not cleared for redistribution. 'almost from scratch' - THE PAPER NAMES ITS OWN HEDGE AND GIVES THE REASON: it calls the approach almost from scratch to emphasise a reduced but still important reliance on a priori NLP knowledge, while the body says 'from scratch' twenty-seven times and 'almost from scratch' once Quotation withheld because this record is not cleared for redistribution. a benchmark - AS AN INDIRECT MEASUREMENT, NOT A GOAL: the scores stand in for the relevance of internal representations that nobody measures directly, and the authors posit those representations are more general than the benchmarks that stand in for them Quotation withheld because this record is not cleared for redistribution. the cost of the design - LINGUISTIC KNOWLEDGE, DELIBERATELY SET ASIDE: avoiding task-specific engineered features is stated to have led the authors to ignore a large body of linguistic knowledge, so the architecture's generality and its ignorance are the same decision Quotation withheld because this record is not cleared for redistribution. contradicts → the reliance on task-specific hand-engineered features in NLP differs in: THE PARAMETER IS WHERE THE FEATURES COME FROM. The span: 'Quotation withheld' 'Instead of' names the axis of disagreement in the same clause as the disagreement. Quotation withheld because this record is not cleared for redistribution. builds-on → the convolutional / Time Delay Neural Network approach of Waibel et al. (1989) Quotation withheld because this record is not cleared for redistribution. extends → Conditional Random Fields (Lafferty et al., 2001) Quotation withheld because this record is not cleared for redistribution. builds-on → graph transformer networks (Bottou et al., 1997; Le Cun et al., 1998) Quotation withheld because this record is not cleared for redistribution. internal representations —operationalised-as→ benchmark internal representations benchmark strength of the relation: 1 Quotation withheld because this record is not cleared for redistribution. 852 — unlabeled corpus used for language modeling Quotation withheld because this record is not cleared for redistribution. 4 — benchmark tasks (POS, CHUNK, NER, SRL) Quotation withheld because this record is not cleared for redistribution. 912,344 — POS training-set size (WSJ sections) Quotation withheld because this record is not cleared for redistribution. one unified network handles many NLP tasks without task-specific engineering about this quote: QUOTE RECOVERED FROM THE PAPER'S OWN CHARACTERS. The recorded quote was LETTERS-ONLY IDENTICAL to this stretch of the staged text but not an exact substring of it - spaces, hyphens, ligature glyphs or mathematical operators differed, so it could never anchor. This is the file's own text between the same two letters: not produced by a rule about the damage, copied out of the file. Round-trip checked (the recovered span is letters-only equal to the original) and length-checked. The text as originally recorded is kept in evidence as extracted. Quotation withheld because this record is not cleared for redistribution. the system learns internal representations from mostly unlabeled data Quotation withheld because this record is not cleared for redistribution. transferring unsupervised representations improves supervised benchmarks Quotation withheld because this record is not cleared for redistribution. avoiding task-specific engineering deliberately disregards prior knowledge Quotation withheld because this record is not cleared for redistribution. internal representations internal representations Quotation withheld because this record is not cleared for redistribution. unlabeled training data unlabeled training data Quotation withheld because this record is not cleared for redistribution. benchmark tasks benchmark tasks Quotation withheld because this record is not cleared for redistribution. ad-hoc features ad-hoc features Quotation withheld because this record is not cleared for redistribution. almost from scratch almost from scratch about this quote: QUOTE RECOVERED FROM THE PAPER'S OWN CHARACTERS. The recorded quote was LETTERS-ONLY IDENTICAL to this stretch of the staged text but not an exact substring of it - spaces, hyphens, ligature glyphs or mathematical operators differed, so it could never anchor. This is the file's own text between the same two letters: not produced by a rule about the damage, copied out of the file. Round-trip checked (the recovered span is letters-only equal to the original) and length-checked. The text as originally recorded is kept in evidence as extracted. Quotation withheld because this record is not cleared for redistribution. Part-Of-Speech tagging Part-Of-Speech tagging Quotation withheld because this record is not cleared for redistribution. unified architecture unified architecture Quotation withheld because this record is not cleared for redistribution. syntactic information syntactic information Quotation withheld because this record is not cleared for redistribution. semantic information semantic information Quotation withheld because this record is not cleared for redistribution. standard-deviation standard-deviation Quotation withheld because this record is not cleared for redistribution.

The full text of this article is not redistributable, so it is not included here. The evidence sentences are withheld with it; the tag layer and its grounding status remain. The article itself is at https://www.jmlr.org/papers/v12/collobert11a.html.