| phenomenon | neural sequence labelling across NLP benchmark tasks— “All the NLP tasks above can be seen as tasks assigning labels to words.” |
| claim-type | method— “We propose a unified neural network architecture and learning algorithm that can be applied to various natural language processing tasks including: part-of-speech tagging, chunking, named entity recognition, and semantic role labeling.” |
| method — evidence | — “We implemented a standalone version of our architecture, written in the C language.” |
| mechanism | THE FIELD'S GOAL IS POSED AS A QUESTION AND THEN REPORTED TO HAVE NO AGREED FORM: will a program ever convert English text into a data structure that UNAMBIGUOUSLY AND COMPLETELY DESCRIBES THE MEANING - and AMONG NUMEROUS PROBLEMS, NO CONSENSUS HAS EMERGED ABOUT THE FORM OF SUCH A DATA STRUCTURE. The obstacle named first is not difficulty but the absence of agreement on what the answer would look like— “Will a computer program ever be able to convert a piece of English text into a data structure that unambiguously and completely describes the meaning of the natural language text? Among numerous problems, no consensus has emerged about the form of such a data structure.” | mechanisms[0] |
| mechanism | THE FIELD'S TASKS ARE NAMED AS REDUCED OBJECTIVES ADOPTED BECAUSE THE REAL QUESTION IS UNRESOLVED: UNTIL SUCH FUNDAMENTAL PROBLEMS ARE RESOLVED, computer scientists MUST SETTLE FOR REDUCED OBJECTIVES - extracting SIMPLER REPRESENTATIONS describing RESTRICTED ASPECTS of the textual information. What the field measures itself on is described as a substitute, in the paper's second sentence— “Until such fundamental Artificial Intelligence problems are resolved, computer scientists must settle for reduced objectives: extracting simpler representations describing restricted aspects of the textual information.” | mechanisms[1] |
| mechanism | THE SAME REPRESENTATIONS ARE GIVEN TWO DIFFERENT WARRANTS, AND ONE OF THEM IS A BELIEF: they are OFTEN MOTIVATED BY SPECIFIC APPLICATIONS, for instance bag-of-words for information retrieval - and they CAN ALSO BE MOTIVATED BY OUR BELIEF THAT THEY CAPTURE SOMETHING MORE GENERAL about natural language. An instrumental justification and an epistemic one, offered for the same objects— “These simpler representations are often motivated by specific applications, for instance, bag-of-words variants for information retrieval. These representations can also be motivated by our belief that they capture something more general about natural language.” | mechanisms[2] |
| mechanism | THE ANNOTATIONS EXIST IN ORDER TO MAKE SYSTEMS COMPARABLE, AND THE BENCHMARKS ARE CREDITED WITH DRIVING THE RESEARCH: text corpora HAVE BEEN MANUALLY ANNOTATED with such data structures IN ORDER TO COMPARE THE PERFORMANCE of various systems, and THE AVAILABILITY OF STANDARD BENCHMARKS HAS STIMULATED RESEARCH. The ground truth's purpose is comparison, and the comparison is what made the field move— “Text corpora have been manually annotated with such data structures in order to compare the performance of various systems. The availability of standard benchmarks has stimulated research in Natural Language Processing (NLP)” | mechanisms[3] |
| mechanism | THE METHOD IS DEFINED BY WHAT IT REFUSES TO USE, AND THE COST IS STATED IN THE SAME SENTENCE: the versatility is achieved BY TRYING TO AVOID TASK-SPECIFIC ENGINEERING AND THEREFORE DISREGARDING A LOT OF PRIOR KNOWLEDGE - instead of exploiting MAN-MADE INPUT FEATURES carefully optimized for each task. What is given up is named as the price of what is gained— “This versatility is achieved by trying to avoid task-specific engineering and therefore disregarding a lot of prior knowledge. Instead of exploiting man-made input features carefully optimized for each task, our system learns internal representations on the basis of vast amounts of mostly unlabeled training data.” | mechanisms[4] |
| mechanism | THE GROUND TRUTH ITSELF COMES IN TWO VOCABULARIES, AND THE FIX IS TO PICK ONE: the ground truth for three of the tasks IS PROVIDED USING TWO DIFFERENT TAGGING SCHEMES, and IN ORDER TO ELIMINATE THIS ADDITIONAL SOURCE OF VARIATIONS the authors DECIDED TO USE THE MOST EXPRESSIVE scheme FOR ALL TASKS. Two naming systems for the same annotations, normalised by choosing the more expressive - not the more common— “The ground truth for the NER, CHUNK, and SRL tasks is provided using two different tagging schemes. In order to eliminate this additional source of variations, we have decided to use the most expressive IOBES tagging scheme for all tasks.” | mechanisms[5] |
| mechanism | THE CHOICE OF HYPER-PARAMETERS IS CONCEDED TO BE ONE OF MANY EQUALLY GOOD ONES, WITH THE UNEXPLORED SPACE NAMED: although they ended up with A PARTICULAR CHOICE, MANY OTHER CHOICES ARE ALMOST EQUALLY AS GOOD - ALTHOUGH PERHAPS THERE ARE OTHERS THAT ARE BETTER, AS WE COULD NOT DO A FULL GRID SEARCH. The reported configuration is presented as a sample rather than an optimum— “However, we should say we believe that although we ended up with a particular choice of parameters, many other choices are almost equally as good, although perhaps there are others that are better as we could not do a full grid search.” | mechanisms[6] |
| mechanism | THE SCALE OF THE EXPERIMENT IS ATTRIBUTED TO THE HARDWARE RATHER THAN TO THE SCIENCE, WRYLY: very long training times MAKE SUCH STRATEGIES NECESSARY FOR THE FORESEEABLE FUTURE - and IF WE HAD BEEN GIVEN COMPUTERS TEN TIMES FASTER, WE PROBABLY WOULD HAVE FOUND USES FOR DATASETS TEN TIMES BIGGER. The design is described as a function of the machines available— “Very long training times make such strategies necessary for the foreseeable future: if we had been given computers ten times faster, we probably would have found uses for datasets ten times bigger.” | mechanisms[7] |
| mechanism | TWO NEGATIVE RESULTS FROM OTHER GROUPS ARE CITED SIDE BY SIDE: one group obtained results SOMEWHAT BELOW THE STATE-OF-THE-ART using joint decoding for two tasks, and another DESCRIBED A NEGATIVE RESULT AT THE SAME JOINT TASK. Failures are gathered as evidence about a practice rather than about the groups that attempted it— “On the other hand, Sutton and McCallum (2005a) obtain results somewhat below the state-of-the-art using joint decoding for SRL and syntactic parsing. Musillo and Merlo (2006) also describe a negative result at the same joint task.” | mechanisms[8] |
| contradicts | the reliance on task-specific hand-engineered features in NLP— “Instead of exploiting man-made input features carefully optimized for each task, our system learns internal representations on the basis of vast amounts of mostly unlabeled training data.” | claim_relations[0] |
| builds-on | the convolutional / Time Delay Neural Network approach of Waibel et al. (1989)— “When using neural networks, the natural choice to tackle this problem becomes a convolutional approach, first introduced by Waibel et al. (1989) and also called Time Delay Neural Networks (TDNNs) in the literature.” | claim_relations[1] |
| extends | Conditional Random Fields (Lafferty et al., 2001)— “Compared to such CRFs, we take advantage of the nonlinear network to learn appropriate features for each task of interest.” | claim_relations[2] |
| builds-on | graph transformer networks (Bottou et al., 1997; Le Cun et al., 1998)— “Our approach is a particular case of the discriminative forward training for graph transformer networks (GTNs) (Bottou et al., 1997; Le Cun et al., 1998).” | claim_relations[3] |
| sense | 'almost from scratch' - THE PAPER NAMES ITS OWN HEDGE AND GIVES THE REASON: it calls the approach almost from scratch to emphasise a reduced but still important reliance on a priori NLP knowledge, while the body says 'from scratch' twenty-seven times and 'almost from scratch' once— “We call this approach “almost from scratch” to emphasize the reduced (but still important) reliance on a priori NLP knowledge” | senses[0] |
| sense | a benchmark - AS AN INDIRECT MEASUREMENT, NOT A GOAL: the scores stand in for the relevance of internal representations that nobody measures directly, and the authors posit those representations are more general than the benchmarks that stand in for them— “In fact we view the benchmarks as indirect measurements of the relevance of the internal representations discovered by the learning procedure, and we posit that these intermediate representations are more general than any of the benchmarks” | senses[1] |
| sense | the cost of the design - LINGUISTIC KNOWLEDGE, DELIBERATELY SET ASIDE: avoiding task-specific engineered features is stated to have led the authors to ignore a large body of linguistic knowledge, so the architecture's generality and its ignorance are the same decision— “Our desire to avoid task-specific engineered features led us to ignore a large body of linguistic knowledge” | senses[2] |
| control | the word embedding dimension held constant at 50 across the tasks, so the language models of the earlier section could be reused unchanged — held — rules out a task-to-task difference that is really a difference in embedding capacity— “The word embedding dimension was kept constant d0 = 50 in order to reuse the language models of Section 4.5.” | controls[0] |
| control | performance also reported with the embeddings FROZEN during task-specific training, beside the figures where they are allowed to move — held — rules out attributing to the pre-trained embeddings a gain that actually comes from fine-tuning them on the task— “We also report performance when our embeddings are kept fixed during task-specific training.” | controls[1] |
| differs in | the reliance on task-specific hand-engineered features in NLP — THE PARAMETER IS WHERE THE FEATURES COME FROM. The span: 'Instead of exploiting man-made input features carefully optimized for each task, our system learns internal representations on the basis of vast amounts of mostly unlabeled training data.' 'Instead of' names the axis of disagreement in the same clause as the disagreement.— “Instead of exploiting man-made input features carefully optimized for each task, our system learns internal representations on the basis of vast amounts of mostly unlabeled training data.”— “Instead of exploiting man-made input features carefully optimized for each task” | claim_relations[0] |
| edge | internal representations —operationalised-as→ benchmark— “In fact we view the benchmarks as indirect measurements of the relevance of the internal representations discovered by the learning procedure, and we posit that these intermediate representations are more general than any of the benchmarks” | concept_edges[0] |
| outcome (stated by the tagger, no quotation) | control — held | controls[0] |
| outcome (stated by the tagger, no quotation) | control — held | controls[1] |
| for which use | limitation — the from-scratch claim: IDEALLY, AN EVEN MORE RAW INPUT WOULD BE TO LEARN FROM LETTER SEQUENCES RATHER THAN WORDS - words are lowercased and capitalisation encoded separately— “Ideally, an even more raw input would be to learn from letter sequences rather than words, however we felt that this was beyond the scope of this work.” | limitations[1] |
| for which use | limitation — every hyperparameter-dependent result: TRAINING TIMES FOR SUCH LARGE SCALE SYSTEMS ARE COUNTED IN WEEKS, [SO] IT IS NOT FEASIBLE TO TRY MANY COMBINATIONS— “Since training times for such large scale systems are counted in weeks, it is not feasible to try many combinations of hyperparameters.” | limitations[2] |
| for which use | limitation — the multi-task claim: NO (OR ONLY MARGINAL) IMPROVEMENTS WERE OBTAINED with the unified architecture over separately trained ones— “While we find worth mentioning that MTL can produce a single unified architecture that performs well for all these tasks, no (or only marginal) improvements were obtained with this approach compared to training separate architectures per task (which still use semisupervised learning, which is somehow the most important MTL task).” | limitations[3] |
| for which use | limitation — any syntactic reading of the language model: ITS STRUCTURE IS PROBABLY TOO RESTRICTIVE FOR SUCH GOALS, and only the word embeddings are exploited— “On the other hand, the structure of our language models is probably too restrictive for such goals, and our current approach only exploits the word embeddings discovered during training.” | limitations[4] |
| for which use | limitation — the benchmark tables reported here: the authors view the benchmarks as indirect measurements of the relevance of the internal representations, and posit that those representations are more general than any of the benchmarks— “In fact we view the benchmarks as indirect measurements of the relevance of the internal representations discovered by the learning procedure, and we posit that these intermediate representations are more general than any of the benchmarks” | limitations[5] |
| for which use | limitation — the comparison of these results against other systems: the authors note there is a great temptation to optimize a system's performance for a specific benchmark— “On the other hand, there is a great temptation to optimize the performance of a system for a specific benchmark” | limitations[6] |
| for which use | limitation — the claim that this system works from raw input: the authors state they did some pre-processing, namely lowercasing and encoding capitalization as a separate feature, and say that with enough unlabeled data they could presumably learn without it— “We did some pre-processing, namely lowercasing and encoding capitalization as another feature. With enough (unlabeled) training data, presumably we could learn a model without this processing” | limitations[7] |
| for which use | limitation — the best figures quoted from this paper: a later section evaluates how much further improvement comes from incorporating standard NLP task-specific engineering, which the main architecture avoids— “Section 6 then evaluates how much further improvement can be achieved by incorporating standard NLP task-specific engineering into our systems” | limitations[8] |
| evidence status note | mechanism — RE-SLICED, AND THE GUARD THAT CAUGHT IT WAS TWENTY MINUTES OLD. My end key was "...Natural Language Processing (NLP) and" - ending on "and" - because the page furniture ("Keywords: ... arXiv 1. Introduction") is spliced into that sentence and its real continuation is on the far side of the header. The dangling-word check refused it at drafting time.— “Text corpora have been manually annotated with such data structures in order to compare the performance of various systems. The availability of standard benchmarks has stimulated research in Natural Language Processing (NLP)” | mechanisms[3] |
| replication | untested |
| limitation | avoiding task-specific features meant forgoing a large body of linguistic knowledge— “Our desire to avoid task-specific engineered features led us to ignore a large body of linguistic knowledge.” | limitations[0] |
| limitation | The system is only 'almost' from scratch: words are lowercased and capitalization encoded, and learning from letter sequences was left out of scope.— “Ideally, an even more raw input would be to learn from letter sequences rather than words, however we felt that this was beyond the scope of this work.” | limitations[1] |
| limitation | Week-long training times made a proper hyperparameter search infeasible.— “Since training times for such large scale systems are counted in weeks, it is not feasible to try many combinations of hyperparameters.” | limitations[2] |
| limitation | Multi-task learning gave no or only marginal gains over separately trained per-task architectures.— “While we find worth mentioning that MTL can produce a single unified architecture that performs well for all these tasks, no (or only marginal) improvements were obtained with this approach compared to training separate architectures per task (which still use semisupervised learning, which is somehow the most important MTL task).” | limitations[3] |
| limitation | The language model structure is probably too restrictive to capture hierarchical syntax; only the word embeddings are exploited downstream.— “On the other hand, the structure of our language models is probably too restrictive for such goals, and our current approach only exploits the word embeddings discovered during training.” | limitations[4] |
| limitation | The benchmark scores are treated as indirect measurements of something the paper does not measure directly, and the claim about it is a posit.— “In fact we view the benchmarks as indirect measurements of the relevance of the internal representations discovered by the learning procedure, and we posit that these intermediate representations are more general than any of the benchmarks” | limitations[5] |
| limitation | The paper names the failure mode its own design is exposed to.— “On the other hand, there is a great temptation to optimize the performance of a system for a specific benchmark” | limitations[6] |
| limitation | The pre-processing the architecture does keep is named, with a presumably attached to removing it.— “We did some pre-processing, namely lowercasing and encoding capitalization as another feature. With enough (unlabeled) training data, presumably we could learn a model without this processing” | limitations[7] |
| limitation | Task-specific engineering is added back in a later section, so the headline architecture and the best numbers are not the same system.— “Section 6 then evaluates how much further improvement can be achieved by incorporating standard NLP task-specific engineering into our systems” | limitations[8] |