Deep Learning in Neural Networks: An Overview

Deep Learning in Neural Networks: An Overview Jürgen Schmidhuber

Tag layer of the DHSS Meta-Tagging Project. The article text is reproduced from the staged copy named in sourceDesc and is not altered; the tags are held separately in standOff so that overlapping and nested spans are preserved.

Deep Learning in Neural Networks: An Overview 2015 Neural Networks 10.1016/j.neunet.2014.09.003 1404.7828 schmidhuber2015-deep-learning schmidhuber2015-deep-learning.txt, the project's staged copy e43cac731c36159079fa044ccd80e05d0e7df073 arxiv-open artificial-intelligence

the paper's own words

the tag layer; a sentence written by the tagger about the paper, always anchored to a quotation but not itself a quotation

0 spans located by searching the stored text for the quote's exact words, allowing any whitespace between them method Quotation withheld because this record is not cleared for redistribution. deep learning in artificial neural networks Quotation withheld because this record is not cleared for redistribution. review review Quotation withheld because this record is not cleared for redistribution. mechanisms deep learning = long credit-assignment paths: assign credit or blame for outcomes back through many layers of causal links (done by backpropagation) Quotation withheld because this record is not cleared for redistribution. fully and fairly assigning credit to all contributors is not achievable what this limits: the credit-assignment aim of the survey: I ACKNOWLEDGE THE LIMITATIONS OF ATTEMPTING TO ACHIEVE THIS GOAL Quotation withheld because this record is not cleared for redistribution. The author states that the credit-assignment goal of the survey cannot be fully achieved. what this limits: the same scope as limitations[0] - the goal is stated as not fully achievable Quotation withheld because this record is not cleared for redistribution. Coverage may be incomplete because of expert selection bias. what this limits: the coverage of the survey: THROUGH AN EXPERT SELECTION BIAS I MAY HAVE MISSED IMPORTANT WORK Quotation withheld because this record is not cleared for redistribution. The survey is biased toward the author's own research group's work. what this limits: the balance of attribution: A RELATED BIAS WAS SURELY INTRODUCED BY MY SPECIAL FAMILIARITY WITH THE WORK OF MY OWN DL RESEARCH GROUP IN THE PAST QUARTER-CENTURY Quotation withheld because this record is not cleared for redistribution. The boundary between shallow and deep learning is set by stipulation, not by a settled criterion. what this limits: every use of the shallow/deep distinction: Quotation withheld DISCUSSIONS WITH DL EXPERTS HAVE NOT YET YIELDED A CONCLUSIVE RESPONSE Quotation withheld because this record is not cleared for redistribution. The reinforcement-learning part is explicitly not a general survey of that field. what this limits: the reinforcement-learning sections: THEY CANNOT SERVE AS A GENERAL RL SURVEY Quotation withheld because this record is not cleared for redistribution. incorporating neurophysiological insights into deep neural-network architectures (e.g. the Neocognitron) Quotation withheld because this record is not cleared for redistribution. deep learning deep learning Quotation withheld because this record is not cleared for redistribution. credit assignment credit assignment Quotation withheld because this record is not cleared for redistribution. shallow learning shallow learning Quotation withheld because this record is not cleared for redistribution. solution depth solution depth Quotation withheld because this record is not cleared for redistribution. problem depth problem depth Quotation withheld because this record is not cleared for redistribution. credit assignment path - chains of possibly learnable causal links between actions and effects, whose DEPTH is what separates shallow from deep learners Quotation withheld because this record is not cleared for redistribution. the shallow/deep boundary - not a settled criterion: the author reports that experts could not agree and then stipulates a threshold for this overview Quotation withheld because this record is not cleared for redistribution. Very Deep Learning - a threshold defined by stipulation for this survey alone, at problem depth greater than ten Quotation withheld because this record is not cleared for redistribution. credit assignment - also what the survey itself performs, on researchers rather than network links, and which the author says cannot be fully achieved Quotation withheld because this record is not cleared for redistribution. 'solution depth' / 'problem depth' - A STIPULATIVE CASCADE THAT MAKES 'DEPTH' WELL-DEFINED WHERE THE FIELD'S OWN BOUNDARY IS NOT. Three definitions in one breath: a 'solution' is the set of used weights; the 'solution depth' is the depth of the deepest CAP in that sequence; and the 'problem depth' is 'the smallest depth of any solution'. THE THIRD IS A MINIMUM OVER SOLUTIONS, NOT A PROPERTY OF A NETWORK - so two networks solving one problem can have different solution depths while the problem depth is a single number about the PROBLEM. This record already records, in another of its senses, that experts could not agree where shallow ends and deep begins; this is the same author making the quantity precise by fiat instead of waiting for the field. Quotation withheld because this record is not cleared for redistribution. `deep learning` - THE SURVEY MARKS ITS OWN TERM'S OUTER EDGE AND DECLINES TO CROSS IT: `universal RL methods, which go beyond what is usually called DL`. A scope statement about the EXTENSION of the term rather than its content, and it is deferred to ordinary usage - `what is USUALLY called` - by an author who elsewhere stipulates his own thresholds. THE TERM IS FIXED BY STIPULATION AT ITS CENTRE AND BY CONVENTION AT ITS EDGE. Quotation withheld because this record is not cleared for redistribution. builds-on → half a century of prior neural-network and machine-learning research Quotation withheld because this record is not cleared for redistribution. deep learning —distinct-from→ shallow learning deep learning shallow learning strength of the relation: 1 Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but the span says experts have NOT YET given a conclusive answer where shallow learning ends; hedged. Quotation withheld because this record is not cleared for redistribution. 88 / 888 — length and citation count of the survey Quotation withheld because this record is not cleared for redistribution. half century — history traced back through the ideas Quotation withheld because this record is not cleared for redistribution. 1940s — earliest neural networks reviewed Quotation withheld because this record is not cleared for redistribution. deep networks including recurrent ones won numerous contests Quotation withheld because this record is not cleared for redistribution. shallow and deep learners are distinguished Quotation withheld because this record is not cleared for redistribution. the survey spans supervised, unsupervised, RL and evolutionary methods Quotation withheld because this record is not cleared for redistribution. much of the relevant work dates from the previous millennium Quotation withheld because this record is not cleared for redistribution. credit assignment paths (CAPs) credit assignment paths (CAPs) Quotation withheld because this record is not cleared for redistribution. Depth of Credit Assignment Paths Depth of Credit Assignment Paths Quotation withheld because this record is not cleared for redistribution. backpropagation backpropagation Quotation withheld because this record is not cleared for redistribution. GPUs GPUs Quotation withheld because this record is not cleared for redistribution. credit assignment path credit assignment path Quotation withheld because this record is not cleared for redistribution. hierarchical representation learning hierarchical representation learning Quotation withheld because this record is not cleared for redistribution. Minimum Description Length (MDL) Minimum Description Length (MDL) Quotation withheld because this record is not cleared for redistribution. recurrent neural networks recurrent neural networks Quotation withheld because this record is not cleared for redistribution. Bayesian Bayesian about this quote: QUOTE REPAIRED: the extractor split a hyphen-space and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted. Quotation withheld because this record is not cleared for redistribution. regression regression Quotation withheld because this record is not cleared for redistribution.

The full text of this article is not redistributable, so it is not included here. The evidence sentences are withheld with it; the tag layer and its grounding status remain. The article itself is at https://doi.org/10.1016/j.neunet.2014.09.003.