Deep Learning in Neural Networks: An Overview

Jürgen Schmidhuber · Neural Networks · 2015 · artificial-intelligence · 10.1016/j.neunet.2014.09.003 · record schmidhuber2015-deep-learning
tag layer · index · concepts · authors · עברית

Where this record's text came from. faceted 2026-08 by reading the locally held text (pdftext); every evidence string verified as an exact substring of that text before merging. Claim relations left empty on purpose: as a survey the paper relates hundreds of prior works to each other, but it nowhere positions its own contribution against a specific named prior overview, so every candidate relation would be an inference from a bare citation. Claim type is 'method' for the CAP-depth measure, which is the only novel apparatus in the paper; the paper's own self-description is a 'historical survey', and it also carries an implicit critique of the field's citation practice ('Since not all DL publications properly acknowledge earlier relevant work'), which the six-value claim type vocabulary cannot express. Quotes taken only from the body; the reference list (roughly the last 160,000 characters of the file) was excluded. 2026-08-14 and 2026-08-26 schema-0.3 passes: every quote verified as an exact substring of the staged text and anchored at merge time; concept terms verified present; every edge endpoint an existing node.

This record’s own method is review, and the tags below follow from that. A review’s findings, quantities and statistics are things stated IN this paper, and some of them report results obtained elsewhere. No tag on this page names a different author. Measured across every review in this corpus, 81% of a review’s tags carry no citation at all — most of them are the review speaking. The corpus does not adjudicate the rest: read an unmarked tag as a claim this paper makes, not necessarily one it made first.

How to read this page. The article itself is not reproduced here — this paper is not cleared for redistribution, so what follows is the tag layer alone. The evidence sentences are withheld with the article. The table at the foot lists every tag the record carries; each row names the facet it belongs to, and the whose words column says whether the tag records the paper's own wording or the tagger's sentence about the paper. Every tag links to each other paper in the corpus carrying it. The machine-readable edition of the same thing is the TEI file beside this one, where the tags live in standOff, and the article is at its publisher via the link below. When we have changed our mind. Some entries in the table begin with a word in capitals and the reason for it. EVIDENCE CORRECTED — the finding still stands, but we had quoted the wrong sentence from the paper, and the quotation has been replaced with the right one. RETRACTED — the entry itself was wrong; it stays on the page, marked, with the reason, rather than being deleted. WITHDRAWN AND REPLACED — the entry was wrong, and a corrected version of it appears in the entries below. Each mark carries its own reason, in brackets, because the reasons differ from one entry to the next. An entry we have WITHDRAWN is removed from this page and kept in the record, with its reason. A line reading Audit and a date is a later re-reading of the entry against its whole sentence: DOUBTFUL means it was kept, and the reason says what is loose about it. Quotation marks inside the tagger's sentence. Where a tag is the tagger's own sentence, a phrase in quotation marks inside it is borrowed from the paper — the tagger's wording around it is not. The paper's own sentence is the evidence quote for that row; on this page it is withheld with the article, and the article is at its publisher. Nothing inside quotation marks anywhere on this page is ours: if it is quoted, it is the paper's. Whose words a concept carries. A concept entry records a term the paper itself uses, so the term, the sense given for it, and its evidence are the paper's own words, quoted where the paper fixes them. This is the opposite of a sense entry in the tag table, which is the tagger's sentence about how the paper uses a term. Where a concept entry could not be grounded in the paper's words, the entry says so on its own row.

Other fields define these differently the same word, defined another way in another discipline
deep learning 11 fields · neural networks 3 fields · supervised learning 3 fields · linear regression 2 fields · recurrent neural networks 2 fields

Tags on this paper

Every tag the record carries, each linking to every other paper in the corpus that carries it. A tag appears here whether or not its words occur in the article — the spans below are only the ones a sentence in this paper licenses. Of the 7 concept tags on this paper, 5 are grounded in a quoted sentence. 2 carry the tagger's gloss of what the term means in this paper, which is not something the article can license.

deep learning neural network backpropagation credit assignment shallow learning solution depth problem depth deep learning = long credit-assignment paths: assign credit or blame for outcomes back through many layers of causal links (done by backpropagation) neural networks supervised learning unsupervised learning reinforcement learning pattern recognition recurrent depth credit assignment path hierarchical representation learning Minimum Description Length (MDL) recurrent neural networks credit assignment paths (CAPs) Depth of Credit Assignment Paths GPUs Bayesian regression linear regression review

Legend

relation to other work	`claim_relations`	how this paper stands to another: agrees, contradicts, extends, qualifies
claim type	`claim_type`	the shape of the claim: existence, mechanism, method, critique, prediction, null
concept edge	`concept_edges`	an asserted relation between two concepts, with the sentence asserting it
concept	`concepts`	an idea the paper works with, quoted where the paper uses it
content tag	`content_tags`	a finding, definition, quantity, statistic or variable named in the paper
limitation	`limitations`	what the paper says its evidence cannot support
mechanism	`mechanisms`	how the reported effect is said to come about
method	`method`	how the paper produced its evidence
phenomenon	`phenomenon`	the effect or regularity under study
sense	`senses`	a distinct use of a term, quoted where the paper fixes it
transfer	`transfers`	a method or concept carried in from another field

The paper

The full text is not reproduced here. This article is not cleared for redistribution, so the page carries the tag layer only. Its evidence sentences are withheld; the grounding status remains with each tag, and the article itself is at https://doi.org/10.1016/j.neunet.2014.09.003. The reading view exists for the papers whose licence permits it.

Every tag on this page

claim_relations 1 · claim_type 1 · concept_edges 1 · concepts 5 · content_tags 17 · limitations 6 · mechanisms 1 · method 1 · phenomenon 1 · senses 6 · transfers 1

Relations on this page carry a quotation for the claim. Where a relation was separately evidenced — and so tested against the vocabulary for its type — the row says so; none of 1 relation here carry that mark. The rest are the tagger’s reading of the sentence quoted beside them.

This record holds no controls. A layer is empty when the tagger recorded nothing in it — which is not the same as the paper having nothing to record.

facet	tag	field	conf.	whose words
claim_type	method Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_type`		the paper's words
phenomenon	deep learning in artificial neural networks Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`phenomenon`		the paper's words
method	review Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: review	`method`		the paper's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: deep learning = long credit-assignment paths: assign credit or blame for outcomes back through many layers of causal links (done by backpropagation)	`mechanisms[0]`		the tagger's words
limitations	fully and fairly assigning credit to all contributors is not achievable Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the credit-assignment aim of the survey: I ACKNOWLEDGE THE LIMITATIONS OF ATTEMPTING TO ACHIEVE THIS GOAL	`limitations[0]`		the tagger's words
limitations	The author states that the credit-assignment goal of the survey cannot be fully achieved. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the same scope as limitations[0] - the goal is stated as not fully achievable	`limitations[1]`		the tagger's words
limitations	Coverage may be incomplete because of expert selection bias. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the coverage of the survey: THROUGH AN EXPERT SELECTION BIAS I MAY HAVE MISSED IMPORTANT WORK	`limitations[2]`		the tagger's words
limitations	The survey is biased toward the author's own research group's work. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the balance of attribution: A RELATED BIAS WAS SURELY INTRODUCED BY MY SPECIAL FAMILIARITY WITH THE WORK OF MY OWN DL RESEARCH GROUP IN THE PAST QUARTER-CENTURY	`limitations[3]`		the tagger's words
limitations	The boundary between shallow and deep learning is set by stipulation, not by a settled criterion. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: every use of the shallow/deep distinction: Quotation withheld DISCUSSIONS WITH DL EXPERTS HAVE NOT YET YIELDED A CONCLUSIVE RESPONSE	`limitations[4]`		the tagger's words
limitations	The reinforcement-learning part is explicitly not a general survey of that field. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the reinforcement-learning sections: THEY CANNOT SERVE AS A GENERAL RL SURVEY	`limitations[5]`		the tagger's words
transfers	incorporating neurophysiological insights into deep neural-network architectures (e.g. the Neocognitron) Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`transfers[0]`		the tagger's words
concepts	deep learning Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: deep learning	`concepts[0]`		the paper's words
concepts	credit assignment Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: credit assignment	`concepts[3]`		the paper's words
concepts	shallow learning Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: shallow learning	`concepts[4]`		the paper's words
concepts	solution depth Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: solution depth	`concepts[5]`	1.0	the paper's words
concepts	problem depth Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: problem depth	`concepts[6]`	1.0	the paper's words
senses	credit assignment path - chains of possibly learnable causal links between actions and effects, whose DEPTH is what separates shallow from deep learners Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[0]`	1.0	the tagger's words
senses	the shallow/deep boundary - not a settled criterion: the author reports that experts could not agree and then stipulates a threshold for this overview Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[1]`	1.0	the tagger's words
senses	Very Deep Learning - a threshold defined by stipulation for this survey alone, at problem depth greater than ten Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[2]`	1.0	the tagger's words
senses	credit assignment - also what the survey itself performs, on researchers rather than network links, and which the author says cannot be fully achieved Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[3]`	1.0	the tagger's words
senses	'solution depth' / 'problem depth' - A STIPULATIVE CASCADE THAT MAKES 'DEPTH' WELL-DEFINED WHERE THE FIELD'S OWN BOUNDARY IS NOT. Three definitions in one breath: a 'solution' is the set of used weights; the 'solution depth' is the depth of the deepest CAP in that sequence; and the 'problem depth' is 'the smallest depth of any solution'. THE THIRD IS A MINIMUM OVER SOLUTIONS, NOT A PROPERTY OF A NETWORK - so two networks solving one problem can have different solution depths while the problem depth is a single number about the PROBLEM. This record already records, in another of its senses, that experts could not agree where shallow ends and deep begins; this is the same author making the quantity precise by fiat instead of waiting for the field. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[4]`	1.0	the tagger's words
senses	`deep learning` - THE SURVEY MARKS ITS OWN TERM'S OUTER EDGE AND DECLINES TO CROSS IT: `universal RL methods, which go beyond what is usually called DL`. A scope statement about the EXTENSION of the term rather than its content, and it is deferred to ordinary usage - `what is USUALLY called` - by an author who elsewhere stipulates his own thresholds. THE TERM IS FIXED BY STIPULATION AT ITS CENTRE AND BY CONVENTION AT ITS EDGE. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[5]`	0.95	the tagger's words
claim_relations	builds-on → half a century of prior neural-network and machine-learning research Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[0]`		the tagger's words
concept_edges	deep learning —distinct-from→ shallow learning Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. strength of the relation: 1 Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but the span says experts have NOT YET given a conclusive answer where shallow learning ends; hedged. also in: deep learning · shallow learning	`concept_edges[0]`		the tagger's words
content_tags	88 / 888 — length and citation count of the survey Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.quantities[0]`		the tagger's words
content_tags	half century — history traced back through the ideas Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.quantities[1]`		the tagger's words
content_tags	1940s — earliest neural networks reviewed Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.quantities[2]`		the tagger's words
content_tags	deep networks including recurrent ones won numerous contests Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.findings[0]`		the tagger's words
content_tags	shallow and deep learners are distinguished Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.findings[1]`		the tagger's words
content_tags	the survey spans supervised, unsupervised, RL and evolutionary methods Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.findings[2]`		the tagger's words
content_tags	much of the relevant work dates from the previous millennium Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.findings[3]`		the tagger's words
content_tags	credit assignment paths (CAPs) Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: credit assignment paths (CAPs)	`content_tags.variables[0]`		the tagger's words
content_tags	Depth of Credit Assignment Paths Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: Depth of Credit Assignment Paths	`content_tags.variables[1]`		the tagger's words
content_tags	backpropagation Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: backpropagation	`content_tags.variables[2]`		the tagger's words
content_tags	GPUs Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: GPUs	`content_tags.variables[3]`		the tagger's words
content_tags	credit assignment path Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: credit assignment path	`content_tags.definitions[0]`		the tagger's words
content_tags	hierarchical representation learning Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: hierarchical representation learning	`content_tags.definitions[1]`		the tagger's words
content_tags	Minimum Description Length (MDL) Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: Minimum Description Length (MDL)	`content_tags.definitions[2]`		the tagger's words
content_tags	recurrent neural networks Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: recurrent neural networks	`content_tags.definitions[3]`		the tagger's words
content_tags	Bayesian Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. about this quote: QUOTE REPAIRED: the extractor split a hyphen-space and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted. also in: Bayesian	`content_tags.statistics[0]`		the tagger's words
content_tags	regression Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: regression	`content_tags.statistics[2]`		the tagger's words

Interpretive choices

Tags that could not be placed in the text:

none — every tag was located in the text

What was deliberately not tagged, and why, is recorded per paper by the tagger. Where a paper has no such record, this page says so rather than implying that nothing was declined.