Deep Residual Learning for Image Recognition

Deep Residual Learning for Image Recognition Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun

Tag layer of the DHSS Meta-Tagging Project. The article text is reproduced from the staged copy named in sourceDesc and is not altered; the tags are held separately in standOff so that overlapping and nested spans are preserved.

Deep Residual Learning for Image Recognition 2015 IEEE CVPR / arXiv 1512.03385 he2015-resnet he2015-resnet.txt, the project's staged copy 5a027378e84009d441919a67edee439ff6bfbc8f arxiv-open artificial-intelligence

the paper's own words

the tag layer; a sentence written by the tagger about the paper, always anchored to a quotation but not itself a quotation

0 spans located by searching the stored text for the quote's exact words, allowing any whitespace between them method Quotation withheld because this record is not cleared for redistribution. the degradation problem in very deep neural networks Quotation withheld because this record is not cleared for redistribution. system-or-tool-building system-or-tool-building Quotation withheld because this record is not cleared for redistribution. mechanisms reformulate each layer to learn a residual function via identity shortcut connections, so very deep networks are easier to optimize and avoid the depth-degradation problem Quotation withheld because this record is not cleared for redistribution. why residual mappings are easier to optimize remains an open question what this limits: the explanation of why residual learning works: THIS HYPOTHESIS, HOWEVER, IS STILL AN OPEN QUESTION Quotation withheld because this record is not cleared for redistribution. The cause of the optimisation difficulty in deep plain nets is not explained. what this limits: the account of what goes wrong in plain deep nets: THE REASON FOR SUCH OPTIMIZATION DIFFICULTIES WILL BE STUDIED IN THE FUTURE Quotation withheld because this record is not cleared for redistribution. Extremely deep (1202-layer) models remain problematic. what this limits: the very deep regime: THERE ARE STILL OPEN PROBLEMS ON SUCH AGGRESSIVELY DEEP MODELS Quotation withheld because this record is not cleared for redistribution. The 1202-layer net's worse test error is attributed to overfitting from excess capacity on a small dataset. what this limits: the 1202-layer result specifically: THE 1202-LAYER NETWORK MAY BE UNNECESSARILY LARGE (19.4M) FOR THIS SMALL DATASET Quotation withheld because this record is not cleared for redistribution. The premise that multiple nonlinear layers can asymptotically approximate complicated functions is unproven. what this limits: the approximation premise rather than the optimisation question of item 0: THE PREMISE THAT MULTIPLE NONLINEAR LAYERS CAN ASYMPTOTICALLY APPROXIMATE COMPLICATED FUNCTIONS is stated as a hypothesis that is STILL AN OPEN QUESTION Quotation withheld because this record is not cleared for redistribution. Stronger regularisation was not explored and might improve the CIFAR-10 results. what this limits: the CIFAR-10 results: COMBINING WITH STRONGER REGULARIZATION MAY IMPROVE RESULTS, WHICH WE WILL STUDY IN THE FUTURE Quotation withheld because this record is not cleared for redistribution. The scenario that motivates the design is conceded to be unlikely in practice. what this limits: the intuition given for residual learning: the authors concede that in real cases Quotation withheld because this record is not cleared for redistribution. The paper's central premise is labelled a hypothesis by its authors. what this limits: the claim that residual mappings are easier to optimize: the authors present it as a hypothesis rather than as a demonstrated result Quotation withheld because this record is not cleared for redistribution. overfitting as the cause of the degradation problem eliminates: the reading that deeper networks degrade because they overfit · outcome (stated by the tagger, no quotation): held Quotation withheld because this record is not cleared for redistribution. vanishing gradients as the cause of the optimisation difficulty in plain nets eliminates: the standard explanation that deep plain nets fail because signals vanish · outcome (stated by the tagger, no quotation): partial Quotation withheld because this record is not cleared for redistribution. degradation degradation Quotation withheld because this record is not cleared for redistribution. overfitting overfitting Quotation withheld because this record is not cleared for redistribution. vanishing gradients vanishing gradients Quotation withheld because this record is not cleared for redistribution. optimization difficulty optimization difficulty Quotation withheld because this record is not cleared for redistribution. residual mapping residual mapping Quotation withheld because this record is not cleared for redistribution. identity mapping identity mapping Quotation withheld because this record is not cleared for redistribution. the degradation problem - IDENTIFIED BY TRAINING ERROR, NOT TEST ERROR: accuracy saturates then degrades rapidly with depth, and the authors' point is that this is NOT overfitting because adding layers raises the TRAINING error - the distinction is what makes it an optimisation problem rather than a generalisation one Quotation withheld because this record is not cleared for redistribution. an identity mapping - AN IDEALISATION THE PAPER WITHDRAWS: the argument for residual learning runs through the case where identity mappings are optimal and the solver can drive the weights to zero, and the authors then state in their own words that in real cases this is unlikely Quotation withheld because this record is not cleared for redistribution. the residual mapping - A REFORMULATION, PRESENTED AS A HYPOTHESIS: instead of fitting the desired underlying mapping the layers fit its residual, and the claim that this is easier to optimise is given in the authors' own words as something they hypothesize Quotation withheld because this record is not cleared for redistribution. extends → plain deep convolutional networks (the degradation problem) Quotation withheld because this record is not cleared for redistribution. replicates → prior reports [11, 42] that adding layers to a deep model raises training error Quotation withheld because this record is not cleared for redistribution. builds-on → VGG nets [41] Quotation withheld because this record is not cleared for redistribution. builds-on → Faster R-CNN [32] Quotation withheld because this record is not cleared for redistribution. extends → highway networks [42, 43] Quotation withheld because this record is not cleared for redistribution. vanishing gradients —causes→ optimization difficulty vanishing gradients optimization difficulty direction on the measure: no-change · strength of the relation: 0.8 Quotation withheld because this record is not cleared for redistribution. 152 — depth of the residual nets evaluated (8x deeper than VGG, lower complexity) Quotation withheld because this record is not cleared for redistribution. 3.57% — ImageNet test-set error of the ResNet ensemble (1st place ILSVRC 2015) Quotation withheld because this record is not cleared for redistribution. 100 and 1000 — network depths analysed on CIFAR-10 Quotation withheld because this record is not cleared for redistribution. An ensemble of residual nets reached 3.57% error on the ImageNet test set, winning 1st place in the ILSVRC 2015 classification task. Quotation withheld because this record is not cleared for redistribution. network depth network depth Quotation withheld because this record is not cleared for redistribution. residual learning residual learning about this quote: QUOTE REPAIRED: the extractor split a hyphen-space and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted. Quotation withheld because this record is not cleared for redistribution. shortcut connection shortcut connection Quotation withheld because this record is not cleared for redistribution. degradation problem degradation problem Quotation withheld because this record is not cleared for redistribution. mean mean Quotation withheld because this record is not cleared for redistribution. standard-deviation standard-deviation Quotation withheld because this record is not cleared for redistribution.

The full text of this article is not redistributable, so it is not included here. The evidence sentences are withheld with it; the tag layer and its grounding status remain. The article itself is at https://arxiv.org/abs/1512.03385.