Review of deep learning: concepts, CNN architectures, challenges, applications, future directions

Review of deep learning: concepts, CNN architectures, challenges, applications, future directions Laith Alzubaidi Jinglan Zhang Amjad J. Humaidi Ayad Q. Al-Dujaili Ye Duan Omran Al-Shamma José Santamaría Mohammed A. Fadhel Muthana Al‐Amidie Laith Farhan

Tag layer of the DHSS Meta-Tagging Project. The article text is reproduced from the staged copy named in sourceDesc and is not altered; the tags are held separately in standOff so that overlapping and nested spans are preserved.

Review of deep learning: concepts, CNN architectures, challenges, applications, future directions 2021 Journal Of Big Data 10.1186/s40537-021-00444-8 alzubaidi2021-review-of-deep-learning-concepts-c alzubaidi2021-review-of-deep-learning-concepts-c.txt, the project's staged copy 2a4a57bf85032245e99477483cce9077e78503e2 cc-by artificial-intelligence

the paper's own words

the tag layer; a sentence written by the tagger about the paper, always anchored to a quotation but not itself a quotation

0 spans located by searching the stored text for the quote's exact words, allowing any whitespace between them deep learning as a field - its concepts, network types, CNN architectures, named challenges, applications and computational substrate Quotation withheld because this record is not cleared for redistribution. whether one review can supply a full understanding of DL, against a review literature in which each paper covers a single aspect Quotation withheld because this record is not cleared for redistribution. a holistic survey of DL with twelve named challenges and their proposed solutions, a CNN-architecture lineage from AlexNet to HRNet, and a declared search methodology Quotation withheld because this record is not cleared for redistribution. critique Quotation withheld because this record is not cleared for redistribution. a field whose knowledge is fragmented across partial reviews, so that assembling a working understanding requires reading past the reviews Quotation withheld because this record is not cleared for redistribution. review review Quotation withheld because this record is not cleared for redistribution. back-propagation multiplies each layer's derivative down the network, so with a squashing activation like the sigmoid, N small derivatives multiplied together make the gradient decline exponentially - and the first layers, which recognise the essential elements of the input, are the ones that stop updating vanishing gradient through repeated multiplication Quotation withheld because this record is not cleared for redistribution. incorporating new information into a trained model interferes with what it already learned; fine-tuning on only the new class leaves performance on the old classes unsuccessful, so continually renewed data forces retraining from scratch catastrophic forgetting by fine-tuning Quotation withheld because this record is not cleared for redistribution. pre-training transfers representations learned from the source distribution; where the source is natural images and the target is raw medical images the learned representation does not apply, so the transferred model performs no better than one trained from scratch domain mismatch defeating transfer Quotation withheld because this record is not cleared for redistribution. many parameter settings satisfy the training objective equally well, so the pipeline does not determine which one is chosen; small modifications force the model toward a completely different solution and hence different predictions in deployment underspecification selecting among equivalent solutions Quotation withheld because this record is not cleared for redistribution. the paper recommends transfer learning for scarce data and then reports that cross-domain TL does not significantly affect medical imaging performance - its own central recommendation carries a negative result three paragraphs later Quotation withheld because this record is not cleared for redistribution. `RL` is defined as representation learning in the introduction and used for reinforcement learning in the classification section; a reader carrying the first definition forward misreads the second Quotation withheld because this record is not cleared for redistribution. underspecification puts major constraints on the credibility of ML predictions and the proposed remedy - stress tests - requires already knowing how the model can fail Quotation withheld because this record is not cleared for redistribution. CNN performance is greatly influenced by hyper-parameter selection, so any small change affects the result - which limits how much a reported architecture comparison can be attributed to architecture Quotation withheld because this record is not cleared for redistribution. labelling remains the binding constraint: crowdsourced workers cannot annotate medical or biological images, expert labelling is costly and slow, so producing the volume of labels deep networks need is called unfeasible Quotation withheld because this record is not cleared for redistribution. despite the stated transition toward semi-supervised and unsupervised learning, most current deep-learning models still use supervised learning - so the data constraint above is not being relieved Quotation withheld because this record is not cleared for redistribution. the architecture of the mammalian visual cortex, carried into network design - the CNN's local receptive fields are presented as a simulation of cells that sense only small regions of a scene Quotation withheld because this record is not cleared for redistribution. pre-trained representations carried from natural-image datasets into medical imaging, and the paper's own report that this particular transfer does not work Quotation withheld because this record is not cleared for redistribution. a declared corpus for the review - a stated floor on papers read and their distribution by year - so that 'we focused on recent work' is a number rather than a claim eliminates: a review's coverage claim resting on the reviewers' sense of having read enough · outcome: held Quotation withheld because this record is not cleared for redistribution. declared review objectives - list and define approaches, list architectures, present challenges, assess applications and computational approaches - fixed in advance of the reading eliminates: a review whose scope is settled after the reading, by fixing the five objectives in advance · outcome (stated by the tagger, no quotation): held Quotation withheld because this record is not cleared for redistribution. cross-domain transfer learning checked against models trained from scratch - the control that decides whether transfer contributed anything, run by others and reported here eliminates: that the reported performance is owed to the pre-trained source domain rather than to the architecture and the target data - models trained from scratch are the comparison that decides it · outcome: absent-stated-about-other Quotation withheld because this record is not cleared for redistribution. representation learning representation learning Quotation withheld because this record is not cleared for redistribution. RL RL Quotation withheld because this record is not cleared for redistribution. deep reinforcement learning deep reinforcement learning Quotation withheld because this record is not cleared for redistribution. DL DL Quotation withheld because this record is not cleared for redistribution. ML ML Quotation withheld because this record is not cleared for redistribution. TL TL Quotation withheld because this record is not cleared for redistribution. Same-domain TL Same-domain TL Quotation withheld because this record is not cleared for redistribution. underspecification underspecification Quotation withheld because this record is not cleared for redistribution. stress tests stress tests Quotation withheld because this record is not cleared for redistribution. Sparse Connectivity Sparse Connectivity Quotation withheld because this record is not cleared for redistribution. Weight Sharing Weight Sharing Quotation withheld because this record is not cleared for redistribution. Dropout Dropout Quotation withheld because this record is not cleared for redistribution. Dying ReLU Dying ReLU Quotation withheld because this record is not cleared for redistribution. catastrophic forgetting catastrophic forgetting Quotation withheld because this record is not cleared for redistribution. vanishing gradient problem vanishing gradient problem Quotation withheld because this record is not cleared for redistribution. overfitting overfitting Quotation withheld because this record is not cleared for redistribution. black box black box Quotation withheld because this record is not cleared for redistribution. imbalanced imbalanced Quotation withheld because this record is not cleared for redistribution. hyper-parameter selection hyper-parameter selection Quotation withheld because this record is not cleared for redistribution. lightweight models trained from scratch lightweight models trained from scratch Quotation withheld because this record is not cleared for redistribution. feature engineering feature engineering Quotation withheld because this record is not cleared for redistribution. histogram of oriented gradients (HOG) histogram of oriented gradients (HOG) Quotation withheld because this record is not cleared for redistribution. scale-invariant feature transform (SIFT) scale-invariant feature transform (SIFT) Quotation withheld because this record is not cleared for redistribution. bag of words (BoW) bag of words (BoW) Quotation withheld because this record is not cleared for redistribution. feature extraction feature extraction Quotation withheld because this record is not cleared for redistribution. multi-layer data representation architecture multi-layer data representation architecture Quotation withheld because this record is not cleared for redistribution. low-level features low-level features Quotation withheld because this record is not cleared for redistribution. high-level features high-level features Quotation withheld because this record is not cleared for redistribution. unsupervised unsupervised Quotation withheld because this record is not cleared for redistribution. partially supervised (semi-supervised) partially supervised (semi-supervised) Quotation withheld because this record is not cleared for redistribution. supervised supervised Quotation withheld because this record is not cleared for redistribution. labeled data labeled data Quotation withheld because this record is not cleared for redistribution. recurrent neural networks (RNNs) recurrent neural networks (RNNs) Quotation withheld because this record is not cleared for redistribution. convolutional neural networks (CNNs) convolutional neural networks (CNNs) Quotation withheld because this record is not cleared for redistribution. deep neural networks (DNNs) deep neural networks (DNNs) Quotation withheld because this record is not cleared for redistribution. gated recurrent units (GRUs) gated recurrent units (GRUs) Quotation withheld because this record is not cleared for redistribution. long short-term memory (LSTM) approaches long short-term memory (LSTM) approaches Quotation withheld because this record is not cleared for redistribution. decision boundary decision boundary Quotation withheld because this record is not cleared for redistribution. generative networks generative networks Quotation withheld because this record is not cleared for redistribution. dimensionality reduction dimensionality reduction Quotation withheld because this record is not cleared for redistribution. clustering clustering Quotation withheld because this record is not cleared for redistribution. restricted Boltzmann machines restricted Boltzmann machines Quotation withheld because this record is not cleared for redistribution. auto-encoders auto-encoders Quotation withheld because this record is not cleared for redistribution. GANs GANs Quotation withheld because this record is not cleared for redistribution. reward function reward function Quotation withheld because this record is not cleared for redistribution. environment environment Quotation withheld because this record is not cleared for redistribution. supervised learning supervised learning Quotation withheld because this record is not cleared for redistribution. RNN RNN Quotation withheld because this record is not cleared for redistribution. sequential data sequential data Quotation withheld because this record is not cleared for redistribution. short-term memory short-term memory Quotation withheld because this record is not cleared for redistribution. state (hidden) layer state (hidden) layer Quotation withheld because this record is not cleared for redistribution. exploding gradient exploding gradient Quotation withheld because this record is not cleared for redistribution. LSTM LSTM Quotation withheld because this record is not cleared for redistribution. memory blocks memory blocks Quotation withheld because this record is not cleared for redistribution. memory cells memory cells Quotation withheld because this record is not cleared for redistribution. gated units gated units Quotation withheld because this record is not cleared for redistribution. conventional networks conventional networks Quotation withheld because this record is not cleared for redistribution. CNN CNN Quotation withheld because this record is not cleared for redistribution. equivalent representations equivalent representations Quotation withheld because this record is not cleared for redistribution. sparse interactions sparse interactions Quotation withheld because this record is not cleared for redistribution. parameter sharing parameter sharing Quotation withheld because this record is not cleared for redistribution. shared weights shared weights Quotation withheld because this record is not cleared for redistribution. local connections local connections Quotation withheld because this record is not cleared for redistribution. 2D input-data structures 2D input-data structures Quotation withheld because this record is not cleared for redistribution. convolution layers convolution layers Quotation withheld because this record is not cleared for redistribution. sub-sampling (pooling) layers sub-sampling (pooling) layers Quotation withheld because this record is not cleared for redistribution. FC layers FC layers Quotation withheld because this record is not cleared for redistribution. feature maps feature maps Quotation withheld because this record is not cleared for redistribution. softmax softmax Quotation withheld because this record is not cleared for redistribution. support vector machines (SVMs) support vector machines (SVMs) Quotation withheld because this record is not cleared for redistribution. kernel weight kernel weight Quotation withheld because this record is not cleared for redistribution. convolutional operation convolutional operation Quotation withheld because this record is not cleared for redistribution. feature map feature map Quotation withheld because this record is not cleared for redistribution. stride stride Quotation withheld because this record is not cleared for redistribution. padding padding Quotation withheld because this record is not cleared for redistribution. convolutional layer convolutional layer Quotation withheld because this record is not cleared for redistribution. convolutional filters convolutional filters Quotation withheld because this record is not cleared for redistribution. pooling layer pooling layer Quotation withheld because this record is not cleared for redistribution. pooling methods pooling methods Quotation withheld because this record is not cleared for redistribution. max pooling max pooling Quotation withheld because this record is not cleared for redistribution. min pooling min pooling Quotation withheld because this record is not cleared for redistribution. global average pooling (GAP) global average pooling (GAP) Quotation withheld because this record is not cleared for redistribution. global max pooling global max pooling Quotation withheld because this record is not cleared for redistribution. activation function activation function Quotation withheld because this record is not cleared for redistribution. Sigmoid Sigmoid Quotation withheld because this record is not cleared for redistribution. Tanh Tanh Quotation withheld because this record is not cleared for redistribution. ReLU ReLU Quotation withheld because this record is not cleared for redistribution. Leaky ReLU Leaky ReLU Quotation withheld because this record is not cleared for redistribution. Noisy ReLU Noisy ReLU Quotation withheld because this record is not cleared for redistribution. Parametric Linear Units Parametric Linear Units Quotation withheld because this record is not cleared for redistribution. Fully Connected Layer Fully Connected Layer Quotation withheld because this record is not cleared for redistribution. loss functions loss functions Quotation withheld because this record is not cleared for redistribution. Cross-Entropy or Softmax Loss Function Cross-Entropy or Softmax Loss Function Quotation withheld because this record is not cleared for redistribution. log loss function log loss function Quotation withheld because this record is not cleared for redistribution. Euclidean Loss Function Euclidean Loss Function Quotation withheld because this record is not cleared for redistribution. mean square error mean square error Quotation withheld because this record is not cleared for redistribution. Hinge Loss Function Hinge Loss Function Quotation withheld because this record is not cleared for redistribution. Drop-Weights Drop-Weights Quotation withheld because this record is not cleared for redistribution. Data Augmentation Data Augmentation Quotation withheld because this record is not cleared for redistribution. Batch Normalization Batch Normalization Quotation withheld because this record is not cleared for redistribution. internal covariance shift internal covariance shift Quotation withheld because this record is not cleared for redistribution. over-fitted over-fitted Quotation withheld because this record is not cleared for redistribution. under-fitted model under-fitted model Quotation withheld because this record is not cleared for redistribution. just-fitted just-fitted Quotation withheld because this record is not cleared for redistribution. learning rate learning rate Quotation withheld because this record is not cleared for redistribution. training epoch training epoch Quotation withheld because this record is not cleared for redistribution. Batch Gradient Descent Batch Gradient Descent Quotation withheld because this record is not cleared for redistribution. Stochastic Gradient Descent Stochastic Gradient Descent Quotation withheld because this record is not cleared for redistribution. Mini-batch Gradient Descent Mini-batch Gradient Descent Quotation withheld because this record is not cleared for redistribution. Adaptive Moment Estimation (Adam) Adaptive Moment Estimation (Adam) Quotation withheld because this record is not cleared for redistribution. Adam Adam Quotation withheld because this record is not cleared for redistribution. learning strategy learning strategy Quotation withheld because this record is not cleared for redistribution. residual links residual links Quotation withheld because this record is not cleared for redistribution. deep network convergence deep network convergence Quotation withheld because this record is not cleared for redistribution. highway network highway network Quotation withheld because this record is not cleared for redistribution. ResNet ResNet Quotation withheld because this record is not cleared for redistribution. DenseNet DenseNet Quotation withheld because this record is not cleared for redistribution. CapsuleNet CapsuleNet Quotation withheld because this record is not cleared for redistribution. CapsNet CapsNet Quotation withheld because this record is not cleared for redistribution. High-Resolution Network (HRNet) High-Resolution Network (HRNet) Quotation withheld because this record is not cleared for redistribution. HRNet HRNet Quotation withheld because this record is not cleared for redistribution. training data training data Quotation withheld because this record is not cleared for redistribution. simulated data simulated data Quotation withheld because this record is not cleared for redistribution. data augmentation techniques data augmentation techniques Quotation withheld because this record is not cleared for redistribution. data-space solutions data-space solutions Quotation withheld because this record is not cleared for redistribution. model interpretability model interpretability Quotation withheld because this record is not cleared for redistribution. perturbation-based approaches perturbation-based approaches Quotation withheld because this record is not cleared for redistribution. back-propagation-based techniques back-propagation-based techniques Quotation withheld because this record is not cleared for redistribution. score of confidence score of confidence Quotation withheld because this record is not cleared for redistribution. temperature scaling temperature scaling Quotation withheld because this record is not cleared for redistribution. incorporating new information into a plain DL model incorporating new information into a plain DL model Quotation withheld because this record is not cleared for redistribution. parameter pruning parameter pruning Quotation withheld because this record is not cleared for redistribution. knowledge distillation knowledge distillation Quotation withheld because this record is not cleared for redistribution. compact convolution filters compact convolution filters Quotation withheld because this record is not cleared for redistribution. low-rank factorization low-rank factorization Quotation withheld because this record is not cleared for redistribution. lack of training data lack of training data Quotation withheld because this record is not cleared for redistribution. stress tests stress tests Quotation withheld because this record is not cleared for redistribution. classification classification Quotation withheld because this record is not cleared for redistribution. localization localization Quotation withheld because this record is not cleared for redistribution. detection detection Quotation withheld because this record is not cleared for redistribution. segmentation segmentation Quotation withheld because this record is not cleared for redistribution. registration registration Quotation withheld because this record is not cleared for redistribution. Target Selection Target Selection Quotation withheld because this record is not cleared for redistribution. Feature Extraction Feature Extraction Quotation withheld because this record is not cleared for redistribution. Feature Matching Feature Matching Quotation withheld because this record is not cleared for redistribution. Pose Optimization Pose Optimization Quotation withheld because this record is not cleared for redistribution. CPU CPU Quotation withheld because this record is not cleared for redistribution. GPU GPU Quotation withheld because this record is not cleared for redistribution. FPGA FPGA Quotation withheld because this record is not cleared for redistribution. Accuracy Accuracy Quotation withheld because this record is not cleared for redistribution. Sensitivity or Recall Sensitivity or Recall Quotation withheld because this record is not cleared for redistribution. Specificity Specificity Quotation withheld because this record is not cleared for redistribution. Precision Precision Quotation withheld because this record is not cleared for redistribution. F1-Score F1-Score Quotation withheld because this record is not cleared for redistribution. J Score J Score Quotation withheld because this record is not cleared for redistribution. Youdens J statistic Youdens J statistic Quotation withheld because this record is not cleared for redistribution. Area Under the ROC Curve Area Under the ROC Curve Quotation withheld because this record is not cleared for redistribution. RNN category RNN category Quotation withheld because this record is not cleared for redistribution. increasing the stride value increasing the stride value Quotation withheld because this record is not cleared for redistribution. a feature map of lower dimensions a feature map of lower dimensions Quotation withheld because this record is not cleared for redistribution. sub-sampling layers sub-sampling layers Quotation withheld because this record is not cleared for redistribution. a reduction in the network parameters a reduction in the network parameters Quotation withheld because this record is not cleared for redistribution. how confident the model is in its prediction how confident the model is in its prediction Quotation withheld because this record is not cleared for redistribution. overfitting problem overfitting problem Quotation withheld because this record is not cleared for redistribution. visual cortex visual cortex Quotation withheld because this record is not cleared for redistribution. exploding gradient and vanishing problems exploding gradient and vanishing problems Quotation withheld because this record is not cleared for redistribution. model compression model compression Quotation withheld because this record is not cleared for redistribution. DNN training DNN training Quotation withheld because this record is not cleared for redistribution. inference accelerators inference accelerators Quotation withheld because this record is not cleared for redistribution. Sensitivity Sensitivity Quotation withheld because this record is not cleared for redistribution. Recall Recall Quotation withheld because this record is not cleared for redistribution. CNN → visual cortex CNN structure is described as inspired by visual-cortex organization; inspired-by is absent from the eight governed types. — the relation stated: Convolutional neural networks Quotation withheld Quotation withheld because this record is not cleared for redistribution. LSTM → exploding gradient and vanishing problems LSTM is presented as handling the recurrent network's exploding/vanishing-gradient issue; mitigates is absent from the eight governed types. — the relation stated: Recurrent neural networks Typical unfolded RNN diagram Quotation withheld Quotation withheld because this record is not cleared for redistribution. parameter pruning → model compression Parameter pruning is one of four methods used for model compression; method-for is absent, and the prose does not support forcing an ontological part-of edge. — the relation stated: Model compression Quotation withheld Quotation withheld because this record is not cleared for redistribution. knowledge distillation → model compression Knowledge distillation trains a compact model from a larger model for compression; method-for is absent from the governed types. — the relation stated: Model compression Quotation withheld Quotation withheld because this record is not cleared for redistribution. registration → Target Selection Target Selection is the first named stage of image registration; stage-of is absent from the eight governed types. — the relation stated: Registration Quotation withheld Quotation withheld because this record is not cleared for redistribution. registration → Pose Optimization Pose Optimization is the distance-minimization stage of image registration; stage-of is absent from the eight governed types. — the relation stated: Registration Quotation withheld Quotation withheld because this record is not cleared for redistribution. GPU → DNN training The GPU is described as optimized for DNN training; optimized-for is absent from the eight governed types. — the relation stated: CPU-based approach Quotation withheld Quotation withheld because this record is not cleared for redistribution. FPGA → inference accelerators FPGAs are commonly used to implement inference accelerators; used-for is absent from the governed relation types. — the relation stated: FPGA-based approach Quotation withheld Quotation withheld because this record is not cleared for redistribution. Sensitivity → Recall Sensitivity and Recall are presented as alternate names for one metric; the slash/or naming construction lacks a governed same-as cue, so the alias is filed unhoused. — the relation stated: Evaluation metrics Quotation withheld Quotation withheld because this record is not cleared for redistribution. CapsuleNet → CapsNet CapsNet is the short form used for CapsuleNet; abbreviation expansion is absent from the eight governed types. — the relation stated: CapsuleNet Quotation withheld Quotation withheld because this record is not cleared for redistribution. RL - REPRESENTATION LEARNING. Given in the introduction as another name for deep learning itself Quotation withheld because this record is not cleared for redistribution. RL - REINFORCEMENT LEARNING. Given 16,670 characters later as another name for deep reinforcement learning, a distinct technique the same paper classifies separately. Neither definition acknowledges the other Quotation withheld because this record is not cleared for redistribution. transfer learning - AS RECOMMENDED: enhances accuracy across pattern recognition and computer vision, and is the paper's answer to insufficient training data Quotation withheld because this record is not cleared for redistribution. transfer learning - AS MEASURED ACROSS DISSIMILAR DOMAINS: does not significantly affect performance, with models trained from scratch performing nearly as well. Same technique, opposite verdict, and the disambiguator is source-target similarity Quotation withheld because this record is not cleared for redistribution. Feature engineering versus deep representation learning - conventional ML constructs task-specific features manually, whereas DL extracts layered discriminative features automatically Quotation withheld because this record is not cleared for redistribution. Recurrent state - the hidden layer functions as short-term memory carrying information from earlier elements of a sequence Quotation withheld because this record is not cleared for redistribution. CNN locality - sparse local connections and shared weights exploit image structure while reducing parameter and memory requirements Quotation withheld because this record is not cleared for redistribution. Pooling trade-off - subsampling reduces parameters and computation but can discard the location of a detected feature Quotation withheld because this record is not cleared for redistribution. Activation-function family - ReLU reduces computation but can permanently deactivate neurons; leaky, noisy, and parametric variants modify that behavior Quotation withheld because this record is not cleared for redistribution. Fitting-state distinction - over-fitting fails on unseen data, under-fitting fails to learn enough, and just-fitting succeeds on both training and test sets Quotation withheld because this record is not cleared for redistribution. Dense connectivity - DenseNet concatenates all preceding feature maps, improving information flow at the cost of more feature maps and parameter expense Quotation withheld because this record is not cleared for redistribution. Uncertainty calibration - raw softmax probabilities may be overconfident and require post-scaling before they function as reliable confidence estimates Quotation withheld because this record is not cleared for redistribution. Compression taxonomy - parameter pruning, knowledge distillation, compact filters, and low-rank factorization reduce different sources of model cost Quotation withheld because this record is not cleared for redistribution. Application-task boundary - localization uses one bounding box, detection may use many, segmentation outlines pixels, and registration aligns images Quotation withheld because this record is not cleared for redistribution. Hardware specialization - CPU favors memory and generality, GPU favors parallel DNN training, and FPGA favors configurable low-latency inference Quotation withheld because this record is not cleared for redistribution. Metric-role distinction - evaluation metrics guide classifier selection during training and assess efficiency on hidden data during testing Quotation withheld because this record is not cleared for redistribution. critiques → existing DL review papers, each of which addresses a single application or topic and so leaves an overall lack of knowledge about the field Quotation withheld because this record is not cleared for redistribution. builds-on → the 2020 Google identification of underspecification as a distinct failure mode of ML pipelines Quotation withheld because this record is not cleared for redistribution. qualifies → transfer learning from ImageNet as the standard remedy for insufficient training data in medical imaging Quotation withheld because this record is not cleared for redistribution. DL —part-of→ ML DL ML the relation itself is separately evidenced · strength of the relation: 0.95 · the relation stated: deep learning is a subset of machine learning Quotation withheld because this record is not cleared for redistribution. representation learning —same-as→ RL representation learning RL the relation itself is separately evidenced · strength of the relation: 0.9 · the relation stated: given in apposition in the introduction Quotation withheld because this record is not cleared for redistribution. deep reinforcement learning —same-as→ RL deep reinforcement learning RL the relation itself is separately evidenced · strength of the relation: 0.9 · the relation stated: given as another name 16,670 characters later, for a different concept Quotation withheld because this record is not cleared for redistribution. TL —contrasted-with→ lightweight models trained from scratch TL lightweight models trained from scratch differs in: performance on medical imaging tasks when source and target domains differ · the relation itself is separately evidenced · strength of the relation: 0.9 · differs in (stated by the tagger, no quotation): performance on medical imaging tasks when source and target domains differ · the relation stated: cross-domain transfer does not beat training from scratch on medical imaging Quotation withheld because this record is not cleared for redistribution. RNN —contrasted-with→ conventional networks RNN conventional networks the relation itself is separately evidenced · strength of the relation: 1 · note: The review explicitly contrasts recurrent use of sequential state with conventional networks. Quotation withheld because this record is not cleared for redistribution. CNN —contrasted-with→ RNN CNN RNN the relation itself is separately evidenced · strength of the relation: 1 · note: The paper directly compares their power and feature compatibility. Quotation withheld because this record is not cleared for redistribution. gated recurrent units (GRUs) —part-of→ RNN category gated recurrent units (GRUs) RNN category the relation itself is separately evidenced · strength of the relation: 1 · note: The RNN category is explicitly said to include GRUs. Quotation withheld because this record is not cleared for redistribution. long short-term memory (LSTM) approaches —part-of→ RNN category long short-term memory (LSTM) approaches RNN category the relation itself is separately evidenced · strength of the relation: 1 · note: The RNN category is explicitly said to include LSTM approaches. Quotation withheld because this record is not cleared for redistribution. convolutional filters —part-of→ convolutional layer convolutional filters convolutional layer the relation itself is separately evidenced · strength of the relation: 1 · note: The convolutional layer is stated to consist of a collection of filters. Quotation withheld because this record is not cleared for redistribution. max pooling —part-of→ pooling methods max pooling pooling methods the relation itself is separately evidenced · strength of the relation: 1 · note: Max pooling is explicitly included among pooling methods. Quotation withheld because this record is not cleared for redistribution. global average pooling (GAP) —part-of→ pooling methods global average pooling (GAP) pooling methods the relation itself is separately evidenced · strength of the relation: 1 · note: GAP is explicitly included among pooling methods. Quotation withheld because this record is not cleared for redistribution. increasing the stride value —causes→ a feature map of lower dimensions increasing the stride value a feature map of lower dimensions the relation itself is separately evidenced · strength of the relation: 1 · note: The lower-dimensional map is stated to result from increasing stride. Quotation withheld because this record is not cleared for redistribution. sub-sampling layers —causes→ a reduction in the network parameters sub-sampling layers a reduction in the network parameters the relation itself is separately evidenced · strength of the relation: 1 · note: Down-sampling is stated to lead to fewer parameters. Quotation withheld because this record is not cleared for redistribution. Cross-Entropy or Softmax Loss Function —same-as→ log loss function Cross-Entropy or Softmax Loss Function log loss function the relation itself is separately evidenced · strength of the relation: 1 · note: The review explicitly gives log loss as another name. Quotation withheld because this record is not cleared for redistribution. Drop-Weights —contrasted-with→ Dropout Drop-Weights Dropout the relation itself is separately evidenced · strength of the relation: 1 · note: Weights rather than neurons are dropped; the paper calls this the only difference. Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but Drop-Weights is 'highly similar to dropout', differing in what is dropped; contrasted-with is loose. Quotation withheld because this record is not cleared for redistribution. Adam —is-a→ learning strategy Adam learning strategy the relation itself is separately evidenced · strength of the relation: 1 · note: The paper directly classifies Adam as a learning strategy. Quotation withheld because this record is not cleared for redistribution. ResNet —contrasted-with→ highway network ResNet highway network the relation itself is separately evidenced · strength of the relation: 1 · note: The review contrasts ResNet's always-open parameter-free shortcuts with highway gates. Quotation withheld because this record is not cleared for redistribution. High-Resolution Network (HRNet) —same-as→ HRNet High-Resolution Network (HRNet) HRNet the relation itself is separately evidenced · strength of the relation: 1 · note: The long name is explicitly followed by the short name. Quotation withheld because this record is not cleared for redistribution. catastrophic forgetting —operationalised-as→ incorporating new information into a plain DL model catastrophic forgetting incorporating new information into a plain DL model the relation itself is separately evidenced · strength of the relation: 1 · note: The paper gives an explicit definition. · the relation stated: Catastrophic forgetting Quotation withheld Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but endpoint by anaphora: 'This' = catastrophic forgetting (the heading immediately before); the definition is stated. Quotation withheld because this record is not cleared for redistribution. score of confidence —operationalised-as→ how confident the model is in its prediction score of confidence how confident the model is in its prediction the relation itself is separately evidenced · strength of the relation: 1 · note: The paper explicitly defines the confidence score. Quotation withheld because this record is not cleared for redistribution. FPGA —contrasted-with→ GPU FPGA GPU the relation itself is separately evidenced · strength of the relation: 1 · note: The processors are directly compared on floating-point performance and configuration. Quotation withheld because this record is not cleared for redistribution. J Score —same-as→ Youdens J statistic J Score Youdens J statistic the relation itself is separately evidenced · strength of the relation: 1 · note: The paper explicitly states the alternate metric name. Quotation withheld because this record is not cleared for redistribution. transfer learning from a dissimilar domain does not significantly affect medical imaging performance; models trained from scratch do nearly as well transfer learning from a dissimilar domain does not significantly affect medical imaging performance; models trained from scratch do nearly as well Quotation withheld because this record is not cleared for redistribution. underspecification means small modifications force a model toward a completely different solution and different deployment predictions underspecification means small modifications force a model toward a completely different solution and different deployment predictions Quotation withheld because this record is not cleared for redistribution. the remedy proposed for underspecification requires already understanding how the model can fail the remedy proposed for underspecification requires already understanding how the model can fail Quotation withheld because this record is not cleared for redistribution. CNN performance is greatly influenced by hyper-parameter selection; any small change affects it. CNN performance is greatly influenced by hyper-parameter selection; any small change affects it. Quotation withheld because this record is not cleared for redistribution. most current deep-learning models still use supervised learning despite the stated transition toward semi- and unsupervised methods most current deep-learning models still use supervised learning despite the stated transition toward semi- and unsupervised methods Quotation withheld because this record is not cleared for redistribution. a ranking-type metric that, in contrast to probability and threshold metrics, exposes the entire classifier ranking performance AUC Quotation withheld because this record is not cleared for redistribution. a subset of ML, inspired by the information processing patterns of the human brain, which requires no human-designed rules and maps input to labels from data DL Quotation withheld because this record is not cleared for redistribution. only a few weights between two adjacent layers, against fully connected networks where every neuron links to every neuron in the next layer Sparse Connectivity Quotation withheld because this record is not cleared for redistribution. no weights allocated between any two neurons of neighbouring layers; one set of weights operates over every pixel of the input matrix Weight Sharing Quotation withheld because this record is not cleared for redistribution. neurons randomly dropped during each training epoch, distributing feature-selection power across the whole group and forcing independent features Dropout Quotation withheld because this record is not cleared for redistribution. incorporating new information into a plain DL model by interfering with the learned information - fine-tuning on a new class breaks the old ones catastrophic forgetting Quotation withheld because this record is not cleared for redistribution. the reason ML models show surprisingly poor behaviour when tested in real-world applications despite passing their own validation underspecification Quotation withheld because this record is not cleared for redistribution. using images that look similar to the target dataset for training - e.g. X-rays of other chest diseases before fine-tuning on COVID-19 chest X-rays Same-domain TL Quotation withheld because this record is not cleared for redistribution. papers reviewed from 2020, of more than 300 total 108 Quotation withheld because this record is not cleared for redistribution. the floor on the number of papers reviewed 300 Quotation withheld because this record is not cleared for redistribution. AUC AUC Quotation withheld because this record is not cleared for redistribution. Sensitivity or Recall Sensitivity or Recall Quotation withheld because this record is not cleared for redistribution. Specificity Specificity Quotation withheld because this record is not cleared for redistribution. hyper-parameter selection hyper-parameter selection Quotation withheld because this record is not cleared for redistribution. amount of data amount of data Quotation withheld because this record is not cleared for redistribution.

The full text of this article is not redistributable, so it is not included here. The evidence sentences are withheld with it; the tag layer and its grounding status remain. The article itself is at https://doi.org/10.1186/s40537-021-00444-8.