Deep Learning for Computer Vision: A Brief Review — TEI-tagged view

Athanasios VoulodimospersName, Nikolaos DoulamispersName, Anastasios DoulamispersName, Eftychios ProtopapadakispersName (2018date). “Deep Learning for Computer Vision: A Brief Review.” Computational Intelligence and Neurosciencetitle·venue.

Object of study — deep learningterm architectures used in computer vision - CNNsterm, the Boltzmann familyterm, and stacked denoising autoencodersterm

Research question — what the main deep learningterm schemes for computer vision are, what they can and cannot do, and what remains unsolved

Contribution — a comparison of three model families by advantage and disadvantage, closing with the statement that the field has no theoretical account of WHY a given architecture works

Full text — tagged

The reference list is not part of the text we hold for this paper. Our copy was extracted from the publisher's page and the bibliography did not survive that extraction, so the article below ends before its references. Nothing has been removed by us.

# Abstract Over the last years deep learningterm methods have been shown to outperform previous state-of-the-art machine learning techniques in several fields, with computer vision being one of the most prominent cases. This reviewterm·mth paper provides a brief overview of some of the most significant deep learningterm schemes used in computer vision problems, that is, Convolutional Neural Networksterm, Deep Boltzmann Machinesterm and Deep Belief Networksterm, and Stacked Denoising Autoencodersterm. A brief account of their history, structure, advantages, and limitations is given, followed by a description of their applications in various computer vision tasks, such as object detectionterm, face recognition, action and activity recognition, and human pose estimation. Finally, a brief overview is given of future directions in designing deep learningterm schemes for computer vision problems and the challenges involved therein.

1. Introduction Deep learningterm allows computational models of multiple processing layers to learn and represent data with multiple levels of abstraction mimicking how the brain perceives and understands multimodal information, thus implicitly capturing intricate structures of large‐scale data. Deep learningterm is a rich family of methods, encompassing neural networksterm, hierarchical probabilistic models, and a variety of unsupervised and supervised feature learningterm algorithms. The recent surge of interest in deep learningterm methods is due to the fact that they have been shown to outperform previous state-of-the-art techniques in several tasks, as well as the abundance of complex data from different sources (e.g., visual, audio, medical, social, and sensor).

The ambition to create a system that simulates the human brain fueled the initial development of neural networksterm. In 1943, McCulloch and Pitts [1] tried to understand how the brain could produce highly complex patterns by using interconnected basic cells, called neurons. The McCulloch and Pitts model of a neuron, called a MCP model, has made an important contribution to the development of artificial neural networksterm. A series of major contributions in the field is presented in Table 1, including LeNet [2] and Long Short-Term Memory [3], leading up to today's “era of deep learningterm.” One of the most substantial breakthroughs in deep learningterm came in 2006, when Hinton et al. [4] introduced the Deep Belief Networkterm, with multiple layers of Restricted Boltzmann Machinesterm, greedily training one layer at a timeterm in an unsupervised way. Guiding the training of intermediate levels of representationterm using unsupervised learning, performed locally at each level, was the main principle behind a series of developments that brought about the last decade's surge in deep architectures and deep learningterm algorithms.

Table 1.

Important milestones in the history of neural networksterm and machine learning, leading up to the era of deep learningterm.

| Milestone/contribution | Contributor, year | | :-- | :-: | | MCP model, regarded as the ancestor of the Artificial Neural Networkterm | McCulloch & Pitts, 1943 | | Hebbian learning rule | Hebb, 1949 | | First perceptron | Rosenblatt, 1958 | | Backpropagation | Werbos, 1974 | | Neocognitronterm, regarded as the ancestor of the Convolutional Neural Networkterm | Fukushima, 1980 | | Boltzmann Machineterm | Ackley, Hinton & Sejnowski, 1985 | | Restricted Boltzmann Machineterm (initially known as Harmonium) | Smolensky, 1986 | | Recurrent Neural Networkterm | Jordan, 1986 | | Autoencodersterm | Rumelhart, Hinton & Williams, 1986 | | Ballard, 1987 | | LeNet, starting the era of Convolutional Neural Networksterm | LeCun, 1990 | | LSTM | Hochreiter & Schmidhuber, 1997 | | Deep Belief Networkterm, ushering the “age of deep learningterm” | Hinton, 2006 | | Deep Boltzmann Machineterm | Salakhutdinov & Hinton, 2009 | | AlexNet, starting the age of CNNterm used for ImageNet classification | Krizhevsky, Sutskever, & Hinton, 2012 | Among the most prominent factors that contributed to the huge boost of deep learningterm are the appearance of large, high-quality, publicly available labelled datasetsterm, along with the empowerment of parallel GPU computingterm, which enabled the transition from CPU-basedterm to GPU-based trainingterm thus allowing for significant acceleration in deep models' training. Additional factors may have played a lesser role as well, such as the alleviation of the vanishing gradient problemterm owing to the disengagement from saturating activation functionsterm (such as hyperbolic tangentterm and the logistic functionterm), the proposal of new regularization techniquesterm (e.g., dropoutterm, batch normalizationterm, and data augmentationterm), and the appearance of powerful frameworks like TensorFlowterm [5], theanoterm [6], and mxnetterm [7], which allow for faster prototyping.

Deep learningterm has fueled great strides in a variety of computer vision problems, such as object detectionterm (e.g., [8, 9]), motion tracking (e.g., [10, 11]), action recognition (e.g., [12, 13]), human pose estimation (e.g., [14, 15]), and semantic segmentationterm (e.g., [16, 17]). In this overview, we will concisely reviewterm·mth the main developments in deep learningterm architectures and algorithms for computer vision applications. In this context, we will focus on three of the most important types of deep learningterm models with respect to their applicability in visual understanding, that is, Convolutional Neural Networks (CNNs)term, the “Boltzmann familyterm” including Deep Belief Networksterm (DBNs) and Deep Boltzmann Machinesterm (DBMs) and Stacked (Denoising) Autoencodersterm. Needless to say, the current coverage is by no means exhaustive; for example, Long Short-Term Memory (LSTM), in the category of Recurrent Neural Networksterm, although of great significance as a deep learningterm scheme, is not presented in this reviewterm·mth, since it is predominantly applied in problems such as language modeling, text classification, handwriting recognition, machine translation, speech/music recognition, and less so in computer vision problems. The overview is intended to be useful to computer vision and multimedia analysis researchers, as well as to general machine learning researchers, who are interested in the state of the art in deep learningterm for computer vision tasks, such as object detectionterm and recognition, face recognition, action/activity recognition, and human pose estimation.

The remainder of this paper is organized as follows. In Section 2, the three aforementioned groups of deep learningterm model are reviewed: Convolutional Neural Networksterm, Deep Belief Networksterm and Deep Boltzmann Machinesterm, and Stacked Autoencodersterm. The basic architectures, training processes, recent developments, advantages, and limitations of each group are presented. In Section 3, we describe the contribution of deep learningterm algorithms to key computer vision tasks, such as object detectionterm and recognition, face recognition, action/activity recognition, and human pose estimation; we also provide a list of important datasets and resources for benchmarking and validation of deep learningterm algorithms. Finally, Section 4 concludes the paper with a summary of findings.

2. Deep Learningterm Methods and Developments 2.1. Convolutional Neural Networksterm Convolutional Neural Networks (CNNs)term were inspired by the visual system's structureterm, and in particular by the models of it proposed in [18]. The first computational models based on these local connectivities between neuronsterm and on hierarchically organized transformations of the imageterm are found in Neocognitronterm [19], which describes that when neurons with the same parameters are applied on patches of the previous layer at different locations, a form of translational invarianceterm is acquired. Yann LeCun and his collaborators later designed Convolutional Neural Networksterm employing the error gradient and attaining very good results in a variety of pattern recognition tasks [20– 22].

A CNNterm comprises three main types of neural layers, namely, (i) convolutional layersterm, (ii) pooling layersterm, and (iii) fully connected layersterm. Each type of layer plays a different role. Figure 1 shows a CNNterm architecture for an object detectionterm in image task. Every layer of a CNNterm transforms the input volumeterm to an output volume of neuron activationterm, eventually leading to the final fully connected layersterm, resulting in a mapping of the input data to a 1D feature vectorterm. CNNsterm have been extremely successful in computer vision applications, such as face recognition, object detectionterm, powering vision in robotics, and self-driving cars.

Figure 1.

 Example architecture of a CNNterm for a computer vision task (object detectionterm).

_(i) Convolutional Layersterm._ In the convolutional layersterm, a CNNterm utilizes various kernelsterm to convolve the whole imageterm as well as the intermediate feature mapsterm, generating various feature mapsterm. Because of the advantages of the convolution operationterm, several works (e.g., [23, 24]) have proposed it as a substitute for fully connected layersterm with a view to attaining faster learning timesterm.

_(ii) Pooling Layersterm._ Pooling layersterm are in charge of reducing the spatial dimensionsterm (width × height) of the input volumeterm for the next convolutional layerterm. The pooling layerterm does not affect the depth dimensionterm of the volume. The operation performed by this layer is also called subsamplingterm or downsamplingterm, as the reduction of size leads to a simultaneous loss of informationterm. However, such a loss is beneficial for the network because the decrease in size leads to less computational overheadterm for the upcoming layers of the network, and also it works against overfittingterm. Average poolingterm and max poolingterm are the most commonly used strategies. In [25] a detailed theoretical analysis of max poolingterm and average poolingterm performances is given, whereas in [26] it was shown that max poolingterm can lead to faster convergenceterm, select superior invariant featuresterm, and improve generalizationterm. Also there are a number of other variations of the pooling layerterm in the literature, each inspired by different motivations and serving distinct needs, for example, stochastic poolingterm [27], spatial pyramid poolingterm [28, 29], and def-poolingterm [30].

_(iii) Fully Connected Layersterm._ Following several convolutional and pooling layersterm, the high-level reasoningterm in the neural networkterm is performed via fully connected layersterm. Neurons in a fully connected layerterm have full connectionsterm to all activation in the previous layer, as their name implies. Their activation can hence be computed with a matrix multiplicationterm followed by a bias offsetterm. Fully connected layersterm eventually convert the 2D feature mapsterm into a 1D feature vectorterm. The derived vector either could be fed forward into a certain number of categoriesterm for classification [31] or could be considered as a feature vector for further processing [32].

The architecture of CNNsterm employs three concrete ideas: (a) local receptive fieldsterm, (b) tied weightsterm, and (c) spatial subsamplingterm. Based on local receptive fieldterm, each unit in a convolutional layerterm receives inputs from a set of neighboring unitsterm belonging to the previous layer. This way neurons are capable of extracting elementary visual featuresterm such as edges or cornersterm. These features are then combined by the subsequent convolutional layersterm in order to detect higher order featuresterm. Furthermore, the idea that elementary feature detectorsterm, which are useful on a part of an image, are likely to be useful across the entire image is implemented by the concept of tied weightsterm. The concept of tied weightsterm constraints a set of units to have identical weightsterm. Concretely, the units of a convolutional layerterm are organized in planesterm. All units of a planeterm share the same set of weights. Thus, each planeterm is responsible for constructing a specific featureterm. The outputs of planesterm are called feature mapsterm. Each convolutional layerterm consists of several planesterm, so that multiple feature mapsterm can be constructed at each location.

During the construction of a feature mapterm, the entire image is scanned by a unit whose states are stored at corresponding locations in the feature mapterm. This construction is equivalent to a convolution operationterm, followed by an additive bias term and sigmoid function: | | | | --- | --- | | | (1) | where d stands for the depth of the convolutional layerterm, W is the weight matrix, and b is the bias term. For fully connected neural networksterm, the weight matrix is full, that is, connects every input to every unit with different weights. For CNNsterm, the weight matrix W is very sparse due to the concept of tied weightsterm. Thus, W has the form of | | | | --- | --- | | | (2) | where w are matrices having the same dimensions with the units' receptive fields. Employing a sparse weight matrix reduces the number of network's tunable parameters and thus increases its generalizationterm ability. Multiplying W with layer inputs is like convolving the input with w, which can be seen as a trainable filter. If the input to d − 1 convolutional layerterm is of dimension N × N and the receptive field of units at a specific planeterm of convolutional layerterm d is of dimension m × m, then the constructed feature mapterm will be a matrix of dimensions ( N − m + 1)×( N − m + 1). Specifically, the element of feature mapterm at ( i, j) location will be | | | | --- | --- | | | (3) | with | | | | --- | --- | | | (4) | where the bias term b is scalar. Using ( 4) and ( 3) sequentially for all ( i, j) positions of input, the feature mapterm for the corresponding planeterm is constructed.

One of the difficulties that may arise with training of CNNsterm has to do with the large number of parametersterm that have to be learned, which may lead to the problem of overfittingterm. To this end, techniques such as stochastic poolingterm, dropoutterm, and data augmentationterm have been proposed. Furthermore, CNNsterm are often subjected to pretrainingterm, that is, to a process that initializes the network with pretrained parametersterm instead of randomly set onesterm. Pretrainingterm can accelerate the learning processterm and also enhance the generalization capabilityterm of the network.

Overall, CNNsterm were shown to significantly outperform traditional machine learning approaches in a wide range of computer vision and pattern recognition tasks [33], examples of which will be presented in Section 3. Their exceptional performance combined with the relative easiness in training are the main reasons that explain the great surge in their popularity over the last few years.

2.2. Deep Belief Networksterm and Deep Boltzmann Machinesterm Deep Belief Networksterm and Deep Boltzmann Machinesterm are deep learningterm models that belong in the “Boltzmann familyterm,” in the sense that they utilize the Restricted Boltzmann Machine (RBM)term as learning moduleterm. The Restricted Boltzmann Machine (RBM)term is a generative stochastic neural networkterm. DBNs have undirected connectionsterm at the top two layers which form an RBM and directed connectionsterm to the lower layers. DBMs have undirected connectionsterm between all layers of the network. A graphic depiction of DBNs and DBMs can be found in Figure 2. In the following subsections, we will describe the basic characteristics of DBNs and DBMs, after presenting their basic building block, the RBM.

Figure 2.

 Deep Belief Networkterm (DBN) and Deep Boltzmann Machineterm (DBM). The top two layers of a DBN form an undirected graph and the remaining layers form a belief network with directed, top-down connections. In a DBM, all connections are undirected.

2.2.1. Restricted Boltzmann Machinesterm A Restricted Boltzmann Machineterm ([34, 35]) is an undirected graphical modelterm with stochastic visible variablesterm v ∈ {0, 1}D and stochastic hidden variablesterm h ∈ {0, 1}F, where each visible variableterm is connected to each hidden variableterm. An RBM is a variant of the Boltzmann Machineterm, with the restriction that the visible unitsterm and hidden unitsterm must form a bipartite graphterm. This restriction allows for more efficient training algorithms, in particular the gradient-based contrastive divergence algorithmterm [36].

The model defines the energy function E: {0, 1}D × {0, 1}F → _ℝ_: | | | | --- | --- | | | (5) | where _θ_ = { a, b, W} are the model parameters; that is, W ij represents the symmetric interaction term between visible unitterm i and hidden unitterm j, and b i, a j are bias terms.

The joint distributionterm over the visible and hidden unitsterm is given by | | | | --- | --- | | | (6) | where _𝒵_( _θ_) is the normalizing constant. The conditional distributions over hidden h and visible v vectors can be derived by ( 5) and ( 6) as | | | | --- | --- | | | (7) | Given a set of observations { v n}n =1_N_ the derivative of the log-likelihood with respect to the model parameters can be derived by ( 6) as | | | | --- | --- | | | (8) | where _𝔼_ P data denotes an expectation with respect to the data distribution P data( h, v; _θ_) = P( h ∣ v; _θ_) P data( v), with P data( v) = (1/ N)∑n _δ_( v − v n) representing the empirical distribution and _𝔼_ P model is an expectation with respect to the distribution defined by the model, as in ( 6).

A detailed explanation along with the description of a practical way to train RBMs was given in [37], whereas [38] discusses the main difficulties of training RBMs and their underlying reasons and proposes a new algorithm with an adaptive learning rate and an enhanced gradient, so as to address the aforementioned difficulties.

2.2.2. Deep Belief Networksterm Deep Belief Networksterm (DBNs) are probabilistic generative modelsterm which provide a joint probability distributionterm over observable dataterm and labelsterm. They are formed by stacking RBMsterm and training them in a greedy manner, as was proposed in [39]. A DBN initially employs an efficient layer-by-layer greedy learning strategyterm to initialize the deep network, and, in the sequel, fine-tunes all weights jointly with the desired outputs. DBNs are graphical models which learn to extract a deep hierarchical representationterm of the training dataterm. They model the joint distributionterm between observed vector x and the l hidden layers h k as follows: | | | | --- | --- | | | (9) | where x = h 0, P( h k∣ h k +1) is a conditional distribution for the visible unitsterm at level k conditioned on the hidden unitsterm of the RBM at level k + 1, and P( h l −1∣ h l) is the visible-hidden joint distributionterm in the top-level RBM.

The principle of greedy layer-wise unsupervised training can be applied to DBNs with RBMs as the building blocks for each layer [33, 39]. A brief description of the process follows: 1. Train the first layer as an RBM that models the raw input x = h 0 as its visible layer.

2. Use that first layer to obtain a representationterm of the input that will be used as data for the second layer. Two common solutions exist. This representationterm can be chosen as being the mean activation P( h 1 = 1∣ h 0) or samples of P( h 1∣ h 0).

3. Train the second layer as an RBM, taking the transformed data (samples or mean activation) as training examplesterm (for the visible layer of that RBM).

4. Iterate steps ((2) and (3)) for the desired number of layers, each time propagating upward either samples or mean values.

5. Fine-tune all the parameters of this deep architecture with respect to a proxy for the DBN log- likelihood, or with respect to a supervised training criterion (after adding extra learning machinery to convert the learned representationterm into supervised predictions, e.g., a linear classifier).

There are two main advantages in the above-described greedy learning processterm of the DBNs [40]. First, it tackles the challenge of appropriate selection of parameters, which in some cases can lead to poor local optimaterm, thereby ensuring that the network is appropriately initialized. Second, there is no requirement for labelled dataterm·var since the process is unsupervised. Nevertheless, DBNs are also plagued by a number of shortcomings, such as the computational costterm associated with training a DBN and the fact that the steps towards further optimization of the network based on maximum likelihood training approximationterm are unclear [41]. Furthermore, a significant disadvantage of DBNs is that they do not account for the two-dimensional structureterm of an input image, which may significantly affect their performance and applicability in computer vision and multimedia analysis problems. However, a later variation of the DBN, the Convolutional Deep Belief Network (CDBN)term ([42, 43]), uses the spatial information of neighboring pixelsterm by introducing convolutional RBMs, thus producing a translation invariant generative modelterm that successfully scales when it comes to high dimensional imagesterm, as is evidenced in [44].

2.2.3. Deep Boltzmann Machinesterm Deep Boltzmann Machinesterm (DBMs) [45] are another type of deep model using RBM as their building block. The difference in architecture of DBNs is that, in the latter, the top two layers form an undirected graphical modelterm and the lower layers form a directed generative modelterm, whereas in the DBM all the connections are undirected. DBMs have multiple layers of hidden unitsterm, where units in odd-numbered layers are conditionally independentterm of even-numbered layers, and vice versa. As a result, inference in the DBMterm is generally intractable. Nonetheless, an appropriate selection of interactions between visible and hidden unitsterm can lead to more tractable versions of the model. During network training, a DBM jointly trains all layers of a specific unsupervised model, and instead of maximizing the likelihood directly, the DBM uses a stochastic maximum likelihood (SML)term [46] based algorithm to maximize the lower bound on the likelihoodterm. Such a process would seem vulnerable to falling in poor local minimaterm [45], leaving several units effectively dead. Instead, a greedy layer-wise training strategyterm was proposed [47], which essentially consists in pretrainingterm the layers of the DBM, similarly to DBN, namely, by stacking RBMsterm and training each layer to independently model the output of the previous layer, followed by a final joint fine-tuningterm.

Regarding the advantages of DBMs, they can capture many layers of complex representationsterm of input data and they are appropriate for unsupervised learning since they can be trained on unlabeled dataterm, but they can also be fine-tuned for a particular task in a supervised fashionterm. One of the attributes that sets DBMs apart from other deep models is that the approximate inference process of DBMs includes, apart from the usual bottom-up process, a top-down feedbackterm, thus incorporating uncertainty about inputsterm in a more effective manner. Furthermore, in DBMs, by following the approximate gradient of a variational lower boundterm on the likelihood objective, one can jointly optimize the parameters of all layers, which is very beneficial especially in cases of learning models from heterogeneous dataterm originating from different modalities [48].

As far as the drawbacks of DBMs are concerned, one of the most important ones is, as mentioned above, the high computational cost of inferenceterm, which is almost prohibitive when it comes to joint optimizationterm in sizeable datasetsterm. Several methods have been proposed to improve the effectiveness of DBMs. These include accelerating inference by using separate models to initialize the values of the hidden unitsterm in all layers [47, 49], or other improvements at the pretrainingterm stage [50, 51] or at the training stage [52, 53].

2.3. Stacked (Denoising) Autoencodersterm Stacked Autoencodersterm use the autoencoderterm as their main building block, similarly to the way that Deep Belief Networksterm use Restricted Boltzmann Machinesterm as component. It is therefore important to briefly present the basics of the autoencoderterm and its denoising version, before describing the deep learningterm architecture of Stacked (Denoising) Autoencodersterm.

2.3.1. Autoencodersterm An autoencoderterm is trained to encode the input x into a representationterm r( x) in a way that input can be reconstructed from r( x) [33]. The target outputterm of the autoencoderterm is thus the autoencoderterm input itself. Hence, the output vectors have the same dimensionality as the input vectorterm. In the course of this process, the reconstruction errorterm is being minimized, and the corresponding code is the learned featureterm. If there is one linear hidden layer and the mean squared error criterionterm is used to train the network, then the k hidden unitsterm learn to project the input in the span of the first k principal componentsterm of the data [54]. If the hidden layer is nonlinearterm, the autoencoderterm behaves differently from PCA, with the ability to capture multimodal aspects of the input distributionterm [55]. The parameters of the model are optimized so that the average reconstruction errorterm is minimized. There are many alternatives to measure the reconstruction errorterm, including the traditional squared error: | | | | --- | --- | | | (10) | where function f is the decoder and f( r( x)) is the reconstruction produced by the model.

If the input is interpreted as bit vectors or vectors of bit probabilities, then the loss function of the reconstruction could be represented by cross-entropy; that is, | | | | --- | --- | | | (11) | The goal is for the representationterm (or code) r( x) to be a distributed representationterm that manages to capture the coordinates along the main variations of the dataterm, similarly to the principle of Principal Componentsterm Analysis (PCA). Given that r( x) is not losslessterm, it is impossible for it to constitute a successful compression for all input x. The aforementioned optimization process results in low reconstruction errorterm on test examples from the same distribution as the training examplesterm but generally high reconstruction errorterm on samples arbitrarily chosen from the input space.

2.3.2. Denoising Autoencodersterm The denoising autoencoderterm [56] is a stochastic version of the autoencoderterm where the input is stochastically corruptedterm, but the uncorrupted inputterm is still used as target for the reconstruction. In simple terms, there are two main aspects in the function of a denoising autoencoderterm: first it tries to encode the input (namely, preserve the information about the input), and second it tries to undo the effect of a corruption processterm stochastically applied to the input of the autoencoderterm (see Figure 3). The latter can only be done by capturing the statistical dependencies between the inputsterm. It can be shown that the denoising autoencoderterm maximizes a lower bound on the log-likelihood of a generative modelterm.

##### Figure 3.

!Figure 3 Denoising autoencoderterm [56].

In [56], the stochastic corruption processterm arbitrarily sets a number of inputs to zeroterm. Then the denoising autoencoderterm is trying to predict the corrupted valuesterm from the uncorrupted onesterm, for randomly selected subsets of missing patternsterm. In essence, the ability to predict any subset of variables from the remaining ones is a sufficient condition for completely capturing the joint distributionterm between a set of variables. It should be mentioned that using autoencodersterm for denoising was introduced in earlier works (e.g., [57]), but the substantial contribution of [56] lies in the demonstration of the successful use of the method for unsupervised pretrainingterm of a deep architecture and in linking the denoising autoencoderterm to a generative modelterm.

2.3.3. Stacked (Denoising) Autoencodersterm It is possible to stack denoising autoencodersterm in order to form a deep network by feeding the latent representationterm (output code) of the denoising autoencoderterm of the layer below as input to the current layer. The unsupervised pretrainingterm of such an architecture is done one layer at a timeterm. Each layer is trained as a denoising autoencoderterm by minimizing the error in reconstructing its input (which is the output code of the previous layer). When the first k layers are trained, we can train the ( k + 1)th layer since it will then be possible compute the latent representationterm from the layer underneath.

When pretrainingterm of all layers is completed, the network goes through a second stage of training called fine-tuningterm. Here supervised fine-tuningterm is considered when the goal is to optimize prediction error on a supervised taskterm. To this end, a logistic regression layerterm is added on the output code of the output layer of the network. The derived network is then trained like a multilayer perceptron, considering only the encoding parts of each autoencoderterm at this point. This stage is supervised, since the target classterm is taken into account during training.

As is easily seen, the principle for training stacked autoencodersterm is the same as the one previously described for Deep Belief Networksterm, but using autoencodersterm instead of Restricted Boltzmann Machinesterm. A number of comparative experimental studies show that Deep Belief Networksterm tend to outperform stacked autoencodersterm ([58, 59]), but this is not always the case, especially when DBNs are compared to Stacked Denoising Autoencodersterm [56].

One strength of autoencodersterm as the basic unsupervised component of a deep architecture is that, unlike with RBMs, they allow almost any parametrization of the layers, on condition that the training criterion is continuous in the parameters. In contrast, one of the shortcomings of SAs is that they do not correspond to a generative modelterm, when with generative modelsterm like RBMs and DBNs, samples can be drawn to check the outputs of the learning processterm.

2.4. Discussion Some of the strengths and limitations of the presented deep learningterm models were already discussed in the respective subsections. In an attempt to compare these models (for a summary see Table 2), we can say that CNNsterm have generally performed better than DBNs in current literature on benchmark computer vision datasets such as MNIST. In cases where the input is nonvisual, DBNs often outperform other models, but the difficulty in accurately estimating joint probabilities as well as the computational costterm in creating a DBN constitutes drawbacks. A major positive aspect of CNNsterm is “feature learningterm,” that is, the bypassing of handcrafted featuresterm, which are necessary for other types of networks; however, in CNNsterm features are automatically learned. On the other hand, CNNsterm rely on the availability of ground truthterm, that is, labelled training dataterm, whereas DBNs/DBMs and SAs do not have this limitation and can work in an unsupervised manner. On a different note, one of the disadvantages of autoencodersterm lies in the fact that they could become ineffective if errors are present in the first layers. Such errors may cause the network to learn to reconstruct the average of the training dataterm. Denoising autoencodersterm [56], however, can retrieve the correct input from a corrupted version, thus leading the network to grasp the structure of the input distribution. In terms of the efficiency of the training process, only in the case of SAs is real-time trainingterm possible, whereas CNNsterm and DBNs/DBMs training processes are time-consuming. Finally, one of the strengths of CNNsterm is the fact that they can be invariant to transformationsterm such as translation, scale, and rotationterm. Invariance to translation, rotation, and scale is one of the most important assets of CNNsterm, especially in computer vision problems, such as object detectionterm, because it allows abstracting an object's identity or categoryterm from the specifics of the visual input (e.g., relative positions/orientation of the camera and the object), thus enabling the network to effectively recognize a given object in cases where the actual pixel valuesterm on the image can significantly differ.

Table 2.

Comparison of CNNsterm, DBNs/DBMs, and SdAsterm with respect to a number of properties. + denotes a good performance in the property and − denotes bad performance or complete lack thereof.

| Model properties | CNNsterm | DBNs/DBMs | SdAsterm | | :-- | :-: | :-: | :-: | | Unsupervised learning | − | + | + | | Training efficiency | − | − | + | | Feature learningterm | + | − | − | | Scale/rotation/translation invariance | + | − | − | | Generalizationterm | + | + | + | 3. Applications in Computer Vision In this section, we survey works that have leveraged deep learningterm methods to address key tasks in computer vision, such as object detectionterm, face recognition, action and activity recognition, and human pose estimation.

3.1. Object Detectionterm Object detectionterm is the process of detecting instances of semantic objectsterm of a certain class (such as humans, airplanes, or birds) in digital images and video ( Figure 4). A common approach for object detectionterm frameworks includes the creation of a large set of candidate windowsterm that are in the sequel classified using CNNterm features. For example, the method described in [32] employs selective searchterm [60] to derive object proposalsterm, extracts CNNterm features for each proposal, and then feeds the features to an SVM classifierterm to decide whether the windows include the object or not. A large number of works is based on the concept of Regions with CNNterm features proposed in [32]. Approaches following the Regions with CNNterm paradigm usually have good detection accuracies (e.g., [61, 62]); however, there is a significant number of methods trying to further improve the performance of Regions with CNNterm approaches, some of which succeed in finding approximate object positions but often cannot precisely determine the exact position of the object [63]. To this end, such methods often follow a joint object detectionterm—semantic segmentationterm approach [64– 66], usually attaining good results.

Figure 4.

!Figure 4 Object detectionterm results comparison from [66]. (a) Ground truthterm; (b) bounding boxes obtained with [32]; (c) bounding boxes obtained with [66].

A vast majority of works on object detectionterm using deep learningterm apply a variation of CNNsterm, for example, [8, 67, 68] (in which a new def-poolingterm layer and new learning strategy are proposed), [9] (weakly supervised cascaded CNNsterm), and [69] (subcategory-aware CNNsterm). However, there does exist a relatively small number of object detectionterm attempts using other deep models. For example, [70] proposes a coarse object locating method based on a saliency mechanism in conjunction with a DBN for object detectionterm in remote sensing images; [71] presents a new DBN for 3D object recognition, in which the top-level model is a third-order Boltzmann machineterm, trained using a hybrid algorithm that combines both generative and discriminative gradients; [72] employs a fused deep learningterm approach, while [73] explores the representationterm capabilities of a deep model in a semisupervised paradigm. Finally, [74] leverages stacked autoencodersterm for multiple organ detection in medical images, while [75] exploits saliency-guided stacked autoencodersterm for video-based salient object detectionterm.

3.2. Face Recognition Face recognition is one of the hottest computer vision applications with great commercial interest as well. A variety of face recognition systems based on the extraction of handcrafted featuresterm have been proposed [76– 79]; in such cases, a feature extractor extracts features from an aligned face to obtain a low-dimensional representationterm, based on which a classifier makes predictions. CNNsterm brought about a change in the face recognition field, thanks to their feature learningterm and transformation invariance properties. The first work employing CNNsterm for face recognition was [80]; today light CNNsterm [81] and VGG Face Descriptor [82] are among the state of the art. In [44] a Convolutional DBN achieved a great performance in face verification.

Moreover, Google's FaceNetterm [83] and Facebook's DeepFaceterm [84] are both based on CNNsterm. DeepFaceterm [84] models a face in 3D and aligns it to appear as a frontal face. Then, the normalized input is fed to a single convolution-pooling-convolution filter, followed by three locally connected layers and two fully connected layersterm used to make final predictions. Although DeepFaceterm attains great performance rates, its representationterm is not easy to interpret because the faces of the same person are not necessarily clustered during the training process. On the other hand, FaceNetterm defines a triplet loss functionterm on the representationterm, which makes the training process learn to cluster the face representationterm of the same person. Furthermore, CNNsterm constitute the core of OpenFaceterm [85], an open-source face recognition tool, which is of comparable (albeit a little lower) accuracy, is open-source, and is suitable for mobile computingterm, because of its smaller size and fast execution time.

3.3. Action and Activity Recognition Human action and activity recognition is a research issue that has received a lot of attention from researchers [86, 87]. Many works on human activity recognition based on deep learningterm techniques have been proposed in the literature in the last few years [88]. In [89] deep learningterm was used for complex event detection and recognition in video sequences: first, saliency maps were used for detecting and localizing events, and then deep learningterm was applied to the pretrained features for identifying the most important frames that correspond to the underlying event. In [90] the authors successfully employ a CNN-based approach for activity recognition in beach volleyball, similarly to the approach of [91] for event classification from large-scale video datasets; in [92], a CNNterm model is used for activity recognition based on smartphone sensor data. The authors of [12] incorporate a radius–margin bound as a regularization term into the deep CNNterm model, which effectively improves the generalizationterm performance of the CNNterm for activity classification. In [13], the authors scrutinize the applicability of CNNterm as joint feature extraction and classification model for fine-grained activities; they find that due to the challenges of large intraclass variances, small interclass variances, and limited training samples per activity, an approach that directly uses deep features learned from ImageNet in an SVM classifierterm is preferable.

Driven by the adaptability of the models and by the availability of a variety of different sensors, an increasingly popular strategy for human activity recognition consists in fusing multimodal features and/or data. In [93], the authors mixed appearance and motion features for recognizing group activities in crowded scenes collected from the web. For the combination of the different modalities, the authors applied multitask deep learningterm. The work of [94] explores combination of heterogeneous features for complex event recognition. The problem is viewed as two different tasks: first, the most informative features for recognizing events are estimated, and then the different features are combined using an AND/OR graph structure. There is also a number of works combining more than one type of model, apart from several data modalities. In [95], the authors propose a multimodal multistream deep learningterm framework to tackle the egocentric activity recognition problem, using both the video and sensor data and employing a dual CNNsterm and Long Short-Term Memory architecture. Multimodal fusion with a combined CNNterm and LSTM architecture is also proposed in [96]. Finally, [97] uses DBNs for activity recognition using input video sequences that also include depth information.

3.4. Human Pose Estimation The goal of human pose estimation is to determine the position of human joints from images, image sequences, depth images, or skeleton data as provided by motion capturing hardware [98]. Human pose estimation is a very challenging task owing to the vast range of human silhouettes and appearances, difficult illumination, and cluttered background. Before the era of deep learningterm, pose estimation was based on detection of body parts, for example, through pictorial structures [99].

Moving on to deep learningterm methods in human pose estimation, we can group them into holistic and part-based methods, depending on the way the input images are processed. The holistic processing methods tend to accomplish their task in a global fashion and do not explicitly define a model for each individual part and their spatial relationships. DeepPose [14] is a holistic model that formulates the human pose estimation method as a joint regression problem and does not explicitly define the graphical model or part detectors for the human pose estimation. Nevertheless, holistic-based methods tend to be plagued by inaccuracy in the high-precision region due to the difficulty in learning direct regression of complex pose vectors from images.

On the other hand, the part-based processing methods focus on detecting the human body parts individually, followed by a graphic model to incorporate the spatial information. In [15], the authors, instead of training the network using the whole image, use the local part patches and background patches to train a CNNterm, in order to learn conditional probabilities of the part presence and spatial relationships. In [100] the approach trains multiple smaller CNNsterm to perform independent binary body-part classification, followed with a higher-level weak spatial model to remove strong outliers and to enforce global pose consistency. Finally, in [101], a multiresolution CNNterm is designed to perform heat-map likelihood regression for each body part, followed with an implicit graphic model to further promote joint consistency.

3.5. Datasets The applicability of deep learningterm approaches has been evaluated on numerous datasets, whose content varied greatly, according the application scenario. Regardless of the investigated case, the main application domain is (natural) images. A brief description of utilized datasets (traditional and new ones) for benchmarking purposes is provided below.

_(1) Grayscale Images._ The most used grayscale images dataset is MNIST [20] and its variations, that is, NIST and perturbed NIST. The application scenario is the recognition of handwritten digits.

_(2) RGB Natural Images._ Caltech RGB image datasets [102], for example, Caltech 101/Caltech 256 and the Caltech Silhouettes, contain pictures of objects belonging to 101/256 categoriesterm. CIFAR datasets [103] consist of thousands of 32 × 32 color images in various classes. COIL datasets [104] consist of different objects imaged at every angle in a 360 rotation.

_(3) Hyperspectral Images._ SCIEN hyperspectral image data [105] and AVIRIS sensor based datasets [106], for example, contain hyperspectral images.

_(4) Facial Characteristics Images._ Adience benchmark dataset [107] can be used for facial attributes identification, that is, age and gender, from images of faces. Face recognition in unconstrained environments [108] is another commonly used dataset.

_(5) Medical Images._ Chest X-ray dataset [109] comprises 112120 frontal-view X-ray images of 30805 unique patients with the text-mined fourteen disease image labelsterm (where each image can have multilabels). Lymph Node Detection and Segmentation datasets [110] consist of Computed Tomography images of the mediastinum and abdomen.

_(6) Video Streams._ The WR datasets [111, 112] can be used for video-based activity recognition in assembly lines [113], containing sequences of 7 categoriesterm of industrial tasks. YouTube-8M [114] is a dataset of 8 million YouTube video URLs, along with video-level labelsterm from a diverse set of 4800 Knowledge Graph entities.

4. Conclusions The surge of deep learningterm over the last years is to a great extent due to the strides it has enabled in the field of computer vision. The three key categoriesterm of deep learningterm for computer vision that have been reviewed in this paper, namely, CNNsterm, the “Boltzmann familyterm” including DBNs and DBMs, and SdAsterm, have been employed to achieve significant performance rates in a variety of visual understanding tasks, such as object detectionterm, face recognition, action and activity recognition, human pose estimation, image retrieval, and semantic segmentationterm. However, each category has distinct advantages and disadvantages. CNNsterm have the unique capability of feature learningterm, that is, of automatically learning features based on the given dataset. CNNsterm are also invariant to transformationsterm, which is a great asset for certain computer vision applications. On the other hand, they heavily rely on the existence of labelled dataterm·var, in contrast to DBNs/DBMs and SdAsterm, which can work in an unsupervised fashion. Of the models investigated, both CNNsterm and DBNs/DBMs are computationally demanding when it comes to training, whereas SdAsterm can be trained in real time under certain circumstances.

As a closing note, in spite of the promising—in some cases impressive—results that have been documented in the literature, significant challenges do remain, especially as far as the theoretical groundwork that would clearly explain the ways to define the optimal selection of model typeterm·var and structure for a given task or to profoundly comprehend the reasons for which a specific architecture or algorithm is effective in a given task or not. These are among the most important issues that will continue to attract the interest of the machine learning research community in the years to come.

Acknowledgments This research is implemented through IKY scholarships programme and cofinanced by the European Union (European Social Fund—ESF) and Greek national funds through the action titled “Reinforcement of Postdoctoral Researchers,” in the framework of the Operational Programme “Human Resources Development Program, Education and Lifelong Learning” of the National Strategic Reference Framework (NSRF) 2014–2020.

Conflicts of Interest The authors declare that there are no conflicts of interest regarding the publication of this paper.