Computer-Aided Diagnosis with Deep Learning Architecture: Applications to Breast Lesions in US Images and Pulmonary Nodules in CT Scans — TEI-tagged view

Jie-Zhi ChengpersName, Dong NipersName, Yi-Hong ChoupersName, Jing QinpersName, Chui-Mei TiupersName, Yeun-Chung ChangpersName, Chiun-Sheng HuangpersName, Dinggang ShenpersName, Chung-Ming ChenpersName (2016date). “Computer-Aided Diagnosis with Deep Learning Architecture: Applications to Breast Lesions in US Images and Pulmonary Nodules in CT Scans.” Scientific Reportstitle·j.

Object of study — deep-learning-based computer-aided diagnosisterm·mth of benign and malignant lesions

Research question — whether a deep architecture can replace hand-designed features in CADxterm·mth

Contribution — an SDAE-based CADxterm·mth scheme validated on breast ultrasound and lung CT

Full text — tagged

Computer-Aided Diagnosisterm·mth with Deep Learningterm·mth Architecture: Applications to Breast Lesionsterm·key in US Images and Pulmonary Nodulesterm·key in CT Scans Abstract This paper performs a comprehensive study on the deep-learning-based computer-aided diagnosisterm·mth (CADxterm·mth) for the differential diagnosis of benign and malignant nodules/lesions by avoiding the potential errors caused by inaccurate image processing results (e.g., boundary segmentation), as well as the classification bias resulting from a less robust feature set, as involved in most conventional CADxterm·mth algorithms. Specifically, the stacked denoising auto-encoderterm·mth (SDAEterm·mth) is exploited on the two CADxterm·mth applications for the differentiation of breast ultrasound lesions and lung CT nodules. The SDAEterm·mth architecture is well equipped with the automatic feature exploration mechanism and noise tolerance advantage, and hence may be suitable to deal with the intrinsically noisy property of medical image data from various imaging modalities. To show the outperformance of SDAE-based CADxterm·mth over the conventional scheme, two latest conventional CADxterm·mth algorithms are implemented for comparison. 10 times of 10-fold cross-validationsterm·key are conducted to illustrate the efficacy of the SDAE-based CADxterm·mth algorithm. The experimental results show the significant performance boost by the SDAE-based CADxterm·mth algorithm over the two conventional methods, suggesting that deep learningterm·mth techniques can potentially change the design paradigm of the CADxterm·mth systems without the need of explicit design and selection of problem-oriented features.

Computer-aided diagnosisterm·mth (CADxterm·mth) is a computerized procedure to provide a second objective opinion for the assistance of medical image interpretation and diagnosis 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 . One of the major CADxterm·mth applications is the differentiation of malignancy/benignancy for tumors/lesions 3 , 4 , 11 , 12 , 13 , 14 , 15 . Several studies have suggested that the incorporation of the CADxterm·mth system into the diagnostic process can improve the performance of image diagnosis by decreasing inter-observer variationterm·dom 16 , 17 and providing the quantitative support for the clinical decision like biopsy recommendations 5 , etc. Specifically, the CADxterm·mth systems were shown to be effective to assist the diagnostic workup for the reduction of unnecessary false-positive biopsies 6 and thoracotomy 10 .

To achieve malignancy identification, the conventional design of CADxterm·mth is often composed of three main steps: feature extractionterm·key 4 , 6 , 7 , 12 , 13 , 14 , 15 , 18 , feature selection 18 , 19 , 20 , 21 , and classification. These three steps need to be well-addressed separately and then integrated together for the overall CADxterm·mth performance tuning. In most previous works, engineering on effective feature extractionterm·key step for each specific problem was regarded as the one of the most important issues 4 , 6 , 7 , 12 , 13 , 14 , 15 , 18 . Extraction of discriminative features could potentially ease the latter steps of feature selection and classification. Nevertheless, the engineering of effective features is problem-oriented and still needs assistance from the latter steps of feature selection and feature integration by classifier, to achieve accurate lesion/nodule differentiation. In general, the diagnostic image features can be categorized into morphological and textural featuresterm·dom.

The extraction of effective features is per se a complicated task that requires a series of image processing steps. These image processing steps involve 1) image segmentationterm·key 4 , 6 , 14 , 15 , 22 , 23 , 24 for the morphological feature computing 4 , 6 , 7 , 14 , 15 , 22 , which unfortunately remains quite difficult to address 25 , and 2) image decomposition 12 , 19 followed with statistical summarizations and presentations 12 , 19 , 20 , 26 for the textural feature calculation 27 . Accordingly, the extraction of useful features highly depends on the quality of each intermediate result in the image processing steps 4 , 6 , 14 , 22 , 27 , which often needs many passes of trial-and-error design to find satisfactory intermediate results. Meanwhile, the computerized image segmentationterm·key results often require various case-by-case user interventions, e.g., parameter adjustment 6 , 28 , manual refinement 6 , 9 , 28 , 29 , 30 , 31 , solution selection 4 , 23 , etc., to improve the contour correctness 4 , 6 , 13 , 15 , 23 . However, with the user’s interventionterm·dom on the segmentation results, the differential diagnostic outcome from the CADxterm·mth system will be biased and the objectiveness will no longer hold. In summary, the design and tuning of the overall performance of the conventional CADxterm·mth framework tends to be very arduous, as many image processing issues need to be well resolved.

Deep learningterm·mth for CADxterm·mth Recently, the deep learningterm·mth techniques have been introduced to the medical image analysis domain with promising results on various applications, like the computerized prognosis for Alzheimer’s disease and mild cognitive impairment 32 , organ segmentations 33 and detection 34 , ultrasound standard plane selection 35 , etc., on 3D or 4D image data, etc. In the context of CAD, most works focused on the problem of abnormality detection (CADe) 36 , 37 , 38 . For the problem of CADxterm·mth, a specific convolutional neural network model, OverFeat 39 , was employed in the work 40 to classify the specific type of peri-fissural nodules with the ensemble fashion in AUC performance around 0.86. In this study, we further exploit the deep learningterm·mth model of the stacked denoising autoencoderterm·mth (SDAEterm·mth) 41 for the differentiation of distinctive types of lesions and nodules depicted with different imaging modalities.

The deep learningterm·mth techniques could potentially change the design paradigm of the CADxterm·mth framework for several advantages over the old conventional frameworks. The advantages can be three-fold. First , deep learningterm·mth can directly uncover features from the training data, and hence the effort of explicit elaboration on feature extractionterm·key can be significantly alleviated. The neuron-crafted features may compensate and even surpass the discriminative power of the conventional feature extractionterm·key methods. Second , feature interaction and hierarchy can be exploited jointly within the intrinsic deep architecture of a neural network. Consequently, the feature selection process will be significantly simplified. Third , the three steps of feature extractionterm·key, selection and supervised classification can be realized within the optimization of the same deep architecture. With such a design, the performance can be tuned more easily in a systematic fashion.

The SDAEterm·mth is a denosing version of the stacked autoencoderterm·mth (SAEterm·mth) 42 . The SAEterm·mth/SDAEterm·mth architecture can automatically discover the diverse representative patterns from the data with the intrinsic data reconstruction mechanism. Accordingly, the SAEterm·mth/SDAEterm·mth architecture can potentially address the issues of high variation in either shape or appearance of lesions/tumors. Meanwhile, with the advantages of automatic feature extractionterm·key mechanism and noise tolerance, the SDAE-based CADxterm·mth scheme can circumvent the potential inaccurate image processing results that could lead to unreliable features in the conventional CADxterm·mth framework. To show the wide scope of applicability, the SDAE-based CADxterm·mth model is applied to the differential diagnosis of breast lesionsterm·key in ultrasound (US) images and pulmonary nodulesterm·key in CT images. Two CADxterm·mth methods, i.e., RANK 19 and CURVE 12 that were specifically developed for the breast and lung applications, respectively, and clinical morphological features (MORPH) are implemented for performance comparison. The MORPH features are computed from either experts’ drawings or the computer-generated boundaries by level set 43 (DRLSE) and grow-cut 44 (GC) methods. Meanwhile, the effect of feature combination of RANK and CURVE with the MORPH is also explored. The experimental results show that the deep-learning-based CADxterm·mth can achieve better differentiation performance than the comparison methods across different modalities and diseases. To illustrate the challenges of the two CADxterm·mth applications, Fig. 1 lists several cases of breast lesionsterm·key and pulmonary nodulesterm·key depicted in US and CT images, respectively. The cases demonstrated in Fig. 1 may not be easy to differentiate for a person without a medical background and even for a junior medical doctor.

Exhibition of breast lesionsterm·key and lung nodules in US and CT images.

Experiment Results Experiment setup We compare the performances of the SDAE-based CADxterm·mth and the two compared algorithms with the six assessment metrics: 1) area under receiver operating characteristic curve (AUC), 2) accuracy (ACCterm·var), 3) sensitivity (SENS), 4) specificity (SPEC), 5) positive predictive value (PPV), and 6) negative predictive value (NPV). For the robustness to data dependence, we perform 10 times of 10-fold cross validations for the SDAEterm·mth and other baseline algorithms on both the breast US and lung CT datasets. In each cross validationterm·key, the partition of training and testing data is randomly determined. For the illustration of the performance dependency of the conventional CADxterm·mth frameworks on the specific problems, all baseline algorithms are applied to all datasets.

Two slice selection strategies, i.e., SINGLE and ALL, are implemented for the lung CT CADxterm·mth problem. For the SINGLE strategy, there are totally 1400 training and testing ROI samples. Each ROI sample stands for a distinctive nodule. In the ALL strategy, member slices of each nodule are involved in the training and testing. Specifically, the member slices of all 1360 training nodules in the 10-fold cross validationterm·key scheme are randomly permutated as the training samples. For the 140 testing nodules, the final differentiation of a testing nodule is determined with the majority voting from the member slices. In the ALL strategy, there are 10133 involved slices. Because slice thickness variation is very high (0.6–5 mm), fully 3D image features for the lung CADxterm·mth may not be suitable.

The distinctiveness of each set of the 10-fold partition is assured in the random selection process. In each 10-fold cross validationterm·key, all algorithms share the same sample partition setting on each fold for fair comparison. For both lung and breast cases, we use the same SDAEterm·mth architecture with 2 hidden layers, each containing 200 and 100 neurons, respectively. The true positive of the lung CT and breast US data is defined as the malignant class.

Results To illustrate the patterns automatically discovered by SDAEterm·mth, Fig. 2 lists several patterns of pulmonary nodulesterm·key in CT images and breast lesionsterm·key in US images, respectively, at the pre-training step. In the subfigures (a) and (c), the diverse appearance patterns of various nodules/lesions summarized at the first hidden layer are shown. The patterns at the second hidden layers that automatically encode the intertwined and hierarchical relations to the first hidden layer patterns are demonstrated in the subfigures (b) and (d).

Examples of constructed patterns in the first and second hidden layers at the pre-training step: ( a,b ) patterns of the first and second hidden layers for pulmonary nodulesterm·key; ( c,d ) patterns of the first and second hidden layers for breast lesionsterm·key. SDAEterm·mth architecture with two hidden layers is used in this study for the differentiation of pulmonary nodulesterm·key and breast lesionsterm·key. It is worth noting that the patterns of the second hidden layers are constructed as the weighted sums from all patterns in the first layer. In the reconstruction, the first layer neurons are simply all assumed activated. The neuron activation can be more complicated with the feed-in of real image data. In ( b , d ) the example patterns enclosed by the yellow rectangles hold the positive weightings to the RN nodule and benignant lesion classes in the supervised training step, whereas the patterns in blue regions are connected to the RM nodule and malignant lesion classes with positive weightings. It can be observed from ( b,d ) that the second hidden layer patterns appear fuzzier due to the effect of weighted sum. All patterns are normalized for clearer presentation.

Tables 1 and 2 summarize the statistics of six performance metrics for all algorithms on the lung CT and breast US datasets, respectively. To visualize the performance agreement between each pair of the comparing algorithms (SDAEterm·mth, CURVE, RANK, and MORPH) over all 100 folds, Bland and Altman plots 45 of the ACCterm·var metrics on the lung and breast datasets are drawn in Figs 3 and 4 , respectively. In a Bland and Altman plot, the horizontal axis stands for the mean value of the assessment metrics between the two comparing methods, whereas the difference values of the two methods are coordinated in the vertical axis. The red line in the Bland and Altman plot suggests the mean difference of the assessment values, whereas the top and bottom blue lines delimit the ±1.96 standard deviation lines of the difference values.

Performance summary of the SDAEterm·mth, CURVE and RANK algorithms on the lung CT dataset.

Performance summary of the SDAEterm·mth, CURVE and RANK algorithms along with the clinical MORPH features on the breast US dataset.

ACCterm·var Bland and Altman plots for six algorithm comparing pairs of “SDAE-CURVE”, “SDAE-RANK” , “SDAE-MORPH”, “CURVE-RANK”, “CURVE-MORPH”, and “RANK-MORPH” on the lung CT dataset. The comparing pairs with ending tag “ALL” are the results with the strategy of using all member slices of a nodule for the training and testing of the three algorithms. The pairs with tag “SINGLE” compare the computerized results with the slice selection strategy of using middle slice.

ACCterm·var Bland and Altman plots for performance comparison of the pairs “SDAE-CURVE”, “SDAE-RANK”, “SDAE-MORPH”, “CURVE-RANK”, “CURVE-MORPH”, and “RANK-MORPH” on the breast dataset.

As can be observed from Figs 3 and 4 and Tables 1 and 2 , the SDAEterm·mth algorithm performs better than the other three algorithms. For the lung nodule differentiation, the SDAEterm·mth algorithm at least achieves 0.15 and 0.1 higher ACCterm·var values than the CURVE, RANK and MORPH methods with respect to the ALL and SINGLE strategies in average. The CURVE algorithm achieves better than the RANK algorithm with the mean ACCterm·var differences for the ALL and SINGLE strategies, respectively. The mean ACCterm·var differences between the CURVE and MORPH are near to 0 with both strategies. The mean ACCterm·var differences between the RANK and MORPH are all lower than 0, suggesting that the RANK is averagely not able to outperform MORPH. For the breast application, the SDAEterm·mth can achieve at least higher ACCterm·var values 0.04 than the RANK, CURVE, and MORPH in average. The CURVE perform worse than RANK with the mean differences of the ACCterm·var values around −0.02, but better than the MORPH with mean differences w.r.t. ACCterm·var values around 0.05. The RANK on the other hand can perform the classification of breast lesionsterm·key better than using the clinical MORPH features with mean differences more than 0.05 on the ACCterm·var metrics. To further illustrate the ACCterm·var and AUC performance distributions of SDAEterm·mth, CURVE, RANK and MORPH, the corresponding box-plots on the lung and breast datasets are also shown in Fig. 5 . Meanwhile, it can be found from Tables 1 and 2 that the morphological features from the DRLSE and GC methods are not very helpful for the boosting of performance. It is because that satisfactory lesion/nodule boundaries are quite difficult to obtain without user intervention. The discussion about the image segmentationterm·key can be found in the second section of the supplementary .

Box plots for performance for the lung and breast datasets with respect to the ACCterm·var and AUC metrics.

Referring to the Fig. 5 , it can be found that performance distributions appear substantially different in most cases, except the pairs “CURVE-MORPH-ALL” and “CURVE-MORPH-SINGLE”. To further investigate the significance of differences for these two pairs, the two sample t-test is applied. The ACCterm·var p -values of the pairs “CURVE-MORPH-ALL” and “CURVE-MORPH-SINGLE” are 0.69 and 0.32 respectively, whereas the AUC p -values w.r.t. each pair are 0.0001 and 0.088. It is thus suggested that ACCterm·var performance differences are not significant between the methods CURVE and MORPH on both strategy and the AUC performance between CURVE and MORPH is not significantly different on the SINGLE strategy.

Discussion and Conclusion Remarks The experimental results show that the SDAEterm·mth algorithm outperforms the conventional CADxterm·mth algorithms on both applications. Specifically, as can be observed in Table 1 , Figs 3 and 5 , the SDAEterm·mth algorithm can differentiate lung nodules significantly better than the two baseline algorithms and the simple clinical MORPH features do in terms of all six assessment metrics. Particularly, it can also be found that the SDAEterm·mth algorithm can achieve a much better differentiation performance with the ALL strategy. Therefore, the involvement of more member slices as training data can be very helpful to boost the performance of a deep learningterm·mth CADxterm·mth scheme, even if some of the training slices only partially depict the nodules of interest. It may be because the inclusion of more nodule member slices as training data can provide richer image contexts for the SDAEterm·mth model to augment the differentiation capability. For breast lesionterm·key classification, the SDAEterm·mth algorithm still outperforms the two texture-based algorithms as shown in Table 2 , Figs 4 and 5 . It is also worth noting that the data size of lung nodules is almost three times of the breast data size.

Referring to Tables 1 and 2 , the CURVE and RANK algorithms perform better at their original problem than the other conventional CADxterm·mth methods. For the performance comparison on the two slice selection strategies of the lung CADxterm·mth problem in the Table 1 , Figs 3 , 4 and 5 , the involvement of all slices for the CURVE and RANK algorithms doesn’t help much. In particular, the RANK algorithm with ALL strategy performs slightly worse than the RANK model trained with SINGLE strategy. This may be because the features used in the RANK algorithm are not only ineffective for the nodule differentiation but also confusing with rich image contexts presented in various member slices of nodules for the classifier. Accordingly, it can be suggested that features for the conventional CADxterm·mth framework may need to be specifically designed for each problem at hand. Thus far, there is barely general features that can be effective for all kinds of diseases and modalities.

Referring to Table 1 and relevant Figs 3 and 5 , it can also be found that the nodule classification performance of the CURVE algorithm with either the ALL or SINGLE strategy is not significantly different to the performance attained by the MORPH features from experts’ drawings in ACCterm·var metrics. Therefore, it suggests that the elaboration of feature extractionterm·key for a CADxterm·mth may sometimes not significantly outperform the simple morphological features commonly used in clinical practice. Accordingly, the design of effective features for a specific CADxterm·mth problem can be an arduous problem. On the other hand, the SDAE-based CADxterm·mth algorithm can do well on both problems with the same architecture setting and doesn’t require the explicit elaboration on feature extractionterm·key and feature relation establishment. Meanwhile, the SAEterm·mth/SDAEterm·mth models are equipped with a visualization mechanism of the learnt patterns encoded in the neurons at the unsupervised phase, as illustrated in Fig. 2 , to facilitate the developer and even medical doctors to understand the machine learning model easier. These advantages and the natural end-to-end training manner of the deep learningterm·mth methods can potentially benefit the inter-discipline application like CADxterm·mth to enable people with engineering and clinical background work closer and come up with more effective solutions.

Although the SDAEterm·mth algorithm cannot attain a performance as good as the earlier morphology-based framework 4 on the breast dataset, it doesn’t need the image segmentationterm·key process to obtain a nodule/lesion boundary. The work 4 requires a user to manually give the definition of a ROI and select one segmentation result from the five lesion boundary proposals generated by the segmentation algorithm. So far, there is no algorithm that can guarantee perfect automatic segmentation results for any object of interest and imaging modality. Note that image segmentationterm·key for pulmonary nodulesterm·key in CT scans may sometimes include the nearby tissues, e.g., vessel and airway, into the segmentation results due to similar intensity distribution, whereas the accuracy of breast lesionterm·key segmentation in US images can be easily degraded with the presence of serious shadowing effect. The effect of combination of morphological and textural featuresterm·dom on the conventional CADxterm·mth has also been shown in the Tables 1 and 2 . To avoid the user intervention on the segmentation process, the initialization and parameters of the DRLSE and GC methods are fixed. As it can be found in the Tables 1 and 2 , the MORPH features from contours with the DRLSE and GC methods are not very helpful. It might be because the image segmentationterm·key results are imperfect and therefore the derived MORPH features are less reliable. Accordingly, this may reflect the conclusion of the work 4 that the quality of image segmentationterm·key result matters for the CADxterm·mth scheme based on the morphological features. For a better quality of morphological feature computing, the computed lesion/nodule boundaries usually require manual intervention, including model initialization 13 , 15 , parameter adjustment 6 , 13 , 15 , boundary refinement 6 and selection 4 , 23 , to improve the segmentation results. Nevertheless, the user intervention will bias the computerized differentiation results because the process like boundary refinement and selection involves subjective judgment on the lesion/nodule. Accordingly, the pure objectiveness of computerized diagnosis may no longer hold. In contrast, the SDAEterm·mth algorithm can circumvent this complex image segmentationterm·key step, but can still achieve an objective and satisfactory performance without any user intervention. The SDAEterm·mth algorithm can potentially attain an even better performance if the data size of breast US is larger.

One of the major research lines on textural feature computation is based on the statistics of the Grey Level Co-occurrence Matrixterm·mth (GLCMterm·mth). To boost the differentiation performance, most promising texture-based CADxterm·mth methods, e.g., the works 12 , 19 , decompose the raw image into independent components with a bank of filters, and then compute the GLCMterm·mth from each component in multiple scales and orientations. With each GLCMterm·mth, quite a few texture features can be further derived, e.g., 12 in the work 19 and 14 in the work 12 . In such a framework, the intertwined combination of parameter settings on the filter bank, scale and orientation settings of the GLCMterm·mth, and the number of derived features from each GLCMterm·mth could lead to a large number of features. In this case, the feature selection step has to be applied 19 , 20 to find a subset of the most useful features. Since the feature selection in itself is very tedious and needs to repeat the training process for many times, the whole CADxterm·mth learning process will be rendered into a very long cycle. On the other hand, the training process of the deep architecture is relatively simple, but can achieve a better performance, see Fig. 1 for comparison.

In summary, the deep learningterm·mth architecture holds the advantages of 1) automatic discovery of object features, 2) automatic exploration of feature hierarchy and interaction, 3) relatively simple end-to-end training process, and 4) systematic performance tuning. Although deep learningterm·mth has been successfully applied to many image- and acoustic-related problems 46 , it is less explored in the context of CADxterm·mth. In this study, the efficacy of a deep learning-based CADxterm·mth scheme has been corroborated with extensive experiments. Comparing to the conventional CADxterm·mth framework, the training procedure of deep learningterm·mth is relatively simple but effective, without the need of explicit design of the problem-oriented features. To our best knowledge, this is the first deep-learning-based CADxterm·mth study that demonstrates outperformance over the state-of-the-art CADxterm·mth algorithms across different imaging modalities and diseases. This work may, thus, be referential for further deep learningterm·mth studies on other CADxterm·mth problems and the analysis of higher dimensional image data and images from heterogeneous imaging modalities for more accurate and reliable computerized diagnostic support.

Methods Datasets The methods were carried out by in “accordance” with the approved guidelines. All experimental protocols were approved by Taipei Veterans General Hospital and the Lung Image Database Consortium (LIDC) 47 , 48 . Informed consent was obtained from all subjects. The breast US images adopted here were acquired at Taipei Veterans General Hospital, Taipei, Taiwan 4 with proper IRB approvals. Totally, 520 breast sonograms were scanned from 520 patients. The data involves 275 benign and 245 malignant lesions. All breast lesionsterm·key were histopathologically proved by means of biopsy, mastectomy, etc. More details about the involved US breast lesionsterm·key can be found in the work 4 . The Lung Image Database Consortium (LIDC) 47 , 48 retrospectively collected lung CT image data from 1,010 patients (with appropriate local IRB approvals) at seven academic institutions with various CT machines in the United States. The slice thicknesses of the CT scans ranged diversely from 0.6 mm to 5 mm. 12 radiologists from 5 sites were involved in the annotation process. Nodules with diameters larger than 3 mm were further annotated with rating of malignancy by the radiologists. In this study, we randomly select 700 malignant and 700 benign nodules as the experimental dataset for data balanced. Specifically, the benign set includes the annotated nodules with scores of 1 and 2 in the “likelihood of malignancy” rating, whereas nodules scored as 4 and 5 are included in the malignant set.

SDAE-based CADxterm·mth framework The training of a SDAE-based CADxterm·mth framework can be realized in two steps: the pre-training and supervised training steps. Figure 6 illustrates the flowchart of the SDAE-based CADxterm·mth framework. To facilitate the training and differentiation tasks on the SDAEterm·mth architecture, the image ROIs are resized into smaller patches of 28 × 28, where all pixels in each patch are treated as the input neurons. To preserve the original information, the resized scale factors of the two ROI dimensions and the aspect ratios of the original ROIs are later added into the input layer in the supervised training step.

Flow-chart of our deep-learning-based CADxterm·mth training framework. The pixels of resized ROIs are fed into the network architecture at the pre-training step. The pre-trained network is then refined with the supervised training by adding three neurons carrying aspect ratio of the original ROI and also the resizing factors at the input layer. The final identification result can be made with the softmax classification.

At the pre-training step, the input ROIs are degraded with random corruption and then the noise-tolerance representative patterns for the lesion/tumor can be further identified with the network architecture. Meanwhile, higher semantic levels of the representative patterns and the composite/interaction relations from the low semantic levels can be further sought by constructing autoencoders layer by layer, see Fig. 2 . The constructed architecture at the pre-training step can be served as reliable network initialization for the latter supervised training.

The supervised training of the SDAE-based CADxterm·mth framework performs the fine-tuning on the network architecture to yield the desirable discriminative performance. The scaling factors in the x and y dimensions and the aspect ratio are treated as three individual neurons as new inputs, whereas two extra output neurons of benign and malignant classes are also added on the top of the network for the supervised training, see Fig. 6 . The modified network is then equipped with the initialization from the pre-trained architecture, three new input neurons, and the two outputs. Afterward, the supervised training is performed with the conventional back-propagation for the fine-tuning of the whole network. With such network architecture and two steps of training, the feature extractionterm·key and selection can be systematically and jointly realized with less need of explicit and ad-hoc elaborations. The technical explanation about the SDAEterm·mth model can be found in the first section of the supplementary .

Conventional CADxterm·mth Algorithms for Comparison Two texture-based CADxterm·mth schemes for breast US 19 and lung CT 12 are implemented for comparison. Note that the morphological features are not considered here, since it needs an image segmentationterm·key process, which is very hard and often requires manual refinement on the automated segmentation results. On the other hand, textural features can be computed directly from the ROI. Accordingly the texture-based CADxterm·mth schemes are relatively automatic and objective, and hence more suitable to serve as the comparison baselines. The implemented breast US CADxterm·mth algorithm 19 , called RANK, is the state-of-the-art texture-based method for breast lesionsterm·key classification. The RANK algorithm carries out the ranklet transform, which was shown to be robust to speckle noise, and to decompose the US image data into several independent image components. The GLCM-based texture features are computed from the ranklet components, whereas feature selection is conducted with the bootstrap method 19 , 20 to find an effective subset of features for classification. We adopted the reported best selected features 19 as the input to SVM to reach the final classification result for each US ROI.

For lung nodule differentiation, the latest lung CT CADxterm·mth algorithm 12 , denoted as CURVE, is implemented. The CURVE algorithm applies the curvelet transform to decompose the raw image data into several sub-band components and then computes the GLCMterm·mth on each component to derive texture features. To reduce feature dimensionality, the CURVE algorithm averages the textural features over all angle parameter settings of each GLCMterm·mth for the later classification step with SVM. It is worth noting that the RANK and CURVE algorithms share some common steps such as image decomposition with transform techniques, computing of GLCMterm·mth features, and SVM-based classification, though their feature selection and integration processes are different.

Classification with Clinical Features To further compare the effectiveness of the SDAE-based CADxterm·mth algorithm, the simple clinical (MORPH) features, which are commonly used as preliminary reference for differential diagnosis in clinical practice, are also implemented. Specifically, the size and diameter features of the breast and nodules are computed from the US images and CT scans, respectively. For pulmonary nodulesterm·key, we compute the nodule volume, the maximum major diameter of the approximate ellipsoid of the nodule contour over all member slices, and the maximum area of the nodule over all member slices as the quantitative features. It is worth noting that the computed nodule volume can only approximate the real value due to the slice thickness effect and segmentation errors. The three quantitative features are further classified with SVM for the benignancy/malignancy classification. Similarly, we also compute the simple morphological features for the US breast lesionsterm·key for comparison. The specific simple morphological features are the lesion area and the length of the major axis of the approximated ellipsoid of each breast lesionterm·key. These two features are further served as input of a SVM for the breast lesionterm·key classification. All the MORPH features are derived from manual outlines from experienced medical doctors and image segmentationterm·key results from the DRLSE level set 43 and grow-cut 44 methods.

Training and Testing of CADxterm·mth Algorithms Since the US breast data are 2D images, the training and testing of CADxterm·mth algorithms is relatively simple. On the other hand, the image resolution of lung CT scan is anisotropic between the z (slice thickness) and x - y directions. The slice thickness varies widely. As the CT scan is sliced into 2D transversal views, the computation of a 3D feature on the z direction is less reliable than on the other x and y directions. In such a case, there will be an issue of slice selection for the representation of a nodule in the design of the CADxterm·mth. For a pulmonary noduleterm·key, because in some member CT slices only small portions of the nodule are depicted, not every member CT slice of a nodule can be useful. Accordingly, the efficacy of the involvement of all CT slices for the lung CADxterm·mth is unknown. Two slice selection, i.e., SINGLE and ALL, strategies are implemented in this study for the lung CT CADxterm·mth problem. In SINGLE strategy, the middle slice of the nodule is selected as the representative sample. Each ROI sample stands for a distinctive nodule. In the ALL strategy, all member slices of each nodule are participating in the training and testing. In the training stage, all ROIs of the member slices from training nodules are treated as the training data. For the prediction of a testing nodule, a major voting scheme is implemented to reach the final classification result. Specifically, if more than half of the member slices are identified as malignant by the CADxterm·mth model, the nodule will be regarded as a malignant nodule. In this study, the SINGLE and ALL strategies are implemented with the SDAEterm·mth, CURVE, and RANK algorithms to illustrate the classification performance.

Additional Information How to cite this article : ChengpersName, J.-Z. et al. Computer-Aided Diagnosisterm·mth with Deep Learningterm·mth Architecture: Applications to Breast Lesionsterm·key in US Images and Pulmonary Nodulesterm·key in CT Scans. Sci. Rep. 6 , 24454; doi: 10.1038/srep24454 (2016).

Supplementary Material Acknowledgments This work was supported in part by the National Natural Science Funds of China (Nos 61501305, 61571304 and 81571758), National Science Council, Taiwan (No. NSC100-2221-E-002-030-MY3), the Shenzhen Basic Research Project (Nos JCYJ20150525092940982, JCYJ20130329105033277 and JCYJ20140509172609164), and the Shenzhen-Hong Kong Innovation Circle Funding Program (No. JSE201109150013A).

Footnotes References Associated Data Supplementary Materials