Chintan ParmarpersName, Patrick GrossmannpersName, Johan BussinkpersName, Philippe LambinpersName, Hugo J. W. L. AertspersName (2015date). “Machine Learning methods for Quantitative Radiomic Biomarkers.” Scientific Reportstitle·j.
Object of study — feature selectionterm·mth and classificationterm·mth methods for radiomicterm·dom survival predictionterm·key in lung cancerterm·key
Research question — which machine-learningterm·mth methods give accurate and stable radiomicterm·dom survival predictionterm·key
Contribution — a comparison identifying WLCX feature selectionterm·mth and random foreststerm·key as accurate and stable
phenomenon · claim-type · cross-domain-transfer · mechanism · claim-relation · replication-status · limitation — grounded in the full text| phenomenon | variation in radiomic prediction performance across machine-learning methods — “Our variability analysis indicated that the choice of classification method is the most dominant source of performance variation (34.21% of total variance).” | |
| claim-type | method — “In this radiomic study, fourteen feature selection methods and twelve classification methods were examined in terms of their performance and stability for predicting overall survival.” | |
| method — evidence | — “A total of 440 radiomic features were extracted from pre-treatment computed tomography (CT) images of 464 lung cancer patients.” | |
| object of study — evidence | — “In this radiomic study, fourteen feature selection methods and twelve classification methods were examined in terms of their performance and stability for predicting overall survival.” | |
| research question — evidence | — “In this radiomic study, fourteen feature selection methods and twelve classification methods were examined in terms of their performance and stability for predicting overall survival.” | |
| contribution — evidence | — “For classification methods, random forest (RF) displayed highest predictive performance (AUC: 0.66 ± 0.03) (median ± std), whereas decision tree (DT) (AUC: 0.54 ± 0.04) showed the lowest predictive performance.” | |
| mechanism | dimensionality reduction Feature selection addresses the curse of dimensionality in high-throughput radiomic features. — “Like any high-throughput data-mining field, radiomics also underlies the curse of dimensionality 17 , which should be addressed by appropriate feature selection strategies.” | mechanisms[0] |
| sense | stability - similarity of selected feature sets across data perturbations, or low RSD of classifier AUC — “We observed that MIM was the most stable method (stability = 0.94 ± 0.02) (median ± std) followed by RELIEF (stability = 0.91 ± 0.05) and WLCX (stability = 0.84 ± 0.05)” | senses[0] |
| relation | RSD —same-as→ relative standard deviation — BST had the highest relative standard deviation in AUC scores (RSD = 8.23%) — “BST had the highest relative standard deviation in AUC scores (RSD = 8.23%) and hence the lowest stability among the classification methods.” — “BST had the highest relative standard deviation in AUC scores (RSD = 8.23%)” | concept_edges[0] |
| weight | RSD —same-as→ relative standard deviation — 0.9 — “BST had the highest relative standard deviation in AUC scores (RSD = 8.23%) and hence the lowest stability among the classification methods.” | concept_edges[0] |
| relation | GLCM —same-as→ gray level co-occurrence — the gray level co-occurrence (GLCM) 21 — “Textural features, derived from the gray level co-occurrence (GLCM) 21 and run length matrices (GLRLM) 22 , quantified the intra-tumor heterogeneity.” — “the gray level co-occurrence (GLCM) 21” | concept_edges[1] |
| weight | GLCM —same-as→ gray level co-occurrence — 0.95 — “Textural features, derived from the gray level co-occurrence (GLCM) 21 and run length matrices (GLRLM) 22 , quantified the intra-tumor heterogeneity.” | concept_edges[1] |
| effect direction | feature selection —causes→ overfitting — reduces — “Moreover, feature selection also helps in reducing overfitting of models (increasing the generalizability).” — “reducing overfitting of models” | concept_edges[2] |
| relation | feature selection —causes→ overfitting — feature selection also helps in reducing overfitting of models — “Moreover, feature selection also helps in reducing overfitting of models (increasing the generalizability).” — “feature selection also helps in reducing overfitting of models” | concept_edges[2] |
| weight | feature selection —causes→ overfitting — 0.8 — “Moreover, feature selection also helps in reducing overfitting of models (increasing the generalizability).” | concept_edges[2] |
| relation | medical imaging —contrasted-with→ biopsies — Unlike biopsies, medical imaging is non-invasive — “Unlike biopsies, medical imaging is non-invasive and can provide information regarding the entire tumor phenotype, including the intra-tumor heterogeneity.” — “Unlike biopsies, medical imaging is non-invasive” | concept_edges[3] |
| weight | medical imaging —contrasted-with→ biopsies — 0.85 — “Unlike biopsies, medical imaging is non-invasive and can provide information regarding the entire tumor phenotype, including the intra-tumor heterogeneity.” | concept_edges[3] |
| replication | original | |
| limitation | Different parameter configurations might further improve classifier performance. — “One could argue that with different parameter configurations, the performance of classification methods may improve further.” | limitations[0] |
| limitation | Results need testing in other imaging modalities and cancer types. — “These results should be further tested in other radiomics based predictive studies, with different imaging modalities and in different cancer types.” | limitations[1] |
| limitation | Survival was dichotomised at two years. — “We dichotomized the censored continuous survival data using a cutoff time of 2 years.” | limitations[2] |
| limitation | A prior comparison was limited to 40 patients without independent validation. — “This study, however, was limited by the small cohort size as the final results were obtained on only 40 patients.” | limitations[3] |
finding · quantity · variable · definition · key-term — grounded in the full text| finding | Random forest had the highest predictive performance; decision trees the lowest. — “For classification methods, random forest (RF) displayed highest predictive performance (AUC: 0.66 ± 0.03) (median ± std), whereas decision tree (DT) (AUC: 0.54 ± 0.04) showed the lowest predictive performance.” | content_tags.findings[0] |
| finding | Classification method is the dominant source of performance variation. — “Our variability analysis indicated that the choice of classification method is the most dominant source of performance variation (34.21% of total variance).” | content_tags.findings[1] |
| finding | MIM was the most stable feature selection method. — “We observed that MIM was the most stable method (stability = 0.94 ± 0.02) (median ± std) followed by RELIEF (stability = 0.91 ± 0.05) and WLCX (stability = 0.84 ± 0.05)” | content_tags.findings[2] |
| finding | Boosting had the lowest classifier stability. — “BST had the highest relative standard deviation in AUC scores (RSD = 8.23%) and hence the lowest stability among the classification methods.” | content_tags.findings[3] |
| quantity | 440 features, 464 patients — — “A total of 440 radiomic features were extracted from pre-treatment computed tomography (CT) images of 464 lung cancer patients.” | content_tags.quantities[0] |
| quantity | 34.21% — — “Our variability analysis indicated that the choice of classification method is the most dominant source of performance variation (34.21% of total variance).” | content_tags.quantities[1] |
| quantity | 6.25% and 23.03% — — “Feature selection accounted for the 6.25%, whereas interaction of classifier & feature selection explained 23.03% of the total variation.” | content_tags.quantities[2] |
| variable | feature relevancy — “Feature relevancy is a measure of feature’s association with the target/outcome variable, whereas feature redundancy is the amount of redundancy present in a particular feature with respect to the set of already selected features.” | content_tags.variables[0] |
| definition | precision oncology — “‘Precision oncology’ refers to the customization of cancer care, where practices and/or therapies are being tailored to individual patients.” | content_tags.definitions[0] |
| definition | Filter methods — “Filter methods are feature-ranking methods, which rank the features using a scoring criterion.” | content_tags.definitions[1] |
| key-terms | radiomics, machine learning, feature selection, random forest, stability, overfitting, precision oncology, lung cancer, survival prediction |
| concept | Radiomics — not defined in this paper's own words; extracting and mining large numbers of medical imaging features quantifying tumor phenotypic characteristics — “Radiomics extracts and mines large number of medical imaging features quantifying tumor phenotypic characteristics.” | concepts[0] |
| concept | precision oncology — the customization of cancer care, where practices and/or therapies are being tailored to individual patients — “‘Precision oncology’ refers to the customization of cancer care, where practices and/or therapies are being tailored to individual patients.” | concepts[1] |
| concept | Machine-learning — computational methods/models using experience (data) to improve performance or make accurate predictions — ““Machine-learning” can be broadly defined as computational methods/models using experience (data) to improve performance or make accurate predictions” | concepts[2] |
| concept | curse of dimensionality — not defined in this paper's own words; the difficulty of learning from very many features — “Like any high-throughput data-mining field, radiomics also underlies the curse of dimensionality 17 , which should be addressed by appropriate feature selection strategies.” | concepts[3] |
| concept | feature selection — not defined in this paper's own words; choosing a subset of features for a predictive model — “Like any high-throughput data-mining field, radiomics also underlies the curse of dimensionality 17 , which should be addressed by appropriate feature selection strategies.” | concepts[4] |
| concept | overfitting — not defined in this paper's own words; a model fitting training data at the cost of generalizability — “Moreover, feature selection also helps in reducing overfitting of models (increasing the generalizability).” | concepts[5] |
| concept | Filter methods — feature-ranking methods, which rank the features using a scoring criterion — “Filter methods are feature-ranking methods, which rank the features using a scoring criterion.” | concepts[6] |
| concept | wrapper — not defined in this paper's own words; feature selection that searches subsets using the classifier itself — “We only used filter-based approaches because they are computationally more efficient and less prone to overfitting than the wrapper and embedded methods” | concepts[7] |
| concept | Feature relevancy — a measure of feature’s association with the target/outcome variable — “Feature relevancy is a measure of feature’s association with the target/outcome variable, whereas feature redundancy is the amount of redundancy present in a particular feature with respect to the set of already selected features.” | concepts[8] |
| concept | feature redundancy — the amount of redundancy present in a particular feature with respect to the set of already selected features — “Feature relevancy is a measure of feature’s association with the target/outcome variable, whereas feature redundancy is the amount of redundancy present in a particular feature with respect to the set of already selected features.” | concepts[9] |
| concept | classification — a supervised learning task of inferring a function from labeled training data — “In machine-learning, the classification is considered as a supervised learning task of inferring a function from labeled training data” | concepts[10] |
| concept | RSD — relative standard deviation — “BST had the highest relative standard deviation in AUC scores (RSD = 8.23%) and hence the lowest stability among the classification methods.” | concepts[11] |
| concept | relative standard deviation — the absolute value of the coefficient of variation — “RSD is the absolute value of the coefficient of variation and is often expressed in percentage.” | concepts[12] |
| concept | GLCM — gray level co-occurrence — “Textural features, derived from the gray level co-occurrence (GLCM) 21 and run length matrices (GLRLM) 22 , quantified the intra-tumor heterogeneity.” | concepts[13] |
| concept | gray level co-occurrence — not defined in this paper's own words; a texture matrix of how often grey levels co-occur — “Textural features, derived from the gray level co-occurrence (GLCM) 21 and run length matrices (GLRLM) 22 , quantified the intra-tumor heterogeneity.” | concepts[14] |
| concept | intra-tumor heterogeneity — not defined in this paper's own words; variation of tumor characteristics within one tumor — “Textural features, derived from the gray level co-occurrence (GLCM) 21 and run length matrices (GLRLM) 22 , quantified the intra-tumor heterogeneity.” | concepts[15] |
| concept | biopsies — not defined in this paper's own words; tissue extraction from tumors — “Unlike biopsies, medical imaging is non-invasive and can provide information regarding the entire tumor phenotype, including the intra-tumor heterogeneity.” | concepts[16] |
| concept | medical imaging — not defined in this paper's own words; non-invasive imaging of the body such as CT — “Unlike biopsies, medical imaging is non-invasive and can provide information regarding the entire tumor phenotype, including the intra-tumor heterogeneity.” | concepts[17] |
Full text — tagged
Machine Learningterm·key methods for Quantitative Radiomicterm·dom Biomarkers Abstract Radiomicsterm·dom extracts and mines large number of medical imagingterm·mth features quantifying tumor phenotypic characteristics. Highly accurate and reliable machine-learningterm·mth approaches can drive the success of radiomicterm·dom applications in clinical care. In this radiomicterm·dom study, fourteen feature selectionterm·mth methods and twelve classificationterm·mth methods were examined in terms of their performance and stabilityterm·key for predicting overall survival. A total of 440 radiomicterm·dom features were extracted from pre-treatment computed tomography (CT) images of 464 lung cancerterm·key patients. To ensure the unbiased evaluation of different machine-learningterm·mth methods, publicly available implementations along with reported parameter configurations were used. Furthermore, we used two independent radiomicterm·dom cohorts for training (n = 310 patients) and validation (n = 154 patients). We identified that Wilcoxon test based feature selectionterm·mth method WLCX (stabilityterm·key = 0.84 ± 0.05, AUC = 0.65 ± 0.02) and a classificationterm·mth method random forestterm·key RF (RSDterm·mth = 3.52%, AUC = 0.66 ± 0.03) had highest prognostic performance with high stabilityterm·key against data perturbation. Our variability analysis indicated that the choice of classificationterm·mth method is the most dominant source of performance variation (34.21% of total variance). Identification of optimal machine-learningterm·mth methods for radiomicterm·dom applications is a crucial step towards stable and clinically relevant radiomicterm·dom biomarkers, providing a non-invasive way of quantifying and monitoring tumor-phenotypic characteristics in clinical practice.
‘Precision oncologyterm·dom’ refers to the customization of cancer care, where practices and/or therapies are being tailored to individual patients. Such customization process can maximize the success of preventive and therapeutic interventions with minimum side effects. Most of the precision oncologyterm·dom related research has centered on the molecular characterization of tumors using genomics based approaches, which require tissue extraction by tumor biopsiesterm·mth. Although several genomics based approaches have successfully been applied in clinical oncology 1 , there are inherent limitations to biopsy based assays. Tumors are spatially and temporally heterogeneous, and repeated tumor biopsiesterm·mth, which increase the risk for a patient, are often required to capture the molecular heterogeneity of tumors. These ethical and clinical challenges related to biopsy-based assays, can be addressed by medical imagingterm·mth, which is a routine practice for cancer diagnosis and staging in clinical oncology. Unlike biopsiesterm·mth, medical imagingterm·mth is non-invasive and can provide information regarding the entire tumor phenotype, including the intra-tumor heterogeneityterm·dom. Furthermore, recent advances in high-resolution image acquisition machines and computational hardware allow the detailed and efficient quantification of tumor phenotypic characteristics. Therefore, medical imagingterm·mth provides unprecedented opportunities for precision oncologyterm·dom.
“Radiomicsterm·dom”, an emerging and promising field, hypothesizes that medical imagingterm·mth provides crucial information regarding tumor physiology, which could be exploited to enhance cancer diagnostics 2 . It provides a comprehensive quantification of tumor phenotypes by extracting and mining large number of quantitative imaging features 3 . Several studies have investigated various radiomicterm·dom features in terms of their prognostic or predictive abilities and reliability across different clinical settings 4 , 5 , 6 , 7 , 8 , 9 , 10 . Different studies have shown the discriminating capabilities of radiomicterm·dom features for the stratification of tumor histology 6 , tumor grades or stages 11 , and clinical outcomes 8 , 12 , 13 . Moreover, some studies have reported the association between radiomicterm·dom features and the underlying gene expression patterns 8 , 14 , 15 .
“Machine-learningterm·mth” can be broadly defined as computational methods/models using experience (data) to improve performance or make accurate predictions 16 . These programmable computational methods are capable of “learning” from data and hence can automate and improve the prediction process. Predictive and prognostic models with high accuracy, reliability, and efficiency are vital factors driving the success of radiomicsterm·dom. Therefore, it is essential to compare different machine-learningterm·mth models for radiomicsterm·dom based clinical biomarkers. Like any high-throughput data-mining field, radiomicsterm·dom also underlies the curse of dimensionalityterm·dom 17 , which should be addressed by appropriate feature selectionterm·mth strategies. Moreover, feature selectionterm·mth also helps in reducing overfittingterm·dom of models (increasing the generalizability). Thus, in order to reduce the dimensionality of radiomicterm·dom feature space and enhance the performance of radiomicsterm·dom based predictive models, different feature selectionterm·mth methods 18 should be thoroughly investigated. However, as radiomicsterm·dom is an emerging research field, most of the published studies have only assessed the predictive capabilities of radiomicterm·dom features without putting much emphasis on the comparison of different feature selectionterm·mth and predictive modeling methods. Only few recent studies have investigated the effect of different feature selectionterm·mth and machine learningterm·key classificationterm·mth methods on radiomicsterm·dom based clinical predictions 19 , 20 , but with limited sample sizes. Furthermore, these studies lacked independent validation of the results, which may restrict the generalizability of their conclusions.
In this study, we investigated a large panel of machine-learningterm·mth approaches for radiomicsterm·dom based survival predictionterm·key. We evaluated 14 feature selectionterm·mth methods and 12 classificationterm·mth methods in terms of their predictive performance and stabilityterm·key against data perturbation. These methods were chosen because of their popularity in literature. Furthermore, publicly available implementations along with reported parameter configurations were used in the analysis, which ensured an unbiased evaluation of these methods. Two independent lung cancerterm·key cohorts were used for training and validation, with in total image and clinical outcome data of 464 patients. Feature selectionterm·mth and predictive modeling are considered as the important building blocks for high throughput data driven radiomicsterm·dom. Therefore, our investigation could help in the identification of optimal machine-learningterm·mth approaches for radiomicsterm·dom based predictive studies, which could enhance the applications of non-invasive and cost-effective radiomicsterm·dom in clinical oncology.
Methods Radiomicterm·dom Features A total of 440 radiomicterm·dom features were used in the analysis. These radiomicterm·dom features quantified tumor phenotypic characteristics on CT images and are divided into four feature groups: I) tumor intensity, II) shape, III) texture and IV) wavelet features. Tumor intensity based features estimated the first order statistics of the intensity histogram, whereas shape features described the 3D geometric properties of the tumor. Textural features, derived from the gray level co-occurrenceterm·mth (GLCMterm·mth) 21 and run length matrices (GLRLM) 22 , quantified the intra-tumor heterogeneityterm·dom. These textural features were computed by averaging their values over all thirteen directions. Wavelet features are the transformed domain representations of the intensity and textural features. These features were computed on different wavelet decompositions of the original image using a coiflet wavelet transformation. Matlab R2012b (The Mathworks, Natick, MA) was used for the image analysis. Radiomicterm·dom features were automatically extracted by our in-house developed radiomicsterm·dom image analysis software, which uses an adapted version of CERR (Computational Environment for Radiotherapy Research) 23 and Matlab for the preprocessing of medical images. Mathematical definitions of all radiomicterm·dom features, as well as the extraction methods, were previously described 8 .
Datasets In this study, we employed two NSCLC cohorts from the two different institutes of Netherlands: (1) Lung1:422 NSCLC patients treated at MAASTRO Clinic in Maastricht. (2) Lung2:225 NSCLC patients treated at Radboud University Medical Center in Nijmegen. CT-scans, manual delineations and clinical data were available for all included patients. More details on the included datasets are described in Supplementary-A . We dichotomized the censored continuous survival data using a cutoff time of 2 years. The patients who lived beyond the cutoff time were labeled as 1, whereas the deceased ones were labeled as 0. The objective of the study was to stratify patients into these two labeled survival classes. Two-years is considered as a relevant survival time for NSCLC patients and several other studies have designed their prediction models using a survival cutoff of 2 years 24 , 25 , 26 . We excluded the patients, which were followed for less than 2 years. It resulted in 310 patients in training cohort (Lung1) and 154 patients in validation cohort (Lung2). All the features were normalized using Z-score normalization.
Feature Selectionterm·mth Methods Fourteen feature selectionterm·mth methods based on filter approaches were used in the analysis (Fisher score (FSCR), Relief (RELF), T-score (TSCR), Chi-square (CHSQ), Wilcoxon (WLCX), Gini index (GINI), Mutual information maximization (MIM), Mutual information feature selectionterm·mth (MIFS), Minimum redundancy maximum relevance (MRMR), Conditional infomax feature extraction (CIFE), Joint mutual information (JMI), Conditional mutual information maximization (CMIM), Interaction capping (ICAP), Double input symmetric relevance (DISR)). In order to improve the readability of this manuscript, we have defined all the acronyms related to feature selectionterm·mth methods in Table 1 . We chose these methods mainly because of their popularity in literature, simplicity and computational efficiency. Furthermore, publicly available implementations were readily available for these methods 27 , 28 , which increases their reusability. Filter methodsterm·mth are feature-ranking methods, which rank the features using a scoring criterion. All filter based feature selectionterm·mth methods can be divided into two categories: univariate methods and multivariate methods. In case of univariate methods, the scoring criterion only depends on the feature relevancyterm·mth ignoring the feature redundancyterm·mth, whereas multivariate methods investigate the multivariate interaction within the features and the scoring criterion is a weighted sum of feature relevancyterm·mth and redundancy. Feature relevancyterm·mth is a measure of feature’s association with the target/outcome variable, whereas feature redundancyterm·mth is the amount of redundancy present in a particular feature with respect to the set of already selected features. Further description regarding the theoretical formulation of feature selectionterm·mth problem and each of the used feature selectionterm·mth methods can be obtained from Supplementary-B online .
Table defining the acronyms related to the used feature selectionterm·mth and classificationterm·mth methods.
Classifiers In machine-learningterm·mth, the classificationterm·mth is considered as a supervised learning task of inferring a function from labeled training data 16 . The training data consists of a set of examples, where each example is represented as a pair of an input vector (features) and a desired output value (target or category label). The classificationterm·mth algorithm (classifier) analyzes the training data and infers a hypothesis (function), which can be used for predicting the labels of unseen observations. Many classifiers belonging to different areas of computer science and statistics have been proposed in machine-learningterm·mth literature 29 . In our study, we used 12 machine-learningterm·mth classifiers arising from 12 classifier families (Bagging (BAG), Bayesian (BY), Boosting (BST), Decision trees (DT), Discriminant analysis (DA), Generalized linear models (GLM), Multiple adaptive regression splines (MARS), Nearest neighbors (NN), Neural networks (Nnet), Partial least square and principle component regression (PLSR), Random foreststerm·key (RF), and Support vector machines (SVM)). The acronyms related to classifiers are defined in Table 1 . All classifiers were implemented using R package caret 30 , which provides a nice interface to access many machine-learningterm·mth algorithms in R. Furthermore, it also provides a user-friendly framework for training different machine-learningterm·mth models. Classifiers were trained using the repeated (3 repeat iterations) 10 fold cross validation of training cohort (Lung1) and their predictive performance was evaluated in the validation cohort (Lung2) using area under ROC curve (AUC). We used parameter configurations that were previously defined by Fernandez-Delgado et al. 31 in a comprehensive comparative study of 179 classifiers and 121 different datasets. We have listed the classificationterm·mth methods along with their parameters and corresponding R packages in Supplementary-C online .
Analysis Predictive Performance of Feature Selectionterm·mth and Classificationterm·mth Methods In order to investigate and compare different feature selectionterm·mth and classificationterm·mth methods, we created a three-dimensional parameter grid for the analysis. For each of the 14 feature selectionterm·mth methods, we incrementally selected features ranging from 5 up to 50, with an increment of 5 features (n = 5, 10, 15, 20, … , 50). These subsets of selected features were then evaluated by using each of the 12 machine-learningterm·mth classifiers and area under ROC curves (AUC).
Stabilityterm·key of Feature Selectionterm·mth and Classificationterm·mth Methods In order to assess the stabilityterm·key of feature selectionterm·mth methods, we used a stabilityterm·key measure proposed by Yu et al. 32 under the hard data perturbation settings 33 . We quantified the stabilityterm·key of a method as the similarity between the results obtained by the same feature selectionterm·mth method, when applied on the two non-overlapping partitions (of size N/2) of the training cohort (Lung1). To compute similarity between the two resultant feature sets, a weighted complete bipartite graph was constructed, where the two node sets corresponded to the two sets of selected features. The edge weights were assigned as the absolute Spearman correlation coefficient between the features at the nodes. We then applied the Hungarian algorithm 34 to identify the maximum weighted matching between the two node sets, and then similarity (stabilityterm·key) was quantified as the final matching cost. For each feature selectionterm·mth method, we computed the stabilityterm·key 100 times using a bootstrap approach and reported the median ± std values in the results.
The empirical stabilityterm·key of a classifier was quantified using the relative standard deviationterm·mth (RSDterm·mth %) and a bootstrap approach. We first selected 30 representative features using the Wilcoxon based feature selectionterm·mth method WLCX and used them to compute the classifier stabilityterm·key. For each classificationterm·mth method, we trained the model on the subsampled training cohort (size N/2) and validated the performance on the validation cohort using AUC. Subsampling of the training cohort was done 100 times using a bootstrap approach. RSDterm·mth is the absolute value of the coefficient of variation and is often expressed in percentage. Here, it was defined as where and were the standard deviation and mean of the 100 AUC values respectively. It should be noted that higher stabilityterm·key in the case of classifiers corresponds to lower RSDterm·mth values.
Stabilityterm·key and Predictive Performance In order to identify the highly reliable and accurate methods, we used the median values of AUC and stabilityterm·key as thresholds. We created two rank lists based on AUC & stabilityterm·key and cited the methods as highly accurate and reliable, which ranked in the top half of both the ranked lists. Feature selectionterm·mth methods having stabilityterm·key ≥0.735 (median stabilityterm·key of all feature selectionterm·mth methods) and AUC ≥ 0.615 (median AUC of all feature selectionterm·mth methods) are considered as highly reliable and accurate methods. Similarly, classificationterm·mth methods having RSDterm·mth ≤ 5.97 (median RSDterm·mth of all classifiers) and AUC ≥ 0.61 (median AUC of all classifiers) are considered as highly reliable and accurate ones.
Experimental Factors Affecting the Radiomicsterm·dom Based Survival Predictionterm·key There are three main experimental factors, which can potentially affect the prediction of radiomicsterm·dom based survival predictionterm·key: feature selectionterm·mth method, classificationterm·mth method and the number of selected features. Multifactor ANOVA was used to quantify the variability in AUC scores contributed by these factors and their interactions. In order to compare the variability contributed by each factor, the estimated variance components were divided by the total variance.
All the analysis was done using R software (R Core Team, Vienna, Austria) version 3.1.2 and Matlab R2012b (The Mathworks, Natick, MA) with Windows 7.
Results To investigate the machine-learningterm·mth approaches for prognostic radiomicterm·dom biomarkers, a total of 440 radiomicterm·dom features were extracted from the segmented tumor regions of the pre-treatment CT images of two independent NSCLC cohorts. Feature selectionterm·mth and classificationterm·mth training was done using the training cohort Lung1 (n = 310 patients), whereas the validation cohort Lung2 (n = 154 patients) was used to assess the predictive performance [see Fig. 1 ].
A total of 440 radiomicterm·dom features were extracted from the segmented tumor regions of the pre-treatment CT images of 464 NSCLC patients. Feature selectionterm·mth and classificationterm·mth training was done using the training cohort Lung1 (n = 310), whereas Lung2 (n = 154) cohort was used as a validation cohort.
Predictive Performance of the Feature Selectionterm·mth and Classificationterm·mth Methods Predictive performance of different feature selectionterm·mth and classificationterm·mth methods was assessed using the area under receiver operator characteristic curve (AUC). Figure 2 depicts the performance of feature selectionterm·mth (in rows) and classificationterm·mth methods (in columns) using 30 selected features, which are the 30 top ranked features, resulted in feature selectionterm·mth. For each classificationterm·mth method, there are 14 AUC values corresponding to the 14 different feature selectionterm·mth methods. We used a median of all 14 AUC values as a representative AUC of a classifier. Similarly, for each feature selectionterm·mth method, a median of 12 AUCs (corresponding to 12 classificationterm·mth methods) is used as a representative AUC. These representative AUC values for the classificationterm·mth and feature selectionterm·mth methods are given in Table 2 . For classificationterm·mth methods, random forestterm·key (RF) displayed highest predictive performance (AUC: 0.66 ± 0.03) (median ± std), whereas decision tree (DT) (AUC: 0.54 ± 0.04) showed the lowest predictive performance. As far as feature selectionterm·mth methods are concerned, the Wilcoxon test based method WLCX showed highest predictive performance (AUC: 0.65 ± 0.02), whereas method CHSQ (AUC: 0.60 ± 0.03) and CIFE (AUC: 0.60 ± 0.04) had the lowest median AUCs. We repeated the above experiment by varying the number of selected features (range 5–50). Results corresponding to 10, 20, 40 and 50 representative (top ranked) features are reported in Supplementary Figures S1, S2, S3 and S4 online. Furthermore, median AUC values over each of the experimental factors (feature selectionterm·mth methods, classificationterm·mth methods and number of selected features) are depicted by the heatmaps in Supplementary Figures S5, S6 and S7 online. Here as well, random forestterm·key (RF) (classifier) and Wilcoxon test based method WLCX (feature selectionterm·mth) showed highest median AUCs in majority of cases.
Heatmap depicting the predictive performance (AUC) of feature selectionterm·mth (in rows) and classificationterm·mth (in columns) methods. It can be observed that RF, BAG and BY classificationterm·mth methods and feature selectionterm·mth methods WLCX, MRMR and MIFS shows relatively high predictive performance in many cases.
Table describing the median values of AUC and stabilityterm·key for different Classificationterm·mth and Feature Selectionterm·mth methods.
Stabilityterm·key of the Feature Selectionterm·mth and Classificationterm·mth Methods We assessed the feature selectionterm·mth methods in terms of their stabilityterm·key against data resampling using the hard data perturbation settings 33 . We observed that MIM was the most stable method (stabilityterm·key = 0.94 ± 0.02) (median ± std) followed by RELIEF (stabilityterm·key = 0.91 ± 0.05) and WLCX (stabilityterm·key = 0.84 ± 0.05), whereas GINI (stabilityterm·key = 0.68 ± 0.10), JMI (stabilityterm·key = 0.68 ± 0.05), CHSQ (stabilityterm·key = 0.69 ± 0.09), DISR (stabilityterm·key = 0.69 ± 0.05) and CIFE (stabilityterm·key = 0.69 ± 0.05) showed relatively low stabilityterm·key [ Table 2 ].
Empirical stabilityterm·key of classificationterm·mth methods was quantified using the relative standard deviationterm·mth (RSDterm·mth) and a bootstrap approach. We observed that BY was the most stable classificationterm·mth method (RSDterm·mth = 0.86%) followed by GLM (RSDterm·mth = 2.19%), PLSR (RSDterm·mth = 2.24%) and RF (RSDterm·mth = 3.52%). BST had the highest relative standard deviationterm·mth in AUC scores (RSDterm·mth = 8.23%) and hence the lowest stabilityterm·key among the classificationterm·mth methods. RSDterm·mth (%) values corresponding to all 12 classifiers are reported in Table 2 .
Stabilityterm·key and Predictive Performance Scatterplots in Fig. 3 assesses the stabilityterm·key and prediction performance. It can be observed that feature selectionterm·mth methods WLCX (stabilityterm·key = 0.84 ± 0.05, AUC = 0.65 ± 0.02), MIFS (stabilityterm·key = 0.8 ± 0.03, AUC = 0.63 ± 0.03), MRMR (stabilityterm·key = 0.74 ± 0.03, AUC = 0.63 ± 0.03) and FSCR (stabilityterm·key = 0.78 ± 0.08, AUC = 0.62 ± 0.04) should be preferred as their stabilityterm·key and predictive performance was higher than the corresponding median values across all feature selectionterm·mth methods (stabilityterm·key = 0.735, AUC = 0.615). Similarly for classificationterm·mth methods, RF (RSDterm·mth = 3.52%, AUC = 0.66 ± 0.03), BY (RSDterm·mth = 0.86%, AUC = 0.64 ± 0.05), BAG (RSDterm·mth = 5.56%, AUC = 0.64 ± 0.03), GLM (RSDterm·mth = 2.19%, AUC = 0.63 ± 0.02), and PLSR (RSDterm·mth = 2.24%, AUC = 0.63 ± 0.02), the stabilityterm·key and predictive performance was higher than the corresponding median values (RSDterm·mth = 5.93%, AUC = 0.61).
Scatterplots between the stabilityterm·key and predictive performance (AUC) of feature selectionterm·mth (FS) (Left) and classificationterm·mth methods (CF) (right). Feature selectionterm·mth methods having stabilityterm·key ≥0.735 (median stabilityterm·key of FS) and AUC ≥ 0.615 (median AUC of FS) are considered as highly reliable and predictive methods. Similarly, classificationterm·mth methods having RSDterm·mth ≤ 5.97 (median RSDterm·mth of CF) and AUC ≥ 0.61 (median AUC of CF) are considered as highly reliable and accurate ones. Highly reliable and predictive methods are displayed in a gray square region.
Experimental Factors Affecting the Radiomicsterm·dom Based Survival Predictionterm·key To quantify the effects of the three experimental factors (feature selectionterm·mth methods, classificationterm·mth methods and the number of selected features), we performed multifactor analysis of variance (ANOVA) on AUC scores. We observed that all three experimental parameters and their interactions are the significant factors affecting the prediction performance [ Fig. 4 ]. Classificationterm·mth method was the most dominant source of variability as it explained 34.21% of the total variance in AUC scores. Feature selectionterm·mth accounted for the 6.25%, whereas interaction of classifier & feature selectionterm·mth explained 23.03% of the total variation. Size of the selected (representative) feature subset only shared 1.65% of the total variance [ Fig. 4 ].
Variation of AUC explained by the experimental factors and their interactions. It can be observed that classificationterm·mth method was the most dominant source of variability. Size of the selected (representative) feature subset shared the least of the total variance.
Discussion Medical imagingterm·mth is a routinely used and easily accessible source of information in clinical oncology. It serves as a non-invasive and cost-effective cancer diagnostic tool. Radiomicsterm·dom employs the medical imagingterm·mth data for the customization of cancer care and hence adds a new and promising dimension to precision oncologyterm·dom 2 , 3 , 8 . Moreover, it can also capture the intra-tumor heterogeneityterm·dom, which is often considered as an important biomarker in oncology 12 , 35 , 36 , 37 . A number of studies have built radiomicsterm·dom based predictive models for various clinical factors (tumor grades, survival outcomes, treatment response, etc.) 12 . For the successful realization of radiomicsterm·dom based predictive analyses, it is required to evaluate and compare different feature selectionterm·mth and predictive modeling methods, which was the primary objective of this study.
Various feature selectionterm·mth methods have been employed for high-throughput data mining problems 38 . In general, feature selectionterm·mth methods are categorized into three main categories: (1) filter methodsterm·mth (2) wrapperterm·mth methods and (3) embedded methods. In this study, we investigated 14 different filter based approaches for radiomicsterm·dom based survival predictionterm·key. We only used filter-based approaches because they are computationally more efficient and less prone to overfittingterm·dom than the wrapperterm·mth and embedded methods 18 , 27 . Furthermore, unlike wrapperterm·mth and embedded methods, filter methodsterm·mth are classifier independent. Thus, they allow separation of the modeling and feature selectionterm·mth component of the predictive analysis, which increases the generalizability of each component and hence the overall analysis.
We also investigated 12 machine-learningterm·mth classificationterm·mth methods belonging to 12 different classifier families. Many classifiers have been proposed in the machine-learningterm·mth literature. Theoretically speaking, these classifiers belong to different fields (classifier families) of computer science and statistics. Therefore, it could really be difficult to understand the underlying assumptions of each and every classifier and tune the parameters in an unbiased manner. The parameter tuning could be biased by user’s more (or lack of) expertise with some classifiers over the others. Usually, the studies, which propose a new classifier, only compare it to the reference classifiers of same family excluding the other classifier families. Even if classifiers belonging to different families are considered for comparison, these reference classifiers are usually implemented using simple tools and with limited parameter configurations while carefully tuning the proposed classifier. These could consequently bias the results in favor of the proposed classifiers 31 . In our study, we are not proposing any new classifier and we have used the same implementation tool (R package caret) for all the classifiers. Furthermore, to ensure unbiased usage of classifiers, we used parameter configurations that were previously defined by Fernandez-Delgado et al. 31 , in an exhaustive study of comparing 179 classifiers over 121 different datasets. These parameter configurations were selected from the literature and have been previously validated on a large number (121) of datasets belonging to different fields. Furthermore, in our study, the parameters were tuned using the repeated cross validation of training data only. Hence, our experimental design allowed us to evaluate different classificationterm·mth methods in an unbiased manner.
Our results show that the Wilcoxon test based feature selectionterm·mth method WLCX yields the highest predictive performance with the majority of classifiers. Interestingly, WLCX is a simple univariate method based on ranks, which does not take into account the redundancy of selected features during feature ranking. The majority of feature selectionterm·mth methods gave highest predictive performance when used with the random forestterm·key (RF) classifier. One could argue that with different parameter configurations, the performance of classificationterm·mth methods may improve further. An exhaustive parameter tuning could be investigated for evaluating the improvement of prediction performance. However, the required computational resources and high time complexity can hinder the exhaustive search. We expect that future radiomicterm·dom studies focusing on different clinical outcomes and similar analysis framework could provide better understanding in this regard. A limited number of methods, which are consistently high performing across different radiomicterm·dom studies, could be further assessed with an exhaustive parameter tuning. Nevertheless, It should be noted that random foreststerm·key (RF) have displayed high predictive performance in several other biomedical and other domain applications as well 31 . These results indicate that choosing the WLCX feature selectionterm·mth method and/or RF classificationterm·mth method increases predictive performance in radiomicsterm·dom.
Results related to our stabilityterm·key analysis provide another dimension for choosing the feature selectionterm·mth and classificationterm·mth methods. Depending upon the applications, one may give importance to the predictive performance or stabilityterm·key and accordingly opt for the required method. Results related to multifactor ANOVA indicated that the classificationterm·mth method is the most dominant source of variation in the prediction performance (AUC) and hence should be chosen carefully. Size of the selected feature subset contributed the least in the total variation of AUC.
Only few studies have investigated and compared different feature selectionterm·mth and machine-learningterm·mth modeling methods for radiomicsterm·dom based clinical predictions 19 , 20 . Recently, Hawkins et al. 19 have compared four different feature selectionterm·mth and classificationterm·mth methods for CT based survival predictionterm·key of NSCLC patients. This study, however, was limited by the small cohort size as the final results were obtained on only 40 patients. Furthermore, it also lacked an independent validation of the results. On the contrary, two independent radiomicterm·dom cohorts of sizes 310 and 154 patients were used in our analysis and an independent validation of the results was reported.
Our radiomicterm·dom analysis is focused on the prediction of two-year patient survival in NSCLC patients. It provides an unbiased evaluation of different machine-learningterm·mth methods of feature selectionterm·mth and classificationterm·mth. It could be considered as a reference for the future radiomicsterm·dom based predictive studies. Our results indicated that choosing Wilcoxon test based feature selectionterm·mth method WLCX and/or random forestterm·key (RF) classificationterm·mth method gives highest performance for radiomicsterm·dom based survival predictionterm·key. Furthermore, these methods also turned out reasonably stable against data perturbation and hence they could be preferred for radiomicsterm·dom based predictive studies. These results should be further tested in other radiomicsterm·dom based predictive studies, with different imaging modalities and in different cancer types.
It has been previously shown that for NSCLC patients, statistical models based on patient’s tumor and treatment characteristics provide significantly better predictions than the human expert 24 . Moreover, several other studies have highlighted the limitation of doctors’ prognostic capability for terminally ill cancer patients 39 , 40 , 41 . The predictions of human experts can suffer from inter-observer variability. On the contrary, statistical models could make the prediction system more deterministic if the parameter configurations and the training framework are fixed.
The potential clinical utility of radiomicsterm·dom based prognostic models has been stated in previous study 8 . With expanding radiomicsterm·dom cohorts and feature dimensions, we expect higher prediction performance in future radiomicterm·dom studies. Furthermore, the integrative studies like radiomics-genomics in combination with standard clinical covariates could also improvise the prediction performance and further validate the utility of these methods in clinical practice. Overall, our analysis is a step forward towards the enhancements of radiomicsterm·dom based clinical predictions.
Additional Information How to cite this article : ParmarpersName, C. et al. Machine Learningterm·key methods for Quantitative Radiomicterm·dom Biomarkers. Sci. Rep. 5 , 13087; doi: 10.1038/srep13087 (2015).
Supplementary Material Acknowledgments Authors acknowledge financial support from the National Institute of Health (NIH-USA U24CA194354, and NIH-USA U01CA190234), EU 7th framework program (EURECA, ARTFORCE), Kankeronderzoekfonds Limburg from the Health Foundation Limburg and the Dutch Cancer Society (KWF UM 2009–4454, KWF MAC 2013–6425).
Footnotes References Associated Data Supplementary Materials