Geert LitjenspersName, Clara I. SánchezpersName, Nadya TimofeevapersName, Meyke HermsenpersName, Iris NagtegaalpersName, Iringo KovacspersName, Christina Hulsbergen - van de KaapersName, Peter BultpersName, Bram van GinnekenpersName, Jeroen van der LaakpersName (2016date). “Deep learning as a tool for increased accuracy and efficiency of histopathological diagnosis..” Scientific Reportstitle·j.
Object of study — convolutional neural networksterm·key applied to whole-slide histopathologyterm·key images
Research question — whether CNNs can improve the efficiency of cancer diagnosis in H&Eterm·mth whole-slide images
Contribution — CNN cancer likelihood mapsterm·mth that exclude 30-44% of negative slides without missing positives
phenomenon · claim-type · cross-domain-transfer · mechanism · claim-relation · replication-status · limitation — grounded in the full text| phenomenon | automatic exclusion of tumour-negative slides by deep learning — “For the sentinel lymph nodes, specificity was even higher at 44%, without missing any slide containing micro- or macro-metastases.” | |
| claim-type | method — “In this paper we introduce ‘deep learning’ as a technique to improve the objectivity and efficiency of histopathologic slide analysis.” | |
| method — evidence | — “For the sentinel lymph node procedure, we assessed how well the system was capable of identifying individual micro- and macro-metastases using free-response ROC (FROC) analysis and if it is capable of excluding slides which do not contain any metastases using ROC analysis.” | |
| object of study — evidence | — “This study investigates the general applicability of CNNs to improve the efficiency of cancer diagnosis in H&E images by applying it to two novel tasks: the detection of prostate cancer in biopsy specimens and the detection of breast cancer metastases in resected sentinel lymph nodes.” | |
| research question — evidence | — “This study investigates the general applicability of CNNs to improve the efficiency of cancer diagnosis in H&E images by applying it to two novel tasks: the detection of prostate cancer in biopsy specimens and the detection of breast cancer metastases in resected sentinel lymph nodes.” | |
| contribution — evidence | — “We found that all slides containing prostate cancer and micro- and macro-metastases of breast cancer could be identified automatically while 30–40% of the slides containing benign and normal tissue could be excluded without the use of any additional immunohistochemical markers or human intervention.” | |
| mechanism | learned features The networks learn relevant features directly from large image databases. — “These systems function by learning relevant features directly from huge image databases (typically millions of images).” | mechanisms[0] |
| mechanism | data augmentation Rotation and flipping of patches keep the network from learning rotation-dependent features. — “During patch extraction, patches were rotated (0, 90, 180 and 270 degrees) and subsequently flipped to make sure the network would not learn rotation-dependent features.” | mechanisms[1] |
| sense | deep learning - learning features directly from large image sets rather than hand-crafted extractors — “These systems function by learning relevant features directly from huge image databases (typically millions of images).” | senses[0] |
| sense | slide exclusion - removing tumour-negative slides from human review at full sensitivity — “In the prostate cancer slides, up to 32% of the slides not containing disease could be excluded.” | senses[1] |
| relation | H&E —same-as→ hematoxylin and eosin — ( ) — “Microscopic analysis of hematoxylin and eosin (H&E) stained sections has been the basis for cancer diagnosis and grading for the past century 1 .” — “hematoxylin and eosin (H&E)” | concept_edges[0] |
| weight | H&E —same-as→ hematoxylin and eosin — 0.95 — “Microscopic analysis of hematoxylin and eosin (H&E) stained sections has been the basis for cancer diagnosis and grading for the past century 1 .” | concept_edges[0] |
| relation | WSI —same-as→ whole slide images — ( ) — “Digital whole slide images (WSI) allow the application of image analysis techniques to aid pathologists in the examination and quantification of slides 5 .” — “Digital whole slide images (WSI)” | concept_edges[1] |
| weight | WSI —same-as→ whole slide images — 0.95 — “Digital whole slide images (WSI) allow the application of image analysis techniques to aid pathologists in the examination and quantification of slides 5 .” | concept_edges[1] |
| relation | ROC —same-as→ receiver-operator curve — ( ) — “For prostate cancer detection the CNNs were evaluated on a per-slide level using receiver-operator curve (ROC)-analysis.” — “receiver-operator curve (ROC)-analysis” | concept_edges[2] |
| weight | ROC —same-as→ receiver-operator curve — 0.95 — “For prostate cancer detection the CNNs were evaluated on a per-slide level using receiver-operator curve (ROC)-analysis.” | concept_edges[2] |
| relation | FROC —same-as→ free-response ROC — ( ) — “For the sentinel lymph node procedure, we assessed how well the system was capable of identifying individual micro- and macro-metastases using free-response ROC (FROC) analysis and if it is capable of excluding slides which do not contain any metastases using ROC analysis.” — “free-response ROC (FROC) analysis” | concept_edges[3] |
| weight | FROC —same-as→ free-response ROC — 0.95 — “For the sentinel lymph node procedure, we assessed how well the system was capable of identifying individual micro- and macro-metastases using free-response ROC (FROC) analysis and if it is capable of excluding slides which do not contain any metastases using ROC analysis.” | concept_edges[3] |
| relation | ITC —same-as→ isolated tumor cells — ( ) — “For the second batch no on-slide annotations were available, only the per-case outcome (presence of macro- and/or micro-metastases and isolated tumor cells (ITC)).” — “isolated tumor cells (ITC)” | concept_edges[4] |
| weight | ITC —same-as→ isolated tumor cells — 0.95 — “For the second batch no on-slide annotations were available, only the per-case outcome (presence of macro- and/or micro-metastases and isolated tumor cells (ITC)).” | concept_edges[4] |
| relation | CLM —same-as→ cancer likelihood map — ( ) — “A cancer likelihood map (CLM), the output of the CNN indicating cancer likelihood per pixel, for a representative WSI from the test set with cancer covering 30% of the tissue area is shown in Fig. 2 .” — “A cancer likelihood map (CLM)” | concept_edges[5] |
| weight | CLM —same-as→ cancer likelihood map — 0.95 — “A cancer likelihood map (CLM), the output of the CNN indicating cancer likelihood per pixel, for a representative WSI from the test set with cancer covering 30% of the tissue area is shown in Fig. 2 .” | concept_edges[5] |
| relation | AUC —same-as→ area under the ROC curve — ( ) — “The average bootstrapped area under the ROC curve (AUC) for the median analysis was 0.99 (0.95–1.0) and 0.98 (0.94–0.99) for the 90 th -percentile analysis.” — “area under the ROC curve (AUC)” | concept_edges[6] |
| weight | AUC —same-as→ area under the ROC curve — 0.95 — “The average bootstrapped area under the ROC curve (AUC) for the median analysis was 0.99 (0.95–1.0) and 0.98 (0.94–0.99) for the 90 th -percentile analysis.” | concept_edges[6] |
| relation | Bootstrap —is-a→ statistical technique — is a — “Bootstrap is a statistical technique which enables estimation of confidence intervals by repeated resampling of a representative population 18 .” — “Bootstrap is a statistical technique” | concept_edges[7] |
| weight | Bootstrap —is-a→ statistical technique — 0.9 — “Bootstrap is a statistical technique which enables estimation of confidence intervals by repeated resampling of a representative population 18 .” | concept_edges[7] |
| relation | patch-by-patch classification —contrasted-with→ fully convolutional networks — in contrast to — “In contrast to these two papers, which perform patch-by-patch classification, we use fully convolutional networks to obtain per-pixel cancer likelihood maps and segmentations in whole-slide images.” | concept_edges[8] |
| weight | patch-by-patch classification —contrasted-with→ fully convolutional networks — 0.8 — “In contrast to these two papers, which perform patch-by-patch classification, we use fully convolutional networks to obtain per-pixel cancer likelihood maps and segmentations in whole-slide images.” | concept_edges[8] |
| effect direction | prostate specific antigen —causes→ number of prostate biopsy sections — increases — “The number of prostate biopsy sections has strongly increased in the past decades due to the advent of prostate specific antigen (PSA) testing 16 .” — “has strongly increased” | concept_edges[9] |
| relation | prostate specific antigen —causes→ number of prostate biopsy sections — increased due to — “The number of prostate biopsy sections has strongly increased in the past decades due to the advent of prostate specific antigen (PSA) testing 16 .” | concept_edges[9] |
| weight | prostate specific antigen —causes→ number of prostate biopsy sections — 0.7 — “The number of prostate biopsy sections has strongly increased in the past decades due to the advent of prostate specific antigen (PSA) testing 16 .” | concept_edges[9] |
| replication | original | |
| limitation | Data come from a single center. — “One further limitation is the fact that we only investigated data from a single center.” | limitations[0] |
| limitation | False positives occur at tissue boundaries from tearing and deformation. — “In the prostate cancer experiment, some detection errors (i.e. false positive detections) still occur at the boundaries of the tissue, mostly due to tearing and tissue deformation.” | limitations[1] |
| limitation | Including isolated tumor cells drops performance to 0.74 AUC. — “When also including slides only containing ITC, performance drops to an area under the ROC curve of 0.74 and 0.02 specificity at 0.999 sensitivity.” | limitations[2] |
finding · quantity · variable · definition · key-term — grounded in the full text| finding | All positive slides were found while 30-40% of negative slides could be excluded. — “We found that all slides containing prostate cancer and micro- and macro-metastases of breast cancer could be identified automatically while 30–40% of the slides containing benign and normal tissue could be excluded without the use of any additional immunohistochemical markers or human intervention.” | content_tags.findings[0] |
| finding | Prostate slide-level AUC was 0.99 for median analysis. — “The average bootstrapped area under the ROC curve (AUC) for the median analysis was 0.99 (0.95–1.0) and 0.98 (0.94–0.99) for the 90 th -percentile analysis.” | content_tags.findings[1] |
| finding | 90-93% of micro- and macro-metastases were localised at one or two false positives per image. — “At the expense of one or two false positive detections per tumor-negative image 90% or 93% of all individual micro- and macro-metastases could be identified, respectively.” | content_tags.findings[2] |
| finding | Up to 32% of prostate and 44% of sentinel node negative slides could be excluded. — “For the sentinel lymph nodes, specificity was even higher at 44%, without missing any slide containing micro- or macro-metastases.” | content_tags.findings[3] |
| quantity | 0.99 (0.95–1.0) — — “The average bootstrapped area under the ROC curve (AUC) for the median analysis was 0.99 (0.95–1.0) and 0.98 (0.94–0.99) for the 90 th -percentile analysis.” | content_tags.quantities[0] |
| quantity | 90% / 93% — — “At the expense of one or two false positive detections per tumor-negative image 90% or 93% of all individual micro- and macro-metastases could be identified, respectively.” | content_tags.quantities[1] |
| quantity | 32% / 44% — — “For the sentinel lymph nodes, specificity was even higher at 44%, without missing any slide containing micro- or macro-metastases.” | content_tags.quantities[2] |
| quantity | 0.74 — — “When also including slides only containing ITC, performance drops to an area under the ROC curve of 0.74 and 0.02 specificity at 0.999 sensitivity.” | content_tags.quantities[3] |
| quantity | 60–70% — — “Furthermore, around 60–70% of the sentinel lymph nodes do not contain any metastases 17 .” | content_tags.quantities[4] |
| variable | AUC — “The average bootstrapped area under the ROC curve (AUC) for the median analysis was 0.99 (0.95–1.0) and 0.98 (0.94–0.99) for the 90 th -percentile analysis.” | content_tags.variables[0] |
| definition | Bootstrap — “Bootstrap is a statistical technique which enables estimation of confidence intervals by repeated resampling of a representative population 18 .” | content_tags.definitions[0] |
| definition | cancer likelihood map — “A cancer likelihood map (CLM), the output of the CNN indicating cancer likelihood per pixel, for a representative WSI from the test set with cancer covering 30% of the tissue area is shown in Fig. 2 .” | content_tags.definitions[1] |
| key-terms | deep learning, convolutional neural networks, histopathology, whole slide images, prostate cancer, sentinel lymph nodes, metastases, ROC, FROC, cancer likelihood map |
| concept | deep learning — the application of multi-layered artificial neural networks to a wide range of problems — “While ‘deep learning’ cannot be considered a single technique, it can roughly be described as the application of multi-layered artificial neural networks to a wide range of problems, from speech recognition to image analysis.” | concepts[0] |
| concept | H&E — hematoxylin and eosin — “Microscopic analysis of hematoxylin and eosin (H&E) stained sections has been the basis for cancer diagnosis and grading for the past century 1 .” | concepts[1] |
| concept | hematoxylin and eosin — not defined in this paper's own words; the standard histology stain — “Microscopic analysis of hematoxylin and eosin (H&E) stained sections has been the basis for cancer diagnosis and grading for the past century 1 .” | concepts[2] |
| concept | WSI — whole slide images — “Digital whole slide images (WSI) allow the application of image analysis techniques to aid pathologists in the examination and quantification of slides 5 .” | concepts[3] |
| concept | whole slide images — not defined in this paper's own words; high-resolution digitised glass slides — “Digital whole slide images (WSI) allow the application of image analysis techniques to aid pathologists in the examination and quantification of slides 5 .” | concepts[4] |
| concept | ROC — receiver-operator curve — “For prostate cancer detection the CNNs were evaluated on a per-slide level using receiver-operator curve (ROC)-analysis.” | concepts[5] |
| concept | receiver-operator curve — not defined in this paper's own words; the sensitivity-specificity trade-off curve — “For prostate cancer detection the CNNs were evaluated on a per-slide level using receiver-operator curve (ROC)-analysis.” | concepts[6] |
| concept | FROC — free-response ROC — “For the sentinel lymph node procedure, we assessed how well the system was capable of identifying individual micro- and macro-metastases using free-response ROC (FROC) analysis and if it is capable of excluding slides which do not contain any metastases using ROC analysis.” | concepts[7] |
| concept | free-response ROC — not defined in this paper's own words; ROC variant scoring localisation per false positive — “For the sentinel lymph node procedure, we assessed how well the system was capable of identifying individual micro- and macro-metastases using free-response ROC (FROC) analysis and if it is capable of excluding slides which do not contain any metastases using ROC analysis.” | concepts[8] |
| concept | ITC — isolated tumor cells — “For the second batch no on-slide annotations were available, only the per-case outcome (presence of macro- and/or micro-metastases and isolated tumor cells (ITC)).” | concepts[9] |
| concept | isolated tumor cells — not defined in this paper's own words; tumour cell clusters smaller than micro-metastases — “For the second batch no on-slide annotations were available, only the per-case outcome (presence of macro- and/or micro-metastases and isolated tumor cells (ITC)).” | concepts[10] |
| concept | CLM — the output of the CNN indicating cancer likelihood per pixel — “A cancer likelihood map (CLM), the output of the CNN indicating cancer likelihood per pixel, for a representative WSI from the test set with cancer covering 30% of the tissue area is shown in Fig. 2 .” | concepts[11] |
| concept | cancer likelihood map — the output of the CNN indicating cancer likelihood per pixel — “A cancer likelihood map (CLM), the output of the CNN indicating cancer likelihood per pixel, for a representative WSI from the test set with cancer covering 30% of the tissue area is shown in Fig. 2 .” | concepts[12] |
| concept | AUC — area under the ROC curve — “The average bootstrapped area under the ROC curve (AUC) for the median analysis was 0.99 (0.95–1.0) and 0.98 (0.94–0.99) for the 90 th -percentile analysis.” | concepts[13] |
| concept | area under the ROC curve — not defined in this paper's own words; the ROC summary statistic — “The average bootstrapped area under the ROC curve (AUC) for the median analysis was 0.99 (0.95–1.0) and 0.98 (0.94–0.99) for the 90 th -percentile analysis.” | concepts[14] |
| concept | Bootstrap — a statistical technique which enables estimation of confidence intervals by repeated resampling of a representative population — “Bootstrap is a statistical technique which enables estimation of confidence intervals by repeated resampling of a representative population 18 .” | concepts[15] |
| concept | statistical technique — not defined in this paper's own words; a method of statistical inference — “Bootstrap is a statistical technique which enables estimation of confidence intervals by repeated resampling of a representative population 18 .” | concepts[16] |
| concept | patch-by-patch classification — not defined in this paper's own words; classifying image patches independently — “In contrast to these two papers, which perform patch-by-patch classification, we use fully convolutional networks to obtain per-pixel cancer likelihood maps and segmentations in whole-slide images.” | concepts[17] |
| concept | fully convolutional networks — not defined in this paper's own words; CNNs producing per-pixel predictions — “In contrast to these two papers, which perform patch-by-patch classification, we use fully convolutional networks to obtain per-pixel cancer likelihood maps and segmentations in whole-slide images.” | concepts[18] |
| concept | prostate specific antigen — not defined in this paper's own words; the blood marker whose testing drives biopsies — “The number of prostate biopsy sections has strongly increased in the past decades due to the advent of prostate specific antigen (PSA) testing 16 .” | concepts[19] |
| concept | number of prostate biopsy sections — not defined in this paper's own words; how many prostate biopsy sections pathologists must read — “The number of prostate biopsy sections has strongly increased in the past decades due to the advent of prostate specific antigen (PSA) testing 16 .” | concepts[20] |
| area under the curve AUC | as written: area under the ROC curve (AUC) — “The average bootstrapped area under the ROC curve (AUC) for the median analysis was 0.99 (0.95–1.0) and 0.98 (0.94–0.99) for the 90 th -percentile analysis.” | content_tags.statistics[0] |
Full text — tagged
The reference list is not part of the text we hold for this paper. Our copy was extracted from the publisher's page and the bibliography did not survive that extraction, so the article below ends before its references. Nothing has been removed by us.
Deep learningterm·mth as a tool for increased accuracy and efficiency of histopathological diagnosis.
Abstract Pathologists face a substantial increase in workload and complexity of histopathologic cancer diagnosis due to the advent of personalized medicine. Therefore, diagnostic protocols have to focus equally on efficiency and accuracy. In this paper we introduce ‘deep learningterm·mth’ as a technique to improve the objectivity and efficiency of histopathologic slide analysis. Through two examples, prostate cancerterm·key identification in biopsy specimens and breast cancer metastasis detection in sentinel lymph nodesterm·key, we show the potential of this new methodology to reduce the workload for pathologists, while at the same time increasing objectivity of diagnoses. We found that all slides containing prostate cancerterm·key and micro- and macro-metastases of breast cancer could be identified automatically while 30–40% of the slides containing benign and normal tissue could be excluded without the use of any additional immunohistochemical markers or human intervention. We conclude that ‘deep learningterm·mth’ holds great promise to improve the efficacy of prostate cancerterm·key diagnosis and breast cancer staging.
Microscopic analysis of hematoxylin and eosinterm·mth (H&Eterm·mth) stained sections has been the basis for cancer diagnosis and grading for the past century 1 . Protocols for the complete workup of biopsies or resected tissue specimens, including microscopic analysis, exist for many of the most common cancer types (e.g. lung, breast, prostate). Use of these protocols has led to strong prognostic and widely used grading strategies (e.g. the Gleason grading system) 2 .
Due to the rise in cancer incidence and patient-specific treatment options, diagnosis and grading of cancer has become increasingly complex. Pathologists nowadays have to go over a large number of slides, often including additional immunohistochemical stains, to come to a complete diagnosis. Moreover, there is an increase in the amount of quantitative parameters pathologists have to extract for commonly used grading systems (e.g. lengths, surface areas, mitotic counts) 3 . Due to these difficulties, analysis protocols have been adapted and fine-tuned to offer the best balance between prognostic power and feasibility in daily clinical routine 4 .
The recent introduction of whole-slide scanning systems offers an opportunity to quantify and improve histopathologic procedures. These systems digitize glass slides with stained tissue sections at high resolution. Digital whole slide imagesterm·mth (WSIterm·mth) allow the application of image analysis techniques to aid pathologists in the examination and quantification of slides 5 . One such technique which has gained prominence in the last five years in other fields is ‘deep learningterm·mth’ 6 . While ‘deep learningterm·mth’ cannot be considered a single technique, it can roughly be described as the application of multi-layered artificial neural networks to a wide range of problems, from speech recognition to image analysis. In recent years, ‘deep learningterm·mth’ techniques have quickly become the state of the art in computer vision. A specific neural network subtype (convolutional neural networksterm·key; CNN 7 8 has become the de facto standard in image recognition and is approaching human performance in a number of tasks 6 . These systems function by learning relevant features directly from huge image databases (typically millions of images). This is in contrast to more traditional pattern recognition techniques, which strongly rely on manually crafted quantitative feature extractors.
In spite of these huge successes, ‘deep learningterm·mth’ techniques have not yet made a big impact on the field of medical imaging. One of the main reasons is that for the traditional imaging based specialties (e.g. radiology) the large numbers of images that are needed to train complex ‘deep learningterm·mth’ systems are not readily available. In digital histopathologyterm·key this is easier: one WSIterm·mth typically contains trillions of pixels from which hundreds of examples of cancerous glands (in the case of prostate or breast cancer) can be extracted.
Some initial work has been published over the last five years discussing the application of ‘deep learningterm·mth’ techniques to microscopic and histopathologic images. Ciresan et al . were the first to apply convolutional neural networksterm·key to the task of mitosis counting for primary breast cancer grading 9 . Furthermore, in a different publication, they showed the applicability of patch-driven convolutional neural networksterm·key to segmentation tasks 10 . Wang et al . later expanded the work on mitosis detection by combining hand-crafted features and convolutional neural networksterm·key 11 . Other applications of convolutional networks include primary breast cancer detection 12 , glioma grading 13 and epithelium and stroma segmentation 14 . Last, Su et al . used another ‘deep learningterm·mth’ technique, called stacked denoising auto-encoders to perform cell detection and segmentation in lung cancer and brain tumors 15 .
This study investigates the general applicability of CNNs to improve the efficiency of cancer diagnosis in H&Eterm·mth images by applying it to two novel tasks: the detection of prostate cancerterm·key in biopsy specimens and the detection of breast cancer metastasesterm·key in resected sentinel lymph nodesterm·key.
The number of prostate biopsy sectionsterm·dom has strongly increased in the past decades due to the advent of prostate specific antigenterm·dom (PSA) testing 16 . Because of the nature of the standard biopsy procedure (eight to twelve random biopsies under ultrasound-guidance), each procedure results in several slides. The majority of these slides typically do not contain cancer. The histopathological analysis could be streamlined significantly if these normal slides could automatically be excluded without expelling any slides containing cancer. We collected consecutive single-center biopsy specimens of 254 patients who underwent MR-guided biopsy procedures for prostate cancerterm·key at our institution. These specimens were prepared according to standard histopathologic protocol and subsequently digitized using an Olympus VS120-S5 system (Olympus, Tokyo, Japan).
The sentinel lymph nodeterm·key procedure is well known for its tedious inspection protocol 4 . Several sections of the lymph node have to be investigated for micro-metastases (0.2–2 mm) and macro-metastases (>2 mm). Furthermore, around 60–70% of the sentinel lymph nodesterm·key do not contain any metastasesterm·key 17 . In this paper we focus on the sentinel lymph nodeterm·key procedure for breast cancer with the aim of identifying slides which do not contain micro- or macro-metastases. Also, we tried to identify the correct location of the metastasesterm·key within a specific slide. In total 271 patients were included from our institution. Specimens were prepared according to the standard histopathologic protocol and subsequently digitized using a 3DHistech Pannoramic 250 Flash II scanner (3DHistech, Budapest, Hungary).
After digitization of the H&E-stained slides cancer and metastasesterm·key were manually delineated using a computer mouse by a resident of pathology (I.K., prostate cancerterm·key experimentterm·mth) and a lab technician (M.H., sentinel lymph nodeterm·key experimentterm·mth), under the supervision of experienced pathologists (C. A. H. K., P. B.). From these annotated areas small prototype image regions (‘patches’) were extracted to train CNNs to detect cancer areas in validation data sets (schematic overview in Fig. 1 ). These validation data sets were used to optimize the network parameters. After training, the CNN was converted to a fully convolutional networkterm·mth which gave per-pixel predictions on the presence of cancer and metastasesterm·key in separate, not previously used, test data sets. For prostate cancerterm·key detection the CNNs were evaluated on a per-slide level using receiver-operator curveterm·mth (ROCterm·mth)-analysis. We also investigated how well the system could exclude slides without cancer from further diagnostic processing. For the sentinel lymph nodeterm·key procedure, we assessed how well the system was capable of identifying individual micro- and macro-metastases using free-response ROCterm·mth (FROCterm·mth) analysis and if it is capable of excluding slides which do not contain any metastasesterm·key using ROCterm·mth analysis.
Results Subjects Prostate cancerterm·key From the initial set of 254 patients, eleven were excluded because the glass slides were not available. Four were excluded because no biopsy was taken during the procedure and one was excluded as the tissue sample was too small for pathologic analysis. Out of the remaining 238, we randomly selected 225 glass slides for digitization, of which 100 were assigned to the training set, 50 to the validation set and 75 to the test set. The training set sampling was stratified such that a near-50/50-distribution between slides containing cancer and slides not containing cancer was obtained. All slides were successfully digitized and annotated. Further details on the selected slides can be found in Table 1 .
Breast cancer sentinel lymph nodesterm·key Data collection for the sentinel lymph nodeterm·key experimentsterm·mth was performed in two batches. The first batch was obtained by including 173 slides from the case files of an experienced breast pathologist (P.B). These initial slides were split into a training (98), validation (33) and test (42) set. These slides were subsequently digitized and every metastasis was annotated. To make sure our results were not biased to a single pathologist’s case selection, we acquired a second set of data by including all the consecutive sentinel lymph nodeterm·key cases for breast cancer from October 2014 to April 2015, resulting in an additional 98 whole-slide images. For the second batch no on-slide annotations were available, only the per-case outcome (presence of macro- and/or micro-metastases and isolated tumor cellsterm·dom (ITCterm·dom)). Further details on the included cases can be found in Table 2 . Of cases with only ITCterm·dom, 22 out of 24 had additional immunohistochemistry ordered by the pathologist.
Prostate cancerterm·key detection A cancer likelihood mapterm·mth (CLMterm·mth), the output of the CNN indicating cancer likelihood per pixel, for a representative WSIterm·mth from the test set with cancer covering 30% of the tissue area is shown in Fig. 2 . The cancerous glands indicated by the pathologist’s outline (in magenta) are correctly identified with high likelihood. The stroma within the annotation areas is correctly identified as a low cancer likelihood region (in green, most easily identifiable in the high-resolution sub-images).
Several other examples are presented in Fig. 3 . In Fig. 3b , we show a high-resolution sub-image of a false positive region. Due to cutting and histopathologic processing, tissue at the edges of the biopsy specimens often deforms and tears, resulting in abnormal appearance. If we examine this area closely, we can see that the false positive glands indeed show some features which are comparable to those of cancer (e.g. fusing glands, irregular shape). In general, we can clearly see a distinct separation between malignant ( Figs 2 and 3a ) and benign biopsy specimens ( Fig. 3b,c ) based on the CLMsterm·mth.
Quantitatively, the result of performing histogram analysis on the CLMsterm·mth can be best represented using ROCterm·mth analysis. In Fig. 4 we present the ROCterm·mth curves for both median and 90 th -percentile analysis of the cumulative histogram of the CLMterm·mth over the independent test set. Indicated with the dashed lines are the raw ROCterm·mth curves, the solid line and the shaded areas represent the mean bootstrapped ROCterm·mth curve and the 95 th -percentiles. Bootstrapterm·mth is a statistical techniqueterm·mth which enables estimation of confidence intervals by repeated resampling of a representative population 18 . The average bootstrapped area under the ROC curve (AUC)term·stat for the median analysis was 0.99 (0.95–1.0) and 0.98 (0.94–0.99) for the 90 th -percentile analysis. However, the 90 th -percentile analysis has higher specificity at a sensitivity level of 0.999 (0.32, 95%-CI: 0.29–0.97) compared to median analysis (0.17, 95%-CI: 0.15–1.0).
Identification of breast cancer metastasesterm·key in sentinel lymph nodesterm·key Representative examples of sentinel lymph nodeterm·key specimens are presented in Fig. 5 (without metastasesterm·key) and Fig. 6 (with metastasesterm·key). The metastasesterm·key are correctly detected with very high likelihood (red color). Areas containing only lymphocytes are mostly transparent (likelihood close to zero), whereas areas containing histiocytes or mixtures of histiocytes and lymphocytes are transparent – green (low likelihoods). Distinction between histiocyte–rich regions and metastasesterm·key is also a well-known difficulty for residents in pathology.
Quantitatively, results were analyzed in two ways. FROCterm·mth analysis was used to assess localization accuracy, whereas ROCterm·mth analysis was used to assess performance at the slide level. FROCterm·mth analysis was only performed on the test set, as annotations are required to assess localization accuracy. The FROCterm·mth and ROCterm·mth curves are shown in Fig. 7 . Results of the FROCterm·mth and ROCterm·mth analyses are also summarized in Table 3 . At the expense of one or two false positive detections per tumor-negative image 90% or 93% of all individual micro- and macro-metastases could be identified, respectively. If we also include all isolated tumor cellterm·dom (ITCterm·dom) instances, 71% was found at the expensive of one false positive detection per tumor-negative image and 74% was found at the expense of two false positive detections per image.
The ROCterm·mth analysis shows that in both the test and consecutive data sets, an area under the ROC curveterm·mth of close to 0.90 can be obtained on the slide level when discriminating slides without from slides with micro- and macro-metastases. Furthermore, at 0.999 sensitivity up to 0.44 specificity could be obtained in the consecutive set. When also including slides only containing ITCterm·dom, performance drops to an area under the ROC curveterm·mth of 0.74 and 0.02 specificity at 0.999 sensitivity.
Discussion Although deep learningterm·mth is an active research field, the application of deep learningterm·mth to histopathologyterm·key is relatively new. Most already published work has focused on the detection of mitotic figures 9 11 or identification and segmentation of individual cells 15 19 . One paper used a convolutional auto-encoder to segment basal-cell carcinoma in H&E-images of breast cancer 20 .However, this model is only evaluated on images from pre-selected regions of interest and not on whole slides, making it difficult to assess its practical value.
The two papers most closely related to our work have focused on different entities. Cruz-Roa et al . used a CNN to detect and segment primary breast cancer 12 and Ertosun et al . investigated the grading of gliomas 13 . We explored the applicability of CNNs to digitized histopathologyterm·key through two different experimentsterm·mth: prostate cancerterm·key detection in H&E-stained biopsy specimens and identification of metastasesterm·key in sentinel lymph nodesterm·key obtained from breast cancer patients. In contrast to these two papers, which perform patch-by-patch classificationterm·mth, we use fully convolutional networksterm·mth to obtain per-pixel cancer likelihood mapsterm·mth and segmentations in whole-slide images. Furthermore, we are the first to report slide-level accuracies for cancer detection.
In both experimentsterm·mth we were able to successfully train convolutional neural networksterm·key, although the amount of case data was less than what is generally typical in ‘deep learningterm·mth’ experimentsterm·mth. The fact that we performed extensive data augmentation and boosting in combination with the relatively limited domain (i.e. H&E-stained histopathologic images compared to natural images) made this possible.
In both applications we investigated whether it was possible to identify slides not containing disease without overlooking any slides containing disease. In the prostate cancerterm·key slides, up to 32% of the slides not containing disease could be excluded. For the sentinel lymph nodesterm·key, specificity was even higher at 44%, without missing any slide containing micro- or macro-metastases. This indicates that substantial gains in efficiency are possible by using CNNs to exclude tumor-negative slides from further human analysis.
Next to the performance of the CNN at high sensitivity, area under the ROC curveterm·mth was also high in both cases, with an AUCterm·mth of 0.99 for the prostate cancerterm·key experimentterm·mth (median analysis) and 0.88 for the sentinel lymph nodeterm·key experimentterm·mth (consecutive set). Furthermore, localization accuracy was high for micro- and macro-metastases in the sentinel lymph nodeterm·key experimentterm·mth (90% sensitivity at 1 false positive per normal image).
There are some limitations to the application of the CNNs, especially for the sentinel lymph nodesterm·key. Although the accuracy of detecting micro- and macro-metastases is high, adding the requirement of having to identify all clusters of isolated tumor cellsterm·dom lowers performance significantly (0.74 AUCterm·mth for the consecutive set). However, the importance of ITCsterm·dom is debated. Some have found no prognostic implication of ITCsterm·dom 4 21 at all or when the ITCsterm·dom are visible through immunohistochemistry only 15 16 . Others did find ITCsterm·dom having a negative prognostic impact, albeit effect sizes differ 22 23 24 . However, for the clinical application of CNNs this is of limited importance. If the application of the CNNs can detect the micro- and macro-metastases with high accuracy, and we have shown this, the ITCsterm·dom can be detected by immunohistochemistry, without having a pathologist looking at the H&Eterm·mth stained slides. In The Netherlands, according to the national guideline for breast cancer, immunohistochemistry is mandatory when no tumor is found in the H&E-stained slides.
In the prostate cancerterm·key experimentterm·mth, some detection errors (i.e. false positive detections) still occur at the boundaries of the tissue, mostly due to tearing and tissue deformation. These are expected artifacts that occur during histopathologic processing, and ideally our CNN would be robust to this. However, due to the fact that these artifacts can have a wide range of appearances and only occur sporadically, this is not yet the case with the size of the training set used in this study. For the slide level analysis, these spurious detections are not problematic; they occur equally in slides containing and not containing cancer, making their separation still possible.
One further limitation is the fact that we only investigated data from a single center. Although we included data from distinctly different tissue types and used digitization equipment from two different vendors, it is important that these results are confirmed in future, multi-center studies.
As far as the authors are aware, this is the first paper describing the general applicability of a ‘deep learningterm·mth’ technique to the diagnostic analysis of whole slide imagesterm·mth of sentinel lymph nodesterm·key and prostate biopsies. We have shown that this technique is potentially highly suitable to improve the efficiency of the diagnostic process in histopathologyterm·key. This could in turn lead to adapted protocols, where pathologists perform a more detailed analysis on the difficult samples, as the easy samples are already handled by a computer system.
Although we specifically looked at clinical diagnosis in this study, the potential of these ‘deep learningterm·mth’ techniques reaches further. They could also be used to quickly analyze huge clinical trial databases to extract relevant cases, or automatically annotate areas of disease to allow fast quantification (e.g. area, diameter). Furthermore, the technique is not limited to H&E-stained images and could readily be applied to immunohistochemistry, which might be of interest when researching the efficacy of drugs or the expression of genes. Both are worthwhile avenues for future research.
Methods Materials For all patients in this study the institutional review board waived the need for informed consent.
Prostate cancerterm·key A search was performed on our institution’s PACS system to identify all patients who underwent MR-guided biopsy of the prostate after an initial multi-parametric MRI suspicious for cancer in 2012. This resulted in a total of 254 patients who were initially included in this study. After assessing suitability for inclusion, patients were randomly assigned to one of three sets: training, validation and test.
The biopsy specimens were previously stained using standard protocols for H&E-staining used in routine clinical care, after which the specimens were analyzed and reported on by an experienced pathologist as part of routine diagnostics. For this study these glass slides were obtained from the archive of the Department of Pathology for subsequent digitization.
Sentinel lymph nodeterm·key The sentinel lymph nodeterm·key case files of an experienced pathologist (P.B.) were used to identify subjects to include in the initial training set. The sentinel node specimens were previously stained using standard protocols for H&E-staining used in routine clinical care, after which the specimens were analyzed and reported by the experienced pathologist as part of routine diagnostics.
To make sure our results were not biased incidentally by only evaluating on the cases obtained from one pathologist’s case file, we included all sentinel lymph nodeterm·key subjects from October 2014–April 2015 as an extra testing set.
One H&E-stained slide per subject was subsequently obtained from the archive at the Department of Pathology for digitization. Slides were selected such that they contained the largest area of metastasesterm·key, if metastasesterm·key were present.
Digitization and annotation Prostate cancerterm·key Prostate cancerterm·key slides were digitized using an Olympus VS120-S5 slide scanning system. Slides were digitized using a 40× objective (resultant pixel resolution of 0.16 microns). After digitization the digital slides were annotated for cancer using an in-house developed freehand drawing tool. Annotation was performed by a resident of pathology (I.K.) under the supervision of an experienced pathologist (C.H.-v.d.K.). Sometimes two consecutive sections were included on one glass slide. In those cases, only one of the two sections was annotated and the other excluded from further analysis.
Sentinel lymph nodeterm·key Sentinel lymph nodeterm·key slides were digitized using a 3DHistech Pannoramic 250 Flash II slide scanner. Slides were digitized using a 20× objective (resultant pixel resolution of 0.24 microns). After digitization the slides were annotated using the Aperio ImageScope software using a freehand drawing tool. Annotation was performed by a lab technician (M.H.). Annotations were subsequently checked by an experienced pathologist for correctness and completeness (P.B.).
The extra testing set was not annotated, only the pathologist’s report was available for this set.
Pre-processing steps The annotations were used to generate binary mask images at the same resolution as the original slides. Any pixel inside an annotated region was labeled as cancer (label 1), whereas all other regions were left blank (label 0).
In addition to the binary annotation mask, we also generated binary tissue masks to separate background from tissue. To this end we performed a simple thresholding procedure on the optical density of the RGB channels. Optical density of a channel is obtained through: Here OD c is the optical density of the channel c (Red, Green or Blue), I is the intensity of the channel and I max is the maximum intensity, which is 255 due to 8-bit quantization. By thresholding the optical densities at 0.2, all background could be removed resulting in a binary mask where tissue is labeled 1 and background is labeled 0.
Convolutional neural networkterm·key training and application To train the convolutional neural networkterm·key we made use of the open-source ‘deep learningterm·mth’ libraries Theano 0.7 and pylearn2 0.1 25 26 .
As it is impossible to feed entire whole-slide images to the network at once, we randomly extracted small patches from the whole-slide image for training. Whole-slide results can then be obtained by applying the network to every pixel in the image.
Patch size in pixels was determined empirically during initial experimentsterm·mth. We tried 64 × 64, 128 × 128 and 256 × 256 pixel patches. The 64 × 64 sized patches performed substantially worse on patch-based accuracy and 256 × 256 sized patches limited convolutional networks depth due to memory limitations of the GPU. As such, we settled on a patch size of 128 × 128.
To allow the network to learn the appearance of cancer (prostate or metastasized breast cancer in this paper), it is important that the small patches contain enough information to allow discrimination between patches with and without cancer. In contrast, when one selects patches which are too large, it is harder for the network to identify the relevant discriminatory features. As such, the correct resolution at which to extract the information had to be determined.
For the prostate cancerterm·key experimentterm·mth, guidelines suggest that initial inspection of biopsy specimens should be performed using a microscope with a 5× objective, already indicating that this is a suitable resolution. Subsequently, we confirmed this in a small observer experimentterm·mth where an untrained researcher was able to separate cancer and non-cancer patches at 5× (equivalent resolution 0.60 microns per pixel). As such, for this experimentterm·mth patches were extracted at 5× objective magnification. For the sentinel lymph nodeterm·key slides a similar strategy was used, and here a 10× objective magnification was selected (0.48 microns per pixel).
Patches of 128 × 128 pixels were extracted at the determined resolution level from both cancerous regions and non-cancerous tissue regions using the masks obtained in the pre-processing stage. Extraction was performed such that an equal number of patches for both classes was obtained. During patch extraction, patches were rotated (0, 90, 180 and 270 degrees) and subsequently flipped to make sure the network would not learn rotation-dependent features. This resulted in eight variations of each single patch and was performed for both positive and negative samples. In total this resulted in a total of 920,000 patches for the prostate cancerterm·key experimentterm·mth and 2.6 million patches for the lymph node experimentterm·mth. Patch-extraction was performed for both the training and the validation sets in both experimentsterm·mth. Test sets in both experimentsterm·mth were only used for final evaluation and were untouched during the training procedure.
After patch extraction, a convolutional neural networkterm·key was trained where the performance of the network was monitored by assessing the miss-classification rate on the validation set patches. Training was stopped when validation set error did not improve for five epochs. Network structure (e.g. number of layers, filters per layer, number of nodes in fully connected layers) and parameters (e.g. learning rate, momentum) were continuously tweaked to obtain maximum performance on the validation set. In the end, very similar network structures and parameter settings were obtained for both experimentsterm·mth. Training time per epoch was around 80 and 200 minutes per epoch for the prostate cancerterm·key and lymph node experimentsterm·mth respectively (GeForce GTX970). Optimal performance was reached after 5 and 12 epochs respectively. The full network specifications can be found in the supplementary files .
After network training, the CNN was converted to a fully convolutional networkterm·mth to allow fast application to the whole slide imageterm·mth 27 . Applying this fully convolutional networkterm·mth to the whole slide imageterm·mth resulted in a likelihood map where each pixel has a continuous likelihood between 0 and 1 of containing cancer. The likelihood map generation time of a prostate biopsy slide is between 5–10 minutes and that of a lymph node slide between 30–40 minutes. However, the average time to analyze a slide is highly dependent on the amount of tissue on the slide. Due to the nature of the procedures, prostate biopsies often contained much less tissue and thus were quicker to process.
Due to the nature of the sentinel lymph nodeterm·key slides, where often only a tiny fraction of the slide contains cancer and most of the slide is covered by lymphocytes, certain normal regions which look more similar to cancer are typically underrepresented in the training data. In other words, the network is not capable of correctly identifying these areas as normal. To tackle this problem, we choose to use a boosting approach similar to the one used by Ciresan et al . 9 We used the initial likelihood maps obtained for the training data set to sample new patches for both the cancerous and non-cancerous class. However, this time we increased the likelihood of adding a patch to the training data if the center pixel of the patch was initially classified wrongly by the network. This process results in additional training data which contains more difficult samples. Subsequently, we re-trained the network on the old and enriched patches and obtained the final likelihood maps for the sentinel lymph nodeterm·key slides.
This boosting procedure was not applied to the prostate cancerterm·key detection experimentterm·mth, as in biopsy slides the area covered by cancer and the area covered by normal tissue is more balanced.
Evaluation Prostate cancerterm·key A normalized cumulative histogram is calculated on the final whole-slide likelihood image. The histogram was constructed with 100 bins, equally spaced between 0 and 1. In cases where there is no cancer in the slide, the histogram will rise quickly, whereas in cancer cases, the histogram will rise more slowly. To move from this histogram to a slide level label, we used percentile analysis. A percentile is selected and then we inspect at which likelihood we reach this percentile. By doing this for all cases and the performing receiver-operating characteristic (ROCterm·mth) analysis given this likelihood, we can assess CNN performance at the slide level. We used the validation data set to obtain the optimal percentiles with a step size of 10 for both overall area under the ROC curveterm·mth and highest specificity at 0.999 sensitivity, which were the median and the 90 th -percentile respectively.
To assess the confidence interval of the area under the ROC curveterm·mth, we used bootstrapping using 10,000 bootstrapterm·mth samples 18 .
Sentinel lymph nodeterm·key The final likelihood map for the sentinel lymph nodeterm·key slides is first thresholded at a likelihood of 0.3 to get rid of all the low likelihood false positives. Subsequently, we performed connected component analysis to obtain all the detected lesions. All components with a diameter smaller than 0.02 mm (10% of the minimum diameter for micro-metastases) are subsequently removed to get rid of spurious detection caused by artifacts (e.g. dust, tissue deformation). To obtain a likelihood per component, the median likelihood across all pixels within each component is calculated.
Given the component segmentations and likelihoods, we perform free-receiver operating characteristic (FROCterm·mth) at the metastasis level and ROCterm·mth analysis at the slide level on the test set. On the consecutive set we only performed ROCterm·mth analysis as we did not have annotations for this set. In FROCterm·mth analysis we assess for each individual metastasis, being macro-metastases, micro-metastases or ITCterm·dom, whether it is detected. A metastasis is counted as detected when a component segmentation has a Dice coefficient of at least 0.5 with the annotation. All components which do not coincide with a metastasis are considered false positive detections. The component likelihood is subsequently used to generate the FROCterm·mth curve. The sensitivity in the FROCterm·mth curve is expressed with respect to false positive detections in metastasis-free slides.
For ROCterm·mth analysis, the component with the highest likelihood is used as the slide level score, as this is the component that is actionable. These slide likelihoods were subsequently used to construct the ROCterm·mth curves at the slide level.
All analysis for the sentinel lymph nodeterm·key cases were performed twice: once assuming ITCsterm·dom are not considered metastasesterm·key (i.e. they are ignored) and once assuming ITCsterm·dom are metastasesterm·key and as such have to be detected.
Additional Information How to cite this article : LitjenspersName, G. et al . Deep learningterm·mth as a tool for increased accuracy and efficiency of histopathological diagnosis. Sci. Rep. 6 , 26286; doi: 10.1038/srep26286 (2016).
Supplementary Material