| phenomenon | deep learning in medical image analysis— “Deep learning algorithms, in particular convolutional networks, have rapidly become a methodology of choice for analyzing medical images.” |
| claim-type | existence— “From the 308 papers reviewed in this survey, it is evident that deep learning has pervaded every aspect of medical image analysis.” |
| method — evidence | — “This paper reviews the major deep learning concepts pertinent to medical image analysis and summarizes over 300 contributions to the field, most of which appeared in the last year.” |
| transfer | from artificial intelligence (computer vision / deep learning): deep convolutional networks, developed in computer vision, adopted as the methodology of choice for medical image analysis (classification, detection, segmentation)— “Deep learning algorithms, in particular convolutional networks, have rapidly become a methodology of choice for analyzing medical images” | transfers[0] |
| mechanism | THE SAME TECHNICAL PROBLEM IS REPORTED TO HAVE DIFFERENT SEVERITY IN TWO FIELDS, AND THE CONTRAST IS DRAWN EXPLICITLY: EVEN WHEN DATA IS ANNOTATED BY DOMAIN EXPERT, LABEL NOISE CAN BE A SIGNIFICANT LIMITING FACTOR - WHEREAS IN COMPUTER VISION THE NOISE IN THE LABELING OF IMAGES IS TYPICALLY RELATIVELY LOW. The word `label` names something with a different reliability depending on who produced it— “Even when data is annotated by domain expert, label noise can be a significant limiting factor in developing algorithms, whereas in computer vision the noise in the labeling of images is typically relatively low.” | mechanisms[0] |
| mechanism | A REFERENCE STANDARD IS BUILT WITH CONSENSUS DELIBERATELY NOT FORCED, AND THE DISAGREEMENT THEN OUTNUMBERS THE AGREEMENT THREE TO ONE: nodules were annotated by four radiologists INDEPENDENTLY, the readers REVIEWED EACH OTHERS ANNOTATIONS BUT NO CONSENSUS WAS FORCED - and the number they DID NOT UNANIMOUSLY AGREE ON to be a nodule WAS THREE TIMES LARGER THAN THE NUMBER THEY DID FULLY AGREE ON— “To give an example, a widely used dataset for evaluating image analysis algorithms to detect nodules in lung CT is In this the LIDC-IDRI dataset (Armato et al., 2011). dataset pulmonary nodules were annotated by four radiologists independently. Subsequently the readers reviewed each others annotations but no consensus was forced. It turned… | mechanisms[1] |
| mechanism | THE CONSEQUENCE FOR ANYTHING TRAINED ON IT IS STATED, AND THE FIX IS AN OPEN PROBLEM: training a system on such data REQUIRES CAREFUL CONSIDERATION OF HOW TO DEAL WITH NOISE AND UNCERTAINTY IN THE REFERENCE STANDARD - one COULD THINK OF solutions like incorporating labeling uncertainty directly in the loss function, BUT THIS IS STILL AN OPEN CHALLENGE. The uncertainty is acknowledged and has nowhere to go— “Training a deep learning system on such data requires careful consideration of how to deal with noise and uncertainty in the reference standard. One could think of solutions like incorporating labeling uncertainty directly in the loss function, but this is still an open challenge.” | mechanisms[2] |
| mechanism | A BINARY PRESENTATION OF THE TASK IS CALLED A GROSS SIMPLIFICATION, AND THE REASON IS HETEROGENEITY ON BOTH SIDES: classification is often presented as A BINARY TASK - normal versus abnormal, object versus background - HOWEVER THIS IS OFTEN A GROSS SIMPLIFICATION AS BOTH CLASSES CAN BE HIGHLY HETEROGENEOUS. Not one class hiding structure but both— “In medical imaging often classification or segmentation is presented as a binary task: normal versus abnormal, object versus background. However, this is often a gross simplification as both classes can be highly heterogeneous.” | mechanisms[3] |
| mechanism | THE `NORMAL` CATEGORY IS SHOWN TO CONTAIN THINGS THAT ARE NOT NORMAL, INCLUDING ARTIFACTS OF THE INSTRUMENT: the normal category OFTEN CONSISTS OF COMPLETELY NORMAL TISSUE **BUT ALSO** several categories of BENIGN FINDINGS, WHICH CAN BE RARE, and MAY OCCASIONALLY INCLUDE A WIDE VARIETY OF IMAGING ARTIFACTS. The residual class is defined by what it is not, and collects the measurement's own failures— “For example, the normal category often consists of completely normal tissue but also several categories of benign findings, which can be rare, and may occasionally include a wide variety of imaging artifacts.” | mechanisms[4] |
| mechanism | THE PERFORMANCE PATTERN IS PREDICTED FROM THE CATEGORY'S STRUCTURE: systems end up GOOD AT EXCLUDING THE MOST COMMON NORMAL SUBCLASSES BUT FAIL MISERABLY ON SEVERAL RARE ONES. A single accuracy figure over a heterogeneous class hides exactly the failures that matter, and the shape of the class predicts the shape of the failure— “good at excluding the most common normal subclasses, but fail miserably on several rare ones.” | mechanisms[5] |
| mechanism | THE CORRECT FIX IS NAMED AND DECLINED ON COST, IN ONE SENTENCE: a STRAIGHTFORWARD SOLUTION would be to turn the system into a MULTI-CLASS one by providing DETAILED ANNOTATIONS OF ALL POSSIBLE SUBCLASSES - OBVIOUSLY THIS AGAIN COMPOUNDS THE ISSUE OF LIMITED AVAILABILITY OF EXPERT TIME FOR ANNOTATING AND IS THEREFORE OFTEN SIMPLY NOT FEASIBLE. The right answer is known and unaffordable— “A straightforward solution would be to turn the deep learning system in a multi-class system by providing it with detailed annotations of all possible subclasses. Obviously this again compounds the issue of limited availability of expert time for annotating and is therefore often simply not feasible.” | mechanisms[6] |
| mechanism | THE WORKAROUNDS ARE REPORTED TO FAIL FOR THE SAME REASON THE PROBLEM EXISTS: strategies that incorporate intelligence in the training process - selective sampling, hard negative mining - TYPICALLY FAIL WHEN THERE IS SUBSTANTIAL NOISE IN THE REFERENCE STANDARD - and ADDITIONAL METHODS FOR DEALING WITH WITHIN-CLASS HETEROGENEITY WOULD BE HIGHLY WELCOME. The noise in the labels defeats the corrections for the labels— “However, such strategies typically fail when there is substantial noise in the reference standard. Additional methods for dealing with within-class heterogeneity would be highly welcome.” | mechanisms[7] |
| mechanism | A STAIN IS OFFERED AS A SUBSTITUTE FOR EXPERTISE: in histopathology one can sometimes use SPECIFIC IMMUNOHISTOCHEMICAL STAINS to highlight regions of interest, REDUCING THE NEED FOR EXPERT EXPERIENCE - and non-expert labels can be leveraged VIA CROWD-SOURCING. Two ways to replace a judgement: a reagent and a crowd— “One can also consider leveraging non-expert labels via crowd-sourcing (Rajchl et al., 2016b). Other potential solutions can be found within the medical field itself; in histopathology one can sometimes use specific immunohistochemical stains to highlight regions of interest, reducing the need for expert experience” | mechanisms[8] |
| mechanism | A BENCHMARK IS REPORTED TO HAVE SATURATED, SO THAT IMPROVEMENTS CAN NO LONGER BE ATTRIBUTED: since 2014 the performance on the benchmark HAS SATURATED and IT IS DIFFICULT TO ASSESS WHETHER THE SMALL INCREASES IN PERFORMANCE CAN REALLY BE ATTRIBUTED TO 'BETTER' AND MORE SOPHISTICATED ARCHITECTURES. The scare quotes on `better` are the paper's, and the measuring instrument has stopped discriminating— “Since 2014, the performance on the ImageNet benchmark has saturated and it is difficult to assess whether the small increases in performance can really be attributed to ’better’ and more sophisticated architectures.” | mechanisms[9] |
| mechanism | A PERFORMANCE CRITERION IS DECLARED INSUFFICIENT AND SUPPLEMENTED BY AN ARTICULATION CRITERION, JUSTIFIED BY LEGAL ACCOUNTABILITY: these methods have often been described as BLACK BOXES, and especially in medicine WHERE ACCOUNTABILITY IS IMPORTANT AND CAN HAVE SERIOUS LEGAL CONSEQUENCES, IT IS OFTEN NOT ENOUGH TO HAVE A GOOD PREDICTION SYSTEM - the system ALSO HAS TO BE ABLE TO ARTICULATE ITSELF in a certain way. What counts as success acquires a second component from outside the field— “Finally, deep learning methods have often been described as ‘black boxes’. Especially in medicine, where accountability is important and can have serious legal consequences, it is often not enough to have a good prediction system. This system also has to be able to articulate itself in a certain way.” | mechanisms[10] |
| mechanism | A SURVEY DE-DUPLICATES ITS OWN POPULATION BY A JUDGEMENT IT NEVER DEFINES, AND SUPPLIES A WORKED EXAMPLE INSTEAD OF A CRITERION: when largely overlapping work had been reported in multiple publications, ONLY THE PUBLICATION DEEMED MOST IMPORTANT WAS INCLUDED - A TYPICAL EXAMPLE HERE WAS ARXIV PREPRINTS THAT WERE SUBSEQUENTLY PUBLISHED, or conference contributions later expanded into journal papers. What `most important` means is stated nowhere, the survey's whole 308-paper population rests on it, and the example is doing the work a definition would have done— “When largely overlapping work had been reported in multiple publica- tions, only the publication deemed most important was included. A typical example here was arXiv preprints that were subsequently published or conference contri- butions which were expanded and published in journals.” | mechanisms[11] |
| builds-on | convolutional neural networks (deep learning)— “convolutional networks, have rapidly become a methodology of choice for analyzing medical images” | claim_relations[0] |
| extends | Shen et al. (2017), the earlier dedicated review of deep learning in medical image analysis— “Although they cover a substantial amount of work, we feel that important areas of the field were not represented.” | claim_relations[1] |
| sense | the reference standard - AN ANNOTATION SET WITH ITS DISAGREEMENT LEFT IN: four radiologists annotated independently, reviewed each other, and NO CONSENSUS WAS FORCED, so what every accuracy on this dataset is measured against is a set whose disputed part is three times its agreed part - the standard is not a ground truth but a record of expert disagreement— “To give an example, a widely used dataset for evaluating image analysis algorithms to detect nodules in lung CT is In this the LIDC-IDRI dataset (Armato et al., 2011). dataset pulmonary nodules were annotated by four radiologists independently. Subsequently the readers reviewed each others annotations but no consensus was forced. It turned… | senses[0] |
| sense | label noise - A FIELD-RELATIVE QUANTITY: the survey contrasts medical imaging, where it can be a significant limiting factor even with domain-expert annotation, against computer vision, where it is typically relatively low - so the same word names a small nuisance in one literature and a limiting factor in the other, and the architectures move between them— “Even when data is annotated by domain expert, label noise can be a significant limiting factor in developing algorithms, whereas in computer vision the noise in the labeling of images is typically relatively low” | senses[1] |
| sense | the binary task - A STATED SIMPLIFICATION: normal versus abnormal is called a gross simplification because both classes are highly heterogeneous, which is why systems can excel on common normal subclasses and fail on rare ones; the two-class label is a choice about the data, not a property of it— “In medical imaging often classification or segmentation is presented as a binary task: normal versus abnormal, object versus background. However, this is often a gross simplification as both classes can be highly heterogeneous” | senses[2] |
| control | pre-stated exclusion of papers not reporting results on medical image data or using only standard feed-forward networks with handcrafted features, plus de-duplication where the same work appeared in several publications — held — rules out a survey padded by work that is not deep learning on medical images, and double-counting of one piece of work reported more than once— “We excluded papers that did not report results on medical image data or only used standard feed-forward neural networks with handcrafted features. When overlapping work had been reported in multiple publications, only the publication(s) deemed most important were included.” | controls[0] |
| outcome (stated by the tagger, no quotation) | control — held | controls[0] |
| for which use | limitation — the survey's own account of the field: it offers A CRITICAL DISCUSSION OF OPEN CHALLENGES AND DIRECTIONS FOR FUTURE RESEARCH rather than a settled picture— “a critical discussion of open challenges and directions for future research” | limitations[0] |
| for which use | limitation — any comparison between deep and traditional methods drawn from this survey: WE FOCUS PARTICULARLY ON SUCH DEEP MODELS, AND DO NOT INCLUDE THE MORE TRADITIONAL FEATURE LEARNING APPROACHES THAT HAVE BEEN APPLIED TO MEDICAL IMAGES— “In this survey, we focus particularly on such deep models, and do not include the more traditional feature learning approaches that have been applied to medical images.” | limitations[1] |
| for which use | limitation — every count and trend in the survey: THE LAST UPDATE TO THE INCLUDED PAPERS WAS ON FEBRUARY 1, 2017— “The last update to the included papers was on February 1, 2017.” | limitations[2] |
| for which use | limitation — the completeness of the literature search behind those counts: WE EXPECT THE SEARCH TERMS USED TO COVER MOST, IF NOT ALL, OF THE WORK INCORPORATING DEEP LEARNING METHODS— “We expect the search terms used to cover most, if not all, of the work incorporating deep learning methods.” | limitations[3] |
| evidence status note | limitation — THIS COPY MOVES TWO WORDS INTO THE QUOTE: the 'In this' after 'is' belongs at the start of the next sentence, before 'dataset pulmonary nodules'.— “To give an example, a widely used dataset for evaluating image analysis algorithms to detect nodules in lung CT is In this the LIDC-IDRI dataset (Armato et al., 2011). dataset pulmonary nodules were annotated by four radiologists independently. Subsequently the readers reviewed each others annotations but no consensus was forced. It turned… | limitations[4] |
| for which use | limitation — any accuracy reported on the LIDC-IDRI dataset: no consensus was forced between the four radiologists, and the number of nodules they did not unanimously agree on was three times larger than the number they did fully agree on— “To give an example, a widely used dataset for evaluating image analysis algorithms to detect nodules in lung CT is In this the LIDC-IDRI dataset (Armato et al., 2011). dataset pulmonary nodules were annotated by four radiologists independently. Subsequently the readers reviewed each others annotations but no consensus was forced. It turned… | limitations[4] |
| for which use | limitation — any transfer of a computer-vision result to medical imaging: label noise can be a significant limiting factor even with domain-expert annotation, whereas in computer vision the labelling noise is typically relatively low— “Even when data is annotated by domain expert, label noise can be a significant limiting factor in developing algorithms, whereas in computer vision the noise in the labeling of images is typically relatively low” | limitations[5] |
| for which use | limitation — any system trained on a disputed reference standard: the authors state that handling noise and uncertainty in the reference standard requires careful consideration and that incorporating labelling uncertainty in the loss function is still an open challenge— “Training a deep learning system on such data requires careful consideration of how to deal with noise and uncertainty in the reference standard. One could think of solutions like incorporating labeling uncertainty directly in the loss function, but this is still an open challenge” | limitations[6] |
| evidence status note | mechanism — THIS COPY MOVES TWO WORDS INTO THE QUOTE: the 'In this' after 'is' belongs at the start of the next sentence, before 'dataset pulmonary nodules'.— “To give an example, a widely used dataset for evaluating image analysis algorithms to detect nodules in lung CT is In this the LIDC-IDRI dataset (Armato et al., 2011). dataset pulmonary nodules were annotated by four radiologists independently. Subsequently the readers reviewed each others annotations but no consensus was forced. It turned… | mechanisms[1] |
| evidence status note | mechanism — RE-SLICED off a damage combination I had not seen before today and which NONE of my guards catch: the sentence reads "systems that are ex- 25 tremely good at excluding" - a line-break hyphen RETAINED (shape 11) with a PAGE NUMBER inserted at the break. The page-number guard needs a lowercase letter before the digit and finds "- "; the hyphen guards look for a fused or a surviving hyphen, not one with a number wedged into it. So the sentence's most important word is destroyed and every check passes. The span now starts after it.— “good at excluding the most common normal subclasses, but fail miserably on several rare ones.” | mechanisms[5] |
| evidence status note | mechanism — The span carries this file's surviving line-break hyphens - “publica- tions”, “contri- butions” - quoted verbatim, because a quote is the staged file's bytes. AND THE SPAN IS IN TEXT THAT WAS NOT STAGED UNTIL 2026-08-17: this record held 64% of its source and stopped mid-clause, so these sentences were never available to any earlier pass.— “When largely overlapping work had been reported in multiple publica- tions, only the publication deemed most important was included. A typical example here was arXiv preprints that were subsequently published or conference contri- butions which were expanded and published in journals.” | mechanisms[11] |
| evidence status note | sense — THIS COPY MOVES TWO WORDS INTO THE QUOTE: the 'In this' after 'is' belongs at the start of the next sentence, before 'dataset pulmonary nodules'.— “To give an example, a widely used dataset for evaluating image analysis algorithms to detect nodules in lung CT is In this the LIDC-IDRI dataset (Armato et al., 2011). dataset pulmonary nodules were annotated by four radiologists independently. Subsequently the readers reviewed each others annotations but no consensus was forced. It turned… | senses[0] |
| replication | untested |
| limitation | open challenges and future-research directions remain in the field— “a critical discussion of open challenges and directions for future research” | limitations[0] |
| limitation | The survey deliberately excludes traditional (non-deep) feature-learning approaches applied to medical images.— “In this survey, we focus particularly on such deep models, and do not include the more traditional feature learning approaches that have been applied to medical images.” | limitations[1] |
| limitation | Coverage has a hard cut-off date, so later work is not included.— “The last update to the included papers was on February 1, 2017.” | limitations[2] |
| limitation | Completeness of the literature search is an expectation rather than a guarantee.— “We expect the search terms used to cover most, if not all, of the work incorporating deep learning methods.” | limitations[3] |
| limitation | On a widely used dataset the disputed annotations outnumber the agreed ones three to one.— “To give an example, a widely used dataset for evaluating image analysis algorithms to detect nodules in lung CT is In this the LIDC-IDRI dataset (Armato et al., 2011). dataset pulmonary nodules were annotated by four radiologists independently. Subsequently the readers reviewed each others annotations but no consensus was forced. It turned… | limitations[4] |
| limitation | Label noise is worse in this field than in the field the methods came from, and the survey says so.— “Even when data is annotated by domain expert, label noise can be a significant limiting factor in developing algorithms, whereas in computer vision the noise in the labeling of images is typically relatively low” | limitations[5] |
| limitation | Training on such data is stated to be an open problem, not a solved preprocessing step.— “Training a deep learning system on such data requires careful consideration of how to deal with noise and uncertainty in the reference standard. One could think of solutions like incorporating labeling uncertainty directly in the loss function, but this is still an open challenge” | limitations[6] |