A Survey on Deep Learning in Medical Image Analysis Geert Litjens Thijs Kooi Babak Ehteshami Bejnordi Arnaud Arindra Adiyoso Setio

Tag layer of the DHSS Meta-Tagging Project. The article text is reproduced from the staged copy named in sourceDesc and is not altered; the tags are held separately in standOff so that overlapping and nested spans are preserved.

 A Survey on Deep Learning in Medical Image Analysis 2017 Medical Image Analysis / arXiv 10.1016/j.media.2017.07.005 1702.05747 litjens2017-medical-imaging-survey litjens2017-medical-imaging-survey.txt, the project's staged copy 7c9113fb24dc2baa8d3b0bdd5d131f217ac5e60a arxiv-open medicine

the paper's own words

the tag layer; a sentence written by the tagger about the paper, always anchored to a quotation but not itself a quotation

 0 spans located by searching the stored text for the quote's exact words, allowing any whitespace between them Quotation withheld because this record is not cleared for redistribution. Quotation withheld because this record is not cleared for redistribution. review Quotation withheld because this record is not cleared for redistribution. THE SAME TECHNICAL PROBLEM IS REPORTED TO HAVE DIFFERENT SEVERITY IN TWO FIELDS, AND THE CONTRAST IS DRAWN EXPLICITLY: EVEN WHEN DATA IS ANNOTATED BY DOMAIN EXPERT, LABEL NOISE CAN BE A SIGNIFICANT LIMITING FACTOR - WHEREAS IN COMPUTER VISION THE NOISE IN THE LABELING OF IMAGES IS TYPICALLY RELATIVELY LOW. The word `label` names something with a different reliability depending on who produced it Quotation withheld because this record is not cleared for redistribution. A REFERENCE STANDARD IS BUILT WITH CONSENSUS DELIBERATELY NOT FORCED, AND THE DISAGREEMENT THEN OUTNUMBERS THE AGREEMENT THREE TO ONE: nodules were annotated by four radiologists INDEPENDENTLY, the readers REVIEWED EACH OTHERS ANNOTATIONS BUT NO CONSENSUS WAS FORCED - and the number they DID NOT UNANIMOUSLY AGREE ON to be a nodule WAS THREE TIMES LARGER THAN THE NUMBER THEY DID FULLY AGREE ON about this quote: THIS COPY MOVES TWO WORDS INTO THE QUOTE: the 'In this' after 'is' belongs at the start of the next sentence, before 'dataset pulmonary nodules'. Quotation withheld because this record is not cleared for redistribution. THE CONSEQUENCE FOR ANYTHING TRAINED ON IT IS STATED, AND THE FIX IS AN OPEN PROBLEM: training a system on such data REQUIRES CAREFUL CONSIDERATION OF HOW TO DEAL WITH NOISE AND UNCERTAINTY IN THE REFERENCE STANDARD - one COULD THINK OF solutions like incorporating labeling uncertainty directly in the loss function, BUT THIS IS STILL AN OPEN CHALLENGE. The uncertainty is acknowledged and has nowhere to go Quotation withheld because this record is not cleared for redistribution. A BINARY PRESENTATION OF THE TASK IS CALLED A GROSS SIMPLIFICATION, AND THE REASON IS HETEROGENEITY ON BOTH SIDES: classification is often presented as A BINARY TASK - normal versus abnormal, object versus background - HOWEVER THIS IS OFTEN A GROSS SIMPLIFICATION AS BOTH CLASSES CAN BE HIGHLY HETEROGENEOUS. Not one class hiding structure but both Quotation withheld because this record is not cleared for redistribution. THE `NORMAL` CATEGORY IS SHOWN TO CONTAIN THINGS THAT ARE NOT NORMAL, INCLUDING ARTIFACTS OF THE INSTRUMENT: the normal category OFTEN CONSISTS OF COMPLETELY NORMAL TISSUE **BUT ALSO** several categories of BENIGN FINDINGS, WHICH CAN BE RARE, and MAY OCCASIONALLY INCLUDE A WIDE VARIETY OF IMAGING ARTIFACTS. The residual class is defined by what it is not, and collects the measurement's own failures Quotation withheld because this record is not cleared for redistribution. THE PERFORMANCE PATTERN IS PREDICTED FROM THE CATEGORY'S STRUCTURE: systems end up GOOD AT EXCLUDING THE MOST COMMON NORMAL SUBCLASSES BUT FAIL MISERABLY ON SEVERAL RARE ONES. A single accuracy figure over a heterogeneous class hides exactly the failures that matter, and the shape of the class predicts the shape of the failure about this quote: RE-SLICED off a damage combination I had not seen before today and which NONE of my guards catch: the sentence reads "systems that are ex- 25 tremely good at excluding" - a line-break hyphen RETAINED (shape 11) with a PAGE NUMBER inserted at the break. The page-number guard needs a lowercase letter before the digit and finds "- "; the hyphen guards look for a fused or a surviving hyphen, not one with a number wedged into it. So the sentence's most important word is destroyed and every check passes. The span now starts after it. Quotation withheld because this record is not cleared for redistribution. THE CORRECT FIX IS NAMED AND DECLINED ON COST, IN ONE SENTENCE: a STRAIGHTFORWARD SOLUTION would be to turn the system into a MULTI-CLASS one by providing DETAILED ANNOTATIONS OF ALL POSSIBLE SUBCLASSES - OBVIOUSLY THIS AGAIN COMPOUNDS THE ISSUE OF LIMITED AVAILABILITY OF EXPERT TIME FOR ANNOTATING AND IS THEREFORE OFTEN SIMPLY NOT FEASIBLE. The right answer is known and unaffordable Quotation withheld because this record is not cleared for redistribution. THE WORKAROUNDS ARE REPORTED TO FAIL FOR THE SAME REASON THE PROBLEM EXISTS: strategies that incorporate intelligence in the training process - selective sampling, hard negative mining - TYPICALLY FAIL WHEN THERE IS SUBSTANTIAL NOISE IN THE REFERENCE STANDARD - and ADDITIONAL METHODS FOR DEALING WITH WITHIN-CLASS HETEROGENEITY WOULD BE HIGHLY WELCOME. The noise in the labels defeats the corrections for the labels Quotation withheld because this record is not cleared for redistribution. A STAIN IS OFFERED AS A SUBSTITUTE FOR EXPERTISE: in histopathology one can sometimes use SPECIFIC IMMUNOHISTOCHEMICAL STAINS to highlight regions of interest, REDUCING THE NEED FOR EXPERT EXPERIENCE - and non-expert labels can be leveraged VIA CROWD-SOURCING. Two ways to replace a judgement: a reagent and a crowd Quotation withheld because this record is not cleared for redistribution. A BENCHMARK IS REPORTED TO HAVE SATURATED, SO THAT IMPROVEMENTS CAN NO LONGER BE ATTRIBUTED: since 2014 the performance on the benchmark HAS SATURATED and IT IS DIFFICULT TO ASSESS WHETHER THE SMALL INCREASES IN PERFORMANCE CAN REALLY BE ATTRIBUTED TO 'BETTER' AND MORE SOPHISTICATED ARCHITECTURES. The scare quotes on `better` are the paper's, and the measuring instrument has stopped discriminating Quotation withheld because this record is not cleared for redistribution. A PERFORMANCE CRITERION IS DECLARED INSUFFICIENT AND SUPPLEMENTED BY AN ARTICULATION CRITERION, JUSTIFIED BY LEGAL ACCOUNTABILITY: these methods have often been described as BLACK BOXES, and especially in medicine WHERE ACCOUNTABILITY IS IMPORTANT AND CAN HAVE SERIOUS LEGAL CONSEQUENCES, IT IS OFTEN NOT ENOUGH TO HAVE A GOOD PREDICTION SYSTEM - the system ALSO HAS TO BE ABLE TO ARTICULATE ITSELF in a certain way. What counts as success acquires a second component from outside the field Quotation withheld because this record is not cleared for redistribution. A SURVEY DE-DUPLICATES ITS OWN POPULATION BY A JUDGEMENT IT NEVER DEFINES, AND SUPPLIES A WORKED EXAMPLE INSTEAD OF A CRITERION: when largely overlapping work had been reported in multiple publications, ONLY THE PUBLICATION DEEMED MOST IMPORTANT WAS INCLUDED - A TYPICAL EXAMPLE HERE WAS ARXIV PREPRINTS THAT WERE SUBSEQUENTLY PUBLISHED, or conference contributions later expanded into journal papers. What `most important` means is stated nowhere, the survey's whole 308-paper population rests on it, and the example is doing the work a definition would have done about this quote: The span carries this file's surviving line-break hyphens - “publica- tions”, “contri- butions” - quoted verbatim, because a quote is the staged file's bytes. AND THE SPAN IS IN TEXT THAT WAS NOT STAGED UNTIL 2026-08-17: this record held 64% of its source and stopped mid-clause, so these sentences were never available to any earlier pass. Quotation withheld because this record is not cleared for redistribution. what this limits: the survey's own account of the field: it offers Quotation withheld rather than a settled picture Quotation withheld because this record is not cleared for redistribution. what this limits: any comparison between deep and traditional methods drawn from this survey: WE FOCUS PARTICULARLY ON SUCH DEEP MODELS, AND DO NOT INCLUDE THE MORE TRADITIONAL FEATURE LEARNING APPROACHES THAT HAVE BEEN APPLIED TO MEDICAL IMAGES Quotation withheld because this record is not cleared for redistribution. what this limits: every count and trend in the survey: THE LAST UPDATE TO THE INCLUDED PAPERS WAS ON FEBRUARY 1, 2017 Quotation withheld because this record is not cleared for redistribution. what this limits: the completeness of the literature search behind those counts: WE EXPECT THE SEARCH TERMS USED TO COVER MOST, IF NOT ALL, OF THE WORK INCORPORATING DEEP LEARNING METHODS Quotation withheld because this record is not cleared for redistribution. about this quote: THIS COPY MOVES TWO WORDS INTO THE QUOTE: the 'In this' after 'is' belongs at the start of the next sentence, before 'dataset pulmonary nodules'. · what this limits: any accuracy reported on the LIDC-IDRI dataset: no consensus was forced between the four radiologists, and the number of nodules they did not unanimously agree on was three times larger than the number they did fully agree on Quotation withheld because this record is not cleared for redistribution. what this limits: any transfer of a computer-vision result to medical imaging: label noise can be a significant limiting factor even with domain-expert annotation, whereas in computer vision the labelling noise is typically relatively low Quotation withheld because this record is not cleared for redistribution. what this limits: any system trained on a disputed reference standard: the authors state that handling noise and uncertainty in the reference standard requires careful consideration and that incorporating labelling uncertainty in the loss function is still an open challenge Quotation withheld because this record is not cleared for redistribution. Quotation withheld because this record is not cleared for redistribution. eliminates: a survey padded by work that is not deep learning on medical images, and double-counting of one piece of work reported more than once · outcome (stated by the tagger, no quotation): held Quotation withheld because this record is not cleared for redistribution. reference standard Quotation withheld because this record is not cleared for redistribution. label noise Quotation withheld because this record is not cleared for redistribution. selective sampling Quotation withheld because this record is not cleared for redistribution. within-class heterogeneity Quotation withheld because this record is not cleared for redistribution. binary task Quotation withheld because this record is not cleared for redistribution. about this quote: THIS COPY MOVES TWO WORDS INTO THE QUOTE: the 'In this' after 'is' belongs at the start of the next sentence, before 'dataset pulmonary nodules'. Quotation withheld because this record is not cleared for redistribution. Quotation withheld because this record is not cleared for redistribution. Quotation withheld because this record is not cleared for redistribution. Quotation withheld because this record is not cleared for redistribution. Quotation withheld because this record is not cleared for redistribution. Quotation withheld because this record is not cleared for redistribution. Quotation withheld because this record is not cleared for redistribution. Quotation withheld because this record is not cleared for redistribution. Quotation withheld because this record is not cleared for redistribution. Quotation withheld because this record is not cleared for redistribution. Quotation withheld because this record is not cleared for redistribution. about this quote: QUOTE RECOVERED FROM THE PAPER'S OWN CHARACTERS. The recorded quote was LETTERS-ONLY IDENTICAL to this stretch of the staged text but not an exact substring of it - spaces, hyphens, ligature glyphs or mathematical operators differed, so it could never anchor. This is the file's own text between the same two letters: not produced by a rule about the damage, copied out of the file. Round-trip checked (the recovered span is letters-only equal to the original) and length-checked. The text as originally recorded is kept in evidence as extracted. Quotation withheld because this record is not cleared for redistribution. training data Quotation withheld because this record is not cleared for redistribution. decision boundary Quotation withheld because this record is not cleared for redistribution. discriminant features Quotation withheld because this record is not cleared for redistribution. imaging tasks Quotation withheld because this record is not cleared for redistribution. GOFAI Quotation withheld because this record is not cleared for redistribution. handcrafted features Quotation withheld because this record is not cleared for redistribution. feature learning about this quote: QUOTE RECOVERED FROM THE PAPER'S OWN CHARACTERS. The recorded quote was LETTERS-ONLY IDENTICAL to this stretch of the staged text but not an exact substring of it - spaces, hyphens, ligature glyphs or mathematical operators differed, so it could never anchor. This is the file's own text between the same two letters: not produced by a rule about the damage, copied out of the file. Round-trip checked (the recovered span is letters-only equal to the original) and length-checked. The text as originally recorded is kept in evidence as extracted. Quotation withheld because this record is not cleared for redistribution. Bayesian about this quote: QUOTE REPAIRED: the extractor split a hyphen-space and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted. Quotation withheld because this record is not cleared for redistribution. regression Quotation withheld because this record is not cleared for redistribution. statistical-significance about this quote: QUOTE REPAIRED: the extractor split a hyphen-space and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted. Quotation withheld because this record is not cleared for redistribution.

The full text of this article is not redistributable, so it is not included here. The evidence sentences are withheld with it; the tag layer and its grounding status remain. The article itself is at https://doi.org/10.1016/j.media.2017.07.005.