Attention gated networks: Learning to leverage salient regions in medical images

Attention gated networks: Learning to leverage salient regions in medical images Jo Schlemper Ozan Oktay Michiel Schaap Mattias P. Heinrich‬ Bernhard Kainz Ben Glocker Daniel Rueckert

Tag layer of the DHSS Meta-Tagging Project. The article text is reproduced from the staged copy named in sourceDesc and is not altered; the tags are held separately in standOff so that overlapping and nested spans are preserved.

Attention gated networks: Learning to leverage salient regions in medical images 2019 Medical Image Analysis 10.1016/j.media.2019.01.012 schlemper2019-attention-gated-networks-learning schlemper2019-attention-gated-networks-learning.txt, the project's staged copy 0ea854df39ee8627e4351e3db99424bfbf960f0b cc-by computer-science

the paper's own words

the tag layer; a sentence written by the tagger about the paper, always anchored to a quotation but not itself a quotation

0 spans located by searching the stored text for the quote's exact words, allowing any whitespace between them whether a network can be made to find the organ it is supposed to measure, instead of being handed a cropped region by a second network trained to find it Quotation withheld because this record is not cleared for redistribution. whether a soft attention gate inside a single CNN can replace the cascade - a localisation model followed by a segmentation model - without losing accuracy or adding many parameters Quotation withheld because this record is not cleared for redistribution. a grid-based additive attention gate that drops into VGG or U-Net, learns where to look with no bounding boxes and no extra supervision, improves recall and precision on CT pancreas segmentation and fetal ultrasound plane detection, and is shown to beat simply making the baseline bigger Quotation withheld because this record is not cleared for redistribution. method Quotation withheld because this record is not cleared for redistribution. small, variable-shaped structures that a single-stage network misses, and the cascades built to hand it a cropped view of them - which repeat the same low-level features in every model of the chain Quotation withheld because this record is not cleared for redistribution. system-or-tool-building system-or-tool-building Quotation withheld because this record is not cleared for redistribution. a cascade spends a whole network learning where the organ is and then throws that network's features away; a gate computes the same decision from features the predictor already has, so the localisation costs 8% more parameters instead of a second model, and the low-level features are extracted once localisation folded into the predictor Quotation withheld because this record is not cleared for redistribution. the gating parameters start so that every feature vector passes at every location - the network begins as its own baseline and has to learn to suppress, which is why no multi-stage training schedule is needed and why any difference is attributable to what the gate learned rather than to where it started a gate initialised to do nothing Quotation withheld because this record is not cleared for redistribution. the gating signal is taken from a coarser grid, where the receptive field is large enough to know where the organ is, and applied to a finer map, where the resolution is enough to draw its boundary - which is the same division of labour a cascade makes between its two networks, done inside one coarse context deciding fine attention Quotation withheld because this record is not cleared for redistribution. softmax over the attention map produces sparse, intensity-sensitive attention, which is bad in ultrasound; sigmoid only rescales and lets more information through, and pays for it with gradient saturation. Neither is free and the paper reports both costs a normalisation choice that trades sparsity against gradients Quotation withheld because this record is not cleared for redistribution. the attention map outlines the discriminant region and does not necessarily coincide with the entire object, so it localises without delineating Quotation withheld because this record is not cleared for redistribution. the comparison against 2D cascaded state-of-the-art models is called insufficient to draw a final conclusion, because the 3D model has context and the 2D models have resolution Quotation withheld because this record is not cleared for redistribution. softmax normalisation gives sparse, intensity-sensitive maps and sigmoid gives gradient saturation - the normalisation choice has no free option Quotation withheld because this record is not cleared for redistribution. training the attention mechanism is more complex than the baseline, and the most stable route was per-scale training followed by fine-tuning Quotation withheld because this record is not cleared for redistribution. residual connections around the gate gave no significant improvement in the segmentation experiments Quotation withheld because this record is not cleared for redistribution. additive soft attention, carried from sentence-to-sentence machine translation into a convolutional network over medical images, and re-based on an image grid so the coefficients can be local Quotation withheld because this record is not cleared for redistribution. higher-capacity U-Net models trained and compared against the smaller gated model, rather than only against the same-size baseline eliminates: the extra parameters as the explanation of the gain - a module that adds 8% capacity and improves DSC by 2-3% has to be shown to beat 8% more capacity spread uniformly, and it does at p = .007 · outcome: held Quotation withheld because this record is not cleared for redistribution. the same two models retrained at a smaller training size (30 images instead of 120) eliminates: the improvement being an artefact of one data budget - if attention only helps when data is plentiful it is not solving the localisation problem it claims to · outcome: held Quotation withheld because this record is not cleared for redistribution. models trained on CT-150 applied directly to the public CT-82 benchmark with no fine-tuning, before any adaptation is done eliminates: the result being tied to one institution's data - the transfer is measured before the models are allowed to see the new distribution · outcome: held Quotation withheld because this record is not cleared for redistribution. both models also trained from scratch on 61 CT-82 images, in addition to the fine-tuned and directly-applied conditions eliminates: pre-training on the private dataset as a hidden advantage in the public benchmark comparison · outcome: held Quotation withheld because this record is not cleared for redistribution. 5-fold cross-validation on CT-82 with the Attention U-Net, reported with its standard deviation eliminates: a single favourable train/test split - the spread of 6.23 DSC is the size of the effect being claimed, and reporting it makes that visible · outcome: held Quotation withheld because this record is not cleared for redistribution. conditional random field post-processing deliberately NOT applied, and said so, although the comparison methods use it eliminates: post-processing absorbing or masking the gate's contribution - the measured difference is the module's in an isolated setting, at the cost of a lower absolute number · outcome: held Quotation withheld because this record is not cleared for redistribution. ultrasound frames cropped to the central 208 x 272 so the network cannot read the annotations the scanner prints around the image eliminates: the network reading the annotations the scanner prints around the image instead of the anatomy - a leak that would look like accuracy · outcome (stated by the tagger, no quotation): held Quotation withheld because this record is not cleared for redistribution. gating parameters initialised so that the gates pass every feature vector at every location, making the network start as its own baseline eliminates: that the attention gates help only because they change the initialisation - starting open makes the network its own baseline at step zero · outcome (stated by the tagger, no quotation): held Quotation withheld because this record is not cleared for redistribution. class imbalance handled by weighted sampling for classification and a Dice loss for segmentation, in both the attention and the baseline networks eliminates: the dominant background class carrying the score, and any difference between the two networks being a difference in how each one met the imbalance · outcome (stated by the tagger, no quotation): held Quotation withheld because this record is not cleared for redistribution. attention gates attention gates Quotation withheld because this record is not cleared for redistribution. cascaded cascaded Quotation withheld because this record is not cleared for redistribution. soft attention soft attention Quotation withheld because this record is not cleared for redistribution. Hard attention Hard attention Quotation withheld because this record is not cleared for redistribution. discriminant region discriminant region Quotation withheld because this record is not cleared for redistribution. recall recall Quotation withheld because this record is not cleared for redistribution. trainable parameters trainable parameters Quotation withheld because this record is not cleared for redistribution. DSC DSC Quotation withheld because this record is not cleared for redistribution. conditional random field conditional random field Quotation withheld because this record is not cleared for redistribution. an attention map - AS THE PAPER USES IT: a soft region proposal that suppresses background and lets the network localise without a second model Quotation withheld because this record is not cleared for redistribution. an attention map - AS THE PAPER SAYS IT IS NOT: the object. It outlines the DISCRIMINANT REGION and does not necessarily coincide with the whole structure Quotation withheld because this record is not cleared for redistribution. the gain from attention - AGAINST THE SAME-SIZE BASELINE: 2-3 DSC points for 8% extra parameters Quotation withheld because this record is not cleared for redistribution. the gain from attention - AGAINST A BASELINE GROWN TO MATCH: still significant at p = .007, which is what makes it a claim about the gate rather than about size Quotation withheld because this record is not cleared for redistribution. the state-of-the-art comparison - AS THE TABLE READS: comparable to 2D cascaded models and better than single-model frameworks Quotation withheld because this record is not cleared for redistribution. the state-of-the-art comparison - AS THE AUTHORS QUALIFY IT: not sufficient to draw a final conclusion, because their 3D model trades resolution for context and the 2D models trade context for resolution Quotation withheld because this record is not cleared for redistribution. critiques → cascaded multi-stage CNN frameworks, which reach 81-82% DSC on the TCIA pancreas benchmark but re-extract the same low-level features in every model of the chain Quotation withheld because this record is not cleared for redistribution. critiques → hard-attention approaches, which crop, are non-differentiable, and need reinforcement learning for parameter updates Quotation withheld because this record is not cleared for redistribution. extends → soft attention as used for adaptive feature pooling in natural image classification and captioning, generalised here to grid-based gating so coefficients can be specific to local regions Quotation withheld because this record is not cleared for redistribution. qualifies → its own state-of-the-art comparison, which the authors say is not sufficient to draw a final conclusion because the 3D and 2D models are given different information Quotation withheld because this record is not cleared for redistribution. qualifies → the interpretability claim for attention maps - they outline what the model used, not the object, and the paper says so in the same paragraph that offers them for localisation Quotation withheld because this record is not cleared for redistribution. attention gates —distinct-from→ cascaded attention gates cascaded differs in: whether localisation is a separate model with its own parameters or a module inside the predictor · the relation itself is separately evidenced · strength of the relation: 0.95 · differs in (stated by the tagger, no quotation): whether localisation is a separate model with its own parameters or a module inside the predictor · the relation stated: the gates suppress background progressively inside one network instead of cropping a region of interest between two Audit 2026-09-28: DOUBTFUL - read against its span and kept, but attention gates remove the need for an external localisation module; distinct-from 'cascaded' is terse. Quotation withheld because this record is not cleared for redistribution. soft attention —distinct-from→ Hard attention soft attention Hard attention differs in: whether the attention step can be differentiated through · the relation itself is separately evidenced · strength of the relation: 0.95 · differs in (stated by the tagger, no quotation): whether the attention step can be differentiated through · the relation stated: hard attention crops and is non-differentiable, needing reinforcement learning; soft attention is probabilistic and trains by ordinary backpropagation Quotation withheld because this record is not cleared for redistribution. attention gates —distinct-from→ trainable parameters attention gates trainable parameters differs in: whether the extra parameters are spread across all layers or spent on localisation · the relation itself is separately evidenced · strength of the relation: 0.95 · differs in (stated by the tagger, no quotation): whether the extra parameters are spread across all layers or spent on localisation · the relation stated: a U-Net grown uniformly to the same size does not reach the gated model, so the gain is not capacity Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but the span writes 'AGs' and says they contribute more than simply increasing capacity; distinct-from is loose. Quotation withheld because this record is not cleared for redistribution. attention gates beat a U-Net grown uniformly to the same parameter count, at p = .007, so the gain is the module and not the capacity attention gates beat a U-Net grown uniformly to the same parameter count, at p = .007, so the gain is the module and not the capacity Quotation withheld because this record is not cleared for redistribution. the improvement holds at 30 training images as well as 120, so it is not an artefact of one data budget the improvement holds at 30 training images as well as 120, so it is not an artefact of one data budget Quotation withheld because this record is not cleared for redistribution. Attention gates increase recall in pancreas segmentation (p = .005). Attention gates increase recall in pancreas segmentation (p = .005). Quotation withheld because this record is not cleared for redistribution. the attention map outlines the discriminant region and does not necessarily coincide with the entire object the attention map outlines the discriminant region and does not necessarily coincide with the entire object Quotation withheld because this record is not cleared for redistribution. No post-processing such as a conditional random field was used, so as to isolate the gates' contribution. No post-processing such as a conditional random field was used, so as to isolate the gates' contribution. Quotation withheld because this record is not cleared for redistribution. AG-Sononet performs close to a Sononet with double the capacity. AG-Sononet performs close to a Sononet with double the capacity. Quotation withheld because this record is not cleared for redistribution. each attention gate learns to focus on a particular subset of organs, starting from a uniform distribution and localising over training each attention gate learns to focus on a particular subset of organs, starting from a uniform distribution and localising over training Quotation withheld because this record is not cleared for redistribution. probabilistic, end-to-end differentiable attention using standard backpropagation without posterior sampling - as against hard attention, which crops and needs reinforcement learning soft attention Quotation withheld because this record is not cleared for redistribution. what an attention map outlines - the part of the object the classifier uses, which the authors state does not necessarily coincide with the entire object discriminant region Quotation withheld because this record is not cleared for redistribution. extra capacity added to the standard U-Net by the attention gates, for a 2-3% DSC improvement 8% Quotation withheld because this record is not cleared for redistribution. mean pancreas DSC over 5-fold cross-validation on CT-82, with a standard deviation of 6.23 81.48 Quotation withheld because this record is not cleared for redistribution. the bottom of the atlas-method DSC range on the TCIA pancreas benchmark 69.6% Quotation withheld because this record is not cleared for redistribution. DSC DSC Quotation withheld because this record is not cleared for redistribution. p p Quotation withheld because this record is not cleared for redistribution. recall recall Quotation withheld because this record is not cleared for redistribution. trainable parameters trainable parameters Quotation withheld because this record is not cleared for redistribution.

The full text of this article is not redistributable, so it is not included here. The evidence sentences are withheld with it; the tag layer and its grounding status remain. The article itself is at https://doi.org/10.1016/j.media.2019.01.012.