A. E. ClarkpersName, B. BiffipersName, R. SiverapersName, A. Dall'AstapersName, L. FesseypersName, T.-L. WongpersName, G. ParamasivampersName, D. DunawaypersName, S. SchievanopersName, C. C. LeespersName (2020date). “Developing and testing an algorithm for automatic segmentation of the fetal face from three-dimensional ultrasound images.” Royal Society Open Sciencetitle·j.
Object of study — manual and atlas-based automatic segmentationterm·mth of the fetal faceterm·key from 3D ultrasoundterm·mth volumes
Research question — how variable manual refinementterm·mth of segmentations is between and within operators, and how an optimized automatic algorithm performs
Contribution — technical feasibility of fetal faceterm·key segmentation, with automatic performance improving asterm·mth the atlasterm·mth grows
phenomenon · claim-type · cross-domain-transfer · mechanism · claim-relation · replication-status · limitation — grounded in the full text| phenomenon | segmentation of the fetal face from 3D ultrasound — “Image segmentation and computer analysis of three-dimensional US volumes of the fetal face may provide an objective measure to quantify fetal facial features and identify abnormalities.” | |
| claim-type | method — “We have demonstrated the technical feasibility of fetal facial segmentation using manual refinement of partially automatically segmented US facial volumes.” | |
| method — evidence | — “This study assesses the inter- and intra-operator variability of MS and tests an optimized version of our automatic segmentation (AS) algorithm.” | |
| object of study — evidence | — “This study assesses the inter- and intra-operator variability of MS and tests an optimized version of our automatic segmentation (AS) algorithm.” | |
| research question — evidence | — “This study assesses the inter- and intra-operator variability of MS and tests an optimized version of our automatic segmentation (AS) algorithm.” | |
| contribution — evidence | — “We have demonstrated the technical feasibility of fetal facial segmentation using manual refinement of partially automatically segmented US facial volumes.” | |
| mechanism | label fusion from atlas images The algorithm selects the most relevant parts of the atlas images and fuses them to build the new segmentation. — “It selects the most relevant parts of the different atlas images and fuses them in order to create the new image.” | mechanisms[0] |
| sense | atlas - a bank of already segmented ground-truth images the algorithm draws on to segment a new image — “Multi-atlas propagation segmentation uses a set of already segmented ‘ground-truth’ guide images (i.e. ‘atlas’), as a reference bank for an algorithm tasked with building the segmentation of a new, unseen image.” | senses[0] |
| sense | partially automatic segmentation - automatic atlas-based output that still needs manual refinement — “We have demonstrated the technical feasibility of fetal facial segmentation using manual refinement of partially automatically segmented US facial volumes.” | senses[1] |
| control | leave-one-out cross-validation of the algorithm on the first 20 atlas images — held — “Of the four atlases tested, A_30 was the optimum atlas size with an overall improvement in performance balanced by no excessive increase in computational time.” — “The algorithm was validated by applying a ‘leave-one-out’ cross-validation scheme on the first 20 images of each atlas” | controls[0] |
| relation | US —same-as→ ultrasound — ( ) — “Fetal craniofacial abnormalities are challenging to detect and diagnose on prenatal ultrasound (US).” — “diagnose on prenatal ultrasound (US)” | concept_edges[0] |
| weight | US —same-as→ ultrasound — 0.95 — “Fetal craniofacial abnormalities are challenging to detect and diagnose on prenatal ultrasound (US).” | concept_edges[0] |
| relation | MS —same-as→ manual segmentation — ( ) — “In previous work by this group, manual segmentation (MS) was used to delineate the fetal face surfaces meshes from 20 3D US volumes [ 6 ].” — “by this group, manual segmentation (MS)” | concept_edges[1] |
| weight | MS —same-as→ manual segmentation — 0.95 — “In previous work by this group, manual segmentation (MS) was used to delineate the fetal face surfaces meshes from 20 3D US volumes [ 6 ].” | concept_edges[1] |
| relation | AS —same-as→ automatic segmentation — ( ) — “This study assesses the inter- and intra-operator variability of MS and tests an optimized version of our automatic segmentation (AS) algorithm.” — “our automatic segmentation (AS) algorithm” | concept_edges[2] |
| weight | AS —same-as→ automatic segmentation — 0.95 — “This study assesses the inter- and intra-operator variability of MS and tests an optimized version of our automatic segmentation (AS) algorithm.” | concept_edges[2] |
| relation | ASSD —same-as→ average symmetric surface distance — ( ) — “The average symmetric surface distance (ASSD) calculates how much the surface varies between two segmentations.” — “average symmetric surface distance (ASSD)” | concept_edges[3] |
| weight | ASSD —same-as→ average symmetric surface distance — 0.95 — “The average symmetric surface distance (ASSD) calculates how much the surface varies between two segmentations.” | concept_edges[3] |
| relation | ICC —same-as→ Intraclass correlation coefficient — ( ) — “Intraclass correlation coefficient (ICC) using repeated measures ANOVA was calculated to compare the total segmentation volumes of each operator.” — “Intraclass correlation coefficient (ICC)” | concept_edges[4] |
| weight | ICC —same-as→ Intraclass correlation coefficient — 0.95 — “Intraclass correlation coefficient (ICC) using repeated measures ANOVA was calculated to compare the total segmentation volumes of each operator.” | concept_edges[4] |
| effect direction | atlas size —causes→ manual refinement — reduces — “We have also shown that it is possible to improve the performance of our AS algorithm by increasing the atlas size which in turn reduces the amount of manual refinement required.” — “reduces the amount of manual refinement required” | concept_edges[5] |
| relation | atlas size —causes→ manual refinement — reduces — “We have also shown that it is possible to improve the performance of our AS algorithm by increasing the atlas size which in turn reduces the amount of manual refinement required.” — “by increasing the atlas size which in turn reduces the amount of manual refinement required” | concept_edges[5] |
| weight | atlas size —causes→ manual refinement — 0.8 — “We have also shown that it is possible to improve the performance of our AS algorithm by increasing the atlas size which in turn reduces the amount of manual refinement required.” | concept_edges[5] |
| effect direction | ground-truth images —causes→ computational time — increases — “As expected, average computational time increases by increasing the number of ground-truth images in the atlas, with differences ( p < 0.001) between all comparisons” — “average computational time increases” | concept_edges[6] |
| relation | ground-truth images —causes→ computational time — increases — “As expected, average computational time increases by increasing the number of ground-truth images in the atlas, with differences ( p < 0.001) between all comparisons” — “average computational time increases by increasing the number of ground-truth images in the atlas” | concept_edges[6] |
| weight | ground-truth images —causes→ computational time — 0.85 — “As expected, average computational time increases by increasing the number of ground-truth images in the atlas, with differences ( p < 0.001) between all comparisons” | concept_edges[6] |
| outcome | control — held — “The algorithm was validated by applying a ‘leave-one-out’ cross-validation scheme on the first 20 images of each atlas” — “Of the four atlases tested, A_30 was the optimum atlas size with an overall improvement in performance balanced by no excessive increase in computational time.” | controls[0] |
| replication | original | |
| limitation | The number of segmentations was modest. — “The number of segmentations performed was modest, although sufficient to demonstrate the low variability between segmentations and acceptability of this method.” | limitations[0] |
| limitation | Operators differed in skill, two with no ultrasound experience. — “The operators were of differing skill levels, with two operators having no prior experience of performing or interpreting US images.” | limitations[1] |
| limitation | Manual refinement and its time are the main barrier to clinical use. — “The main limitation of this technique and barrier to use in clinical practice is the need for additional manual refinement and the time taken to perform this.” | limitations[2] |
| limitation | Current methods cannot segment fetal faces robustly in variable contexts and poor image quality. — “At the present time, state-of-art methods are unable to accurately segment fetal faces from 3D US images while being robust to a highly variable context (the fetal hand or umbilical cord in contact with or obstructing the face, contact with the uterine wall or placenta) and poor image quality [ 9 ].” | limitations[3] |
finding · quantity · variable · definition · key-term — grounded in the full text| finding | Manual refinements showed low inter- and intra-operator variability. — “Assessment of the manual refinements showed low inter- and intra-operator variability demonstrating its suitability for optimizing the AS algorithm.” | content_tags.findings[0] |
| finding | Differences in Dice score and ASSD reflect lower intra- than inter-operator variability. — “Post hoc analysis demonstrated that these differences in Dice score ( p = 0.03) and ASSD ( p = 0.002) can both be attributed to a lower intra-operator variability when compared to the three corresponding inter-operator measurements ( table 1 ).” | content_tags.findings[1] |
| finding | Segmentation volume did not correlate with inter-operator differences. — “There was no correlation found between overall segmentation volume and inter-operator differences for the average Dice score ( T -test p = 0.66) or ASSD ( p = 0.25) values.” | content_tags.findings[2] |
| finding | Atlas size affected the Dice score. — “Atlas size had an impact on the Dice score (Kruskal–Wallis p < 0.001).” | content_tags.findings[3] |
| finding | A_30 was the optimum atlas size. — “Of the four atlases tested, A_30 was the optimum atlas size with an overall improvement in performance balanced by no excessive increase in computational time.” | content_tags.findings[4] |
| finding | Increasing atlas size improved performance and reduced manual refinement. — “The algorithm showed improved performance following an increase in the atlas size in turn reducing the need for manual refinement.” | content_tags.findings[5] |
| quantity | Thirty-five — — “Thirty-five 3D US volumes of fetal faces above 24 weeks of gestation (25 +0 –37 +0 ) were acquired for clinical indications between 2016 and 2018 using Voluson E6, E8 or E10 ultrasound machine (GE Healthcare) with a low-frequency probe (4–8 MHz) and retrospectively included in this study.” | content_tags.quantities[0] |
| quantity | 11 abnormal and 24 normal — — “Cases were defined as normal or abnormal based upon the antenatal US diagnosis, with a total of 11 abnormal and 24 normal cases.” | content_tags.quantities[1] |
| quantity | 5 h3 m — — “The average time required to perform the MS was 5 h3 m per face (range 3 h2 m–6h 10 m).” | content_tags.quantities[2] |
| quantity | 0.44 mm — — “While a difference in ASSD was demonstrated, the ‘real-life’ difference was minimal, equating to a median of 0.44 mm (0.16–1.1 mm).” | content_tags.quantities[3] |
| quantity | 0.98 — — “However, this difference is minimal with the differences equal to 0.86 standard deviations for the Dice score and 0.83 standard deviations for the ASSD, and an ICC of 0.98 confirms good overall agreement between operators on the segmentation volume.” | content_tags.quantities[4] |
| variable | atlas size — “The algorithm showed improved performance following an increase in the atlas size in turn reducing the need for manual refinement.” | content_tags.variables[0] |
| variable | computational time — “As expected, average computational time increases by increasing the number of ground-truth images in the atlas, with differences ( p < 0.001) between all comparisons” | content_tags.variables[1] |
| definition | image segmentation — “In order to achieve this, a surface representation or map of the fetal face is extracted from the 3D volume via image segmentation, a process to divide an image or volume into non-overlapping separate parts or ‘segments’ to facilitate volumetric image processing and analysis [ 7 ].” | content_tags.definitions[0] |
| definition | ASSD — “The average symmetric surface distance (ASSD) calculates how much the surface varies between two segmentations.” | content_tags.definitions[1] |
| key-terms | image segmentation, fetal face, three-dimensional ultrasound, atlas, Dice score, manual segmentation, automatic segmentation, craniofacial abnormalities, inter-operator variability, ASSD |
| concept | image segmentation — a process to divide an image or volume into non-overlapping separate parts or ‘segments’ to facilitate volumetric image processing and analysis [ 7 ] — “In order to achieve this, a surface representation or map of the fetal face is extracted from the 3D volume via image segmentation, a process to divide an image or volume into non-overlapping separate parts or ‘segments’ to facilitate volumetric image processing and analysis [ 7 ].” | concepts[0] |
| concept | US — ultrasound — “Fetal craniofacial abnormalities are challenging to detect and diagnose on prenatal ultrasound (US).” | concepts[1] |
| concept | ultrasound — not defined in this paper's own words — “Fetal craniofacial abnormalities are challenging to detect and diagnose on prenatal ultrasound (US).” | concepts[2] |
| concept | MS — manual segmentation — “In previous work by this group, manual segmentation (MS) was used to delineate the fetal face surfaces meshes from 20 3D US volumes [ 6 ].” | concepts[3] |
| concept | manual segmentation — not defined in this paper's own words; segmentation delineated by hand — “In previous work by this group, manual segmentation (MS) was used to delineate the fetal face surfaces meshes from 20 3D US volumes [ 6 ].” | concepts[4] |
| concept | AS — automatic segmentation — “This study assesses the inter- and intra-operator variability of MS and tests an optimized version of our automatic segmentation (AS) algorithm.” | concepts[5] |
| concept | automatic segmentation — not defined in this paper's own words — “This study assesses the inter- and intra-operator variability of MS and tests an optimized version of our automatic segmentation (AS) algorithm.” | concepts[6] |
| concept | Multi-atlas propagation segmentation — uses a set of already segmented ‘ground-truth’ guide images (i.e. ‘atlas’), as a reference bank for an algorithm tasked with building the segmentation of a new, unseen image — “Multi-atlas propagation segmentation uses a set of already segmented ‘ground-truth’ guide images (i.e. ‘atlas’), as a reference bank for an algorithm tasked with building the segmentation of a new, unseen image.” | concepts[7] |
| concept | atlas — a set of already segmented ‘ground-truth’ guide images (i.e. ‘atlas’) — “Multi-atlas propagation segmentation uses a set of already segmented ‘ground-truth’ guide images (i.e. ‘atlas’), as a reference bank for an algorithm tasked with building the segmentation of a new, unseen image.” | concepts[8] |
| concept | ASSD — average symmetric surface distance — “The average symmetric surface distance (ASSD) calculates how much the surface varies between two segmentations.” | concepts[9] |
| concept | average symmetric surface distance — calculates how much the surface varies between two segmentations — “The average symmetric surface distance (ASSD) calculates how much the surface varies between two segmentations.” | concepts[10] |
| concept | ICC — Intraclass correlation coefficient — “Intraclass correlation coefficient (ICC) using repeated measures ANOVA was calculated to compare the total segmentation volumes of each operator.” | concepts[11] |
| concept | Intraclass correlation coefficient — not defined in this paper's own words — “Intraclass correlation coefficient (ICC) using repeated measures ANOVA was calculated to compare the total segmentation volumes of each operator.” | concepts[12] |
| concept | Dice score — not defined in this paper's own words; the overlap between two segmentations, from 0 to 1 — “A good overlap is considered when Dice score ≥ 0.7 [ 19 ].” | concepts[13] |
| concept | atlas size — not defined in this paper's own words; the number of ground-truth images in the atlas — “We have also shown that it is possible to improve the performance of our AS algorithm by increasing the atlas size which in turn reduces the amount of manual refinement required.” | concepts[14] |
| concept | manual refinement — not defined in this paper's own words — “We have also shown that it is possible to improve the performance of our AS algorithm by increasing the atlas size which in turn reduces the amount of manual refinement required.” | concepts[15] |
| concept | computational time — not defined in this paper's own words — “As expected, average computational time increases by increasing the number of ground-truth images in the atlas, with differences ( p < 0.001) between all comparisons” | concepts[16] |
| concept | ground-truth images — not defined in this paper's own words — “As expected, average computational time increases by increasing the number of ground-truth images in the atlas, with differences ( p < 0.001) between all comparisons” | concepts[17] |
| Dice coefficient | as written: Dice score ≥ 0.7 — “A good overlap is considered when Dice score ≥ 0.7 [ 19 ].” | content_tags.statistics[0] |
| intraclass correlation coefficient ICC | as written: ICC of 0.98 — “However, this difference is minimal with the differences equal to 0.86 standard deviations for the Dice score and 0.83 standard deviations for the ASSD, and an ICC of 0.98 confirms good overall agreement between operators on the segmentation volume.” | content_tags.statistics[1] |
| Kruskal–Wallis test | as written: Kruskal–Wallis p < 0.001 — “Atlas size had an impact on the Dice score (Kruskal–Wallis p < 0.001).” | content_tags.statistics[2] |
Full text — tagged
Developing and testing an algorithm for automatic segmentationterm·mth of the fetal faceterm·key from three-dimensional ultrasoundterm·key images Abstract Fetal craniofacial abnormalitiesterm·key are challenging to detect and diagnose on prenatal ultrasoundterm·mth (USterm·mth). Image segmentationterm·mth and computer analysis of three-dimensional USterm·mth volumes of the fetal faceterm·key may provide an objective measure to quantify fetal facial features and identify abnormalities. We have developed and tested an atlas-based partially automated facial segmentation algorithm; however, the volumes require additional manual segmentationterm·mth (MSterm·mth), which is time and labour intensive and may preclude this method from clinical adoption. These manually refined segmentations can then be used asterm·mth a reference (atlasterm·mth) by the partially automated segmentation algorithm to improve algorithmic performance with the aim of eliminating the need for manual refinementterm·mth and developing a fully automated system. This study assesses the inter- and intra-operator variability of MSterm·mth and tests an optimized version of our automatic segmentationterm·mth (ASterm·mth) algorithm. The manual refinementsterm·mth of 15 fetal facesterm·key performed by three operators and repeated by one operator were assessed by Dice scoreterm·mth, average symmetrical surface distance and volume difference. The performance of the partially automatic algorithm with difference size atlasesterm·mth was evaluated by Dice scoreterm·mth and computational timeterm·dom. Assessment of the manual refinementsterm·mth showed low inter- and intra-operator variability demonstrating its suitability for optimizing the ASterm·mth algorithm. The algorithm showed improved performance following an increase in the atlas sizeterm·dom in turn reducing the need for manual refinementterm·mth.
Keywords: three-dimensional ultrasoundterm·key, craniofacial abnormalitiesterm·key, image segmentationterm·mth Introduction Fetal craniofacial abnormalitiesterm·key provide valuable clues in diagnosing genetic conditions and syndromes. However, many of these phenotypical features can be subtle and, therefore, challenging to identify on prenatal two-dimensional (2D) ultrasoundterm·mth (USterm·mth) even for the most expert of investigators. The use of three-dimensional (3D) USterm·mth scans complement 2D findings and can provide additional information, both aiding the identification of abnormalities in utero asterm·mth well asterm·mth providing additional diagnostic clues, thus facilitating targeted genetic testing [ 1 – 5 ].
Traditionally, analysis of 3D USterm·mth scans is subjective, operator dependent and heavily reliant on the breadth of the operator's experience and with rarer conditions evading detection asterm·mth their features are not universally well known. Computerized analysis of 3D USterm·mth volumes could provide an objective method to characterize fetal facial morphology [ 6 ], which may assist in identifying conditions with unknown genetic factors but well-characterized craniofacial features and improve the diagnosis of phenotypically heterogenous conditions.
In order to achieve this, a surface representation or map of the fetal faceterm·key is extracted from the 3D volume via image segmentationterm·mth, a process to divide an image or volume into non-overlapping separate parts or ‘segments’ to facilitate volumetric image processing and analysis [ 7 ]. In previous work by this group, manual segmentationterm·mth (MSterm·mth) was used to delineate the fetal faceterm·key surfaces meshes from 20 3D USterm·mth volumes [ 6 ]. This is a particularly time-consuming process which precludes it from everyday clinical use.
The use of various methods for the automatic reconstruction of the fetal faceterm·key from 3D USterm·mth data have been explored previously. Feng et al . [ 8 ] developed an automatic method for fetal faceterm·key detection; however, their algorithm results in the creation of a mesh rather than a true surface representation of the face. Bonacina et al . [ 9 ] describe an automatic method using histogram processing which they applied to five 3D USterm·mth volumes with good result, although fetal contact with maternal tissue interfered with the algorithmic performance in one of these cases, and Speranza et al . detail a technique to extract 3D data from USterm·mth images which enables 3D models of the fetal faceterm·key to be printed [ 10 ].
In order to overcome the limitations of MSterm·mth and facilitate the clinical adoption of this technology, we implemented an atlas-based partially automatic segmentationterm·mth (ASterm·mth) method, based on algorithms described by Zuluaga et al . [ 11 ]. Multi-atlas propagation segmentationterm·mth usesterm·mth a set of already segmented ‘ground-truth’ guide images (i.e. ‘atlasterm·mth’), asterm·mth a reference bank for an algorithm tasked with building the segmentation of a new, unseen image. It selects the most relevant parts of the different atlasterm·mth images and fuses them in order to create the new image. Preliminary results based on an atlasterm·mth of 20 images have demonstrated its feasibility for the extraction of 3D faces from fetal 3D USterm·mth [ 12 ]. However, in order to further improve this technique and to enable analysis of the segmented volume, additional refinement of the ASterm·mth via MSterm·mth is required ( figure 1 ) in order to clearly define the facial borders, remove extraneous material and complete parts of the face which may be missing ( figure 2 ).
Example of an automatically segmented (ASterm·mth) fetal faceterm·key before (left) and after (right) manual refinementterm·mth of the ASterm·mth has been performed.
Examples of the segmentation produced by the automatic algorithm (top), the corresponding 3D USterm·mth volume (middle) and the final segmentation following manual refinementterm·mth (bottom).
It has been suggested that MSterm·mth is a subjective process, with variation between operators potentially impairing the accuracy of subsequent quantitative analysis. The variability of MSterm·mth has been examined previously in other imaging modalities and anatomical structures [ 13 – 16 ]. We identified one study which compared the MSterm·mth of five utero-fetal unit segmentations to those of an ASterm·mth method [ 17 ]. However, to the best of our knowledge no studies have been conducted examining the variability of MSterm·mth or the use of an automatic atlas-based segmentation algorithm specifically for the fetal faceterm·key.
Aims The aims of this study were to: 1. Evaluate the inter- and intra-operator variability of the manual refinementsterm·mth of automatic segmentationsterm·mth in order to assess the accuracy of this method.
2. Test an optimized version of our partially ASterm·mth algorithm.
Methods Thirty-five 3D USterm·mth volumes of fetal facesterm·key above 24 weeks of gestation (25 +0 –37 +0 ) were acquired for clinical indications between 2016 and 2018 using Voluson E6, E8 or E10 ultrasoundterm·mth machine (GE Healthcare) with a low-frequency probe (4–8 MHz) and retrospectively included in this study. Cases were defined asterm·mth normal or abnormal based upon the antenatal USterm·mth diagnosis, with a total of 11 abnormal and 24 normal cases. The volumes were fully anonymized in order to allow their inclusion in this technical feasibility study. Ethical approval was not required for this study using fully anonymized routinely collected USterm·mth volumes (Prenatal 3D face study: technical feasibility and improving the methodology) based on information provided to the online Health Research Authority tool ( http://www.hra-decisiontools.org.uk/research/ ). An overview of the main steps of our workflow is illustrated in figure 3 .
A schematic of the workflow for this study. The manual refinementsterm·mth of Op_1 #1 were then progressively added to the original set of 20 images in order to increase the atlas sizeterm·dom.
The 3D volumes were acquired in the midsagittal plane of the fetal faceterm·key, using the highest resolution acquisition mode and a 3D volume box matched to the size of the fetal faceterm·key. The volume was acquired where possible without intervening umbilical cord or fetal limbs. The volumes were aligned using multiplanar view along the x- , y- and z -axes and subsequently exported asterm·mth Cartesian volumes (.vol) for further processing. The first 20 facial volumes were manually segmented in Mimics (Materialise NV, Belgium) and used to create our ASterm·mth tool [ 12 ]. A further 15 facial volumes were segmented via our partially ASterm·mth tool with additional manual refinementterm·mth performed in ITK-SNAP ( www.itksnap.org ) [ 18 ]. The mid-points of the posterior border of the widest cross-section of the orbits were used asterm·mth a landmark to define the depth of manual refinementterm·mth required in ITK-SNAP.
Inter- and intra-operator manual segmentationterm·mth assessment Fifteen additional volumes were initially segmented by our partially ASterm·mth tool with manual refinementterm·mth of the automatic segmentationsterm·mth performed on all 15 volumes by three different operators (op_1, op_2 and op_3). Op_1 repeated the segmentation of the 15 volumes twice within a 6-month period denoted by op_1 #1 and op_1 #2 . All operators were blind to one another's manual segmentationsterm·mth.
In order to assess inter-operator variabilityterm·key, a comparison of 15 manual segmentationsterm·mth (MSterm·mth) was performed between the three operators and intra-operator variability assessed by comparing the 15 repeated segmentations of op_1 (op_1 #1 -op_2, op_1 #1 -op_3, op_2-op_3 and op_1 #1 -op_1 #2 ).
We assessed three commonly used parameters which evaluate overlap, surface distance and segmentation volume in order to assess both the accuracy and reproducibility of the manual segmentationsterm·mth.
Overlap assessment The degree of overlap between the MSterm·mth performed by all three operators asterm·mth well asterm·mth between the repeated segmentations of op_1 was calculated using the Dice coefficientterm·stat (0–1). A good overlap is considered when Dice scoreterm·mth ≥ 0.7 [ 19 ].
The Dice scoreterm·mth [ 20 ] was computed asterm·mth where M 1 and M 2 are the segmentation masks and | M 1 | and | M 2 | are the volumes of these segmented regions. Given the complexity of the fetal faceterm·key and how subtle the changes seen in dysmorphic facial features can be, we considered a Dice scoreterm·mth ≥ 0.8 to be acceptable in this context.
Surface assessment The average symmetric surface distanceterm·mth (ASSDterm·mth) calculates how much the surface varies between two segmentations. The Euclidean distance is calculated between each surface voxel of one segmentation to the closest surface voxel of the other segmentation. The average of all of these distances is then calculated in millimetres, with a result of 0 equivalent to two identical segmentations. The ASSDterm·mth was calculated asterm·mth: where S 1 and S 2 are the set of voxels on the surface (in our case the face) of each segmentation, and the distance of a point to a surface is defined by the minimal Euclidean distance from this point to a point on the surface: Volume assessment The volume of each MSterm·mth was calculated asterm·mth the sum of the total segmented voxels. The volumes were then compared by computing the pairwise percentage of volume differences asterm·mth: where V i and V j are the segmentation volumes of the same image asterm·mth produced by operators i and j , respectively.
Automatic segmentationterm·mth algorithm A partially automatic atlas-based segmentation (ASterm·mth) algorithm was implemented, asterm·mth described by Zuluaga et al. [ 11 ], that can build a segmentation of a previously unseen image based on a set of already segmented (i.e. ‘ground-truth’) images or ‘atlasterm·mth’. Default parameters were used for both the non-rigid registration using the Fast Free-Form Deformation (F3D) algorithm [ 21 ] and the label fusion step using the niftySeg implementation of the Similarity and Truth Estimation for Propagated Segmentations (STEPS) algorithm [ 22 ]. To understand the optimum number of ground-truth imagesterm·mth required for the atlasterm·mth to achieve the highest quality segmentation in a reasonable computational timeterm·dom, we evaluated the algorithms' performance for different numbers of manually segmented ground-truth imagesterm·mth: 20, 25, 30 and 35 images. To do so, we created a growing sequence of atlasesterm·mth (respectively named A_20, A_25, A_30 and A_35) by progressively adding the new manually refined segmentations by op_1 #1 to the original set of 20 images . The algorithm was validated by applying a ‘leave-one-out’ cross-validation scheme on the first 20 images of each atlasterm·mth, i.e. each of the images was automatically segmented with the remaining (19 for A_20, 24 for A_25… etc.) adopted asterm·mth atlasterm·mth. The segmentations resulting from the automatic process were compared with the ground-truth segmentations in terms of visual assessment, Dice scoreterm·mth and computational timeterm·dom, to evaluate the tool performance and select the best atlasterm·mth combination.
Statistical analysis Statistical analysis was performed using Scipy statistical module [ 23 ]; where applicable, individual p -values are presented.
In order to examine the inter- and intra-operator variability, a Friedman test was performed for the four pairwise comparisons (op_1 #1 -op_2, op_1 #1 -op_3, op_2-op_3 and op_1 #1 -op_1 #2 ) to assess the consistency of Dice scoreterm·mth, ASSDterm·mth and segmentation volume for all 15 manually refined 3D USterm·mth volumes between operators (significance level p < 0.05). When the null hypothesis was rejected, a post hoc analysis with Bonferoni correction (significance level p < 0.017) was performed to identify whether the difference was in the inter- or intra-operator measurements.
Intraclass correlation coefficientterm·mth (ICCterm·mth) using repeated measures ANOVA was calculated to compare the total segmentation volumes of each operator.
T -test for the linear model between average Dice scoreterm·mth, ASSDterm·mth and the cubic root of the segmentation volume was performed to assess correlation between segmentation size (volume) and inter-operator differences. When assessing the ASterm·mth algorithm, analysis of variance was performed with a non-parametric Kruskal–Wallis ( p = 0.05) followed by post hoc testing with Bonferroni correction (significance level p = 0.0083) in order to assess difference between Dice scoreterm·mth distributions.
Results Manual segmentationterm·mth assessment Median Dice scoresterm·mth, ASSDterm·mth and pairwise percentage of volume differences between operators are represented in table 1 and figure 4 . When comparing the overall similarity parameters for the four pairwise comparisons (op_1 #1 -op_2, op_1 #1 -op_3, op_2-op_3 and op_1 #1 -op_1 #2 ), a difference was found in Dice scoresterm·mth (Friedman p = 0.02) and ASSDterm·mth ( p < 0.001) but not in the percentage of segmentation volume difference ( p = 0.16). While a difference in Dice scoreterm·mth between operators was found, the median values for this parameter were all above the 0.7 acceptable minimum threshold, the minimum values in each comparison also exceeding 0.8 which we chose asterm·mth our internal threshold. Moreover, the central facial region (superimposed for difference operators in figure 4 ) is the most important for feature identification and also has the best agreement even where intra- and inter-operator Dice scoreterm·mth are lowest.
Values represent median Dice scoresterm·mth, average symmetrical surface distance (ASSDterm·mth) and percentage of segmented volume difference of the manual segmentationsterm·mth performed by the three different operators and between the initial and repeated MSterm·mth performed by op_1. Data are expressed asterm·mth median ± IQR (min–max).
Rendering of the lowest (left column) and highest (right column) MSterm·mth agreement between the three operators and repeated segmentation of operator 1 asterm·mth assessed by the Dice scoreterm·mth.
Post hoc analysis demonstrated that these differences in Dice scoreterm·mth ( p = 0.03) and ASSDterm·mth ( p = 0.002) can both be attributed to a lower intra-operator variability when compared to the three corresponding inter-operator measurements ( table 1 ). However, this difference is minimal with the differences equal to 0.86 standard deviations for the Dice scoreterm·mth and 0.83 standard deviations for the ASSDterm·mth, and an ICCterm·mth of 0.98 confirms good overall agreement between operators on the segmentation volume.
There was no correlation found between overall segmentation volume and inter-operator differences for the average Dice scoreterm·mth ( T -test p = 0.66) or ASSDterm·mth ( p = 0.25) values.
The average time required to perform the MSterm·mth was 5 h3 m per face (range 3 h2 m–6h 10 m).
Automatic segmentationterm·mth The ASterm·mth tool returned a segmentation for all images. Dice scoresterm·mth for the four atlasesterm·mth tested are plotted in figure 5 a for all 20 patients. The boxplot of Dice scoreterm·mth distribution is represented in figure 5 b and table 2 .
Dice scoreterm·mth ( a ), segmentation volume ( c ) and computational timeterm·dom ( e ) plotted for each ASterm·mth, and boxplot of their distribution ( b , d , f ) obtained with the different atlasesterm·mth tested.
Values represent median Dice scoresterm·mth when comparing the manual segmentationsterm·mth with the automatic segmentationsterm·mth obtained with each of the four different atlasesterm·mth tested. Data is expressed asterm·mth median ± IQR (min–max).
Atlas sizeterm·dom had an impact on the Dice scoreterm·mth (Kruskal–Wallis p < 0.001). Post hoc testing demonstrated differences between A_20 versus A_30 ( p = 0.001), A_20 versus A_35 ( p = 0.001) and A_25 versus A_30 ( p = 0.007) but no difference for A_20 versus A_25 ( p = 0.32) or A_30 versus A_35 ( p = 0.96) ( table 2 and figure 5 a,b ). The lowest and highest results according to Dice scoreterm·mth are illustrated in figure 6 for each atlasterm·mth, showing that the main facial features could be represented even in the cases of the lowest Dice scoreterm·mth.
Lowest (top) and highest (bottom) Dice scoresterm·mth for the ASterm·mth of each atlasterm·mth: for A_20 (red), A_25 (blue), A_30 (green) and A_35 (orange) compared to the MSterm·mth (white) asterm·mth assessed by the Dice scoreterm·mth.
Segmentations were also compared in terms of volume ( figure 5 c,d ). An effect of the number of ground-truth cases on the resulting segmentation volume was found ( p < 0.001). In the post hoc analysis, differences were found between A_20 and A_30 ( p = 0.001), A_20 and A_35 ( p < 0.001), A_25 and A_35 ( p = 0.001), and to a lesser extent between A_25 and A_30 ( p = 0.005). There was no difference between A_20 and A_25 (p = 0.34) or A_30 and A_35 ( p = 0.42). The average volume size of the resulting segmentation volume increases with the number of ground-truth imagesterm·mth in the atlasterm·mth but for every atlasterm·mth (A_20, A_25, A_30 and A_35), the segmented volume is on average smaller than the ground-truth volume ( p < 0.001).
The computational timeterm·dom is illustrated in figure 5 e,f , and in table 3 . Asterm·mth expected, average computational timeterm·dom increases by increasing the number of ground-truth imagesterm·mth in the atlasterm·mth, with differences ( p < 0.001) between all comparisons (A_20 versus A_25, A_20 versus A_30, A_20 versus A_35, A_25 versus A_30, A_25 versus A_35 and A_30 versus A_35).
Values represent median computational timeterm·dom (in minutes) to obtain the ASterm·mth for each of the four different atlasesterm·mth tested. Data are expressed asterm·mth median ± IQR (min–max).
Of the four atlasesterm·mth tested, A_30 was the optimum atlas sizeterm·dom with an overall improvement in performance balanced by no excessive increase in computational timeterm·dom. When adopting A_35, the computational timeterm·dom significantly increased without a corresponding improvement in performance compared to A_30.
Discussion The aims of this study were to assess the inter- and intra-operator variability of manual refinementterm·mth of partial ASterm·mth of the fetal faceterm·key and to test an optimized version of our ASterm·mth algorithm. We have shown good reproducibility and accuracy of manual segmentationsterm·mth of fetal faceterm·key 3D USterm·mth volumes between operators asterm·mth well asterm·mth demonstrating low intra-operator variability with all Dice scoresterm·mth exceeding our internally set threshold of 0.8. While a difference in ASSDterm·mth was demonstrated, the ‘real-life’ difference was minimal, equating to a median of 0.44 mm (0.16–1.1 mm). This difference is unlikely to be clinically significant and not large enough to negatively impact upon the accuracy of subsequent diagnostic analysis of the segmentation volumes. We have also shown that it is possible to improve the performance of our ASterm·mth algorithm by increasing the atlas sizeterm·dom which in turn reduces the amount of manual refinementterm·mth required. However, beyond a certain point, atlas sizeterm·dom improvement was minimal and outweighed by significantly increased computational timeterm·dom.
A fully automatic and objective method to identify and characterize abnormal fetal faceterm·key morphology in utero has the potential to both increase prenatal diagnosis of genetic syndromes and asterm·mth a result improve patient counselling, allow robust delivery and postnatal management planning and provide additional long-term predictive information for parents. Some of these conditions cannot yet be diagnosed through prenatal invasive or non-invasive testing, and others can be, but parents may opt not to undergo testing [ 24 ]. Many genetic syndromes and conditions have very distinct craniofacial characteristics. Identifying these abnormalities on prenatal USterm·mth is largely subjective, requiring significant USterm·mth operator expertise and there is a propensity to recognize dysmorphic facial features when they coexist with other anatomical abnormalities.
Trisomies are the most identifiable aneuploidies: trisomy 21 fetuses have facial features of brachycephaly, midface hypoplasia, flattened nasal bridge and macroglossia, while trisomy 18 features a prominent occiput, low-set ears, and micrognathia [ 25 ], however, these conditions are readily diagnosable from invasive and non-invasive prenatal diagnosis. By contrast, rarer syndromes cannot always be diagnosed from prenatal testing asterm·mth in many the genetic abnormality is not known and in some a strong clinical suspicion of a diagnosis or differential diagnoses is required for targeted genetic testing [ 24 , 26 , 27 ]. These may also be associated with developmental or physical impairments asterm·mth the child grows older, hence the importance of prenatal diagnosis [ 28 – 30 ].
Such conditions include Crouzon syndrome which is characteristed by brachycephaly, frontal bossing, shallow orbits and maxilla hypoplasia, and Apert syndrome with flat facies, prominent forehead and hypertelorism [ 25 , 30 ]. Frontal bossing, a flattened nasal bridge and micrognathia is a feature of many skeletal dysplasias and although severity can vary significantly, the malar hypoplasia, zygomatic bone cleft and down-slanting palpebral fissures can be very distinct in Treacher Collins syndrome [ 25 ]. The diagnosis of Cornelia de Lange syndrome (CdLS) is rarely made on prenatal USterm·mth but has profound lifelong implications. Even when CdLS is suspected prenatally the diagnosis can remain elusive asterm·mth 30% of cases have unknown genetic aetiology [ 29 ]. The diagnosis is most commonly made clinically in the neonatal and paediatric period with distinct facial characteristics including low-set ears, short upturned nose, synophris, everted nostrils, micrognathia and long philtrum giving the diagnostic clues [ 31 ].
This study has some limitations. The number of segmentations performed was modest, although sufficient to demonstrate the low variability between segmentations and acceptability of this method. The operators were of differing skill levels, with two operators having no prior experience of performing or interpreting USterm·mth images. This had the potential to negatively impact on the variability between manual segmentationsterm·mth; however, our results suggest that this was not the case and rather, reinforce the reliability of the manual refinementterm·mth. There was also a potential for the operators to become more competent in manual segmentationsterm·mth over time, which may have impacted on the variability by reducing it. This was not the case: two of the operators performed the manual segmentationsterm·mth in the same order, with no differences between their performance and that of the other operator demonstrated.
Performing segmentations of USterm·mth volumes, in comparison to other imaging modalities such asterm·mth CT and MRI, is challenging given the variations in USterm·mth image quality through acquisition, operator skill, movement artefact, maternal body habitus, amniotic fluid volume and fetal position. These all result in reduced image quality and, therefore, reduced boundary definition within parts of the USterm·mth image. This then has the potential to cause higher variability between manual segmentationsterm·mth. However, our results suggest that these did not particularly affect the study and, given that USterm·mth remains the imaging modality of choice in anomaly screening and in specialist fetal medicine units, are reassuring in regard to the potential application of this method in clinical practice.
The main limitation of this technique and barrier to use in clinical practice is the need for additional manual refinementterm·mth and the time taken to perform this. The median segmentation time in this study was approximately 5 h per fetal facial volume, which is clearly impractical in everyday clinical practice and would preclude this technique from clinical use in its current form. With any segmentation method used to reconstruct the fetal faceterm·key, asterm·mth noted previously, the main concern would be in ensuring accuracy and avoiding the normalizing of an abnormal face or vice versa. At the present time, state-of-art methods are unable to accurately segment fetal facesterm·key from 3D USterm·mth images while being robust to a highly variable context (the fetal hand or umbilical cord in contact with or obstructing the face, contact with the uterine wall or placenta) and poor image quality [ 9 ]. These limitations are partially overcome by our automatic algorithm with promising results despite the inability to produce a complete segmentation of the face in the most difficult cases resulting in the need for manual corrections. We have demonstrated that the consistency of the manual segmentationsterm·mth is a good foundation for atlas-based segmentation algorithms. Moreover, the improvements seen in our partially automatic algorithm performance in regard to improved Dice scoresterm·mth and therefore accuracy associated with an increased number of reference images (atlasterm·mth) are positive. Further work will focus on more accurate and faster registration algorithms in order to move closer to a fully automated method suitable for clinical ‘point of diagnosis’ use.
Conclusion We have demonstrated the technical feasibility of fetal facial segmentation using manual refinementterm·mth of partially automatically segmented USterm·mth facial volumes. Moreover, the quality of ASterm·mth can be improved by increasing the atlas sizeterm·dom and training the algorithm with a larger number of manually segmented volumes thus reducing the need for manual refinementterm·mth of the automatic segmentationsterm·mth. Though not yet feasible, given the improvements that we have demonstrated in these techniques, it is possible to consider that further refinement would lead to a fully automated method for the reconstruction and quantification of the fetal faceterm·key. This is a prerequisite for clinical utility, asterm·mth to be an adjunct to prenatal diagnosis from fetal facial morphological appearance such a system would need to work almost in real time with the USterm·mth scan itself.
Supplementary Material Data accessibility The dataset used in this study consists of fully anonymized USterm·mth images. Asterm·mth outlined in our methods, ethical approval was not required for this technical feasibility study asterm·mth it did not fulfil the HRA requirements for research ethics consideration. Therefore, patients have not consented for their USterm·mth volumes to be publicly available. With the agreement of the journal's Editorial Office, the code for the fetal faceterm·key segmentation tool used in this paper is openly available on R.S.'s GitHub page: https://github.com/rsivera/fetal-faces-us_segmentation .
Authors' contributions C.C.L., S.S., A.E.C., B.B. and A.D. conceived and designed the study; A.E.C., A.D., C.C.L., L.F., T.-L.W. and G.P. were involved in data collection; A.E.C., B.B. and R.S. analysed the data for publication. A.C., B.B., R.S., A.D., S.S., L.F., T.-L.W., G.P., D.D. and C.C.L. drafted the article and revised it for important intellectual content.
Competing interests The authors declare no competing interests, financial or otherwise.
Funding This work was supported by the National Institute for Health Research (NIHR) Biomedical Research Centre based at Imperial College Healthcare NHS Trust and Imperial College London (grant no. P74419). The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health. This work was also supported by the Engineering and Physical Science Research Council (grant no. EP/N02124X/1) and the European Research Council (grant no. ERC-2017-StG-757923).
References Associated Data Supplementary Materials Data Availability Statement The dataset used in this study consists of fully anonymized USterm·mth images. Asterm·mth outlined in our methods, ethical approval was not required for this technical feasibility study asterm·mth it did not fulfil the HRA requirements for research ethics consideration. Therefore, patients have not consented for their USterm·mth volumes to be publicly available. With the agreement of the journal's Editorial Office, the code for the fetal faceterm·key segmentation tool used in this paper is openly available on R.S.'s GitHub page: https://github.com/rsivera/fetal-faces-us_segmentation .