A review of deep learning based methods for medical image multi-organ segmentation

Deep learning has revolutionized image processing and achieved the-state-of-art performance in many medical image segmentation tasks. Many deep learning-based methods have been published to segment different parts of the body for different medical applications. It is necessary to summarize the current state of development for deep learning in the field of medical image segmentation. In this paper, we aim to provide a comprehensive review with a focus on multi-organ image segmentation, which is crucial for radiotherapy where the tumor and organs-at-risk need to be contoured for treatment planning. We grouped the surveyed methods into two broad categories which are ‘pixel-wise classification’ and ‘end-to-end segmentation’. Each category was divided into subgroups according to their network design. For each type, we listed the surveyed works, highlighted important contributions and identified specific challenges. Following the detailed review, we discussed the achievements, shortcomings and future potentials of each category. To enable direct comparison, we listed the performance of the surveyed works that used thoracic and head-and-neck benchmark datasets.     

Using federated data sources and Varian Learning Portal framework to train a neural network model for automatic organ segmentation

PurposeIn this study we trained a deep neural network model for female pelvis organ segmentation using data from several sites without any personal data sharing. The goal was to assess its prediction power compared with the model trained in a centralized manner.MethodsVarian Learning Portal (VLP) is a distributed machine learning (ML) infrastructure enabling privacy-preserving research across hospitals from different regions or countries, within the framework of a trusted consortium. Such a framework is relevant in the case when there is a high level of trust among the participating sites, but there are legal restrictions which do not allow the actual data sharing between them. We trained an organ segmentation model for the female pelvic region using the synchronous data distributed framework provided by the VLP.ResultsThe prediction performance of the model trained using the federated framework offered by VLP was on the same level as the performance of the model trained in a centralized manner where all training data was pulled together in one centre.ConclusionsVLP infrastructure can be used for GPU-based training of a deep neural network for organ segmentation for the female pelvic region. This organ segmentation instance is particularly difficult due to the high variation in the organs’ shape and size. Being able to train the model using data from several clinics can help, for instance, by exposing the model to a larger range of data variations. VLP framework enables such a distributed training approach without sharing protected health information.     

Using Manifold Learning for Atlas Selection in Multi-Atlas Segmentation

Multi-atlas segmentation has been widely used to segment various anatomical structures. The success of this technique partly relies on the selection of atlases that are best mapped to a new target image after registration. Recently, manifold learning has been proposed as a method for atlas selection. Each manifold learning technique seeks to optimize a unique objective function. Therefore, different techniques produce different embeddings even when applied to the same data set. Previous studies used a single technique in their method and gave no reason for the choice of the manifold learning technique employed nor the theoretical grounds for the choice of the manifold parameters. In this study, we compare side-by-side the results given by 3 manifold learning techniques (Isomap, Laplacian Eigenmaps and Locally Linear Embedding) on the same data set. We assess the ability of those 3 different techniques to select the best atlases to combine in the framework of multi-atlas segmentation. First, a leave-one-out experiment is used to optimize our method on a set of 110 manually segmented atlases of hippocampi and find the manifold learning technique and associated manifold parameters that give the best segmentation accuracy. Then, the optimal parameters are used to automatically segment 30 subjects from the Alzheimer’s Disease Neuroimaging Initiative (ADNI). For our dataset, the selection of atlases with Locally Linear Embedding gives the best results. Our findings show that selection of atlases with manifold learning leads to segmentation accuracy close to or significantly higher than the state-of-the-art method and that accuracy can be increased by fine tuning the manifold learning process.     

Biview Learning for Human Posture Segmentation from 3D Points Cloud

Posture segmentation plays an essential role in human motion analysis. The state-of-the-art method extracts sufficiently high-dimensional features from 3D depth images for each 3D point and learns an efficient body part classifier. However, high-dimensional features are memory-consuming and difficult to handle on large-scale training dataset. In this paper, we propose an efficient two-stage dimension reduction scheme, termed biview learning, to encode two independent views which are depth-difference features (DDF) and relative position features (RPF). Biview learning explores the complementary property of DDF and RPF, and uses two stages to learn a compact yet comprehensive low-dimensional feature space for posture segmentation. In the first stage, discriminative locality alignment (DLA) is applied to the high-dimensional DDF to learn a discriminative low-dimensional representation. In the second stage, canonical correlation analysis (CCA) is used to explore the complementary property of RPF and the dimensionality reduced DDF. Finally, we train a support vector machine (SVM) over the output of CCA. We carefully validate the effectiveness of DLA and CCA utilized in the two-stage scheme on our 3D human points cloud dataset. Experimental results show that the proposed biview learning scheme significantly outperforms the state-of-the-art method for human posture segmentation.     

Determination of causes of death via spectrochemical analysis of forensic autopsies‐based pulmonary edema fluid samples with deep learning algorithm

This study investigated whether infrared spectroscopy combined with a deep learning algorithm could be a useful tool for determining causes of death by analyzing pulmonary edema fluid from forensic autopsies. A newly designed convolutional neural network‐based deep learning framework, named DeepIR and eight popular machine learning algorithms, were used to construct classifiers. The prediction performances of these classifiers demonstrated that DeepIR outperformed the machine learning algorithms in establishing classifiers to determine the causes of death. Moreover, DeepIR was generally less dependent on preprocessing procedures than were the machine learning algorithms; it provided the validation accuracy with a narrow range from 0.9661 to 0.9856 and the test accuracy ranging from 0.8774 to 0.9167 on the raw pulmonary edema fluid spectral dataset and the nine preprocessing protocol‐based datasets in our study. In conclusion, this study demonstrates that the deep learning‐equipped Fourier transform infrared spectroscopy technique has the potential to be an effective aid for determining causes of death.     

Investigating the effects of microstructural changes induced by myocardial infarction on the elastic parameters of the heart

Within this work, we investigate how physiologically observed microstructural changes induced by myocardial infarction impact the elastic parameters of the heart. We use the LMRP model for poroelastic composites (Miller and Penta in Contin Mech Thermodyn 32:1533–1557, 2020) to describe the microstructure of the myocardium and investigate microstructural changes such as loss of myocyte volume and increased matrix fibrosis as well as increased myocyte volume fraction in the areas surrounding the infarct. We also consider a 3D framework to model the myocardium microstructure with the addition of the intercalated disks, which provide the connections between adjacent myocytes. The results of our simulations agree with the physiological observations that can be made post-infarction. That is, the infarcted heart is much stiffer than the healthy heart but with reperfusion of the tissue it begins to soften. We also observe that with the increase in myocyte volume of the non-damaged myocytes the myocardium also begins to soften. With a measurable stiffness parameter the results of our model simulations could predict the range of porosity (reperfusion) that could help return the heart to the healthy stiffness. It would also be possible to predict the volume of the myocytes in the area surrounding the infarct from the overall stiffness measurements.

     

Benchmarking of deep learning algorithms for 3D instance segmentation of confocal image datasets

Segmenting three-dimensional (3D) microscopy images is essential for understanding phenomena like morphogenesis, cell division, cellular growth, and genetic expression patterns. Recently, deep learning (DL) pipelines have been developed, which claim to provide high accuracy segmentation of cellular images and are increasingly considered as the state of the art for image segmentation problems. However, it remains difficult to define their relative performances as the concurrent diversity and lack of uniform evaluation strategies makes it difficult to know how their results compare. In this paper, we first made an inventory of the available DL methods for 3D cell segmentation. We next implemented and quantitatively compared a number of representative DL pipelines, alongside a highly efficient non-DL method named MARS. The DL methods were trained on a common dataset of 3D cellular confocal microscopy images. Their segmentation accuracies were also tested in the presence of different image artifacts. A specific method for segmentation quality evaluation was adopted, which isolates segmentation errors due to under- or oversegmentation. This is complemented with a 3D visualization strategy for interactive exploration of segmentation quality. Our analysis shows that the DL pipelines have different levels of accuracy. Two of them, which are end-to-end 3D and were originally designed for cell boundary detection, show high performance and offer clear advantages in terms of adaptability to new data.     

DeepMIB: User-friendly and open-source software for training of deep learning network for biological image segmentation

We present DeepMIB, a new software package that is capable of training convolutional neural networks for segmentation of multidimensional microscopy datasets on any workstation. We demonstrate its successful application for segmentation of 2D and 3D electron and multicolor light microscopy datasets with isotropic and anisotropic voxels. We distribute DeepMIB as both an open-source multi-platform Matlab code and as compiled standalone application for Windows, MacOS and Linux. It comes in a single package that is simple to install and use as it does not require knowledge of programming. DeepMIB is suitable for everyone interested of bringing a power of deep learning into own image segmentation workflows.     

Nationwide claims data validated for quality assessments in acute myocardial infarction in the Netherlands

Introduction

Since health insurance is compulsory in the Netherlands, the centrally registered medical claims data might pose a unique opportunity to evaluate quality of (cardiac) care on a national level without additional collection of data. However, validation of these claims data has not yet been assessed.

Design

Retrospective cohort study.

Methods

National claims data (‘national registry’) were compared with data collected by patient records reviews in four representative hospitals (‘validation registry’). In both registries, we extracted the national diagnosis codes for ST-segment elevation myocardial infarction and non-ST-segment elevation myocardial infarction of 2012 and 2013. Additionally, data on medication use at one year after acute myocardial infarction (AMI) was extracted from the Dutch pharmacy information systems and also validated by local patient records reviews. The data were compared at three stages: 1) validation of diagnosis and treatment coding; 2) validation of the hospital where follow-up has taken place; 3) validation of follow-up medical treatment after 365 days.

Results

In total, 3,980 patients (‘national registry’) and 4,014 patients (‘validation registry’) were compared at baseline. After one-year follow-up, 2,776 and 2,701 patients, respectively, were evaluated. Baseline characteristics, diagnosis and individual medication were comparable between the two registries. Of all 52,672 AMI patients in the Netherlands in 2012 and 2013, 81% used aspirin, 76% used P2Y12 inhibitors, 85% used statins, 82% used beta-blockers and 74% angiotensin converting enzyme inhibitors/angiotensin II antagonists. Optimal medical treatment was achieved in 49% of the patients with AMI.

Conclusion

Nationwide routinely collected claims data in patients with an acute myocardial infarction are highly accurate. This offers an opportunity for use in quality assessments of cardiac care.

     

Myocardial material parameter estimation: a comparison of invariant based orthotropic constitutive equations

This study investigated a number of invariant based orthotropic and transversely isotropic constitutive equations for their suitability to fit three-dimensional simple shear mechanics data of passive myocardial tissue.

A number of orthotropic laws based on Green strain components and one microstructurally based law have previously been investigated to fit experimental measurements of stress-strain behaviour. Here we extend this investigation to include several recently proposed functional forms, i.e. invariant based orthotropic and transversely isotropic constitutive relations.

These laws were compared on the basis of (i) ‘goodness of fit’: how well they fit a set of six shear deformation tests, (ii) ‘variability’: how well determined the material parameters are over the range of experiments. These criteria were utilised to discuss the advantages and disadvantages of the constitutive laws.

It was found that a specific form of the polyconvex type as well as the exponential Fung-type law from the previous study were most suitable for modelling the orthotropic behaviour of myocardium under simple shear.     

Optimizing cardiac material parameters with a genetic algorithm

Determining the unknown material parameters of intact ventricular myocardium can be challenging due to highly nonlinear material behavior. Previous studies combining a gradient-search optimization procedure with finite element analysis (FEA) were limited to two-dimensional (2D) models or simplified three-dimensional (3D) geometries. Here we present a novel scheme to estimate unknown material parameters for ventricular myocardium by combining a genetic algorithm (GA) with nonlinear finite element analysis. This approach systematically explores the domain of the material parameters. The objective function to minimize was the error between simulated strain data and finite element model strains. The proposed scheme was validated for a 2D problem using a realistic material law for ventricular myocardium. Optimized material parameters were generally within 0.5% of the true values. To demonstrate the robustness of the new scheme, unknown material parameters were also determined for a realistic 3D heart model with an exponential hyperelastic material law. When using strains from two material points, the algorithm converged to parameters within 5% of the true values. We conclude that the proposed scheme is robust when estimating myocardial material parameters in 2D and 3D models.     

A Context-Aware Delayed Agglomeration Framework for Electron Microscopy Segmentation

Electron Microscopy (EM) image (or volume) segmentation has become significantly important in recent years as an instrument for connectomics. This paper proposes a novel agglomerative framework for EM segmentation. In particular, given an over-segmented image or volume, we propose a novel framework for accurately clustering regions of the same neuron. Unlike existing agglomerative methods, the proposed context-aware algorithm divides superpixels (over-segmented regions) of different biological entities into different subsets and agglomerates them separately. In addition, this paper describes a “delayed” scheme for agglomerative clustering that postpones some of the merge decisions, pertaining to newly formed bodies, in order to generate a more confident boundary prediction. We report significant improvements attained by the proposed approach in segmentation accuracy over existing standard methods on 2D and 3D datasets.     

Validation of three-dimensional conduction models using experimental mapping: are we getting closer?

The anisotropic material properties, irregular geometry, and specialized conduction system of the heart all affect the three-dimensional (3D) spread of electrical activation. A limited number of research groups have tried accounting for these features in 3D conduction models to investigate more thoroughly their observations of cardiac electrical activity in 3D experimental preparations. The full potential of these large scale conduction models, however, has not been realized because of a lack of quantitative validation with experiment. Such validation is critical in order to use the models to predict the electrical response of the myocardium to drugs or electrical stimulation. In this paper, a quantitative, experimental validation of paced activation in a 3D conduction model of a 3 cm × 3 cm × 1 cm section of the ventricular wall is presented. Epicardial and intramural pacing stimuli were applied in the center of a 528 channel electrode plaque sutured to the left ventricle in dogs. Unipolar electrograms were recorded at 2 kHz during and after pacing. Fiber directions within the tissue below the electrodes were estimated histologically and from pace-mapping. Simulated epicardial electrograms were computed for surface paced beats using our 3D bidomain model of the mapped tissue volume incorporating the measured fiber directions. Extracellular potentials and isochronal maps resulting from paced activations in both model and experiment were directly compared. Preliminary results demonstrate that our 3D model reproduces qualitatively such key features of the experimental data as electrogram morphologies and epicardial conduction velocities. Though quantitative agreement between model and experiment was only moderate, the validation approach described herein is an essential first step in assessing the predictive capability of present day conduction models.     

A hybrid graph-based approach for right ventricle segmentation in cardiac MRI by long axis information transition

Right ventricle segmentation is a challenging task in cardiac image analysis due to its complex anatomy and huge shape variations. In this paper, we proposed a semi-automatic approach by incorporating the right ventricle region and shape information into livewire framework and using one slice segmentation result for the segmentation of adjacent slices. The region term is created using our previously proposed region growing algorithm combined with the SUSAN edge detector while the shape prior is obtained by forming a signed distance function (SDF) from a set of binary masks of the right ventricle and applying PCA on them. Short axis slices are divided into two groups: primary and secondary slices. A primary slice is segmented by the proposed modified livewire and the livewire seeds are transited to a pre-processed version of upper and lower slices (secondary) to find new seed positions in these slices. The shortest path algorithm is applied on each pair of seeds for segmentation. This method is applied on 48 MR patients (from MICCAI’12 Right Ventricle Segmentation Challenge) and yielded an average Dice Metric of 0.937 ± 0.58 and the Hausdorff Distance of 5.16 ± 2.88 mm for endocardium segmentation. The correlation with the ground truth contours were measured as 0.99, 0.98, and 0.93 for EDV, ESV and EF respectively. The qualitative and quantitative results declare that the proposed method outperforms the state-of-the-art methods that uses the same dataset and the cardiac global functional parameters are calculated robustly by the proposed method.     

Rotationally resliced 3D prostate segmentation of MR images using Bhattacharyya similarity and active band theory

PurposeIn this article, we propose a novel, semi-automatic segmentation method to process 3D MR images of the prostate using the Bhattacharyya coefficient and active band theory with the goal of providing technical support for computer-aided diagnosis and surgery of the prostate.MethodsOur method consecutively segments a stack of rotationally resectioned 2D slices of a prostate MR image by assessing the similarity of the shape and intensity distribution in neighboring slices. 2D segmentation is first performed on an initial slice by manually selecting several points on the prostate boundary, after which the segmentation results are propagated consecutively to neighboring slices. A framework of iterative graph cuts is used to optimize the energy function, which contains a global term for the Bhattacharyya coefficient with the help of an auxiliary function. Our method does not require previously segmented data for training or for building statistical models, and manual intervention can be applied flexibly and intuitively, indicating the potential utility of this method in the clinic.ResultsWe tested our method on 3D T2-weighted MR images from the ISBI dataset and PROMISE12 dataset of 129 patients, and the Dice similarity coefficients were 90.34 ± 2.21% and 89.32 ± 3.08%, respectively. The comparison was performed with several state-of-the-art methods, and the results demonstrate that the proposed method is robust and accurate, achieving similar or higher accuracy than other methods without requiring training.ConclusionThe proposed algorithm for segmenting 3D MR images of the prostate is accurate, robust, and readily applicable to a clinical environment for computer-aided surgery or diagnosis.     

Automated detection and segmentation of synaptic contacts in nearly isotropic serial electron microscopy images

We describe a protocol for fully automated detection and segmentation of asymmetric, presumed excitatory, synapses in serial electron microscopy images of the adult mammalian cerebral cortex, taken with the focused ion beam, scanning electron microscope (FIB/SEM). The procedure is based on interactive machine learning and only requires a few labeled synapses for training. The statistical learning is performed on geometrical features of 3D neighborhoods of each voxel and can fully exploit the high z-resolution of the data. On a quantitative validation dataset of 111 synapses in 409 images of 1948×1342 pixels with manual annotations by three independent experts the error rate of the algorithm was found to be comparable to that of the experts (0.92 recall at 0.89 precision). Our software offers a convenient interface for labeling the training data and the possibility to visualize and proofread the results in 3D. The source code, the test dataset and the ground truth annotation are freely available on the website http://www.ilastik.org/synapse-detection.     

Empirical research in clinical ethics: The ‘committed researcher’ approach

After the ‘empirical turn’ in bioethics, few specific approaches have been developed for doing clinical ethics research in close connection with clinical decision-making on a daily basis. In this paper we describe the ‘committed researcher’ approach to research in clinical ethics that we have developed over the years. After comparing it to two similar research methodological approaches, the ‘embedded researcher’ and ‘deliberative engagement’, we highlight its main features: it is patient-oriented, it is implemented by collegial and multidisciplinary teams, it uses an ethical grid to build the interview guide, and it is geared towards bringing the results to bear on the public debate surrounding the issue at stake. Finally, we position our methodological approach with respect to the ‘is vs. ought’ distinction. We argue that our ‘commitment researcher’ approach to clinical ethics research takes concerned people’s life-building values as the main data, and compares them to the larger normative framework underlying the medical practice at stake.     

The IDIP framework for assessing protein function and its application to the prion protein

The quest to determine the function of a protein can represent a profound challenge. Although this task is the mandate of countless research groups, a general framework for how it can be approached is conspicuously lacking. Moreover, even expectations for when the function of a protein can be considered to be ‘known’ are not well defined. In this review, we begin by introducing concepts pertinent to the challenge of protein function assignments. We then propose a framework for inferring a protein's function from four data categories: ‘inheritance’, ‘distribution’, ‘interactions’ and ‘phenotypes’ (IDIP). We document that the functions of proteins emerge at the intersection of inferences drawn from these data categories and emphasise the benefit of considering them in an evolutionary context. We then apply this approach to the cellular prion protein (PrP^C), well known for its central role in prion diseases, whose function continues to be considered elusive by many investigators. We document that available data converge on the conclusion that the function of the prion protein is to control a critical post-translational modification of the neural cell adhesion molecule in the context of epithelial-to-mesenchymal transition and related plasticity programmes. Finally, we argue that this proposed function of PrP^C has already passed the test of time and is concordant with the IDIP framework in a way that other functions considered for this protein fail to achieve. We anticipate that the IDIP framework and the concepts analysed herein will aid the investigation of other proteins whose primary functional assignments have thus far been intractable.