Effect of segmentation errors on 3D-to-2D registration of implant models in X-ray images

In many biomedical applications, it is desirable to estimate the three-dimensional (3D) position and orientation (pose) of a metallic rigid object (such as a knee or hip implant) from its projection in a two-dimensional (2D) X-ray image. If the geometry of the object is known, as well as the details of the image formation process, then the pose of the object with respect to the sensor can be determined. A common method for 3D-to-2D registration is to first segment the silhouette contour from the X-ray image; that is, identify all points in the image that belong to the 2D silhouette and not to the background. This segmentation step is then followed by a search for the 3D pose that will best match the observed contour with a predicted contour. Although the silhouette of a metallic object is often clearly visible in an X-ray image, adjacent tissue and occlusions can make the exact location of the silhouette contour difficult to determine in places. Occlusion can occur when another object (such as another implant component) partially blocks the view of the object of interest. In this paper, we argue that common methods for segmentation can produce errors in the location of the 2D contour, and hence errors in the resulting 3D estimate of the pose. We show, on a typical fluoroscopy image of a knee implant component, that interactive and automatic methods for segmentation result in segmented contours that vary significantly. We show how the variability in the 2D contours (quantified by two different metrics) corresponds to variability in the 3D poses. Finally, we illustrate how traditional segmentation methods can fail completely in the (not uncommon) cases of images with occlusion.     

Dynamic Lung Modeling and Tumor Tracking Using Deformable Image Registration and Geometric Smoothing

A greyscale-based fully automatic deformable image registration algorithm, based on an optical flow method together with geometric smoothing, is developed for dynamic lung modeling and tumor tracking. In our computational processing pipeline, the input data is a set of 4D CT images with 10 phases. The triangle mesh of the lung model is directly extracted from the more stable exhale phase (Phase 5). In addition, we represent the lung surface model in 3D volumetric format by applying a signed distance function and then generate tetrahedral meshes. Our registration algorithm works for both triangle and tetrahedral meshes. In CT images, the intensity value reflects the local tissue density. For each grid point, we calculate the displacement from the static image (Phase 5) to match with the moving image (other phases) by using merely intensity values of the CT images. The optical flow computation is followed by a regularization of the deformation field using geometric smoothing. Lung volume change and the maximum lung tissue movement are used to evaluate the accuracy of the application. Our testing results suggest that the application of deformable registration algorithm is an effective way for delineating and tracking tumor motion in image-guided radiotherapy.     

Optical-CT Dual-Modality Mapping Base on DRR Registration

Optical-CT dual-modality imaging requires the mapping between 2D fluorescence images and 3D body surface light flux. In this paper, we proposed an optical-CT dual-modality image mapping algorithm based on the Digitally Reconstructed Radiography (DRR) registration. In the process of registration, a series of DRR images were computed from CT data using the ray casting algorithm. Then, the improved HMNI similarity strategy based on Hausdorff distance was used to complete the registration of the white-light optical images and DRR virtual images. According to the corresponding relationship obtained by the image registration and the Lambert’s cosine law based on the pin-hole imaging model, the 3D light intensity distribution on the surface of the object could be solved. The feasibility and effectiveness of the mapping algorithm are verified by the irregular phantom and mouse experiments.     

Registration of on-board X-ray images with 4DCT: A proposed method of phase and setup verification for gated radiotherapy

Current gated radiation therapy starts with simulation 4DCT images of a patient with lung cancer. We propose a method to confirm the phase of 4DCT for planning and setup position at the time of treatment. An intensity-based rigid algorithm was developed in this work to register an orthogonal set of on-board projection X-ray images with each phase of the 4DCT. Multiple DRRs for one of ten 4DCT phases are first generated and the correlation coefficient (CC) between the projection X-ray image and each DRR is computed. The maximum value of CC for the phase is found via a simulated annealing optimization process. The whole process repeats for all ten phases. The 4DCT phase that has the highest CC is identified as the breathing phase of the X-ray. The phase verification process is validated by a moving phantom study. Thus, the method may be used to independently confirm the correspondence between the gating phase at the times of 4DCT simulation and radiotherapy delivery. When the intended X-ray phase and actual gating phase are consistent, the registration of the DRRs and the projection images may also yield the values of patient shifts for treatment setup. This method could serve as the 4D analog of the conventional setup film as it provides both verification of the specific phase at the time of treatment and isocenter positioning shifts for treatment delivery.     

Evaluation of a 3D object registration method for analysis of humeral kinematics

In 3D image-based studies of joint kinematics, 3D registration methods should be automatic, insensitive to segmentation inconsistencies and use coordinate systems that have clinically relevant orientations and locations because this is important for analyzing rotation angles and translation directions. We developed and evaluated a registration method, which is based on the cylindrical geometry of the humerus shaft and an analysis of the inertia moments of the humerus head, in order to consistently and automatically orient the humerus coordinate system according to its anatomy. Registration techniques must be thoroughly evaluated. In this study we used a well-detectable marker as reference, from which coordinate system determination errors of a 3D object could be measured. This allowed us to quantify by means of unique error analysis the translational and rotational errors in terms of how much and about/along which humeral axis errors occurred. The evaluation experiments were performed using virtual rotations of 3D humeral binary image, a humerus model and a 3D image of a volunteer's shoulder. They indicated that the humeral coordinate system determination errors primarily originated from segmentation inconsistencies, which influenced mostly the humeral transverse axes orientation. The error analysis revealed that the developed registration method reduced the effect of manual segmentation inconsistencies on the orientation of the humeral transverse axes up to 37%, in comparison to the commonly used inertia registration.     

Automated image registration for assessing three-dimensional alignment of entire lower extremity and implant position using bi-plane radiography

An automated image-matching technique is presented to assess alignment of the entire lower extremity for normal and implanted knees and the positioning of implants with respect to bone. Sawbone femur and tibia and femoral and tibial components of a total knee arthroplasty system were used. Three spherical markers were attached to each sawbone and each component to define the local coordinate system. Outlines of the three-dimensional (3D) bone models and component computer-aided design (CAD) models were projected onto extracted contours of the femur, tibia, and implants in frontal and oblique X-ray images. Three-dimensional position of each model was recovered by minimizing the difference between the projected outline and the contour. Median values of the absolute error in estimating relative positions were within 0.5 mm and 0.6° for the femur with respect to the tibia, 0.5 mm and 0.5° for the femoral component with respect to the tibial component, 0.6 mm and 0.6° for the femoral component with respect to the femur, and 0.5 mm and 0.4° for the tibial component with respect to the tibia, indicating significant improvements when compared to manually obtained results.     

An ImageJ-based tool for three-dimensional registration between different types of microscopic images

Three-dimensional (3D) registration (i.e., alignment) between two microscopic images is very helpful to study tissues that do not adhere to substrates, such as mouse embryos and organoids, which are often 3D rotated during imaging. However, there is no 3D registration tool easily accessible for experimental biologists. Here we developed an ImageJ-based tool which allows for 3D registration accompanied with both quantitative evaluation of the accuracy and reconstruction of 3D rotated images. In this tool, several landmarks are manually provided in two images to be aligned, and 3D rotation is computed so that the distances between the paired landmarks from the two images are minimized. By simultaneously providing multiple points (e.g., all nuclei in the regions of interest) other than the landmarks in the two images, the correspondence of each point between the two images, i.e., to which nucleus in one image a certain nucleus in another image corresponds, is quantitatively explored. Furthermore, 3D rotation is applied to one of the two images, resulting in reconstruction of 3D rotated images. We demonstrated that this tool successfully achieved 3D registration and reconstruction of images in mouse pre- and post-implantation embryos, where one image was obtained during live imaging and another image was obtained from fixed embryos after live imaging. This approach provides a versatile tool applicable for various tissues and species.     

基于图像配准分析物种进化关系的新方法

图像配准是图像处理的一个重要技术,可用于分析两幅图像之间的相似度。本文提出了一种基于图像配准分析物种进化关系的新方法:首先利用一阶马尔可夫链方法计算不同基因组序列的寡聚核苷酸转移概率矩阵;然后将转移概率矩阵转换为彩色图像矩阵,并绘制物种两两之间彩色图像矩阵的联合直方图;最后分析联合直方图点集的分布情况,引入直方图点集的散度公式,将其作为相似性测度的标准,从而鉴定物种亲缘关系的远近。100种细菌全基因组的计算结果表明,相较于单基因法或基于基因组寡聚核苷酸频率组分差异信息的方法,本文提出的新方法具有更高的准确度和分辨力,它不仅能够很好地分辨科以下的分类单元,而且对科以上的分类单元同样具有较好的区分效果。该方法有望发展成为物种鉴定及系统发育推断的有效手段。     

eHUGS: Enhanced Hierarchical Unbiased Graph Shrinkage for Efficient Groupwise Registration

Effective and efficient spatial normalization of a large population of brain images is critical for many clinical and research studies, but it is technically very challenging. A commonly used approach is to choose a certain image as the template and then align all other images in the population to this template by applying pairwise registration. To avoid the potential bias induced by the inappropriate template selection, groupwise registration methods have been proposed to simultaneously register all images to a latent common space. However, current groupwise registration methods do not make full use of image distribution information for more accurate registration. In this paper, we present a novel groupwise registration method that harnesses the image distribution information by capturing the image distribution manifold using a hierarchical graph with its nodes representing the individual images. More specifically, a low-level graph describes the image distribution in each subgroup, and a high-level graph encodes the relationship between representative images of subgroups. Given the graph representation, we can register all images to the common space by dynamically shrinking the graph on the image manifold. The topology of the entire image distribution is always maintained during graph shrinkage. Evaluations on two datasets, one for 80 elderly individuals and one for 285 infants, indicate that our method can yield promising results.     

Automatic model-based roentgen stereophotogrammetric analysis (RSA) of total knee prostheses

Conventional radiography is insensitive for early and accurate estimation of the mal-alignment and wear of knee prostheses. The two-staged (rough and fine) registration of the model-based RSA technique has recently been developed to in vivo estimate the prosthetic pose (i.e, location and orientation). In the literature, rough registration often uses template match or manual adjustment of the roentgen images. Additionally, possible error induced by the nonorthogonality of taking two roentgen images neither examined nor calibrated prior to fine registration. This study developed two RSA methods for automate the estimation of the prosthetic pose and decrease the nonorthogonality-induced error. The predicted results were validated by both simulative and experimental tests and compared with reported findings in the literature. The outcome revealed that the feature-recognized method automates pose estimation and significantly increases the execution efficiency up to about 50 times in comparison with the literature counterparts. Although the nonorthogonal images resulted in undesirable errors, the outline-optimized method can effectively compensate for the induced errors prior to fine registration. The superiority in automation, efficiency, and accuracy demonstrated the clinical practicability of the two proposed methods especially for the numerous fluoroscopic images of dynamic motion.     

Homographic Patch Feature Transform: A Robustness Registration for Gastroscopic Surgery

Image registration is a key component of computer assistance in image guided surgery, and it is a challenging topic in endoscopic environments. In this study, we present a method for image registration named Homographic Patch Feature Transform (HPFT) to match gastroscopic images. HPFT can be used for tracking lesions and augmenting reality applications during gastroscopy. Furthermore, an overall evaluation scheme is proposed to validate the precision, robustness and uniformity of the registration results, which provides a standard for rejection of false matching pairs from corresponding results. Finally, HPFT is applied for processing in vivo gastroscopic data. The experimental results show that HPFT has stable performance in gastroscopic applications.     

Medical image registration using fuzzy theory

Mutual information (MI)-based registration, which uses MI as the similarity measure, is a representative method in medical image registration. It has an excellent robustness and accuracy, but with the disadvantages of a large amount of calculation and a long processing time. In this paper, by computing the medical image moments, the centroid is acquired. By applying fuzzy c-means clustering, the coordinates of the medical image are divided into two clusters to fit a straight line, and the rotation angles of the reference and floating images are computed, respectively. Thereby, the initial values for registering the images are determined. When searching the optimal geometric transformation parameters, we put forward the two new concepts of fuzzy distance and fuzzy signal-to-noise ratio (FSNR), and we select FSNR as the similarity measure between the reference and floating images. In the experiments, the Simplex method is chosen as multi-parameter optimisation. The experimental results show that this proposed method has a simple implementation, a low computational cost, a fast registration and good registration accuracy. Moreover, it can effectively avoid trapping into the local optima. It is adapted to both mono-modality and multi-modality image registrations.     

Evaluation of a Learning-based Deformable Registration Method on Abdominal CT Images

BackgroundReliable image comparisons, based on fast and accurate deformable registration methods, are recognized as key steps in the diagnosis and follow-up of cancer as well as for radiation therapy planning or surgery. In the particular case of abdominal images, the images to compare often differ widely from each other due to organ deformation, patient motion, movements of gastrointestinal tract or breathing. As a consequence, there is a need for registration methods that can cope with both local and global large and highly non-linear deformations.MethodDeformable registration of medical images traditionally relies on the iterative minimization of a cost function involving a large number of parameters. For complex deformations and large datasets, this process is computationally very demanding, leading to processing times that are incompatible with the clinical routine workflow. Moreover, the highly non-convex nature of these optimization problems leads to a high risk of convergence toward local minima. Recently, deep learning approaches using Convolutional Neural Networks (CNN) have led to major breakthroughs by providing computationally fast unsupervised methods for the registration of 2D and 3D images within seconds. Among all the proposed approaches, the VoxelMorph learning-based framework pioneered to learn in an unsupervised way the complex mapping, parameterized using a CNN, between every couple of 2D or 3D pairs of images and the corresponding deformation field by minimizing a standard intensity-based similarity metrics over the whole learning database. Voxelmorph has so far only been evaluated on brain images. The present study proposes to evaluate this method in the context of inter-subject registration of abdominal CT images, which present a greater challenge in terms of registration than brain images, due to greater anatomical variability and significant organ deformations.ResultsThe performances of VoxelMorph were compared with the current top-performing non-learning-based deformable registration method “Symmetric Normalization” (SyN), implemented in ANTs, on two representative databases: LiTS and 3D-IRCADb-01. Three different experiments were carried out on 2D or 3D data, the atlas-based or pairwise registration, using two different similarity metrics, namely (MSE and CC). Accuracy of the registration was measured by the Dice score, which quantifies the volume overlap for the selected anatomical region.All the three experiments exhibit that the two deformable registration methods significantly outperform the affine registration and that VoxelMorph accuracy is comparable or even better than the reference non-learning based registration method ANTs (SyN), with a drastically reduced computation time.ConclusionBy substituting a time consuming optimization problem, VoxelMorph has made an outstanding achievement in learning-based registration algorithm, where a registration function is trained and thus, able to perform deformable registration almost accurately on abdominal images, while reducing the computation time from minutes to seconds and from seconds to milliseconds in comparison to ANTs (SyN) on a CPU.     

Model-based simulation of dynamic magnetic resonance imaging signals

This paper presents a model-based method to efficiently simulate dynamic magnetic resonance imaging signals. Using an analytical spatiotemporal object model, the method can approximate time-varying k-space signals such as those from objects in motion and/or during dynamic contrast enhancement. Both rigid-body and non-rigid-body motions can be simulated using the proposed method. In addition, it can simulate data with arbitrary data sampling order and/or non-uniform k-space trajectory. A set of simulated images were compared with real data acquired from a rat model on a 4.7 T scanner to verify the model. The efficient simulation method is expected to be useful for rapid testing of various imaging and image analysis algorithms such as image reconstruction, image registration, motion compensation, and kinetic parameter mapping.     

基于配准的插值

断层间图像插值是三维重建的一个关键步骤,因为图像上像素之间的间隔常常小于断层图像之间的距离,而在三维重建需要它们有一致的分辨率.由于是同模态断层图像层间插值,对于解决同模态弹性配准问题,Thirion的demons算法比较适合.所以配准采用Demons方法.Demons算法先判断出待配准图像上各个象素点的运动方法,通过对各个象素点的移动来实现非刚性配准.在这个算法中,每张图像都被视为同灰度值轮廓的集合.该算法可以应用于精度要求比较高的体数据插值重建过程.     

DSA图像中运动伪影的消除方法

由于病人存在着各种运动(如呼吸、肌肉运动、心脏运动、设备噪声),在成像过程中常会造成图像上出现伪影,干扰医生的正常诊断,为消除这种伪影,本文提出一种基于图像配准思想的全自动消除伪影的方法,该方法能够自动消除DSA图像中的大部分运动伪影,使DSA图像得到较好的增强,并为后面的血管分割和三维重建提供便利,是一种快速有效的方法。     

A Fast Image Alignment Approach for 2D Classification of Cryo-EM Images Using Spectral Clustering

Three-dimensional (3D) reconstruction in single-particle cryo-electron microscopy (cryo-EM) is a significant technique for recovering the 3D structure of proteins or other biological macromolecules from their two-dimensional (2D) noisy projection images taken from unknown random directions. Class averaging in single-particle cryo-EM is an important procedure for producing high-quality initial 3D structures, where image alignment is a fundamental step. In this paper, an efficient image alignment algorithm using 2D interpolation in the frequency domain of images is proposed to improve the estimation accuracy of alignment parameters of rotation angles and translational shifts between the two projection images, which can obtain subpixel and subangle accuracy. The proposed algorithm firstly uses the Fourier transform of two projection images to calculate a discrete cross-correlation matrix and then performs the 2D interpolation around the maximum value in the cross-correlation matrix. The alignment parameters are directly determined according to the position of the maximum value in the cross-correlation matrix after interpolation. Furthermore, the proposed image alignment algorithm and a spectral clustering algorithm are used to compute class averages for single-particle 3D reconstruction. The proposed image alignment algorithm is firstly tested on a Lena image and two cryo-EM datasets. Results show that the proposed image alignment algorithm can estimate the alignment parameters accurately and efficiently. The proposed method is also used to reconstruct preliminary 3D structures from a simulated cryo-EM dataset and a real cryo-EM dataset and to compare them with RELION. Experimental results show that the proposed method can obtain more high-quality class averages than RELION and can obtain higher reconstruction resolution than RELION even without iteration.     

Reconstruction of 3-Dimensional Histology Volume and its Application to Study Mouse Mammary Glands

Histology volume reconstruction facilitates the study of 3D shape and volume change of an organ at the level of macrostructures made up of cells. It can also be used to investigate and validate novel techniques and algorithms in volumetric medical imaging and therapies. Creating 3D high-resolution atlases of different organs^1,2,3 is another application of histology volume reconstruction. This provides a resource for investigating tissue structures and the spatial relationship between various cellular features. We present an image registration approach for histology volume reconstruction, which uses a set of optical blockface images. The reconstructed histology volume represents a reliable shape of the processed specimen with no propagated post-processing registration error. The Hematoxylin and Eosin (H&E) stained sections of two mouse mammary glands were registered to their corresponding blockface images using boundary points extracted from the edges of the specimen in histology and blockface images. The accuracy of the registration was visually evaluated. The alignment of the macrostructures of the mammary glands was also visually assessed at high resolution.This study delineates the different steps of this image registration pipeline, ranging from excision of the mammary gland through to 3D histology volume reconstruction. While 2D histology images reveal the structural differences between pairs of sections, 3D histology volume provides the ability to visualize the differences in shape and volume of the mammary glands.     

RAMTaB: robust alignment of multi-tag bioimages

Background

In recent years, new microscopic imaging techniques have evolved to allow us to visualize several different proteins (or other biomolecules) in a visual field. Analysis of protein co-localization becomes viable because molecules can interact only when they are located close to each other. We present a novel approach to align images in a multi-tag fluorescence image stack. The proposed approach is applicable to multi-tag bioimaging systems which (a) acquire fluorescence images by sequential staining and (b) simultaneously capture a phase contrast image corresponding to each of the fluorescence images. To the best of our knowledge, there is no existing method in the literature, which addresses simultaneous registration of multi-tag bioimages and selection of the reference image in order to maximize the overall overlap between the images.

Methodology/Principal Findings

We employ a block-based method for registration, which yields a confidence measure to indicate the accuracy of our registration results. We derive a shift metric in order to select the Reference Image with Maximal Overlap (RIMO), in turn minimizing the total amount of non-overlapping signal for a given number of tags. Experimental results show that the Robust Alignment of Multi-Tag Bioimages (RAMTaB) framework is robust to variations in contrast and illumination, yields sub-pixel accuracy, and successfully selects the reference image resulting in maximum overlap. The registration results are also shown to significantly improve any follow-up protein co-localization studies.

Conclusions

For the discovery of protein complexes and of functional protein networks within a cell, alignment of the tag images in a multi-tag fluorescence image stack is a key pre-processing step. The proposed framework is shown to produce accurate alignment results on both real and synthetic data. Our future work will use the aligned multi-channel fluorescence image data for normal and diseased tissue specimens to analyze molecular co-expression patterns and functional protein networks.