Image-adaptive deconvolution for three-dimensional deep biological imaging

Deconvolution algorithms are widely used in conventional fluorescence microscopy, but they remain difficult to apply to deep imaging systems such as confocal and two-photon microscopy, due to the practical difficulty of measuring the system's point spread function (PSF), especially in biological experiments. Since a separate PSF measurement performed under the design optical conditions of the microscope cannot reproduce the true experimental conditions prevailing in situ, the most natural approach to solve the problem is to extract the PSF from the images themselves. We investigate here the approach of cropping an approximate PSF directly from the images, by exploiting the presence of small structures within the samples under study. This approach turns out to be practical in many cases, allowing significantly better restorations than with a design PSF obtained by imaging fluorescent beads in gel. We demonstrate the advantages of this approach with a number of deconvolution experiments performed both on artificially blurred and noisy test images, and on real confocal images taken within an in vitro preparation of the mouse hearing organ.     

Portable deep learning singlet microscope

Having the least lenses, the significant feature of the singlet imaging system, helps the development of the portable and cost‐effective microscopes. A novel method of monochromatic/color singlet microscopy, which is combined with only one aspheric lens and deep learning computational imaging technology, is proposed in this article. The designed singlet aspheric lens is an approximate linear signal system, which means modulation‐transfer‐function curves on all field‐of‐views (5 mm diagonally) are almost coincident with each other. The purpose of the designed linear signal system is to further improve the resolution of our microscope by using deep learning algorithm. As a proof of concept, we designed a singlet microscopy based on our method, which weighs only 400 g. The experimental data and results of the sample USAF?1951 target and bio‐sample (the Equisetum‐arvense Strobile L.S), prove that the performance of the proposed singlet microscope is competitive to a commercial microscope with the 4X/NA0.1 objective lens. We believe that our idea and method would guide to design more cost‐effective and powerful singlet imaging system.     

Deep learning protocol for improved photoacoustic brain imaging

One of the key limitations for the clinical translation of photoacoustic imaging is penetration depth that is linked to the tissue maximum permissible exposures (MPE) recommended by the American National Standards Institute (ANSI). Here, we propose a method based on deep learning to virtually increase the MPE in order to enhance the signal‐to‐noise ratio of deep structures in the brain tissue. The proposed method is evaluated in an in vivo sheep brain imaging experiment. We believe this method can facilitate clinical translation of photoacoustic technique in brain imaging, especially in transfontanelle brain imaging in neonates.     

High-definition mapping of neural activity using voltage-sensitive dyes

The distribution of patterns of activity in different brain structures has been related to the encoding and processing of sensory information. Consequently, it is important to be able to image the distribution of these patterns to understand basic brain functions. The spatial resolution of voltage-sensitive dye (VSD) methods has recently been enhanced considerably by the use of video imaging techniques. The main factor that now hampers the resolution of VSD patterns is the inherent limitation of the optical systems. Unfortunately, the intrinsic characteristics of VSD images impose important limitations that restrict the use of general deconvolution techniques. To overcomes this problem, in this study an image restoration procedure has been implemented that takes into consideration the limiting characteristics of VSD signals. This technique is based on applying a set of imaging processing steps. First, the signal-to-noise (S/N) ratio of the images was improved to avoid an increase in the noise levels during the deconvolution procedures. For this purpose, a new filter technique was implemented that yielded better results than other methods currently used in optical imaging. Second, focal plane images were deconvolved using a modification of the well-known nearest-neighbor deconvolution algorithm. But to reduce the light exposure of the preparation and simplify image acquisition procedures, adjacent image planes were modeled according to the in-focus image planes and the empirical point spread function (PSF) profiles. Third, resulting focal plane responses were processed to reduce the contribution of optical responses that originate in distant image planes. This method was found to be satisfactory under simulated and real experimental conditions. By comparing the restored and unprocessed images, it was clearly demonstrated that this method can effectively remove the out-of-focus artifacts and produce focal plane images of better quality. Evaluations of the tissue optical properties allowed assessment of the maximum practical optical section thickness using this deconvolution technique in the optical system tested. Determination of the three-dimensional PSF permitted the correct application of deconvolution algorithms and the removal of the contaminating light arising from adjacent as well as distant optical planes. The implementation of this deconvolution approach in salamander olfactory bulb allowed the detailed study of the laminar distribution of voltage-sensitive changes across the bulb layer. It is concluded that (1) this deconvolution procedure is well suited to deconvolved low-contrast images and offers important advantages over other alternatives; (2) this method can be properly used only when the tissue optical properties are first determined; (3) high levels of light scattering in the tissue reduce the optical section capabilities of this technique as well as other deconvolution procedures; and (4) use of the highest numerical aperture in the objectives is advisable because this improves not only the light-collecting efficiency to detect poor-contrast images, but also the spatial frequency differences between adjacent image planes. Under this condition it is possible to overcome some of the limitations imposed by the light scattering/birefringence of the tissue.     

Structured illumination imaging with quasi‐periodic patterns

Structured illumination microscopy (SIM) is a well‐established method for optical sectioning and super‐resolution. The core of structured illumination is using a periodic pattern to excite image signals. This work reports a method for estimating minor pattern distortions from the raw image data and correcting these distortions during SIM image processing. The method was tested with both simulated and experimental image data from two‐photon Bessel light‐sheet SIM. The results proves the method is effective in challenging situations, where strong scattering background exists, signal‐to‐noise ratio (SNR) is low and the sample structure is sparse. Experimental results demonstrate restoring synaptic structures in deep brain tissue, despite the presence of strong light scattering and tissue‐induced SIM pattern distortion.     

DeconvTest: Simulation framework for quantifying errors and selecting optimal parameters of image deconvolution

Deconvolution is an essential step of image processing that aims to compensate for the image blur caused by the microscope's point spread function. With many existing deconvolution methods, it is challenging to choose the method and its parameters most appropriate for particular image data at hand. To facilitate this task, we developed DeconvTest: an open‐source Python‐based framework for generating synthetic microscopy images, deconvolving them with different algorithms, and quantifying reconstruction errors. In contrast to existing software, DeconvTest combines all components required to analyze deconvolution performance in a systematic, high‐throughput and quantitative manner. We demonstrate the power of the framework by using it to identify the optimal deconvolution settings for synthetic and real image data. Based on this, we provide a guideline for (a) choosing optimal values of deconvolution parameters for image data at hand and (b) optimizing imaging conditions for best results in combination with subsequent image deconvolution.     

Cross-domain retinopathy classification with optical coherence tomography images via a novel deep domain adaptation method

Deep learning based retinopathy classification with optical coherence tomography (OCT) images has recently attracted great attention. However, existing deep learning methods fail to work well when training and testing datasets are different due to the general issue of domain shift between datasets caused by different collection devices, subjects, imaging parameters, etc. To address this practical and challenging issue, we propose a novel deep domain adaptation (DDA) method to train a model on a labeled dataset and adapt it to an unlabelled dataset (collected under different conditions). It consists of two modules for domain alignment, that is, adversarial learning and entropy minimization. We conduct extensive experiments on three public datasets to evaluate the performance of the proposed method. The results indicate that there are large domain shifts between datasets, resulting a poor performance for conventional deep learning methods. The proposed DDA method can significantly outperform existing methods for retinopathy classification with OCT images. It achieves retinopathy classification accuracies of 0.915, 0.959 and 0.990 under three cross-domain (cross-dataset) scenarios. Moreover, it obtains a comparable performance with human experts on a dataset where no labeled data in this dataset have been used to train the proposed DDA method. We have also visualized the learnt features by using the t-distributed stochastic neighbor embedding (t-SNE) technique. The results demonstrate that the proposed method can learn discriminative features for retinopathy classification.     

Wide‐field monitoring and real‐time local recording of microvascular networks on small animals with a dual‐raster‐scanned photoacoustic microscope

Photoacoustic microscopy (PAM) provides a new method for the imaging of small‐animals with high‐contrast and deep‐penetration. However, the established PAM systems have suffered from a limited field‐of‐view or imaging speed, which are difficult to both monitor wide‐field activity of organ and record real‐time change of local tissue. Here, we reported a dual‐raster‐scanned photoacoustic microscope (DRS‐PAM) that integrates a two‐dimensional motorized translation stage for large field‐of‐view imaging and a two‐axis fast galvanometer scanner for real‐time imaging. The DRS‐PAM provides a flexible transition from wide‐field monitoring the vasculature of organs to real‐time imaging of local dynamics. To test the performance of DRS‐PAM, clear characterization of angiogenesis and functional detail was illustrated, hemodynamic activities of vasculature in cerebral cortex of a mouse were investigated. Furthermore, response of tumor to treatment were successfully monitored during treatment. The experimental results demonstrate the DRS‐PAM holds the great potential for biomedical research of basic biology.     

三维反卷积显微成像技术浅谈

三维宽场反卷积显微成像技术是应用光学切片方法获取三维标本的二维图像序列,然后通过反卷积图像处理方法进行图像恢复,进而进行三纺重建的一种以光学技术和图像处理技术为核心的业微成橡方法。本讲述了光学切片的基本原理,给出了反卷积处理中点扩展函数的理论模型和实验测试方法,然后对现存的反卷积算法做了对比。对这一领域的发展趋势做了预测。     

三维反卷积显微成像技术浅谈

三维宽场反应卷积显微成像技术是应用光学切片方法获取三维标本的二维图像序列,然后通过反卷积图像处理方法进行图像恢复,进而进行三维重建的一种以光学技术和图像处理技术为核心的显微成像方法。本文讲述了光学切片的基本原理,给出了反卷积处理中点扩展函数的理论模型和实验测试方法,然后对现存的反卷积算法做了对比。最后,文章对这一领域的发展趋势作了预测。     

LV-GAN: A deep learning approach for limited-view optoacoustic imaging based on hybrid datasets

The optoacoustic imaging (OAI) methods are rapidly evolving for resolving optical contrast in medical imaging applications. In practice, measurement strategies are commonly implemented under limited-view conditions due to oversized image objectives or system design limitations. Data acquired by limited-view detection may impart artifacts and distortions in reconstructed optoacoustic (OA) images. We propose a hybrid data-driven deep learning approach based on generative adversarial network (GAN), termed as LV-GAN, to efficiently recover high quality images from limited-view OA images. Trained on both simulation and experiment data, LV-GAN is found capable of achieving high recovery accuracy even under limited detection angles less than 60^°. The feasibility of LV-GAN for artifact removal in biological applications was validated by ex vivo experiments based on two different OAI systems, suggesting high potential of a ubiquitous use of LV-GAN to optimize image quality or system design for different scanners and application scenarios.     

Deep learning a boon for biophotonics?

This review covers original articles using deep learning in the biophotonic field published in the last years. In these years deep learning, which is a subset of machine learning mostly based on artificial neural network geometries, was applied to a number of biophotonic tasks and has achieved state‐of‐the‐art performances. Therefore, deep learning in the biophotonic field is rapidly growing and it will be utilized in the next years to obtain real‐time biophotonic decision‐making systems and to analyze biophotonic data in general. In this contribution, we discuss the possibilities of deep learning in the biophotonic field including image classification, segmentation, registration, pseudostaining and resolution enhancement. Additionally, we discuss the potential use of deep learning for spectroscopic data including spectral data preprocessing and spectral classification. We conclude this review by addressing the potential applications and challenges of using deep learning for biophotonic data.     

Multimodal imaging for tumour characterization from micro‐ to macroscopic level using a newly developed dorsal chamber designed for long‐term follow‐up

Optical imaging of living animals is a unique method of studying the dynamics of physiological and pathological processes at a subcellular level. One‐shot acquisitions at high resolution can be achieved on exteriorized organs before animal euthanasia. For longitudinal follow‐up, intravital imaging can be used and involves imaging windows implanted in cranial, thoracic or dorsal regions. Several imaging window models exist, but none have proven to be applicable for long‐term monitoring and most biological processes take place over several weeks. Moreover, none are compatible with multiple imaging modalities, meaning that different biological parameters cannot be assessed in an individual animal. We developed a new dorsal chamber that was well tolerated by mice (over several months) and allowed individual and collective cell tracking and behaviour analysis by optical imaging, ultrasound and magnetic resonance tomography. This new model broadens potential applications to areas requiring study of long‐term biological processes, as in cancer research.     

Label‐free spectroscopic tissue characterization using fluorescence excitation‐scanning spectral imaging

Spectral imaging approaches provide new possibilities for measuring and discriminating fluorescent molecules in living cells and tissues. These approaches often employ tunable filters and robust image processing algorithms to identify many fluorescent labels in a single image set. Here, we present results from a novel spectral imaging technology that scans the fluorescence excitation spectrum, demonstrating that excitation‐scanning hyperspectral image data can discriminate among tissue types and estimate the molecular composition of tissues. This approach allows fast, accurate quantification of many fluorescent species from multivariate image data without the need of exogenous labels or dyes. We evaluated the ability of the excitation‐scanning approach to identify endogenous fluorescence signatures in multiple unlabeled tissue types. Signatures were screened using multi‐pass principal component analysis. Endmember extraction techniques revealed conserved autofluorescent signatures across multiple tissue types. We further examined the ability to detect known molecular signatures by constructing spectral libraries of common endogenous fluorophores and applying multiple spectral analysis techniques on test images from lung, liver and kidney. Spectral deconvolution revealed structure‐specific morphologic contrast generated from pure molecule signatures. These results demonstrate that excitation‐scanning spectral imaging, coupled with spectral imaging processing techniques, provides an approach for discriminating among tissue types and assessing the molecular composition of tissues. Additionally, excitation scanning offers the ability to rapidly screen molecular markers across a range of tissues without using fluorescent labels. This approach lays the groundwork for translation of excitation‐scanning technologies to clinical imaging platforms.     

Visible‐near infrared‐II skull optical clearing window for in vivo cortical vasculature imaging and targeted manipulation

Skull optical clearing window permits us to perform in vivo cortical imaging without craniotomy, but mainly limits to visible (vis)‐near infrared (NIR)‐I light imaging. If the skull optical clearing window is available for NIR‐II, the imaging depth will be further enhanced. Herein, we developed a vis‐NIR‐II skull optical clearing agents with deuterium oxide instead of water, which could make the skull transparent in the range of visible to NIR‐II. Using a NIR‐II excited third harmonic generation microscope, the cortical vasculature of mice could be clearly distinguished even at the depth of 650 μm through the vis‐NIR‐II skull clearing window. The imaging depth after clearing is close to that without skull, and increases by three times through turbid skull. Furthermore, the new skull optical clearing window promises to realize NIR‐II laser‐induced targeted injury of cortical single vessel. This work enhances the ability of NIR‐II excited nonlinear imaging techniques for accessing to cortical neurovasculature in deep tissue.     

Deep learning of diffraction image patterns for accurate classification of five cell types

Development of label‐free methods for accurate classification of cells with high throughput can yield powerful tools for biological research and clinical applications. We have developed a deep neural network of DINet for extracting features from cross‐polarized diffraction image (p‐DI) pairs on multiple pixel scales to accurately classify cells in five types. A total of 6185 cells were measured by a polarization diffraction imaging flow cytometry (p‐DIFC) method followed by cell classification with DINet on p‐DI data. The averaged value and SD of classification accuracy were found to be 98.9% ± 1.00% on test data sets for 5‐fold training and test. The invariance of DINet to image translation, rotation, and blurring has been verified with an expanded p‐DI data set. To study feature‐based classification by DINet, two sets of correctly and incorrectly classified cells were selected and compared for each of two prostate cell types. It has been found that the signature features of large dissimilarities between p‐DI data of correctly and incorrectly classified cell sets increase markedly from convolutional layers 1 and 2 to layers 3 and 4. These results clearly demonstrate the importance of high‐order correlations extracted at the deep layers for accurate cell classification.      

Restored interlaced volumetric imaging increases image quality and scanning speed during intravital imaging in living mice

Dynamic intravital imaging is essential for revealing ongoing biological phenomena within living organisms and is influenced primarily by several factors: motion artifacts, optical properties and spatial resolution. Conventional imaging quality within a volume, however, is degraded by involuntary movements and trades off between the imaged volume, imaging speed and quality. To balance such trade‐offs incurred by two‐photon excitation microscopy during intravital imaging, we developed a unique combination of interlaced scanning and a simple image restoration algorithm based on biological signal sparsity and a graph Laplacian matrix. This method increases the scanning speed by a factor of four for a field size of 212 μm × 106 μm × 130 μm, and significantly improves the quality of four‐dimensional dynamic volumetric data by preventing irregular artifacts due to the movement observed with conventional methods. Our data suggest this method is robust enough to be applied to multiple types of soft tissue.     

A spatial resolution evaluation method of endoscopic optical coherence tomography system using the annular phantom

As an important biomedical imaging method, endoscopic optical coherence tomography (OCT) is necessary to check its performance regularly. The ordinary plane phantoms are only able to evaluate part of image tangent to the probe. In this research, a spatial resolution estimate method of the endoscope OCT system is proposed. The annular phantom, made by uniformly distributing golden scattered microparticles in polydimethylsiloxane (PDMS), can provide dynamic scanning imaging evaluation of endoscopic OCT system, closer to its actual working status. The point spread function analysis method is used to analyze the imaging results of the annular phantom with the endoscopic OCT system. And many scattered particles are statistically analyzed to determine the spatial resolution of the endoscope OCT system. The method is low in cost, simple and convenient. It is valuable for the development of test standards for endoscope OCT systems.     

An automatic diagnostic system of coronary artery lesions in Kawasaki disease using intravascular optical coherence tomography imaging

Intravascular optical coherence tomography (IV‐OCT) is a light‐based imaging modality with high resolution, which employs near‐infrared light to provide tomographic intracoronary images. Morbidity caused by coronary heart disease is a substantial cause of acute coronary syndrome and sudden cardiac death. The most common intracoronay complications caused by coronary artery disease are intimal hyperplasia, calcification, fibrosis, neovascularization and macrophage accumulation, which require efficient prevention strategies. OCT can provide discriminative information of the intracoronary tissues, which can be used to train a robust fully automatic tissue characterization model based on deep learning. In this study, we aimed to design a diagnostic model of coronary artery lesions. Particularly, we trained a random forest using convolutional neural network features to distinguish between normal and diseased arterial wall structure. Then, based on the arterial wall structure, fully convolutional network is designed to extract the tissue layers in normal cases, and pathological tissues regardless of lesion type in pathological cases. Then, the type of the lesions can be characterized with high precision using our previous model. The results demonstrate the robustness of the model with the approximate overall accuracy up to 90%.