Machine learning of diffraction image patterns for accurate classification of cells modeled with different nuclear sizes

Measurement of nuclear‐to‐cytoplasm (N:C) ratios plays an important role in detection of atypical and tumor cells. Yet, current clinical methods rely heavily on immunofluroescent staining and manual reading. To achieve the goal of rapid and label‐free cell classification, realistic optical cell models (OCMs) have been developed for simulation of diffraction imaging by single cells. A total of 1892 OCMs were obtained with varied nuclear volumes and orientations to calculate cross‐polarized diffraction image (p‐DI) pairs divided into three nuclear size groups of OCM_S, OCM_O and OCM_L based on three prostate cell structures. Binary classifications were conducted among the three groups with image parameters extracted by the algorithm of gray‐level co‐occurrence matrix. The averaged accuracy of support vector machine (SVM) classifier on test dataset of p‐DI was found to be 98.8% and 97.5% respectively for binary classifications of OCM_S vs OCM_O and OCM_O vs OCM_L for the prostate cancer cell structure. The values remain about the same at 98.9% and 97.8% for the smaller prostate normal cell structures. The robust performance of SVM over clustering classifiers suggests that the high‐order correlations of diffraction patterns are potentially useful for label‐free detection of single cells with large N:C ratios.     

Development and evaluation of realistic optical cell models for rapid and label‐free cell assay by diffraction imaging

Methods for rapid and label‐free cell assay are highly desired in life science. Single‐shot diffraction imaging presents strong potentials to achieve this goal as evidenced by past experimental results using methods such as polarization diffraction imaging flow cytometry. We present here a platform of methods toward solving these problems and results of optical cell model (OCM) evaluations by calculations and analysis of cross‐polarized diffraction image (p‐DI) pairs. Four types of realistic OCMs have been developed with two prostate cell structures and adjustable refractive index (RI) parameters to investigate the effects of cell morphology and index distribution on calculated p‐DI pairs. Image patterns have been characterized by a gray‐level co‐occurrence matrix (GLCM) algorithm and four GLCM parameters and linear depolarization ratio δ_L have been selected to compare calculated against measured data of prostate cells. Our results show that the irregular shapes of and heterogeneity in RI distributions for organelles play significant roles in the spatial distribution of scattered light by cells in comparison to the average RI values and their differences among the organelles. Discrepancies in GLCM and δ_L parameters between calculated and measured p‐DI data provide useful insight for understanding light scattering by single cells and improving OCM.      

Deep learning a boon for biophotonics?

This review covers original articles using deep learning in the biophotonic field published in the last years. In these years deep learning, which is a subset of machine learning mostly based on artificial neural network geometries, was applied to a number of biophotonic tasks and has achieved state‐of‐the‐art performances. Therefore, deep learning in the biophotonic field is rapidly growing and it will be utilized in the next years to obtain real‐time biophotonic decision‐making systems and to analyze biophotonic data in general. In this contribution, we discuss the possibilities of deep learning in the biophotonic field including image classification, segmentation, registration, pseudostaining and resolution enhancement. Additionally, we discuss the potential use of deep learning for spectroscopic data including spectral data preprocessing and spectral classification. We conclude this review by addressing the potential applications and challenges of using deep learning for biophotonic data.     

Tissue counter analysis of histologic sections of melanoma: influence of mask size and shape, feature selection, statistical methods and tissue preparation.

BACKGROUND: Tissue counter analysis is an image analysis tool designed for the detection of structures in complex images at the macroscopic or microscopic scale. As a basic principle, small square or circular measuring masks are randomly placed across the image and image analysis parameters are obtained for each mask. Based on learning sets, statistical classification procedures are generated which facilitate an automated classification of new data sets. OBJECTIVE: To evaluate the influence of the size and shape of the measuring masks as well as the importance of feature selection, statistical procedures and technical preparation of slides on the performance of tissue counter analysis in microscopic images. As main quality measure of the final classification procedure, the percentage of elements that were correctly classified was used. STUDY DESIGN: HE-stained slides of 25 primary cutaneous melanomas were evaluated by tissue counter analysis for the recognition of melanoma elements (section area occupied by tumour cells) in contrast to other tissue elements and background elements. Circular and square measuring masks, various subsets of image analysis features and classification and regression trees compared with linear discriminant analysis as statistical alternatives were used. The percentage of elements that were correctly classified by the various classification procedures was assessed. In order to evaluate the applicability to slides obtained from different laboratories, the best procedure was automatically applied in a test set of another 50 cases of primary melanoma derived from the same laboratory as the learning set and two test sets of 20 cases each derived from two different laboratories, and the measurements of melanoma area in these cases were compared with conventional assessment of vertical tumour thickness. RESULTS: Square measuring masks were slightly superior to circular masks, and larger masks (64 or 128 pixels in diameter) were superior to smaller masks (8 to 32 pixels in diameter). As far as the subsets of image analysis features were concerned, colour features were superior to densitometric and Haralick texture features. Statistical moments of the grey level distribution were of least significance. CART (classification and regression tree) analysis turned out to be superior to linear discriminant analysis. In the best setting, 95% of melanoma tissue elements were correctly recognized. Automated measurement of melanoma area in the independent test sets yielded a correlation of r=0.846 with vertical tumour thickness (p<0.001), similar to the relationship reported for manual measurements. The test sets obtained from different laboratories yielded comparable results. CONCLUSIONS: Large, square measuring masks, colour features and CART analysis provide a useful setting for the automated measurement of melanoma tissue in tissue counter analysis, which can also be used for slides derived from different laboratories.     

White blood cell detection using saliency detection and CenterNet: A two-stage approach

White blood cell (WBC) detection plays a vital role in peripheral blood smear analysis. However, cell detection remains a challenging task due to multi-cell adhesion, different staining and imaging conditions. Owing to the powerful feature extraction capability of deep learning, object detection methods based on convolutional neural networks (CNNs) have been widely applied in medical image analysis. Nevertheless, the CNN training is time-consuming and inaccuracy, especially for large-scale blood smear images, where most of the images are background. To address the problem, we propose a two-stage approach that treats WBC detection as a small salient object detection task. In the first saliency detection stage, we use the Itti's visual attention model to locate the regions of interest (ROIs), based on the proposed adaptive center-surround difference (ACSD) operator. In the second WBC detection stage, the modified CenterNet model is performed on ROI sub-images to obtain a more accurate localization and classification result of each WBC. Experimental results showed that our method exceeds the performance of several existing methods on two different data sets, and achieves a state-of-the-art mAP of over 98.8%.     

Species‐level image classification with convolutional neural network enables insect identification from habitus images

1. Changes in insect biomass, abundance, and diversity are challenging to track at sufficient spatial, temporal, and taxonomic resolution. Camera traps can capture habitus images of ground‐dwelling insects. However, currently sampling involves manually detecting and identifying specimens. Here, we test whether a convolutional neural network (CNN) can classify habitus images of ground beetles to species level, and estimate how correct classification relates to body size, number of species inside genera, and species identity.
2. We created an image database of 65,841 museum specimens comprising 361 carabid beetle species from the British Isles and fine‐tuned the parameters of a pretrained CNN from a training dataset. By summing up class confidence values within genus, tribe, and subfamily and setting a confidence threshold, we trade‐off between classification accuracy, precision, and recall and taxonomic resolution.
3. The CNN classified 51.9% of 19,164 test images correctly to species level and 74.9% to genus level. Average classification recall on species level was 50.7%. Applying a threshold of 0.5 increased the average classification recall to 74.6% at the expense of taxonomic resolution. Higher top value from the output layer and larger sized species were more often classified correctly, as were images of species in genera with few species.
4. Fine‐tuning enabled us to classify images with a high mean recall for the whole test dataset to species or higher taxonomic levels, however, with high variability. This indicates that some species are more difficult to identify because of properties such as their body size or the number of related species.
5. Together, species‐level image classification of arthropods from museum collections and ecological monitoring can substantially increase the amount of occurrence data that can feasibly be collected. These tools thus provide new opportunities in understanding and predicting ecological responses to environmental change.

     

Full‐field swept‐source optical coherence tomography and neural tissue classification for deep brain imaging

Optical coherence tomography can differentiate brain regions with intrinsic contrast and at a micron scale resolution. Such a device can be particularly useful as a real‐time neurosurgical guidance tool. We present, to our knowledge, the first full‐field swept‐source optical coherence tomography system operating near a wavelength of 1310 nm. The proof‐of‐concept system was integrated with an endoscopic probe tip, which is compatible with deep brain stimulation keyhole neurosurgery. Neuroimaging experiments were performed on ex vivo brain tissues and in vivo in rat brains. Using classification algorithms involving texture features and optical attenuation, images were successfully classified into three brain tissue types.     

Classifying T cell activity in autofluorescence intensity images with convolutional neural networks

The importance of T cells in immunotherapy has motivated developing technologies to improve therapeutic efficacy. One objective is assessing antigen‐induced T cell activation because only functionally active T cells are capable of killing the desired targets. Autofluorescence imaging can distinguish T cell activity states in a non‐destructive manner by detecting endogenous changes in metabolic co‐enzymes such as NAD(P)H. However, recognizing robust activity patterns is computationally challenging in the absence of exogenous labels. We demonstrate machine learning methods that can accurately classify T cell activity across human donors from NAD(P)H intensity images. Using 8260 cropped single‐cell images from six donors, we evaluate classifiers ranging from traditional models that use previously‐extracted image features to convolutional neural networks (CNNs) pre‐trained on general non‐biological images. Adapting pre‐trained CNNs for the T cell activity classification task provides substantially better performance than traditional models or a simple CNN trained with the autofluorescence images alone. Visualizing the images with dimension reduction provides intuition into why the CNNs achieve higher accuracy than other approaches. Our image processing and classifier training software is available at https://github.com/gitter‐lab/t‐cell‐classification .     

Automated analysis of high‐content microscopy data with deep learning

Existing computational pipelines for quantitative analysis of high‐content microscopy data rely on traditional machine learning approaches that fail to accurately classify more than a single dataset without substantial tuning and training, requiring extensive analysis. Here, we demonstrate that the application of deep learning to biological image data can overcome the pitfalls associated with conventional machine learning classifiers. Using a deep convolutional neural network (DeepLoc) to analyze yeast cell images, we show improved performance over traditional approaches in the automated classification of protein subcellular localization. We also demonstrate the ability of DeepLoc to classify highly divergent image sets, including images of pheromone‐arrested cells with abnormal cellular morphology, as well as images generated in different genetic backgrounds and in different laboratories. We offer an open‐source implementation that enables updating DeepLoc on new microscopy datasets. This study highlights deep learning as an important tool for the expedited analysis of high‐content microscopy data.     

3D-PulCNN: Pulmonary cancer classification from hyperspectral images using convolution combination unit based CNN

Pulmonary cancer is one of the most common malignancies worldwide. Accurate classification of its subtypes is required in differential diagnosis. However, existing algorithms are mostly based on color images, and the improvement of accuracy is quite challenging. In this study, we propose a convolution combination unit (CCU)-based three-dimensional convolutional neural network (3D-PulCNN) for classifying pulmonary cancer presented in microscopic hyperspectral image with both spatial and spectral information. CCU is designed to fuse the features acquired by different convolution scales. Compared with VGGNet, only two fully connected layers are used in this model, reducing the network parameters and model complexity. Experimental results show that 3D-PulCNN achieves overall average (OA) of 0.962 and Precision, Recall, and Kappa of more than 0.920, superior to 2D-VGGNet. Then, 3D-UNet is leveraged to segment cancer cells, and their morphological characteristics are calculated to supply quantitative virtual analysis data for classification results explanation and prognosis assessment.     

Topology-based fluorescence image analysis for automated cell identification and segmentation

Cell segmentation refers to the body of techniques used to identify cells in images and extract biologically relevant information from them; however, manual segmentation is laborious and subjective. We present Topological Boundary Line Estimation using Recurrence Of Neighbouring Emissions (TOBLERONE), a topological image analysis tool which identifies persistent homological image features as opposed to the geometric analysis commonly employed. We demonstrate that topological data analysis can provide accurate segmentation of arbitrarily-shaped cells, offering a means for automatic and objective data extraction. One cellular feature of particular interest in biology is the plasma membrane, which has been shown to present varying degrees of lipid packing, or membrane order, depending on the function and morphology of the cell type. With the use of environmentally-sensitive dyes, images derived from confocal microscopy can be used to quantify the degree of membrane order. We demonstrate that TOBLERONE is capable of automating this task.     

Prediction of cervical neoplasia diagnosis groups. Discriminant analysis on digitized cell images

The purpose of this study was to develop discriminant analysis models for predicting cervical dysplasia/neoplasia case diagnoses using cytometric features derived from the digital image analysis of cell monolayers. The data base consisted of 925 cells from 27 cases diagnosed either as moderate dysplasia (n = 10), severe dysplasia (n = 5), carcinoma in situ (n = 8) or invasive carcinoma (n = 4) on both tissue biopsy and monolayer preparations. Cell features examined were cell diameter, nuclear diameter, nuclear mean optical density (OD), nuclear integrated OD (IOD), nuclear OD standard deviation, normalized IOD, nuclear texture and nuclear-cytoplasmic ratio. Features derived from cells visually classified as moderate dysplasia correctly predicted the case diagnosis of moderate dysplasia versus more severe disease for 85% of the cells. Prediction models using summary measures (mean and variance) derived from all visually classified abnormal cells within each case correctly separated all cases into their respective diagnostic categories. These findings suggest that dysplastic cells in a cytologic sample have features that collectively reflect the tissue diagnosis, regardless of the visual differences among the cells. Such information has potential use for diagnosis and possibly for prognosis.     

Rapid intraoperative diagnosis of gynecological cancer by ATR‐FTIR spectroscopy of fresh tissue biopsy

A rapid and reliable intraoperative diagnostic technique to support clinical decisions was developed using Fourier‐transform infrared (FTIR) spectroscopy. Twenty‐six fresh tissue samples were collected intraoperatively from patients undergoing gynecological surgeries. Frozen section (FS) histopathology aimed to discriminate between malignant and benign tumors was performed, and attenuated total reflection (ATR) FTIR spectra were collected from these samples. Digital dehydration and principal component analysis and linear discriminant analysis (PCA‐LDA) models were developed to classify samples into malignant and benign groups. Two validation schemes were employed: k‐fold and “leave one out.” FTIR absorption spectrum of a fresh tissue sample was obtained in less than 5 minutes. The fingerprint spectral region of malignant tumors was consistently different from that of benign tumors. The PCA‐LDA discrimination model correctly classified the samples into malignant and benign groups with accuracies of 96% and 93% for the k‐fold and “leave one out” validation schemes, respectively. We showed that a simple tissue preparation followed by ATR‐FTIR spectroscopy provides accurate means for very rapid tumor classification into malignant and benign gynecological tumors. With further development, the proposed method has high potential to be used as an adjunct to the intraoperative FS histopathology technique.     

An image analysis method for assessment of prognostic risk in prostate cancer: a pilot study

Fully automated computerized image analysis at medium resolution (1 micron per pixel space) was applied in a study of 17 patients with stage D1 prostate cancer. For this pilot study, patients were selected on the basis of very good or very poor outcome. This selection was made in the hope of identifying morphometric features that are useful in prognostic assessment. Nine patients with good outcome were alive after 7 or more years of follow-up and eight patients with poor prognosis were dead of disease in less than 3 years. All patients were treated with 125I seed implantation to the prostate and pelvic lymph node dissection. Hormone therapy was not administered until the time of distant failure. Routine hematoxylin and eosin tissue sections of lymph nodal tissue bearing metastatic neoplasm were used for this analysis. A minimum of eight scenes per case was analysed. Of 50 measured parameters on each cluster, five (gray level distribution, number of cell clusters per scene, bending energy, average cluster area and cluster polarity) were useful to distinguish patients with good outcome from those with a poor outcome. Thirteen of the 17 patients were correctly classified by image analysis (P = 0.044, Fischer's exact test). By comparison, flow cytometry of the identical tissue samples correctly classified 14 of 17 patients (diploid, good outcome; aneuploid, poor outcome; P = 0.009). Only one patient was incorrectly classified by both image analysis and flow cytometry, implying a complementary prognostic role for the two methods. The encouraging result, successful identification of useful morphometric features, justifies a larger study of unselected patients.     

Structured illumination imaging with quasi‐periodic patterns

Structured illumination microscopy (SIM) is a well‐established method for optical sectioning and super‐resolution. The core of structured illumination is using a periodic pattern to excite image signals. This work reports a method for estimating minor pattern distortions from the raw image data and correcting these distortions during SIM image processing. The method was tested with both simulated and experimental image data from two‐photon Bessel light‐sheet SIM. The results proves the method is effective in challenging situations, where strong scattering background exists, signal‐to‐noise ratio (SNR) is low and the sample structure is sparse. Experimental results demonstrate restoring synaptic structures in deep brain tissue, despite the presence of strong light scattering and tissue‐induced SIM pattern distortion.     

Application of diffractometry and a linear image sensor to measurement of erythrocyte deformability.

We applied laser diffractometry and a linear image sensor to measurement of erythrocyte deformability to detect the light intensity pattern of the diffraction image. Deformability was evaluated as the deformability index (DI), calculated from the width and length of the diffraction pattern ellipse, estimated by the linear image sensor. With the erythrocytes under various shear stresses, the DI was linearly related to results by the geometric method (r = 0.996, p < 0.01). The coefficient of variance of DI at a shear stress of 236 dynes/cm2 was 0.2% (seven human blood samples), which was satisfactory for practical use. The DI was independent of the erythrocyte concentration in the range of 1.5 x 10(7)-5.0 x 10(7) cells/ml of suspension. Correlation between the DI and the logarithm of shear stress was linear in the range of 5 to 350 dynes/cm2 of shear stress in suspension media of different viscosities. Heat-treatment, which decreased membrane flexibility, caused parallel reduction of the DI plotted against the logarithm of shear stress. The method was sensitive and gave reproducible results. It may be useful for clinical applications.     

Rapid analysis of disease state in liquid human serum combining infrared spectroscopy and “digital drying”

In recent years, the diagnosis of brain tumors has been investigated with attenuated total reflection‐Fourier transform infrared (ATR‐FTIR) spectroscopy on dried human serum samples to eliminate spectral interferences of the water component, with promising results. This research evaluates ATR‐FTIR on both liquid and air‐dried samples to investigate “digital drying” as an alternative approach for the analysis of spectra obtained from liquid samples. Digital drying approaches, consisting of water subtraction and least‐squares method, have demonstrated a greater random forest (RF) classification performance than the air‐dried spectra approach when discriminating cancer vs control samples, reaching sensitivity values higher than 93.0% and specificity values higher than 83.0%. Moreover, quantum cascade laser infrared (QCL‐IR) based spectroscopic imaging is utilized on liquid samples to assess the implications of a deep‐penetration light source on disease classification. The RF classification of QCL‐IR data has provided sensitivity and specificity amounting to 85.1% and 75.3% respectively.     

A quantitative analysis of the human bone marrow erythroblastic cell lineage using the SAMBA 200 cell image processor. I. The normal maturation sequence

A quantitative image analysis of the normal maturation sequence for the human bone marrow erythroblastic lineage was performed using the SAMBA 200 cell image processor. The different image analysis steps (image acquisition, preprocessing, segmentation, parametrization and data analysis) are briefly described. Thirty-three parameters related to geometry, color, texture and densitometry were computed on 638 cell images belonging to the five erythroblastic maturation stages. The automated classification of these cells, based upon a stepwise linear discriminant analysis, resulted in 80% correctly classified cells. Acceptance of confusions between successive maturation stages enhanced the rate of correctly classified cells to 100%. Among the ten most discriminating parameters, the nuclear area showed the highest correlation with the changes throughout the maturation process. The projection of the maturation sequence onto the factorial plane resulting from the canonical analysis emphasizes the existence of three phases of the maturation process, a finding that correlates well with the cytologic evolution and the biochemical and functional events during the maturation. The trajectory of cells within this factorial plane is thus regarded as a differentiation path from which a measure of the maturation could be derived.     

Cross-domain retinopathy classification with optical coherence tomography images via a novel deep domain adaptation method

Deep learning based retinopathy classification with optical coherence tomography (OCT) images has recently attracted great attention. However, existing deep learning methods fail to work well when training and testing datasets are different due to the general issue of domain shift between datasets caused by different collection devices, subjects, imaging parameters, etc. To address this practical and challenging issue, we propose a novel deep domain adaptation (DDA) method to train a model on a labeled dataset and adapt it to an unlabelled dataset (collected under different conditions). It consists of two modules for domain alignment, that is, adversarial learning and entropy minimization. We conduct extensive experiments on three public datasets to evaluate the performance of the proposed method. The results indicate that there are large domain shifts between datasets, resulting a poor performance for conventional deep learning methods. The proposed DDA method can significantly outperform existing methods for retinopathy classification with OCT images. It achieves retinopathy classification accuracies of 0.915, 0.959 and 0.990 under three cross-domain (cross-dataset) scenarios. Moreover, it obtains a comparable performance with human experts on a dataset where no labeled data in this dataset have been used to train the proposed DDA method. We have also visualized the learnt features by using the t-distributed stochastic neighbor embedding (t-SNE) technique. The results demonstrate that the proposed method can learn discriminative features for retinopathy classification.     

Hyperspectral Analysis of Columbia Spotted Frog Habitat

Abstract: Wildlife managers increasingly are using remotely sensed imagery to improve habitat delineations and sampling strategies. Advances in remote sensing technology, such as hyperspectral imagery, provide more information than previously was available with multispectral sensors. We evaluated accuracy of high-resolution hyperspectral image classifications to identify wetlands and wetland habitat features important for Columbia spotted frogs (Rana luteiventris) and compared the results to multispectral image classification and United States Geological Survey topographic maps. The study area spanned 3 lake basins in the Salmon River Mountains, Idaho, USA. Hyperspectral data were collected with an airborne sensor on 30 June 2002 and on 8 July 2006. A 12-year comprehensive ground survey of the study area for Columbia spotted frog reproduction served as validation for image classifications. Hyperspectral image classification accuracy of wetlands was high, with a producer's accuracy of 96% (44 wetlands) correctly classified with the 2002 data and 89% (41 wetlands) correctly classified with the 2006 data. We applied habitat-based rules to delineate breeding habitat from other wetlands, and successfully predicted 74% (14 wetlands) of known breeding wetlands for the Columbia spotted frog. Emergent sedge microhabitat classification showed promise for directly predicting Columbia spotted frog egg mass locations within a wetland by correctly identifying 72% (23 of 32) of known locations. Our study indicates hyperspectral imagery can be an effective tool for mapping spotted frog breeding habitat in the selected mountain basins. We conclude that this technique has potential for improving site selection for inventory and monitoring programs conducted across similar wetland habitat and can be a useful tool for delineating wildlife habitats.