Perspectives,potentials and trends of ex vivo and in vivo optical molecular pathology

It is pivotal for medical applications, such as noninvasive histopathologic characterization of tissue, to realize label‐free and molecule‐specific representation of morphologic and biochemical composition in real‐time with subcellular spatial resolution. This unmet clinical need requires new approaches for rapid and reliable real‐time assessment of pathologies to complement established diagnostic tools. Photonic imaging combined with digitalization offers the potential to provide the clinician the requested information both under in vivo and ex vivo conditions. This report summarizes photonic approaches and their use in combination with image processing, machine learning and augmented virtual reality that might solve current challenges in modern medicine. Details are given for pathology, intraoperative diagnosis in head and neck cancer and endoscopic diagnosis in gastroenterology.      

Fingerprint‐to‐CH stretch continuously tunable high spectral resolution stimulated Raman scattering microscope

Stimulated Raman scattering (SRS) microscopy is a label‐free method generating images based on chemical contrast within samples, and has already shown its great potential for high‐sensitivity and fast imaging of biological specimens. The capability of SRS to collect molecular vibrational signatures in bio‐samples, coupled with the availability of powerful statistical analysis methods, allows quantitative chemical imaging of live cells with sub‐cellular resolution. This application has substantially driven the development of new SRS microscopy platforms. Indeed, in recent years, there has been a constant effort on devising configurations able to rapidly collect Raman spectra from samples over a wide vibrational spectral range, as needed for quantitative analysis by using chemometric methods. In this paper, an SRS microscope which exploits spectral shaping by a narrowband and rapidly tunable acousto‐optical tunable filter (AOTF) is presented. This microscope enables spectral scanning from the Raman fingerprint region to the Carbon‐Hydrogen (CH)‐stretch region without any modification of the optical setup. Moreover, it features also a high enough spectral resolution to allow resolving Raman peaks in the crowded fingerprint region. Finally, application of the developed SRS microscope to broadband hyperspectral imaging of biological samples over a large spectral range from 800 to 3600 cm^?1, is demonstrated.     

Hyperspectral scanning laser optical tomography

In order to study physical relationships within tissue volumes or even organism‐level systems, the spatial distribution of multiple fluorescent markers needs to be resolved efficiently in three dimensions. Here, rather than acquiring discrete spectral images sequentially using multiple emission filters, a hyperspectral scanning laser optical tomography system is developed to obtain hyperspectral volumetric data sets with 2‐nm spectral resolution of optically transparent mesoscopic (millimeter‐centimeter) specimens. This is achieved by acquiring a series of point‐scanning hyperspectral extended depth of field images at different angles and subsequently tomographically reconstructing the 3D intensity distribution for each wavelength. This technique is demonstrated to provide robust measurements via the comparison of spectral and intensity profiles of fluorescent bead phantoms. Due to its enhanced spectral resolving ability, this technique is also demonstrated to resolve largely overlapping fluorophores, as demonstrated by the 3D fluorescence hyperspectral reconstruction of a dual‐labeled mouse thymus gland sample and the ability to distinguish tumorous and normal tissues of an unlabeled mouse intestine sample.      

Investigation of human pancreatic cancer tissues by Fourier Transform Infrared Hyperspectral Imaging

Fourier‐transform infrared hyperspectral imaging (FTIR‐HSI) provides hyperspectral images containing both morphological and chemical information. It is widely applied in the biomedical field to detect tumor lesions, even at the early stage, by identifying specific spectral biomarkers. Pancreatic neoplasms present different prognoses and are not always easily classified by conventional analyses. In this study, tissue samples with diagnosis of pancreatic ductal adenocarcinoma and pancreatic neuroendocrine tumor were analyzed by FTIR‐HSI and the spectral data compared with those from healthy and dysplastic samples. Multivariate/univariate approaches were complemented to hyperspectral images, and definite spectral markers of the different lesions identified. The malignant lesions were recognizable both from healthy/dysplastic pancreatic tissues (high values of phospholipids and triglycerides with shorter, more branched and less unsaturated alkyl chains) and between each other (different amounts of total lipids, phosphates and carbohydrates). These findings highlight different metabolic pathways characterizing the different samples, well detectable by FTIR‐HSI.     

Real‐time augmented reality for delineation of surgical margins during neurosurgery using autofluorescence lifetime contrast

Current clinical brain imaging techniques used for surgical planning of tumor resection lack intraoperative and real‐time feedback; hence surgeons ultimately rely on subjective evaluation to identify tumor areas and margins. We report a fluorescence lifetime imaging (FLIm) instrument (excitation: 355 nm; emission spectral bands: 390/40 nm, 470/28 nm, 542/50 nm and 629/53 nm) that integrates with surgical microscopes to provide real‐time intraoperative augmentation of the surgical field of view with fluorescent derived parameters encoding diagnostic information. We show the functionality and safety features of this instrument during neurosurgical procedures in patients undergoing craniotomy for the resection of brain tumors and/or tissue with radiation damage. We demonstrate in three case studies the ability of this instrument to resolve distinct tissue types and pathology including cortex, white matter, tumor and radiation‐induced necrosis. In particular, two patients with effects of radiation‐induced necrosis exhibited longer fluorescence lifetimes and increased optical redox ratio on the necrotic tissue with respect to non‐affected cortex, and an oligodendroglioma resected from a third patient reported shorter fluorescence lifetime and a decrease in optical redox ratio than the surrounding white matter. These results encourage the use of FLIm as a label‐free and non‐invasive intraoperative tool for neurosurgical guidance.     

Detection and delineation of squamous neoplasia with hyperspectral imaging in a mouse model of tongue carcinogenesis

Hyperspectral imaging (HSI) holds the potential for the noninvasive detection of cancers. Oral cancers are often diagnosed at a late stage when treatment is less effective and the mortality and morbidity rates are high. Early detection of oral cancer is, therefore, crucial in order to improve the clinical outcomes. To investigate the potential of HSI as a noninvasive diagnostic tool, an animal study was designed to acquire hyperspectral images of in vivo and ex vivo mouse tongues from a chemically induced tongue carcinogenesis model. A variety of machine‐learning algorithms, including discriminant analysis, ensemble learning, and support vector machines, were evaluated for tongue neoplasia detection using HSI and were validated by the reconstructed pathological gold‐standard maps. The diagnostic performance of HSI, autofluorescence imaging, and fluorescence imaging were compared in this study. Color‐coded prediction maps were generated to display the predicted location and distribution of premalignant and malignant lesions. This study suggests that hyperspectral imaging combined with machine‐learning techniques can provide a noninvasive tool for the quantitative detection and delineation of squamous neoplasia.      

Dual‐polarization Raman spectral imaging to extract overlapping molecular fingerprints of living cells

Raman spectral imaging is gaining more and more attention in biological studies because of its label‐free characteristic. However, the discrimination of overlapping chemical contrasts has been a major challenge. In this study, we introduce an optical method to simultaneously obtain two orthogonally polarized Raman images from a single scan of the sample. We demonstrate how this technique can improve the quality and quantity of the hyperspectral Raman dataset and how the technique is expected to further extend the horizons of Raman spectral imaging in biological studies by providing more detailed chemical information.

The dual‐polarization Raman images of a HeLa cell.     

In vivo label‐free functional photoacoustic monitoring of ischemic reperfusion

Pressure ulcer formation is a common problem among patients confined to bed or restricted to wheelchairs. The ulcer forms when the affected skin and underlying tissues go through repeated cycles of ischemia and reperfusion, leading to inflammation. This theory is evident by intravital imaging studies performed in immune cell–specific, fluorescent reporter mouse skin with induced ischemia‐reperfusion (I‐R) injuries. However, traditional confocal or multiphoton microscopy cannot accurately monitor the progression of vascular reperfusion by contrast agents, which leaks into the interstitium under inflammatory conditions. Here, we develop a dual‐wavelength micro electro mechanical system (MEMS) scanning–based optical resolution photoacoustic microscopy (OR‐PAM) system for continuous label‐free functional imaging of vascular reperfusion in an IR mouse model. This MEMS‐OR‐PAM system provides fast scanning speed for concurrent dual‐wavelength imaging, which enables continuous monitoring of the reperfusion process. During reperfusion, the revascularization of blood vessels and the oxygen saturation (sO₂) changes in both arteries and veins are recorded, from which the local oxygen extraction ratios of the ischemic tissue and the unaffected tissue can be quantified. Our MEMS‐OR‐PAM system provides novel perspectives to understand the I‐R injuries. It solves the problem of dynamic label‐free functional monitoring of the vascular reperfusion at high spatial resolution.      

Imaging depth variations in hyperspectral imaging: Development of a method to detect tumor up to the required tumor‐free margin width

Hyperspectral imaging is a promising technique for resection margin assessment during cancer surgery. Thereby, only a specific amount of the tissue below the resection surface, the clinically defined margin width, should be assessed. Since the imaging depth of hyperspectral imaging varies with wavelength and tissue composition, this can have consequences for the clinical use of hyperspectral imaging as margin assessment technique. In this study, a method was developed that allows for hyperspectral analysis of resection margins in breast cancer. This method uses the spectral slope of the diffuse reflectance spectrum at wavelength regions where the imaging depth in tumor and healthy tissue is equal. Thereby, tumor can be discriminated from healthy breast tissue while imaging up to a similar depth as the required tumor‐free margin width of 2 mm. Applying this method to hyperspectral images acquired during surgery would allow for robust margin assessment of resected specimens. In this paper, we focused on breast cancer, but the same approach can be applied to develop a method for other types of cancer.     

Optical coherence hyperspectral microscopy with a single supercontinuum light source

In the paper, we have developed an optical coherence hyperspectral microscopy with a single supercontinuum light source. The microscopy consists of optical coherence tomography (OCT) and hyperspectral imaging (HSI), which can visualize the structural and functional characteristics of biological tissues. The 500 to 700 nm band is selected for HSI and OCT imaging, where HSI enables imaging of oxygen saturation and hemoglobin (Hb) content, while OCT acquires structural characteristics to assess the morphology of biological tissues. The system performance of the optical coherence hyperspectral microscopy is verified by normal mice ears, and the practical applications of the microscopy is further performed in 4T1 and inflammation Balb/c mice ears in vivo. The experimental results demonstrate that the microscopy has potential to provide complementary information for clinical applications.     

Visualizing the “sandwich” structure of osteocytes in their native environment deep in bone in vivo

Osteocytes are the most abundant cells in bone and always the focus of bone research. They are embedded in the highly scattering mineralized bone matrix. Consequently, visualizing osteocytes deep in bone with subcellular resolution poses a major challenge for in vivo bone research. Here we overcome this challenge by demonstrating 3‐photon imaging of osteocytes through the intact mouse skull in vivo. Through broadband transmittance characterization, we establish that the excitation at the 1700‐nm window enables the highest optical transmittance through the skull. Using label‐free third‐harmonic generation (THG) imaging excited at this window, we visualize osteocytes through the whole 140‐μm mouse skull and 155 μm into the brain in vivo. By developing selective labeling technique for the interstitial space, we visualize the “sandwich” structure of osteocytes in their native environment. Our work provides novel imaging methodology for bone research in vivo.      

Combining hyperspectral imaging and chemometrics to assess and interpret the effects of environmental stressors on zebrafish eye images at tissue level

Changes on an organism by the exposure to environmental stressors may be characterized by hyperspectral images (HSI), which preserve the morphology of biological samples, and suitable chemometric tools. The approach proposed allows assessing and interpreting the effect of contaminant exposure on heterogeneous biological samples monitored by HSI at specific tissue levels. In this work, the model example used consists of the study of the effect of the exposure of chlorpyrifos‐oxon on zebrafish tissues. To assess this effect, unmixing of the biological sample images followed by tissue‐specific classification models based on the unmixed spectral signatures is proposed. Unmixing and classification are performed by multivariate curve resolution‐alternating least squares (MCR‐ALS) and partial least squares‐discriminant analysis (PLS‐DA), respectively. Crucial aspects of the approach are: (1) the simultaneous MCR‐ALS analysis of all images from 1 population to take into account biological variability and provide reliable tissue spectral signatures, and (2) the use of resolved spectral signatures from control and exposed populations obtained from resampling of pixel subsets analyzed by MCR‐ALS multiset analysis as information for the tissue‐specific PLS‐DA classification models. Classification results diagnose the presence of a significant effect and identify the spectral regions at a tissue level responsible for the biological change.      

Portable optical fiber probe-based spectroscopic scanner for rapid cancer diagnosis: a new tool for intraoperative margin assessment

There continues to be a significant clinical need for rapid and reliable intraoperative margin assessment during cancer surgery. Here we describe a portable, quantitative, optical fiber probe-based, spectroscopic tissue scanner designed for intraoperative diagnostic imaging of surgical margins, which we tested in a proof of concept study in human tissue for breast cancer diagnosis. The tissue scanner combines both diffuse reflectance spectroscopy (DRS) and intrinsic fluorescence spectroscopy (IFS), and has hyperspectral imaging capability, acquiring full DRS and IFS spectra for each scanned image pixel. Modeling of the DRS and IFS spectra yields quantitative parameters that reflect the metabolic, biochemical and morphological state of tissue, which are translated into disease diagnosis. The tissue scanner has high spatial resolution (0.25 mm) over a wide field of view (10 cm × 10 cm), and both high spectral resolution (2 nm) and high spectral contrast, readily distinguishing tissues with widely varying optical properties (bone, skeletal muscle, fat and connective tissue). Tissue-simulating phantom experiments confirm that the tissue scanner can quantitatively measure spectral parameters, such as hemoglobin concentration, in a physiologically relevant range with a high degree of accuracy (<5% error). Finally, studies using human breast tissues showed that the tissue scanner can detect small foci of breast cancer in a background of normal breast tissue. This tissue scanner is simpler in design, images a larger field of view at higher resolution and provides a more physically meaningful tissue diagnosis than other spectroscopic imaging systems currently reported in literatures. We believe this spectroscopic tissue scanner can provide real-time, comprehensive diagnostic imaging of surgical margins in excised tissues, overcoming the sampling limitation in current histopathology margin assessment. As such it is a significant step in the development of a platform technology for intraoperative management of cancer, a clinical problem that has been inadequately addressed to date.     

Label‐free and non‐invasive discrimination of HaCaT and melanoma cells in a co‐culture model by hyperspectral confocal reflectance microscopy

A novel hyperspectral confocal microscopy method to separate different cell populations in a co‐culture model is presented here. The described methodological and instrumental approach allows discrimination of different cell types using a non‐invasive, label free method with good accuracy with a single cell resolution. In particular, melanoma cells are discriminated from HaCaT cells by hyperspectral confocal imaging, principal component analysis and optical frequencies signing, as confirmed by fluorescence labelling cross check. The identification seems to be quite robust to be insensitive to the cellular shape within the studied samples, enabling to separate cells according to their cytotype down to a single cell sensitivity.

Set of hyperspectral images of melanoma‐keratinocytes co‐culture model (left), score plot of principal component analysis and spectral analysis of principal components coefficients (center), label‐free spectral identification of cell populations (right).     

Hyperspectral imaging combined with machine learning as a tool to obtain high‐throughput plant salt‐stress phenotyping

The rapid selection of salinity‐tolerant crops to increase food production in salinized lands is important for sustainable agriculture. Recently, high‐throughput plant phenotyping technologies have been adopted that use plant morphological and physiological measurements in a non‐destructive manner to accelerate plant breeding processes. Here, a hyperspectral imaging (HSI) technique was implemented to monitor the plant phenotypes of 13 okra (Abelmoschus esculentus L.) genotypes after 2 and 7 days of salt treatment. Physiological and biochemical traits, such as fresh weight, SPAD, elemental contents and photosynthesis‐related parameters, which require laborious, time‐consuming measurements, were also investigated. Traditional laboratory‐based methods indicated the diverse performance levels of different okra genotypes in response to salinity stress. We introduced improved plant and leaf segmentation approaches to RGB images extracted from HSI imaging based on deep learning. The state‐of‐the‐art performance of the deep‐learning approach for segmentation resulted in an intersection over union score of 0.94 for plant segmentation and a symmetric best dice score of 85.4 for leaf segmentation. Moreover, deleterious effects of salinity affected the physiological and biochemical processes of okra, which resulted in substantial changes in the spectral information. Four sample predictions were constructed based on the spectral data, with correlation coefficients of 0.835, 0.704, 0.609 and 0.588 for SPAD, sodium concentration, photosynthetic rate and transpiration rate, respectively. The results confirmed the usefulness of high‐throughput phenotyping for studying plant salinity stress using a combination of HSI and deep‐learning approaches.     

Fast Evaluation of Unhealthy and Healthy Neonates Using Hyperspectral Features on 700-850 Nm Wavelengths,ROI Extraction,and 3D-CNN

ObjectivesHyperspectral imaging (HSI) has great potential in detecting the health conditions of neonates as it provides diagnostic information about the tissue by avoiding tissue biopsy. HSI gives more features than thermal imaging, which can obtain images in a single wavelength, as it can obtain images in a large number of wavelengths. The data obtained with hyperspectral sensors are 3-dimensional data called hypercube including first two-dimensional spatial information and third-dimensional spectral information.Material and methodsIn this study, hyperspectral data were obtained from 19 different neonates in the Neonatal Intensive Care Unit (NICU) of Selcuk University, Medical Faculty. There are 16 hypercubes from 16 unhealthy neonates, 16 hypercubes from 3 healthy neonates in a period of three months, and 32 hypercubes in total are available. For the training of 3D-CNN model, data augmentation methods, such as rotation, height shifting, width shifting, and shearing were applied to hyperspectral data. A number of 32 hypercubes taken from neonates in NICU were augmented to 160 hypercubes. Spectral signatures were examined and 51 bands in the range of 700-850 nm with distinctive features were used for the classification. The spectral dimension was reduced by applying Principal Component Analysis (PCA) to all hypercubes. In addition, it is aimed to obtain both spectral and spatial features with the 3D-CNN. For increasing the classification efficiency, ROI extraction was made and four datasets were created in different spatial dimensions. These datasets contain 160, 640, 1440, and 5760 hypercubes, respectively.ResultsThe best result was achieved by using 5760 hypercubes of 25x25x51. As a result of the classification of the hypercubes, accuracy 98.00%, sensitivity 97.22%, and specificity 98.78% were obtained. It was determined how many PCs used to achieve the best result. Further, the proposed 3D-CNN model is compared to 2D-CNN model to evaluate the performance of the study.ConclusionIt was aimed to evaluate the health status of neonates fastly by using HSI and 3D-CNN for the first time. The obtained results are an indication that HSI and 3D-CNN are very effective for the evaluation of unhealthy and healthy neonates.     

Development of an image pre‐processor for operational hyperspectral laryngeal cancer detection

Hyperspectral imaging (HSI) is a technology with high potential in the field of non‐invasive detection of cancer. However, in complex imaging situations like HSI of the larynx with a rigid endoscope, various image interferences can disable a proper classification of cancerous tissue. We identified three main problems: i) misregistration of single images in a HS cube due to patient heartbeat ii) image noise and iii) specular reflections (SR). Consequently, an image pre‐processor is developed in the current paper to overcome these image interferences. It encompasses i) image registration ii) noise removal by minimum noise fraction (MNF) transformation and iii) a novel SR detection method. The results reveal that the pre‐processor improves classification performance, while the newly developed SR detection method outperforms global thresholding technique hitherto used by 46%. The novel pre‐processor will be used for future studies towards the development of an operational scheme for HS‐based larynx cancer detection.

RGB image of the larynx derived from the hyperspectral cube and corresponding specular reflections ( a ) manually segmented and ( b ) detected by a novel specular reflection detection method.     

Spectral characterization and unmixing of intrinsic contrast in intact normal and diseased gastric tissues using hyperspectral two-photon microscopy

Background

Living tissues contain a range of intrinsic fluorophores and sources of second harmonic generation which provide contrast that can be exploited for fresh tissue imaging. Microscopic imaging of fresh tissue samples can circumvent the cost and time associated with conventional histology. Further, intrinsic contrast can provide rich information about a tissue''s composition, structure and function, and opens the potential for in-vivo imaging without the need for contrast agents.

Methodology/Principal Findings

In this study, we used hyperspectral two-photon microscopy to explore the characteristics of both normal and diseased gastrointestinal (GI) tissues, relying only on their endogenous fluorescence and second harmonic generation to provide contrast. We obtained hyperspectral data at subcellular resolution by acquiring images over a range of two-photon excitation wavelengths, and found excitation spectral signatures of specific tissue types based on our ability to clearly visualize morphology. We present the two-photon excitation spectral properties of four major tissue types that are present throughout the GI tract: epithelium, lamina propria, collagen, and lymphatic tissue. Using these four excitation signatures as basis spectra, linear unmixing strategies were applied to hyperspectral data sets of both normal and neoplastic tissue acquired in the colon and small intestine. Our results show that hyperspectral unmixing with excitation spectra allows segmentation, showing promise for blind identification of tissue types within a field of view, analogous to specific staining in conventional histology. The intrinsic spectral signatures of these tissue types provide information relating to their biochemical composition.

Conclusions/Significance

These results suggest hyperspectral two-photon microscopy could provide an alternative to conventional histology either for in-situ imaging, or intraoperative ‘instant histology’ of fresh tissue biopsies.     

Wide‐field hyperspectral 3D imaging of functionalized gold nanoparticles targeting cancer cells by reflected light microscopy

We present a new hyperspectral reflected light microscopy system with a scanned broadband supercontinuum light source. This wide‐field and low phototoxic hyperspectral imaging system has been successful for performing spectral three‐dimensional (3D) localization and spectroscopic identification of CD44‐targeted PEGylated AuNPs in fixed cell preparations. Such spatial and spectral information is essential for the improvement of nanoplasmonic‐based imaging, disease detection and treatment in complex biological environment. The presented system can be used for real‐time 3D NP tracking as spectral sensors, thus providing new avenues in the spatio‐temporal characterization and detection of bioanalytes.

3D image of the distribution of functionalized AuNPs attached to CD44‐expressing MDA‐MB‐231 human cancer cells.     

Hyperspectral imaging of mucosal surfaces in patients

The aim of this study was to proof applicability of hyperspectral imaging for the analysis and classification of human mucosal surfaces in vivo. The larynx as a prototypical anatomically well‐defined surgical test area was analyzed by microlaryngoscopy with a polychromatic lightsource and a synchronous triggered monochromatic CCD‐camera. Image stacks (5 benign, 7 malignant tumors) were analyzed by established software (principal component analysis PCA, hyperspectral classification, spectral profiles). Hyperspectral image datacubes were analyzed and classified by conventional software. In PCA, images at 590–680 nm loaded most onto the first PC which typically contained 95% of the total information. Hyperspectral classification clustered the data highlighting altered mucosa. The spectral profiles clearly differed between the different groups. Hyperspectral imaging can be applied to mucosal surfaces. This approach opens the way to analyze spectral characteristics of histologically different lesions in order to build up a spectral library and to allow non‐touch optical biopsy. (© 2012 WILEY‐VCH Verlag GmbH & Co. KGaA, Weinheim)