Intraoperative multispectral and hyperspectral label‐free imaging: A systematic review of in vivo clinical studies

Multispectral and hyperspectral imaging (HSI) are emerging optical imaging techniques with the potential to transform the way surgery is performed but it is not clear whether current systems are capable of delivering real‐time tissue characterization and surgical guidance. We conducted a systematic review of surgical in vivo label‐free multispectral and HSI systems that have been assessed intraoperatively in adult patients, published over a 10‐year period to May 2018. We analysed 14 studies including 8 different HSI systems. Current in‐vivo HSI systems generate an intraoperative tissue oxygenation map or enable tumour detection. Intraoperative tissue oxygenation measurements may help to predict those patients at risk of postoperative complications and in‐vivo intraoperative tissue characterization may be performed with high specificity and sensitivity. All systems utilized a line‐scanning or wavelength‐scanning method but the spectral range and number of spectral bands employed varied significantly between studies and according to the system's clinical aim. The time to acquire a hyperspectral cube dataset ranged between 5 and 30 seconds. No safety concerns were reported in any studies. A small number of studies have demonstrated the capabilities of intraoperative in‐vivo label‐free HSI but further work is needed to fully integrate it into the current surgical workflow.      

Development of an image pre‐processor for operational hyperspectral laryngeal cancer detection

Hyperspectral imaging (HSI) is a technology with high potential in the field of non‐invasive detection of cancer. However, in complex imaging situations like HSI of the larynx with a rigid endoscope, various image interferences can disable a proper classification of cancerous tissue. We identified three main problems: i) misregistration of single images in a HS cube due to patient heartbeat ii) image noise and iii) specular reflections (SR). Consequently, an image pre‐processor is developed in the current paper to overcome these image interferences. It encompasses i) image registration ii) noise removal by minimum noise fraction (MNF) transformation and iii) a novel SR detection method. The results reveal that the pre‐processor improves classification performance, while the newly developed SR detection method outperforms global thresholding technique hitherto used by 46%. The novel pre‐processor will be used for future studies towards the development of an operational scheme for HS‐based larynx cancer detection.

RGB image of the larynx derived from the hyperspectral cube and corresponding specular reflections ( a ) manually segmented and ( b ) detected by a novel specular reflection detection method.     

First experience in clinical application of hyperspectral endoscopy for evaluation of colonic polyps

Early detection and resection of adenomatous polyps prevents their progression to colorectal cancer (CRC), significantly improving patient outcomes. Polyps are typically identified and removed during white-light colonoscopy. Unfortunately, the rate of interval cancers that arise between CRC screening events remains high, linked to poor visualization of polyps during screening and incomplete polyp removal. Here, we sought to evaluate the potential of a hyperspectral endoscope (HySE) to enhance polyp discrimination for detection and resection. We designed, built and tested a new compact HySE in a proof-of-concept clinical study. We successfully collected spectra from three tissue types in seven patients undergoing routine colonoscopy screening. The acquired spectral data from normal tissue and polyps, both pre- and post- resection, were subjected to quantitative analysis using spectral angle mapping and machine learning, which discriminated the data by tissue type, meriting further investigation of HySE as a clinical tool.     

Diagnosis of early gastric cancer based on fluorescence hyperspectral imaging technology combined with partial‐least‐square discriminant analysis and support vector machine

This study investigated the feasibility of using fluorescence hyperspectral imaging technology to diagnose of early‐stage gastric cancer. Fluorescence spectral images of 76 patients who were pathologically diagnosed as non‐atrophic gastritis, premalignant lesions and gastric cancer were collected. Fluorescence spectra at 100‐pixel points were randomly extracted after binarization. Diagnostic models of non‐atrophic gastritis, premalignant lesions and gastric cancer were constructed through partial‐least‐square discriminant analysis (PLS‐DA) and support vector machine (SVM) algorithms. The prediction effects of PLS‐DA and SVM models were compared. Results showed that the average spectra of normal, precancerous and gastric cancer tissues significantly differed at 496, 546, 640 and 670 nm, and regular changes in fluorescence intensity at 546 nm were in the following order: normal > precancerous lesions > gastric cancer. Additionally, the effect of the diagnostic model established by SVM is significantly better than PLS‐DA which accuracy, specificity and sensitivity are above 94%. Experimental results revealed that the fast diagnostic model of early gastric cancer by combining fluorescence hyperspectral imaging technology and improved SVM was effective and feasible, thereby providing an accurate and rapid method for diagnosing early‐stage gastric cancer.      

Differentiating aquatic plant communities in a eutrophic river using hyperspectral and multispectral remote sensing

1. This study evaluates the efficacy of remote sensing technology to monitor species composition, areal extent and density of aquatic plants (macrophytes and filamentous algae) in impoundments where their presence may violate water‐quality standards. 2. Multispectral satellite (IKONOS) images and more than 500 in situ hyperspectral samples were acquired to map aquatic plant distributions. By analyzing field measurements, we created a library of hyperspectral signatures for a variety of aquatic plant species, associations and densities. We also used three vegetation indices. Normalized Difference Vegetation Index (NDVI), near‐infrared (NIR)‐Green Angle Index (NGAI) and normalized water absorption depth (D_H)_, at wavelengths 554, 680, 820 and 977 nm to differentiate among aquatic plant species composition, areal density and thickness in cases where hyperspectral analysis yielded potentially ambiguous interpretations. 3. We compared the NDVI derived from IKONOS imagery with the in situ, hyperspectral‐derived NDVI. The IKONOS‐based images were also compared to data obtained through routine visual observations. Our results confirmed that aquatic species composition alters spectral signatures and affects the accuracy of remote sensing of aquatic plant density. The results also demonstrated that the NGAI has apparent advantages in estimating density over the NDVI and the D_H. 4. In the feature space of the three indices, 3D scatter plot analysis revealed that hyperspectral data can differentiate several aquatic plant associations. High‐resolution multispectral imagery provided useful information to distinguish among biophysical aquatic plant characteristics. Classification analysis indicated that using satellite imagery to assess Lemna coverage yielded an overall agreement of 79% with visual observations and >90% agreement for the densest aquatic plant coverages. 5. Interpretation of biophysical parameters derived from high‐resolution satellite or airborne imagery should prove to be a valuable approach for assessing the effectiveness of management practices for controlling aquatic plant growth in inland waters, as well as for routine monitoring of aquatic plants in lakes and suitable lentic environments.     

Integrated photoacoustic and hyperspectral dual‐modality microscopy for co‐imaging of melanoma and cutaneous squamous cell carcinoma in vivo

Skin carcinoma such as melanoma (MM) and cutaneous squamous cell carcinoma (cSCC) are considered as the highest mortality and the most aggressive skin cancers in dermatology. In view that early diagnosis and treatment can greatly improve the survival rate and life quality of the patients, developing noninvasive and effective evaluation methods is of great significance for the detection and identification of early stage cutaneous cancers. In this article, we propose a hybrid photoacoustic and hyperspectral dual‐modality microscopy to evaluate and differentiate skin carcinoma by structural and multiphysiological parameters. The proposed system's imaging abilities are verified by mimic phantoms and normal mice experiments. Furthermore, in vivo characterization and evaluation results of MM and cSCC mice are obtained successfully, which prove this novel method could be used as a reliable and useful method for skin cancer detection in early stages.     

Label‐free spectroscopic tissue characterization using fluorescence excitation‐scanning spectral imaging

Spectral imaging approaches provide new possibilities for measuring and discriminating fluorescent molecules in living cells and tissues. These approaches often employ tunable filters and robust image processing algorithms to identify many fluorescent labels in a single image set. Here, we present results from a novel spectral imaging technology that scans the fluorescence excitation spectrum, demonstrating that excitation‐scanning hyperspectral image data can discriminate among tissue types and estimate the molecular composition of tissues. This approach allows fast, accurate quantification of many fluorescent species from multivariate image data without the need of exogenous labels or dyes. We evaluated the ability of the excitation‐scanning approach to identify endogenous fluorescence signatures in multiple unlabeled tissue types. Signatures were screened using multi‐pass principal component analysis. Endmember extraction techniques revealed conserved autofluorescent signatures across multiple tissue types. We further examined the ability to detect known molecular signatures by constructing spectral libraries of common endogenous fluorophores and applying multiple spectral analysis techniques on test images from lung, liver and kidney. Spectral deconvolution revealed structure‐specific morphologic contrast generated from pure molecule signatures. These results demonstrate that excitation‐scanning spectral imaging, coupled with spectral imaging processing techniques, provides an approach for discriminating among tissue types and assessing the molecular composition of tissues. Additionally, excitation scanning offers the ability to rapidly screen molecular markers across a range of tissues without using fluorescent labels. This approach lays the groundwork for translation of excitation‐scanning technologies to clinical imaging platforms.     

Scattering based hyperspectral imaging of plasmonic nanoplate clusters towards biomedical applications

A new optical scattering contrast‐agent based on polymer‐nanoparticle encapsulated silver nanoplates (PESNs) is presented. Silver nanoplates were chosen due to the flexibility of tuning their plasmon frequencies. The polymer coating preserves their physical and optical properties and confers other advantages such as controlled contrast agent delivery. Finite difference time domain (FDTD) simulations model the interaction of light with the nanoplates in different orientations in the cluster. Hyperspectral dark field microscopy (HYDFM) observes the scattering spectra of the PESNs. An unsupervised sequential maximum angle convex cone (SMACC) image analysis resolves spectral endmembers corresponding to different stacking orientations of the nanoplates. The orientation‐dependent endmembers qualitatively agree with the FDTD results. For contrast enhancement, the uptake and spatial distribution of PESNs are demonstrated by an HYDFM study of single melanoma cells to result in an enhanced contrast of up to 400%. A supervised spatial mapping of the endmembers obtained by the unsupervised SMACC algorithm reveals spatial distributions of PESNs with various clustering orientations of encapsulated nanoplates. Our study demonstrates tunability in plasmonics properties in clustered metal nanoparticles and its utility for the development of scatter‐based imaging contrast agents for a broad range of applications, including studies of single cells and other biomedical systems.

     

Characterization of blood coagulation dynamics and oxygenation in ex‐vivo retinal vessels by fluorescence hyperspectral imaging

Blood coagulation mechanisms forming a blood clot and preventing hemorrhage have been extensively studied in the last decades. Knowing the mechanisms behind becomes very important particularly in the case of blood vessel diseases. Real‐time and accurate diagnostics accompanied by the therapy are particularly needed, for example, in diseases related to retinal vasculature. In our study, we employ for the first time fluorescence hyperspectral imaging (fHSI) combined with the spectral analysis algorithm concept to assess physical as well as functional information of blood coagulation in real‐time. By laser‐induced local disruption of retinal vessels to mimic blood leaking and subsequent coagulation and a proper fitting algorithm, we were able to reveal and quantify the extent of local blood coagulation through direct identification of the change of oxyhemoglobin concentration within few minutes. We confirmed and illuminated the spatio‐temporal evolution of the essential role of erythrocytes in the coagulation cascade as the suppliers of oxygenated hemoglobin. By additional optical tweezers force manipulation, we showed immediate aggregation of erythrocytes at the coagulation site. The presented fluorescence‐based imaging concept could become a valuable tool in various blood coagulation diagnostics as well as theranostic systems if coupled with the laser therapy.     

Perspectives,potentials and trends of ex vivo and in vivo optical molecular pathology

It is pivotal for medical applications, such as noninvasive histopathologic characterization of tissue, to realize label‐free and molecule‐specific representation of morphologic and biochemical composition in real‐time with subcellular spatial resolution. This unmet clinical need requires new approaches for rapid and reliable real‐time assessment of pathologies to complement established diagnostic tools. Photonic imaging combined with digitalization offers the potential to provide the clinician the requested information both under in vivo and ex vivo conditions. This report summarizes photonic approaches and their use in combination with image processing, machine learning and augmented virtual reality that might solve current challenges in modern medicine. Details are given for pathology, intraoperative diagnosis in head and neck cancer and endoscopic diagnosis in gastroenterology.      

Two‐photon excited fluorescence lifetime measurements through a double‐clad photonic crystal fiber for tissue micro‐endoscopy

This paper presents an endoscopic configuration for measurements of tissue autofluorescence using two–photon excitation and time‐correlated single photon counting detection through a double‐clad photonic crystal fiber (DC‐PCF) without pre‐chirping of laser pulses. The instrument performance was evaluated by measurements of fluorescent standard dyes, biological fluorophores (collagen and elastin), and tissue specimens (muscle, cartilage, tendon). Current results demonstrate the ability of this system to accurately retrieve the fluorescence decay profile and lifetime of these samples. This simple setup, which offers larger penetration depth than one‐photon‐based techniques, may be combined with morphology‐yielding techniques such as photoacoustic and ultrasound imaging. (© 2012 WILEY‐VCH Verlag GmbH & Co. KGaA, Weinheim)     

Resonance Raman quantification of the redox state of cytochromes b and c in‐vivo and in‐vitro

We observe the redox state changes with respiration of cytochromes b and c in mitochondria in a living Saccharomyces cerevisiae cell as well as in isolated mitochondria with the very use of Raman microspectroscopy. The possibility of monitoring the respiration activity of mitochondria in vivo and in vitro by Raman microspectroscopic quantification of the cytochrome redox states is suggested. It will lead to a new means to assess mitochondrial respiration activity in vivo and in vitro without using any labelling or genetic manipulation. (© 2012 WILEY‐VCH Verlag GmbH & Co. KGaA, Weinheim)     

Label‐free and non‐invasive discrimination of HaCaT and melanoma cells in a co‐culture model by hyperspectral confocal reflectance microscopy

A novel hyperspectral confocal microscopy method to separate different cell populations in a co‐culture model is presented here. The described methodological and instrumental approach allows discrimination of different cell types using a non‐invasive, label free method with good accuracy with a single cell resolution. In particular, melanoma cells are discriminated from HaCaT cells by hyperspectral confocal imaging, principal component analysis and optical frequencies signing, as confirmed by fluorescence labelling cross check. The identification seems to be quite robust to be insensitive to the cellular shape within the studied samples, enabling to separate cells according to their cytotype down to a single cell sensitivity.

Set of hyperspectral images of melanoma‐keratinocytes co‐culture model (left), score plot of principal component analysis and spectral analysis of principal components coefficients (center), label‐free spectral identification of cell populations (right).     

Label‐free in vivo imaging of peripheral nerve by multispectral photoacoustic tomography

Unintentional surgical damage to nerves is mainly due to poor visualization of nerve tissue relative to adjacent structures. Multispectral photoacoustic tomography can provide chemical information with specificity and ultrasonic spatial resolution with centimeter imaging depth, making it a potential tool for noninvasive neural imaging. To implement this label‐free imaging approach, a multispectral photoacoustic tomography platform was built. Imaging depth and spatial resolution were characterized. In vivo imaging of the femoral nerve that is 2 mm deep in a nude mouse was performed. Through multivariate curve resolution analysis, the femoral nerve was discriminated from the femoral artery and chemical maps of their spatial distributions were generated.

The femoral nerve was discriminated from the femoral artery by multivariate curve resolution analysis.     

Correlating species and spectral diversities using hyperspectral remote sensing in early‐successional fields

Advances in remote sensing technology can help estimate biodiversity at large spatial extents. To assess whether we could use hyperspectral visible near‐infrared (VNIR) spectra to estimate species diversity, we examined the correlations between species diversity and spectral diversity in early‐successional abandoned agricultural fields in the Ridge and Valley ecoregion of north‐central Virginia at the Blandy Experimental Farm. We established plant community plots and collected vegetation surveys and ground‐level hyperspectral data from 350 to 1,025 nm wavelengths. We related spectral diversity (standard deviations across spectra) with species diversity (Shannon–Weiner index) and evaluated whether these correlations differed among spectral regions throughout the visible and near‐infrared wavelength regions, and across different spectral transformation techniques. We found positive correlations in the visible regions using band depth data, positive correlations in the near‐infrared region using first derivatives of spectra, and weak to no correlations in the red‐edge region using either of the two spectral transformation techniques. To investigate the role of pigment variability in these correlations, we estimated chlorophyll, carotenoid, and anthocyanin concentrations of five dominant species in the plots using spectral vegetation indices. Although interspecific variability in pigment levels exceeded intraspecific variability, chlorophyll was more varied within species than carotenoids and anthocyanins, contributing to the lack of correlation between species diversity and spectral diversity in the red‐edge region. Interspecific differences in pigment levels, however, made it possible to differentiate these species remotely, contributing to the species‐spectral diversity correlations. VNIR spectra can be used to estimate species diversity, but the relationships depend on the spectral region examined and the spectral transformation technique used.     

Adaptive Boosting (AdaBoost)‐based multiwavelength spatial frequency domain imaging and characterization for ex vivo human colorectal tissue assessment

The current gold standard diagnostic test for colorectal cancer remains histological inspections of endoluminal neoplasia in biopsy specimens. However, biopsy site selection requires visual inspection of the bowel, typically with a white‐light endoscope. Therefore, this technique is poorly suited to detect small or innocuous‐appearing lesions. We hypothesize that an alternative modality—multiwavelength spatial frequency domain imaging (SFDI)—would be able to differentiate various colorectal neoplasia from normal tissue. In this ex vivo study of human colorectal tissues, we report the optical absorption and scattering signatures of normal, adenomatous polyp and cancer specimens. An abnormal vs. normal adaptive boosting (AdaBoost) classifier is trained to dichotomize tissue based on SFDI imaging characteristics, and an area under the receiver operating characteristic (ROC) curve (AUC) of 0.95 is achieved. We conclude that AdaBoost‐based multiwavelength SFDI can differentiate abnormal from normal colorectal tissues, potentially improving endoluminal screening of the distal gastrointestinal tract in the future.     

Optical hyperspectral imaging in microscopy and spectroscopy – a review of data acquisition

Rather than simply acting as a photographic camera capturing two‐dimensional (x, y) intensity images or a spectrometer acquiring spectra (λ), a hyperspectral imager measures entire three‐dimensional (x, y, λ) datacubes for multivariate analysis, providing structural, molecular, and functional information about biological cells or tissue with unprecedented detail. Such data also gives clinical insights for disease diagnosis and treatment. We summarize the principles underpinning this technology, highlight its practical implementation, and discuss its recent applications at microscopic to macroscopic scales.

Datacube acquisition strategies in hyperspectral imaging x, y, spatial coordinates; λ, wavelength.     

Real‐time video mosaicking to guide handheld in vivo microscopy

Handheld and endoscopic optical‐sectioning microscopes are being developed for noninvasive screening and intraoperative consultation. Imaging a large extent of tissue is often desired, but miniature in vivo microscopes tend to suffer from limited fields of view. To extend the imaging field during clinical use, we have developed a real‐time video mosaicking method, which allows users to efficiently survey larger areas of tissue. Here, we modified a previous post‐processing mosaicking method so that real‐time mosaicking is possible at >30 frames/second when using a device that outputs images that are 400 × 400 pixels in size. Unlike other real‐time mosaicking methods, our strategy can accommodate image rotations and deformations that often occur during clinical use of a handheld microscope. We perform a feasibility study to demonstrate that the use of real‐time mosaicking is necessary to enable efficient sampling of a desired imaging field when using a handheld dual‐axis confocal microscope.