Journey to the centre of the cell: Virtual reality immersion into scientific data

Visualization of scientific data is crucial not only for scientific discovery but also to communicate science and medicine to both experts and a general audience. Until recently, we have been limited to visualizing the three‐dimensional (3D) world of biology in 2 dimensions. Renderings of 3D cells are still traditionally displayed using two‐dimensional (2D) media, such as on a computer screen or paper. However, the advent of consumer grade virtual reality (VR) headsets such as Oculus Rift and HTC Vive means it is now possible to visualize and interact with scientific data in a 3D virtual world. In addition, new microscopic methods provide an unprecedented opportunity to obtain new 3D data sets. In this perspective article, we highlight how we have used cutting edge imaging techniques to build a 3D virtual model of a cell from serial block‐face scanning electron microscope (SBEM) imaging data. This model allows scientists, students and members of the public to explore and interact with a “real” cell. Early testing of this immersive environment indicates a significant improvement in students’ understanding of cellular processes and points to a new future of learning and public engagement. In addition, we speculate that VR can become a new tool for researchers studying cellular architecture and processes by populating VR models with molecular data.      

Molecular Visualization on the Holodeck

Can virtual reality be useful for visualizing and analyzing molecular structures and three-dimensional (3D) microscopy? Uses we are exploring include studies of drug binding to proteins and the effects of mutations, building accurate atomic models in electron microscopy and x-ray density maps, understanding how immune system cells move using 3D light microscopy, and teaching schoolchildren about biomolecules that are the machinery of life. Virtual reality (VR) offers immersive display with a wide field of view and head tracking for better perception of molecular architectures and uses 6-degree-of-freedom hand controllers for simple manipulation of 3D data. Conventional computer displays with trackpad, mouse and keyboard excel at two-dimensional tasks such as writing and studying research literature, uses for which VR technology is at present far inferior. Adding VR to the conventional computing environment could improve 3D capabilities if new user-interface problems can be solved. We have developed three VR applications: ChimeraX for analyzing molecular structures and electron and light microscopy data, AltPDB for collaborative discussions around atomic models, and Molecular Zoo for teaching young students characteristics of biomolecules. Investigations over three decades have produced an extensive literature evaluating the potential of VR in research and education. Consumer VR headsets are now affordable to researchers and educators, allowing direct tests of whether the technology is valuable in these areas. We survey here advantages and disadvantages of VR for molecular biology in the context of affordable and dramatically more powerful VR and graphics hardware than has been available in the past.     

On the way to building an integrated computational environment for the study of developmental patterns and genetic diseases

Genetic diseases and developmental patterns should be studied on several levels: from macroscale (organs and tissues) to nanoscale (cells, genes, proteins). Due to the overwhelming complexity of the life science data, it is common that disparate data pieces are meticulously stored but never fully analyzed or correlated. We have begun to develop a novel methodology based on virtual reality techniques for the study of these phenomena. Our key approach to knowledge integration is a top-down mapping of data onto visual contexts. For each organism that we want to study, a structural model is created and used as a visual "wireframe" onto which other data types are superimposed in a top-down assembly. Data analysis tools, visual controls, and queries are enabled so that users can interactively explore data. Our visualization technology gives users an opportunity to map disparate data onto a common model, and search visually for hitherto unknown patterns and correlations contained within the data. It is our goal to eventually transform genomics research from measuring various data pieces analytically into a fully interactive exploration of combined data in a 4D immersive visual environment that best matches a researcher's intuition.     

A Functional Magnetic Resonance Imaging (fMRI) Study of Cue-Induced Smoking Craving in Virtual Environments

Smokers who are exposed to smoking-related cues show cardiovascular reactivity and smoking craving compared with their responses to neutral cues, and increased cue reactivity predicts decreased likelihood of successful cessation. Several brain imaging studies suggested four candidate brain regions that might differ in gray matter volumes and densities between smokers and nonsmokers. However, in these studies, smokers were only exposed to smoking-related objects. In our pilot study utilizing a virtual reality (VR) technique, virtual environments (VEs) were more immersive and evoked smoking craving more effectively than traditionally used methods. In this study, we sought to test whether smokers could experience cue-induced smoking craving inside the MRI scanner by using the VR system. The smoking cue reactivity scenario was based in part on our preliminary task and consisted of 2D and 3D (or VE) conditions. The group mean of participants had increased activity in the prefrontal cortex (PFC), left anterior cingulate gyrus (ACC), left supplementary motor area, left uncus, right inferior temporal gyrus, right lingual gyrus, and right precuneus in the 2D condition. Areas of differential activation in the 3D condition were as follows: left superior temporal gyrus, right superior frontal gyrus, and left inferior occipital gyrus in the 3D condition. This finding is consistent with those of previous studies of nicotine craving showing PFC and ACC activation. However, in the 3D condition, the PFC including the superior frontal gyrus as well as the superior temporal gyrus, inferior occipital gyrus, and cerebellum were activated. Therefore, in the 3D condition, participants seemed to have more attention, visual balance, and coordinating movement than in the 2D condition.     

Visualizing biomolecular electrostatics in virtual reality with UnityMol‐APBS

Virtual reality is a powerful tool with the ability to immerse a user within a completely external environment. This immersion is particularly useful when visualizing and analyzing interactions between small organic molecules, molecular inorganic complexes, and biomolecular systems such as redox proteins and enzymes. A common tool used in the biomedical community to analyze such interactions is the Adaptive Poisson‐Boltzmann Solver (APBS) software, which was developed to solve the equations of continuum electrostatics for large biomolecular assemblages. Numerous applications exist for using APBS in the biomedical community including analysis of protein ligand interactions and APBS has enjoyed widespread adoption throughout the biomedical community. Currently, typical use of the full APBS toolset is completed via the command line followed by visualization using a variety of two‐dimensional external molecular visualization software. This process has inherent limitations: visualization of three‐dimensional objects using a two‐dimensional interface masks important information within the depth component. Herein, we have developed a single application, UnityMol‐APBS, that provides a dual experience where users can utilize the full range of the APBS toolset, without the use of a command line interface, by use of a simple graphical user interface (GUI) for either a standard desktop or immersive virtual reality experience.     

Mobile learning applications to improve invertebrate zoology online teaching

The use of new technologies including personal mobile devices has become an indispensable tool in our daily lives, and thus its presence in education is becoming ever more ubiquitous. In the current scenario imposed by the COVID‐19 pandemic, in which in‐person presence in classrooms has been enormously reduced at all educational levels, the use of mobile learning and cutting‐edge methods can greatly improve the way students learn and enhance their online‐learning experience. Mobile applications, combined with extended reality technologies such as virtual reality (VR) and augmented reality (AR), are powerful tools that connect real and virtual environments and allow higher interaction for the user. We have leveraged the advantages of mobile learning and extended reality technologies to develop a series of mobile applications and associated educational activities for university‐level courses involving invertebrate zoology field work. In particular, we have developed (a) a VR SCUBA diving video to explore the diversity of a marine protected area; (b) an AR mobile app to visualize 3D models of marine invertebrates; and (c) a mobile‐based catalogue to explore the terrestrial biodiversity of one of the most diverse regions of Spain. Here we provide detailed information describing the design and creation of these tools, as well as their application in class, to facilitate and encourage their use in higher education. Despite the relatively recent application of these technologies in education, they have an enormous potential: they improve student motivation and learning, can be adapted to different learning styles, reduce social inequalities, and facilitate inclusiveness and diversity practices in the classroom.     

siMBa—a simple graphical user interface for the Bayesian phylogenetic inference program MrBayes

MrBayes is a program that uses a Bayesian framework for inferring phylogenetic relationships. As MrBayes is a command-line-driven program, users acquainted to programs with graphical user interfaces will not find it easy to operate, especially as it requires a complex input format for the data to be analysed. We thus developed siMBa (simple MrBayes), a simple graphical user interface for MrBayes. This tool gives the user interactive control over most of the parameters and also facilitates the input of a multiple sequence alignment, as any widely used format can be used. siMBa is coded in Perl using the Tk module. Executables are provided for Windows, Linux, and Macintosh. The Perl codes, along with executables for different operating system, are freely available to download from [http://www.thines-lab.senckenberg.de/simba].     

Stellate Cells in the Medial Entorhinal Cortex Are Required for Spatial Learning

1. Download high-res image (129KB)
2. Download full-size image

     

KinImmerse: Macromolecular VR for NMR ensembles

Background  

In molecular applications, virtual reality (VR) and immersive virtual environments have generally been used and valued for the visual and interactive experience – to enhance intuition and communicate excitement – rather than as part of the actual research process. In contrast, this work develops a software infrastructure for research use and illustrates such use on a specific case.     

Distributed computing in image analysis using open source frameworks and application to image sharpness assessment of histological whole slide images

Background

Automated image analysis on virtual slides is evolving rapidly and will play an important role in the future of digital pathology. Due to the image size, the computational cost of processing whole slide images (WSIs) in full resolution is immense. Moreover, image analysis requires well focused images in high magnification.

Methods

We present a system that merges virtual microscopy techniques, open source image analysis software, and distributed parallel processing. We have integrated the parallel processing framework JPPF, so batch processing can be performed distributed and in parallel. All resulting meta data and image data are collected and merged. As an example the system is applied to the specific task of image sharpness assessment. ImageJ is an open source image editing and processing framework developed at the NIH having a large user community that contributes image processing algorithms wrapped as plug-ins in a wide field of life science applications. We developed an ImageJ plug-in that supports both basic interactive virtual microscope and batch processing functionality. For the application of sharpness inspection we employ an approach with non-overlapping tiles. Compute nodes retrieve image tiles of moderate size from the streaming server and compute the focus measure. Each tile is divided into small sub images to calculate an edge based sharpness criterion which is used for classification. The results are aggregated in a sharpness map.

Results

Based on the system we calculate a sharpness measure and classify virtual slides into one of the following categories - excellent, okay, review and defective. Generating a scaled sharpness map enables the user to evaluate sharpness of WSIs and shows overall quality at a glance thus reducing tedious assessment work.

Conclusions

Using sharpness assessment as an example, the introduced system can be used to process, analyze and parallelize analysis of whole slide images based on open source software.

     

EzMol: A Web Server Wizard for the Rapid Visualization and Image Production of Protein and Nucleic Acid Structures

EzMol is a molecular visualization Web server in the form of a software wizard, located at http://www.sbg.bio.ic.ac.uk/ezmol/. It is designed for easy and rapid image manipulation and display of protein molecules, and is intended for users who need to quickly produce high-resolution images of protein molecules but do not have the time or inclination to use a software molecular visualization system. EzMol allows the upload of molecular structure files in PDB format to generate a Web page including a representation of the structure that the user can manipulate. EzMol provides intuitive options for chain display, adjusting the color/transparency of residues, side chains and protein surfaces, and for adding labels to residues. The final adjusted protein image can then be downloaded as a high-resolution image. There are a range of applications for rapid protein display, including the illustration of specific areas of a protein structure and the rapid prototyping of images.     

ISFOLD: Structure Prediction of Base Pairs in Non-Helical RNA Motifs from Isostericity Signatures in Their Sequence Alignments

Abstract

The existence and identity of non-Watson-Crick base pairs (bps) within RNA bulges, internal loops, and hairpin loops cannot reliably be predicted by existing algorithms. We have developed the Isfold (Isosteric Folding) program as a tool to examine patterns of nucleotide substitutions from sequence alignments or mutation experiments and identify plausible bp interactions. We infer these interactions based on the observation that each non-Watson-Crick bp has a signature pattern of isosteric substitutions where mutations can be made that preserve the 3D structure. Isfold produces a dynamic representation of predicted bps within defined motifs in order of their probabilities. The software was developed under Windows XP, and is capable of running on PC and MAC with Matlab 7.1 (SP3) or higher. A PC standalone version that does not require Matlab also is available. This software and a user manual are freely available at www.ucsf.edu/frankel/isfold.     

Closed-Loop Control of Active Sensing Movements Regulates Sensory Slip

1. Download high-res image (213KB)
2. Download full-size image

     

SDAR: a practical tool for graphical analysis of two-dimensional data

ABSTRACT: BACKGROUND: Two-dimensional data needs to be processed and analysed in almost any experimental laboratory. Some tasks in this context may be performed with generic software such as spreadsheet programs which are available ubiquitously, others may require more specialised software that requires paid licences. Additionally, more complex software packages typically require more time by the individual user to understand and operate. Practical and convenient graphical data analysis software in Java with a user-friendly interface are rare. RESULTS: We have developed SDAR, a Java application to analyse two-dimensional data with an intuitive graphical user interface. A smart ASCII parser allows import of data into SDAR without particular format requirements. The centre piece of SDAR is the Java class GraphPanel which provides methods for generic tasks of data visualisation. Data can be manipulated and analysed with respect to the most common operations experienced in an experimental biochemical laboratory. Images of the data plots can be generated in SVG-, TIFF- or PNG-format. Data exported by SDAR is annotated with commands compatible with the Grace software. CONCLUSION: Since SDAR is implemented in Java, it is truly cross-platform compatible. The software is easy to install, and very convenient to use judging by experience in our own laboratories. It is freely available to academic users at http://www.structuralchemistry.org/pcsb/. To download SDAR, users will be asked for their name, institution and email address. A manual, as well as the source code of the GraphPanel class can also be downloaded from this site.     

3D imaging of diatoms with ion-abrasion scanning electron microscopy

Ion-abrasion scanning electron microscopy (IASEM) takes advantage of focused ion beams to abrade thin sections from the surface of bulk specimens, coupled with SEM to image the surface of each section, enabling 3D reconstructions of subcellular architecture at 30 nm resolution. Here, we report the first application of IASEM for imaging a biomineralizing organism, the marine diatom Thalassiosira pseudonana. Diatoms have highly patterned silica-based cell wall structures that are unique models for the study and application of directed nanomaterials synthesis by biological systems. Our study provides new insights into the architecture and assembly principles of both the “hard” (siliceous) and “soft” (organic) components of the cell. From 3D reconstructions of developmentally synchronized diatoms captured at different stages, we show that both micro- and nanoscale siliceous structures can be visualized at specific stages in their formation. We show that not only are structures visualized in a whole-cell context, but demonstrate that fragile, early-stage structures are visible, and that this can be combined with elemental mapping in the exposed slice. We demonstrate that the 3D architectures of silica structures, and the cellular components that mediate their creation and positioning can be visualized simultaneously, providing new opportunities to study and manipulate mineral nanostructures in a genetically tractable system.     

Virtual Reality Cue Reactivity Assessment: A Case Study in a Teen Smoker

Cigarette smoking in adolescents is a major public health problem. To address the increasing need for efficacious assessment and treatment methods, we developed and tested a novel virtual reality cue reactivity assessment system. A case study of a controlled virtual reality cue reactivity trial with a 17-year-old adolescent cigarette smoker is presented. During the trial, the participant was exposed to virtual reality (VR) smoking cues and VR neutral cues and assessments of subjective craving and skin conductance response (SCR) were recorded. Upon exposure to VR smoking cues, craving increased. A novel methodology for collecting and analyzing SCR in VR was developed and explored to expand the role of physiological variables in VR research. SCR data indicated specific reactions to smoking cue stimuli, with the subject experiencing increased reactivity to smoking cues (i.e., cigarettes) compared to food or drinks. Based on this case study, further research using VR cue reactivity assessment in adolescent smokers is warranted. The impact of VR in drug research and future applications in research are also discussed.