GPU-based real-time soft tissue deformation with cutting and haptic feedback

This article describes a series of contributions in the field of real-time simulation of soft tissue biomechanics. These contributions address various requirements for interactive simulation of complex surgical procedures. In particular, this article presents results in the areas of soft tissue deformation, contact modelling, simulation of cutting, and haptic rendering, which are all relevant to a variety of medical interventions. The contributions described in this article share a common underlying model of deformation and rely on GPU implementations to significantly improve computation times. This consistency in the modelling technique and computational approach ensures coherent results as well as efficient, robust and flexible solutions.     

Real-time interactive isosurfacing: a new method for improving marching isosurfacing algorithm output and efficiency

Efficient rendering of a changing volumetric data-set is central to the development of effective medical simulations that incorporate haptic feedback. A new method referred to as real-time interactive isosurfacing (RTII) is described in this paper. RTII is an algorithm that can be applied to output from Marching Cubes-like algorithms to improve performance for real-time applications. The approach minimises processing by re-evaluating the isosurface around changing sub-volumes resulting from user interactions. It includes innovations that significantly reduce mesh complexity and improve mesh quality as triangles are created from the Marching Tetrahedra isosurfacing algorithm. Rendering efficiency is further improved over other marching isosurfacing algorithm outputs by maintaining an indexed triangle representation of the mesh. The effectiveness of RTII is discussed within the context of an endoscopic sinus surgery simulation currently being developed by the authors.     

Haptic/Graphic Rehabilitation: Integrating a Robot into a Virtual Environment Library and Applying it to Stroke Therapy

Recent research that tests interactive devices for prolonged therapy practice has revealed new prospects for robotics combined with graphical and other forms of biofeedback. Previous human-robot interactive systems have required different software commands to be implemented for each robot leading to unnecessary developmental overhead time each time a new system becomes available. For example, when a haptic/graphic virtual reality environment has been coded for one specific robot to provide haptic feedback, that specific robot would not be able to be traded for another robot without recoding the program. However, recent efforts in the open source community have proposed a wrapper class approach that can elicit nearly identical responses regardless of the robot used. The result can lead researchers across the globe to perform similar experiments using shared code. Therefore modular "switching out"of one robot for another would not affect development time. In this paper, we outline the successful creation and implementation of a wrapper class for one robot into the open-source H3DAPI, which integrates the software commands most commonly used by all robots.     

Mlab--a mathematical modeling tool

An interactive interpreter called Mlab is described. One uses Mlab by typing commands. In this sense, Mlab is a programming language. It has various mathematical and graphical facilities which make it a useful tool for mathematical modeling. The curve fitting capabilities of Mlab are augmented with differential-equation-handling and matrix-manipulation capabilities which provide a powerful and civilized facility for curve fitting. Many people are engaged in this activity, and, in general, they use programs which are neither sufficiently general nor easy to use. (Some conventional programming is usually required, for example.) Mlab purports to be easier than alternate approaches. The nature of Mlab is discussed with accompanying examples. The main example is the use of curve fitting to determine molecular weight from ultracentrifuge data. This example was chosen because it exhibits a special feature of Mlab, namely the root operator, which appears in the definition of the model function.     

UCSF Chimera, MODELLER, and IMP: an integrated modeling system

Structural modeling of macromolecular complexes greatly benefits from interactive visualization capabilities. Here we present the integration of several modeling tools into UCSF Chimera. These include comparative modeling by MODELLER, simultaneous fitting of multiple components into electron microscopy density maps by IMP MultiFit, computing of small-angle X-ray scattering profiles and fitting of the corresponding experimental profile by IMP FoXS, and assessment of amino acid sidechain conformations based on rotamer probabilities and local interactions by Chimera.     

GenExp: an interactive web-based genomic DAS client with client-side data rendering

Background

The Distributed Annotation System (DAS) offers a standard protocol for sharing and integrating annotations on biological sequences. There are more than 1000 DAS sources available and the number is steadily increasing. Clients are an essential part of the DAS system and integrate data from several independent sources in order to create a useful representation to the user. While web-based DAS clients exist, most of them do not have direct interaction capabilities such as dragging and zooming with the mouse.

Results

Here we present GenExp, a web based and fully interactive visual DAS client. GenExp is a genome oriented DAS client capable of creating informative representations of genomic data zooming out from base level to complete chromosomes. It proposes a novel approach to genomic data rendering and uses the latest HTML5 web technologies to create the data representation inside the client browser. Thanks to client-side rendering most position changes do not need a network request to the server and so responses to zooming and panning are almost immediate. In GenExp it is possible to explore the genome intuitively moving it with the mouse just like geographical map applications. Additionally, in GenExp it is possible to have more than one data viewer at the same time and to save the current state of the application to revisit it later on.

Conclusions

GenExp is a new interactive web-based client for DAS and addresses some of the short-comings of the existing clients. It uses client-side data rendering techniques resulting in easier genome browsing and exploration. GenExp is open source under the GPL license and it is freely available at http://gralggen.lsi.upc.edu/recerca/genexp.     

Visualizing density maps with UCSF Chimera

We describe methods for interactive visualization and analysis of density maps available in the UCSF Chimera molecular modeling package. The methods enable segmentation, fitting, coarse modeling, measuring and coloring of density maps for elucidating structures of large molecular assemblies such as virus particles, ribosomes, microtubules, and chromosomes. The methods are suitable for density maps with resolutions in the range spanned by electron microscope single particle reconstructions and tomography. All of the tools described are simple, robust and interactive, involving computations taking only seconds. An advantage of the UCSF Chimera package is its integration of a large collection of interactive methods. Interactive tools are sufficient for performing simple analyses and also serve to prepare input for and examine results from more complex, specialized, and algorithmic non-interactive analysis software. While both interactive and non-interactive analyses are useful, we discuss only interactive methods here.     

Texture mapping: A new tool for molecular graphics

The real-time texture mapping capabilities of modern graphics workstations are explored with respect to their applications in a variety of relevant scenarios in interactive molecular modeling techniques. The common usage of texture mapping to reduce geometric complexity while enhancing realism is extended, opening new ways to visualize large amounts of molecular data in a comprehensive fashion. Thus, texture mapping may be employed to (1) display and filter multichannel information of structural properties on molecular surfaces, (2) improve the quality and accuracy of highly complex isodensity contours, (3) increase the rendering speed of space-filling atomic representations by two orders of magnitude and (4) apply volume-rendering techniques to large, three-dimensional density distributions in real time. Implementation of these novel techniques requires only moderate modifications or extensions to existing molecular modeling applications.     

人手指柔性触觉感知的记忆特性

触觉再现技术是当前虚拟现实和远程操作机器人领域的前沿,而柔性触觉则是其重要的研究内容。触觉再现接口的设计需要充分研究人手的触觉感知特性。本文在柔性触觉装置上研究了人手指柔性触觉记忆特性。先通过回忆性实验确定人手指的柔性触觉记忆容量,在记忆容量范围内又进行了再认性实验,对人手指的柔性触觉记忆反应时间进行分析。本实验方法简单有效,得出的结论不仅可以用来改进触觉再现装置的设计,而且为触觉再现技术的研究提供了生理学依据。     

基于信息管理的一种虚拟森林景观构建及应用探讨

在分析不同尺度的森林可视化建模内容和技术特点的基础上,提出了一个基于信息管理的虚拟森林景观构造原理和技术体系.把过程建模技术与树木形态结构描述结合,提出了一种交互式、参数化的树木动态建模方法,给出了相应的绘制方法和几何体简化算法以实现加速实时绘制,并以福建省漳浦县为例,建立了典型树种的几何模型库.利用森林调查和遥感动态空间数据,借助地理信息系统ArcObiect组件、图形环境OpenGL和Visual C⁺⁺语言,开发了虚拟森林管理原型系统,实现森林二维/三维交互漫游、查询分析、森林生长仿真模拟,其真实感与模拟精度满足实际森林资源管理需求.最后给出了系统的典型用户界面以及在考虑竞争条件下马尾松自然生长模拟和人工间伐前后的虚拟景观对比的应用例子.     

Exposure render: an interactive photo-realistic volume rendering framework

The field of volume visualization has undergone rapid development during the past years, both due to advances in suitable computing hardware and due to the increasing availability of large volume datasets. Recent work has focused on increasing the visual realism in Direct Volume Rendering (DVR) by integrating a number of visually plausible but often effect-specific rendering techniques, for instance modeling of light occlusion and depth of field. Besides yielding more attractive renderings, especially the more realistic lighting has a positive effect on perceptual tasks. Although these new rendering techniques yield impressive results, they exhibit limitations in terms of their exibility and their performance. Monte Carlo ray tracing (MCRT), coupled with physically based light transport, is the de-facto standard for synthesizing highly realistic images in the graphics domain, although usually not from volumetric data. Due to the stochastic sampling of MCRT algorithms, numerous effects can be achieved in a relatively straight-forward fashion. For this reason, we have developed a practical framework that applies MCRT techniques also to direct volume rendering (DVR). With this work, we demonstrate that a host of realistic effects, including physically based lighting, can be simulated in a generic and flexible fashion, leading to interactive DVR with improved realism. In the hope that this improved approach to DVR will see more use in practice, we have made available our framework under a permissive open source license.     

MMB-GUI: a fast morphing method demonstrates a possible ribosomal tRNA translocation trajectory

Easy-to-use macromolecular viewers, such as UCSF Chimera, are a standard tool in structural biology. They allow rendering and performing geometric operations on large complexes, such as viruses and ribosomes. Dynamical simulation codes enable modeling of conformational changes, but may require considerable time and many CPUs. There is an unmet demand from structural and molecular biologists for software in the middle ground, which would allow visualization combined with quick and interactive modeling of conformational changes, even of large complexes. This motivates MMB-GUI. MMB uses an internal-coordinate, multiscale approach, yielding as much as a 2000-fold speedup over conventional simulation methods. We use Chimera as an interactive graphical interface to control MMB. We show how this can be used for morphing of macromolecules that can be heterogeneous in biopolymer type, sequence, and chain count, accurately recapitulating structural intermediates. We use MMB-GUI to create a possible trajectory of EF-G mediated gate-passing translocation in the ribosome, with all-atom structures. This shows that the GUI makes modeling of large macromolecules accessible to a wide audience. The morph highlights similarities in tRNA conformational changes as tRNA translocates from A to P and from P to E sites and suggests that tRNA flexibility is critical for translocation completion.     

Interactive visual exploration of overlapping similar structures for three-dimensional microscope images

Background

Recent advances in microscopy enable the acquisition of large numbers of tomographic images from living tissues. Three-dimensional microscope images are often displayed with volume rendering by adjusting the transfer functions. However, because the emissions from fluorescent materials and the optical properties based on point spread functions affect the imaging results, the intensity value can differ locally, even in the same structure. Further, images obtained from brain tissues contain a variety of neural structures such as dendrites and axons with complex crossings and overlapping linear structures. In these cases, the transfer functions previously used fail to optimize image generation, making it difficult to explore the connectivity of these tissues.

Results

This paper proposes an interactive visual exploration method by which the transfer functions are modified locally and interactively based on multidimensional features in the images. A direct editing interface is also provided to specify both the target region and structures with characteristic features, where all manual operations can be performed on the rendered image. This method is demonstrated using two-photon microscope images acquired from living mice, and is shown to be an effective method for interactive visual exploration of overlapping similar structures.

Conclusions

An interactive visualization method was introduced for local improvement of visualization by volume rendering in two-photon microscope images containing regions in which linear nerve structures crisscross in a complex manner. The proposed method is characterized by the localized multidimensional transfer function and interface where the parameters can be determined by the user to suit their particular visualization requirements.     

Volumetric feature extraction and visualization of tomographic molecular imaging

Electron tomography is useful for studying large macromolecular complex within their cellular context. The associate problems include crowding and complexity. Data exploration and 3D visualization of complexes require rendering of tomograms as well as extraction of all features of interest. We present algorithms for fully automatic boundary segmentation and skeletonization, and demonstrate their applications in feature extraction and visualization of cell and molecular tomographic imaging. We also introduce an interactive volumetric exploration and visualization tool (Volume Rover), which encapsulates implementations of the above volumetric image processing algorithms, and additionally uses efficient multi-resolution interactive geometry and volume rendering techniques for interactive visualization.     

The MIAmaxent R package: Variable transformation and model selection for species distribution models

The widely used “Maxent” software for modeling species distributions from presence‐only data (Phillips et al., Ecological Modelling, 190, 2006, 231) tends to produce models with high‐predictive performance but low‐ecological interpretability, and implications of Maxent's statistical approach to variable transformation, model fitting, and model selection remain underappreciated. In particular, Maxent's approach to model selection through lasso regularization has been shown to give less parsimonious distribution models—that is, models which are more complex but not necessarily predictively better—than subset selection. In this paper, we introduce the MIAmaxent R package, which provides a statistical approach to modeling species distributions similar to Maxent's, but with subset selection instead of lasso regularization. The simpler models typically produced by subset selection are ecologically more interpretable, and making distribution models more grounded in ecological theory is a fundamental motivation for using MIAmaxent. To that end, the package executes variable transformation based on expected occurrence–environment relationships and contains tools for exploring data and interrogating models in light of knowledge of the modeled system. Additionally, MIAmaxent implements two different kinds of model fitting: maximum entropy fitting for presence‐only data and logistic regression (GLM) for presence–absence data. Unlike Maxent, MIAmaxent decouples variable transformation, model fitting, and model selection, which facilitates methodological comparisons and gives the modeler greater flexibility when choosing a statistical approach to a given distribution modeling problem.     

Three-dimensional models of non-NMDA glutamate receptors.

Structural models have been produced for three types of non-NMDA inotropic glutamate receptors: an AMPA receptor, GluR1, a kainate receptor, GluR6; and a low-molecular-weight kainate receptor from goldfish, GFKAR alpha. Modeling was restricted to the domains of the proteins that bind the neurotransmitter glutamate and that form the ion channel. Model building combined homology modeling, distance geometry, molecular mechanics, interactive modeling, and known constraints. The models indicate new potential interactions in the extracellular domain between protein and agonists, and suggest that the transition from the "closed" to the "open" state involves the movement of a conserved positive residue away from, and two conserved negative residues into, the extracellular entrance to the pore upon binding. As a first approximation, the ion channel domain was modeled with a structure comprising a central antiparallel beta-barrel that partially crosses the membrane, and against which alpha-helices from each subunit are packed; a third alpha-helix packs against these two helices in each subunit. Much, but not all, of the available data were consistent with this structure. Modifying the beta-barrel to a loop-like topology produced a model consistent with available data.     

Haptic perception of object curvature in Parkinson's disease

Background

The haptic perception of the curvature of an object is essential for adequate object manipulation and critical for our guidance of actions. This study investigated how the ability to perceive the curvature of an object is altered by Parkinson''s disease (PD).

Methodology/Principal Findings

Eight healthy subjects and 11 patients with mild to moderate PD had to judge, without vision, the curvature of a virtual “box” created by a robotic manipulandum. Their hands were either moved passively along a defined curved path or they actively explored the curved curvature of a virtual wall. The curvature was either concave or convex (bulging to the left or right) and was judged in two locations of the hand workspace–a left workspace location, where the curved hand path was associated with curved shoulder and elbow joint paths, and a right workspace location in which these joint paths were nearly linear. After exploring the curvature of the virtual object, subjects had to judge whether the curvature was concave or convex. Based on these data, thresholds for curvature sensitivity were established. The main findings of the study are: First, 9 out 11 PD patients (82%) showed elevated thresholds for detecting convex curvatures in at least one test condition. The respective median threshold for the PD group was increased by 343% when compared to the control group. Second, when distal hand paths became less associated with proximal joint paths (right workspace), haptic acuity was reduced substantially in both groups. Third, sensitivity to hand trajectory curvature was not improved during active exploration in either group.

Conclusion/Significance

Our data demonstrate that PD is associated with a decreased acuity of the haptic sense, which may occur already at an early stage of the disease.     

Reconstructing mental object representations: a machine vision approach to human visual recognition

This paper introduces a new approach to assess visual representations underlying the recognition of objects. Human performance is modeled by CLARET, a machine learning and matching system, based on inductive logic programming and graph matching principles. The model is applied to data of a learning experiment addressing the role of prior experience in the ontogenesis of mental object representations. Prior experience was varied in terms of sensory modality, i.e. visual versus haptic versus visuohaptic. The analysis revealed distinct differences between the representational formats used by subjects with haptic versus those with no prior object experience. These differences suggest that prior haptic exploration stimulates the evolution of object representations which are characterized by an increased differentiation between attribute values and a pronounced structural encoding.     

Volumetric modeling and interactive cutting of deformable bodies

A new approach for the interactive simulation of viscoelastic object cutting is presented. Two synchronized geometrical models at different resolutions are used, both derived from medical images. In contrast with most previous approaches, the blade deforms the object, and cutting occurs once a contact pressure threshold is exceeded. Moreover, we achieve interactive simulation rates by embedding a high-resolution geometry within a regular grid with arbitrary resolution. This allows to trade off accuracy for speed in the computation of deformations. The input data is a high-resolution volumetric model of the objects. The surface model of the object, used for rendering as well as collision detection and response, is a polygonal level set of the volumetric data. It is embedded in the volume model using barycentric coordinates.Cutting is performed by removing voxels at the fine level, and updating the surface and volume models accordingly. We introduce a new data structure, which we call a Dynamic Branched Grid, in order to preserve the fine-level topology at the coarse level. When an element of the coarse volumetric model is cut, it is replaced by a number of superimposed elements with the same size and at the same rest position as the original one. Each new element is assigned a part of material contained in the original one, and the mass and stiffness are recomputed accordingly. The well-known problem of creating small, ill-shaped finite elements while remeshing is thus completely avoided.