Development of a Data Management Tool for Investigating Multivariate Space and Free Will Experiences in Virtual Reality

Virtual reality (VR) has become mature enough to be successfully used in clinical applications such as exposure therapy, pain distraction, and neuropsychological assessment. However, we now need to go beyond the outcome data from this research and conduct the detailed scientific investigations required to better understand what factors influence why VR works (or doesn’t) in these types of clinical applications. This knowledge is required to guide the development of VR applications in the key areas of education, training, and rehabilitation and to further evolve existing VR approaches. One of the primary assets obtained with the use of VR is the ability to simulate the complexity of real world environments, within which human performance can be tested and trained. But this asset comes with a price in terms of the capture, quantification and analysis of large, multivariate and concurrent data sources that reflect the naturalistic behavioral interaction that is afforded in a virtual world. As well, while achieving realism has been a main goal in making convincing VR environments, just what constitutes realism and how much is needed is still an open question situated firmly in the research domain. Just as in real “reality,” such factors in virtual reality are complex and multivariate, and the understanding of this complexity presents exceptional challenges to the VR researcher. For certain research questions, good behavioral science often requires consistent delivery of stimuli within tightly controlled lab-based experimental conditions. However, for other important research questions we do not want to constrain naturalistic behavior and limit VR’s ability to replicate real world conditions, simply because it is easier to study human performance with traditional lab-based methodologies. By doing so we may compromise the very qualities that comprise VR’s unique capacity to mimic the experiences and challenges that exist in everyday life. What is really needed to address scientific questions that require natural exploration of a simulated environment are more usable and robust tools to instrument, organize, and visualize the complex data generated by measurements of participant behaviors within a virtual world. This paper briefly describes the rationale and methodology of an initial study in an ongoing research program that aims to investigate human performance within a virtual environment where unconstrained “free will” exploratory behavior is essential to research questions that involve the relationships between physiology, emotion, and memory. After a discussion of the research protocol and the types of data that were collected, we describe a novel tool that was borne from our need to more efficiently capture, manage, and explore the complex data that was generated in this research. An example of a research participant’s annotated display from this data management and visualization tool is then presented. It is our view that this tool provides the capacity to better visualize and understand the complex data relationships that may arise in VR research that investigates naturalistic free will behavior and its impact on other human performance variables.     

Impaired mismatch negativity (MMN) generation in schizophrenia as a function of stimulus deviance,probability, and interstimulus/interdeviant interval

Schizophrenia is a severe mental disorder associated with disturbances in perception and cognition. Event-related potentials (ERP) provide a mechanism for evaluating potential mechanisms underlying neurophysiological dysfunction in schizophrenia. Mismatch negativity (MMN) is a short-duration auditory cognitive ERP component that indexes operation of the auditory sensory (`echoic') memory system. Prior studies have demonstrated impaired MMN generation in schizophrenia along with deficits in auditory sensory memory performance. MMN is elicited in an auditory oddball paradigm in which a sequence of repetitive standard tones is interrupted infrequently by a physically deviant (`oddball') stimulus. The present study evaluates MMN generation as a function of deviant stimulus probability, interstimulus interval, interdeviant interval and the degree of pitch separation between the standard and deviant stimuli. The major findings of the present study are first, that MMN amplitude is decreased in schizophrenia across a broad range of stimulus conditions, and second, that the degree of deficit in schizophrenia is largest under conditions when MMN is normally largest. The pattern of deficit observed in schizophrenia differs from the pattern observed in other conditions associated with MMN dysfunction, including Alzheimer's disease, stroke, and alcohol intoxication.     

Integration of consonant and pitch processing as revealed by the absence of additivity in mismatch negativity

Consonants, unlike vowels, are thought to be speech specific and therefore no interactions would be expected between consonants and pitch, a basic element for musical tones. The present study used an electrophysiological approach to investigate whether, contrary to this view, there is integrative processing of consonants and pitch by measuring additivity of changes in the mismatch negativity (MMN) of evoked potentials. The MMN is elicited by discriminable variations occurring in a sequence of repetitive, homogeneous sounds. In the experiment, event-related potentials (ERPs) were recorded while participants heard frequently sung consonant-vowel syllables and rare stimuli deviating in either consonant identity only, pitch only, or in both dimensions. Every type of deviation elicited a reliable MMN. As expected, the two single-deviant MMNs had similar amplitudes, but that of the double-deviant MMN was also not significantly different from them. This absence of additivity in the double-deviant MMN suggests that consonant and pitch variations are processed, at least at a pre-attentive level, in an integrated rather than independent way. Domain-specificity of consonants may depend on higher-level processes in the hierarchy of speech perception.     

规律短音中极短间隔短音诱发的失匹配负电位

失匹配负电位是听觉事件相关电位的一个成分,它由一系列重复的,同性质的“标准刺激”的物理性质稍有偏离的“偏差刺激”所诱发,在规律性的标准刺激中,偶然的物理性质稍有偏离的刺激,如频率,强度,久度等的些微变化均可诱发ＭＭＮ。偶然地给于时间上“过早出现”的同样刺激,即频率,强度,久度完全相同,只是在规律性的标准刺激中过早地出现的刺激,作为偏差刺激,也可以诱发出ＮＮＭ。本研究在恒定刺激间隔ＩＳＩ＝６００ｍｓ     

Mechanisms of generation of mismatch negativity and their role in discrimination of short acoustical stimuli

The effect of stimulus duration on the mismatch negativity in the auditory event-related potentials was used to study the role of mismatch negativity (MMN) in discrimination of short acoustical stimuli. We compared discrimination of different short acoustical stimuli in active variant of "odd ball" paradigm. It was shown that it is possible to discriminate between standard and deviant acoustical stimuli which do not produce MMN in passive condition. It makes possible to estimate behavioural significance of MMN in active discrimination task. If the MMN had not been recorded in passive condition, that leads to an increase of reaction time in active paradigm approximately by 50 ms.     

Beat Processing Is Pre-Attentive for Metrically Simple Rhythms with Clear Accents: An ERP Study

The perception of a regular beat is fundamental to music processing. Here we examine whether the detection of a regular beat is pre-attentive for metrically simple, acoustically varying stimuli using the mismatch negativity (MMN), an ERP response elicited by violations of acoustic regularity irrespective of whether subjects are attending to the stimuli. Both musicians and non-musicians were presented with a varying rhythm with a clear accent structure in which occasionally a sound was omitted. We compared the MMN response to the omission of identical sounds in different metrical positions. Most importantly, we found that omissions in strong metrical positions, on the beat, elicited higher amplitude MMN responses than omissions in weak metrical positions, not on the beat. This suggests that the detection of a beat is pre-attentive when highly beat inducing stimuli are used. No effects of musical expertise were found. Our results suggest that for metrically simple rhythms with clear accents beat processing does not require attention or musical expertise. In addition, we discuss how the use of acoustically varying stimuli may influence ERP results when studying beat processing.     

Mismatch negativity in the brain evoked potentials induced by short duration acoustic stimuli in adolescents in the norm and those with attention deficit

The effect of stimulus duration on the mismatch negativity of the auditory event-related potentials was studied in healthy children and those with attention-deficit symptoms. The minimum stimulus duration with which a MMN was elicited by frequency deviant was 50 ms in healthy children. The MMN was absent with shorter stimuli (11 ms and 30 ms). These parametres are close to those in adults. As to children with attention-deficit symptoms the MMN was insignificant with all tested stimulus durations (11 ms, 30 ms, 50 ms).     

Auditory stimulus discrimination recorded in dogs, as indicated by mismatch negativity (MMN)

Dog cognition research tends to rely on behavioural response, which can be confounded by obedience or motivation, as the primary means of indexing dog cognitive abilities. A physiological method of measuring dog cognitive processing would be instructive and could complement behavioural response. Electroencephalogram (EEG) has been used in humans to study stimulus processing, which results in waveforms called event-related potentials (ERPs). One ERP component, mismatch negativity (MMN), is a negative deflection approximately 160-200 ms after stimulus onset, which may be related to change detection from echoic sensory memory. We adapted a minimally invasive technique to record MMN in dogs. Dogs were exposed to an auditory oddball paradigm in which deviant tones (10% probability) were pseudo-randomly interspersed throughout an 8 min sequence of standard tones (90% probability). A significant difference in MMN ERP amplitude was observed after the deviant tone in comparison to the standard tone, t₅ = −2.98, p = 0.03. This difference, attributed to discrimination of an unexpected stimulus in a series of expected stimuli, was not observed when both tones occurred 50% of the time, t₁ = −0.82, p > 0.05. Dogs showed no evidence of pain or distress at any point. We believe this is the first illustration of MMN in a group of dogs and anticipate that this technique may provide valuable insights in cognitive tasks such as object discrimination.     

Preattentive auditory information processing under exposure to the 902 MHz GSM mobile phone electromagnetic field: A mismatch negativity (MMN) study

Previous studies on the effects of the mobile phone electromagnetic field (EMF) on various event‐related potential (ERP) components have yielded inconsistent and even contradictory results, and often failed in replication. The mismatch negativity (MMN) is an auditory ERP component elicited by infrequent (deviant) stimuli differing in some physical features from the repetitive frequent (standard) stimuli in a sound sequence. The MMN provides a sensitive measure for cortical auditory stimulus feature discrimination, regardless of attention and other contaminating factors. In this study, MMN responses to duration, intensity, frequency, and gap changes were recorded in healthy young adults (n = 17), using a multifeature paradigm including several types of auditory change in the same stimulus sequence, while a GSM mobile phone was placed on either ear with the EMF (902 MHz pulsed at 217 Hz; SAR_1g = 1.14 W/kg, SAR_10g = 0.82 W/kg, peak value = 1.21 W/kg, measured with an SAM phantom) on or off. An MMN was elicited by all deviant types, while its amplitude and latency showed no significant differences due to EMF exposure for any deviant types. In the present study, we found no conclusive evidence that acute exposure to GSM mobile phone EMF affects cortical auditory change detection processing reflected by the MMN. Bioelectromagnetics 30:241–248, 2009. © 2009 Wiley‐Liss, Inc.     

Mismatch negativity of ERP in cross-modal attention

Event-related potentials were measured in 12 healthy youth subjects aged 19-22 using the paradigm "cross-modal and delayed response" which is able to improve unattended purity and to avoid the effect of task target on the deviant components of ERP. The experiment included two conditions: (i) Attend visual modality, ignore auditory modality; (ii) attend auditory modality, ignore visual modality. The stimuli under the two conditions were the same. The difference wave was obtained by subtracting ERPs of the standard stimuli from that of the deviant stim-uli. The present results showed that mismatch negativity (MMN), N2b and P3 components can be produced in the auditory and visual modalities under attention condition. However, only MMN was observed in the two modalities un-der inattention condition. Auditory and visual MMN have some features in common: their largest MMN wave peaks were distributed respectively over their primary sensory projection areas of the scalp under attention condition, but over front     

Human brain evoked potentials and discrimination of short acoustic stimuli with various frequency

We found that the mismatch negativity (MMM) was absent in evoked potentials in passive condition for frequency deviant acoustical stimuli when stimulus duration was only 11-30 ms. But it was shown that it is possible for participants to fairly well discriminate these stimuli and the component N2b in RP was generated without preceding MMN. Processing negativity also was absent in potentials by the minimal stimulus duration (11 ms). Auditory discrimination nevertheless was still possible, but reaction time and number of committed errors increased significantly.     

Cortical Mapping of Mismatch Negativity with Deviance Detection Property in Rat

Mismatch Negativity (MMN) is an N-methyl-d-aspartic acid (NMDA)-mediated, negative deflection in human auditory evoked potentials in response to a cognitively discriminable change. MMN-like responses have been extensively investigated in animal models, but the existence of MMN equivalent is still controversial. In this study, we aimed to investigate how closely the putative MMN (MMNp) in rats exhibited the comparable properties of human MMN. We used a surface microelectrode array with a grid of 10×7 recording sites within an area of 4.5×3.0 mm to densely map evoked potentials in the auditory cortex of anesthetized rats under the oddball paradigm. Firstly, like human MMN, deviant stimuli elicited negative deflections in auditory evoked potentials following the positive middle-latency response, termed P1. Secondly, MMNp exhibited deviance-detecting property, which could not be explained by simple stimulus specific adaptation (SSA). Thirdly, this MMNp occurred focally in the auditory cortex, including both the core and belt regions, while P1 activation focus was obtained in the core region, indicating that both P1 and MMNp are generated in the auditory cortex, yet the sources of these signals do not completely overlap. Fourthly, MMNp significantly decreased after the application of AP5 (D-(-)-2-amino-5-phosphonopentanoic acid), an antagonist at NMDA receptors. In stark contrast, AP5 affected neither P1 amplitude nor SSA of P1. These results provide compelling evidence that the MMNp we have examined in rats is functionally comparable to human MMN. The present work will stimulate translational research into MMN, which may help bridge the gap between electroencephalography (EEG)/magnetoencephalography (MEG) studies in humans and electrophysiological studies in animals.     

Event-Related Potential Measures of a Violation of an Expected Increase and Decrease in Intensity

Unexpected physical increases in the intensity of a frequently occurring “standard” auditory stimulus are experienced as obtrusive. This could either be because of a physical change, the increase in intensity of the “deviant” stimulus, or a psychological change, the violation of the expectancy for the occurrence of the lower intensity standard stimulus. Two experiments were run in which event-related potentials (ERPs) were recorded to determine whether “psychological” increments (violation of an expectancy for a lower intensity) would be processed differently than psychological decrements (violation of an expectancy for a higher intensity). Event-related potentials (ERPs) were recorded while subjects were presented with auditory tones that alternated between low and high intensity. The subjects ignored the auditory stimuli while watching a video. Deviants were created by repeating the same stimulus. In the first experiment, pairs of stimuli alternating in intensity, were presented in separate increment (H-L…H-L…H-H…H-L, in which H = 80 dB SPL and L = 60 dB SPL) and decrement conditions (L-H…L-H…L-L… L-H, in which H = 90 dB SPL and L = 80 dB SPL). The paradigm employed in the second experiment consisted of an alternating intensity pattern (H-L-H-L-H-H-H-L) or (H-L-H-L-L-L-H-L). Importantly, the stimulus prior to the deviant (the standard) and the actual deviants in both increment and decrement conditions in both experiments were physically identical (80 dB SPL tones). The repetition of the lower intensity tone therefore acted as a psychological rather than a physical decrement (a higher intensity tone was expected) while the repetition of the higher intensity tone acted as a psychological increment (a lower intensity tone was expected). The psychological increments in both experiments elicited a larger amplitude mismatch negativity (MMN) than the decrements. Thus, regardless of whether an acoustic change signals a physical increase in intensity or violates an expected decrease in intensity, a large MMN will be elicited.     

Acoustic versus phonetic representation of speech as reflected by the mismatch negativity event-related potential

The concept of categorical perception of speech and speech-like sounds has been central to models of speech perception for decades. Event-related potentials (ERPs) provide a neurophysiologic perspective of this important phenomenon. In the present experiment the mismatch negativity (MMN) event-related potential, which is sensitive to fine acoustic differences, was recorded in adults. Of interest was whether the MMN reflects the acoustic or categorical perception of speech.The MMN was elicited by stimulus pairs (along a continuum varying in place of articulation from /da/ to /ga/) which had been identified as the same phoneme /da/ (within category condition) and as different phonemes /da/ and /ga/ (across categories condition). The acoustic differences between these two pairs of stimuli were equivalent.The MMN was observed in all subjects both in the within and across category conditions. Furthermore, the MMN did not differ in latency, amplitude or area within and across categories. That is, the MMN indicated equal discrimination both across and within categories. These results suggest that the MMN appears to reflect the processing of acoustic aspects of the speech stimulus, but not phonetic processing into categories. The MMN appears to be an extremely sensitive electrophysiologic index of minimal acoustic differences in speech stimuli.     

Evoked potentials in the brain of adolescents in norm and those with attention deficit during recognition of short-term acoustic stimuli

The effect of stimulus duration on auditory event-related potentials and performance of oddball task was studied in normal children and those with attention-deficit symptoms. Mismatch negativity was absent on presentation of short-term (11 ms) stimuli and present with longer stimuli (50 ms). The adolescents with deficit of attention performed much worse (errors of omission) with the short stimuli. The RT was significantly larger in subjects with attention-deficit with all types of tested stimulus duration. They also manifested a smaller P3b amplitude in response to task-relevant deviant stimuli and larger N2b peaks in response to the standard stimuli. It was possible to differentiate between the MMN and the N2b components owing to the fact that the MMN was absent with shorter stimuli. The findings suggest that there is a deficit in processing of sensory information at the cortical level in subjects with the attention-deficit symptoms.     

Increasing Accessibility to the Blind of Virtual Environments,Using a Virtual Mobility Aid Based On the "EyeCane": Feasibility Study

Virtual worlds and environments are becoming an increasingly central part of our lives, yet they are still far from accessible to the blind. This is especially unfortunate as such environments hold great potential for them for uses such as social interaction, online education and especially for use with familiarizing the visually impaired user with a real environment virtually from the comfort and safety of his own home before visiting it in the real world. We have implemented a simple algorithm to improve this situation using single-point depth information, enabling the blind to use a virtual cane, modeled on the “EyeCane” electronic travel aid, within any virtual environment with minimal pre-processing. Use of the Virtual-EyeCane, enables this experience to potentially be later used in real world environments with identical stimuli to those from the virtual environment. We show the fast-learned practical use of this algorithm for navigation in simple environments.     

Differences in Mismatch Responses to Vowels and Musical Intervals: MEG Evidence

We investigated the electrophysiological response to matched two-formant vowels and two-note musical intervals, with the goal of examining whether music is processed differently from language in early cortical responses. Using magnetoencephalography (MEG), we compared the mismatch-response (MMN/MMF, an early, pre-attentive difference-detector occurring approximately 200 ms post-onset) to musical intervals and vowels composed of matched frequencies. Participants heard blocks of two stimuli in a passive oddball paradigm in one of three conditions: sine waves, piano tones and vowels. In each condition, participants heard two-formant vowels or musical intervals whose frequencies were 11, 12, or 24 semitones apart. In music, 12 semitones and 24 semitones are perceived as highly similar intervals (one and two octaves, respectively), while in speech 12 semitones and 11 semitones formant separations are perceived as highly similar (both variants of the vowel in ‘cut’). Our results indicate that the MMN response mirrors the perceptual one: larger MMNs were elicited for the 12–11 pairing in the music conditions than in the language condition; conversely, larger MMNs were elicited to the 12–24 pairing in the language condition that in the music conditions, suggesting that within 250 ms of hearing complex auditory stimuli, the neural computation of similarity, just as the behavioral one, differs significantly depending on whether the context is music or speech.     

Memory-based mismatch response to frequency changes in rats

Any occasional changes in the acoustic environment are of potential importance for survival. In humans, the preattentive detection of such changes generates the mismatch negativity (MMN) component of event-related brain potentials. MMN is elicited to rare changes ('deviants') in a series of otherwise regularly repeating stimuli ('standards'). Deviant stimuli are detected on the basis of a neural comparison process between the input from the current stimulus and the sensory memory trace of the standard stimuli. It is, however, unclear to what extent animals show a similar comparison process in response to auditory changes. To resolve this issue, epidural potentials were recorded above the primary auditory cortex of urethane-anesthetized rats. In an oddball condition, tone frequency was used to differentiate deviants interspersed randomly among a standard tone. Mismatch responses were observed at 60-100 ms after stimulus onset for frequency increases of 5% and 12.5% but not for similarly descending deviants. The response diminished when the silent inter-stimulus interval was increased from 375 ms to 600 ms for +5% deviants and from 600 ms to 1000 ms for +12.5% deviants. In comparison to the oddball condition the response also diminished in a control condition in which no repetitive standards were presented (equiprobable condition). These findings suggest that the rat mismatch response is similar to the human MMN and indicate that anesthetized rats provide a valuable model for studies of central auditory processing.     

Modulation of Visually Evoked Postural Responses by Contextual Visual,Haptic and Auditory Information: A ‘Virtual Reality Check’

Externally generated visual motion signals can cause the illusion of self-motion in space (vection) and corresponding visually evoked postural responses (VEPR). These VEPRs are not simple responses to optokinetic stimulation, but are modulated by the configuration of the environment. The aim of this paper is to explore what factors modulate VEPRs in a high quality virtual reality (VR) environment where real and virtual foreground objects served as static visual, auditory and haptic reference points. Data from four experiments on visually evoked postural responses show that: 1) visually evoked postural sway in the lateral direction is modulated by the presence of static anchor points that can be haptic, visual and auditory reference signals; 2) real objects and their matching virtual reality representations as visual anchors have different effects on postural sway; 3) visual motion in the anterior-posterior plane induces robust postural responses that are not modulated by the presence of reference signals or the reality of objects that can serve as visual anchors in the scene. We conclude that automatic postural responses for laterally moving visual stimuli are strongly influenced by the configuration and interpretation of the environment and draw on multisensory representations. Different postural responses were observed for real and virtual visual reference objects. On the basis that automatic visually evoked postural responses in high fidelity virtual environments should mimic those seen in real situations we propose to use the observed effect as a robust objective test for presence and fidelity in VR.     

Interstimulus interval and auditory event-related potentials in children: evidence for multiple generators

In the present study, the component structure of auditory event-related potentials (ERP) was studied in children of 7–9 years old by presenting stimuli with different interstimulus intervals (ISI). A short-term auditory sensory memory, as reflected by ISI effects on ERPs, was also studied. Auditory ERPs were recorded to brief unattended 1000 Hz frequent, `standard' and 1100 Hz rare, `deviant' (probability 0.1) tone stimuli with ISIs of 350, 700 and 1400 ms (in separate blocks). With the 350 ms-ISI, the ERP waveform to the standard stimulus consisted of P100-N250 peaks. With the two longer ISIs, in addition, the frontocentral N160 and N460 peaks were observed. Results suggested that N160, found with the longer ISIs, is a correlate of the adult auditory N1. In difference waves, obtained by subtracting ERP to standard stimuli from ERP to deviant stimuli, two negativities were revealed. The first was the mismatch negativity (MMN), which is elicited by any discriminable change in repetitive auditory input. The MMN data suggested that neural traces of auditory sensory memory lasted for at least 1400 ms, probably considerably longer, as no MMN attenuation was found across the ISIs used. The second, later negativity was similar to MMN in all aspects, except for the scalp distribution, which was posterior to that of the MMN.