Comparing the Similarity of Responses Received from Studies in Amazon’s Mechanical Turk to Studies Conducted Online and with Direct Recruitment

Computer and internet based questionnaires have become a standard tool in Human-Computer Interaction research and other related fields, such as psychology and sociology. Amazon’s Mechanical Turk (AMT) service is a new method of recruiting participants and conducting certain types of experiments. This study compares whether participants recruited through AMT give different responses than participants recruited through an online forum or recruited directly on a university campus. Moreover, we compare whether a study conducted within AMT results in different responses compared to a study for which participants are recruited through AMT but which is conducted using an external online questionnaire service. The results of this study show that there is a statistical difference between results obtained from participants recruited through AMT compared to the results from the participant recruited on campus or through online forums. We do, however, argue that this difference is so small that it has no practical consequence. There was no significant difference between running the study within AMT compared to running it with an online questionnaire service. There was no significant difference between results obtained directly from within AMT compared to results obtained in the campus and online forum condition. This may suggest that AMT is a viable and economical option for recruiting participants and for conducting studies as setting up and running a study with AMT generally requires less effort and time compared to other frequently used methods. We discuss our findings as well as limitations of using AMT for empirical studies.     

Collecting Psycholinguistic Response Time Data Using Amazon Mechanical Turk

Researchers in linguistics and related fields have recently begun exploiting online crowd-sourcing tools, like Amazon Mechanical Turk (AMT), to gather behavioral data. While this method has been successfully validated for various offline measures—grammaticality judgment or other forced-choice tasks—its use for mainstream psycholinguistic research remains limited. This is because psycholinguistic effects are often dependent on relatively small differences in response times, and there remains some doubt as to whether precise timing measurements can be gathered over the web. Here we show that three classic psycholinguistic effects can in fact be replicated using AMT in combination with open-source software for gathering response times client-side. Specifically, we find reliable effects of subject definiteness, filler-gap dependency processing, and agreement attraction in self-paced reading tasks using approximately the same numbers of participants and/or trials as similar laboratory studies. Our results suggest that psycholinguists can and should be taking advantage of AMT and similar online crowd-sourcing marketplaces as a fast, low-resource alternative to traditional laboratory research.     

Does Intuition Cause Cooperation?

Recently, researchers claimed that people are intuitively inclined to cooperate with reflection causing them to behave selfishly. Empirical support for this claim came from experiments using a 4-player public goods game with a marginal return of 0.5 showing that people contributed more money to a common project when they had to decide quickly (i.e., a decision based on intuition) than when they were instructed to reflect and decide slowly. This intuitive-cooperation effect is of high scientific and practical importance because it argues against a central assumption of traditional economic and evolutionary models. The first experiment of present study was set up to examine the generality of the intuitive-cooperation effect and to further validate the experimental task producing the effect. In Experiment 1, we investigated Amazon Mechanical Turk (AMT) workers'' contributions to a 4-player public goods game with a marginal return of 0.5 while we manipulated the knowledge about the other players'' contribution to the public goods game (contribution known vs. contribution unknown), the identity of the other players (humans vs. computers randomly generating contributions) and the time constraint (time pressure/intuition vs. forced delay/reflection). However, the results of Experiment 1 failed to reveal an intuitive-cooperation effect. Furthermore, four subsequent direct replications attempts with AMT workers (Experiments 2a, 2b, 2c and Experiment 3, which was conducted with naïve/inexperienced participants) also failed to demonstrate intuitive-cooperation effects. Taken together, the results of the present study could not corroborate the idea that people are intuitively cooperative, hence suggesting that the theoretical relationship between intuition and cooperation should be further scrutinized.     

The application of generalizability theory to surface electromyographic measurements during psychophysiological stress testing: how many measurements are needed?

Generalizability theory is an extension of classical reliability theory that allows multiple sources of measurement error in an experimental design to be investigated simultaneously. In the present study, generalizability theory was used to evaluate measurement error in psychophysiological test procedures used to differentiate tension headache patients from normal controls based upon measures of electromyographic (EMG) responding. Thirty-three subjects who met diagnostic criteria for tension-type headache and 40 normal control subjects who rarely or never experienced headache participated in two laboratory sessions. EMG activity of head and neck muscles was recorded while subjects performed baseline, relaxation, choice reaction time, psychomotor tracking, and cold pressor tasks. Variance components were computed for an experimental design having subjects nested within experimenters and crossed with sessions and replications. Generalizability coefficients were computed for combinations of various numbers of sessions and replications. The generalizability of EMG measures was highly variable, depending on the experimental conditions in force. The largest sources of measurement error were attributed to the unique responsiveness of individual subjects under a particular set of treatment conditions. For some stress tests currently in use, data from several testing sessions may need to be averaged in order to achieve acceptable levels of generalizability. Generalizability greater than 0.80 can be expected only rarely when data are collected during a single session. In the research setting, low generalizability may account for the failure of EMG-based stress tests to differentiate tension headache patients from controls during stressful task performance. In the clinical setting, the generalizability of information derived from "stress profiling" or muscle "scanning" techniques, which depend on results obtained during a single testing session, is doubtful.     

Direct and Conceptual Replications of Burgmer & Englich (2012): Power May Have Little to No Effect on Motor Performance

Burgmer and Englich (2012) have reported that manipulating feelings of power can substantially improve performance on two motor tasks: golf and darts. We conducted two high-powered direct replications of the effects of power on golf, two online conceptual replications using mirror-tracing as a performance measure, and an additional conceptual replication using a cognitive performance measure (word-search). Overall, we found little to no effect of power on motor skill (d = 0.09, 95% CI[-0.07, 0.22], n = 603). We varied task difficulty, re-analyzed data without participants showing weak responses on manipulation checks, and tried adjusting performance scores for age, gender, and initial task skill. None of these secondary analyses revealed a strong effect of power on performance. A meta-analysis integrating our data with Burgmer & Englich leaves open the possibility that manipulating power could provide a modest boost in motor skill (d = 0.19, 95% CI [0.001, 0.38], n = 685). Unfortunately, the pattern of performance changes we observed was unrelated to group differences in perceived and rated power, suggesting that what motor effects do occur with this protocol may not be directly related to the construct of power. [Burgmer, P., &Englich, B. (2012). Bullseye!: How Power Improves Motor Performance. Social Psychological and Personality Science, 4(2), 224–232.]     

Real-Time Subject-Independent Pattern Classification of Overt and Covert Movements from fNIRS Signals

Recently, studies have reported the use of Near Infrared Spectroscopy (NIRS) for developing Brain–Computer Interface (BCI) by applying online pattern classification of brain states from subject-specific fNIRS signals. The purpose of the present study was to develop and test a real-time method for subject-specific and subject-independent classification of multi-channel fNIRS signals using support-vector machines (SVM), so as to determine its feasibility as an online neurofeedback system. Towards this goal, we used left versus right hand movement execution and movement imagery as study paradigms in a series of experiments. In the first two experiments, activations in the motor cortex during movement execution and movement imagery were used to develop subject-dependent models that obtained high classification accuracies thereby indicating the robustness of our classification method. In the third experiment, a generalized classifier-model was developed from the first two experimental data, which was then applied for subject-independent neurofeedback training. Application of this method in new participants showed mean classification accuracy of 63% for movement imagery tasks and 80% for movement execution tasks. These results, and their corresponding offline analysis reported in this study demonstrate that SVM based real-time subject-independent classification of fNIRS signals is feasible. This method has important applications in the field of hemodynamic BCIs, and neuro-rehabilitation where patients can be trained to learn spatio-temporal patterns of healthy brain activity.     

Ranging in Human Sonar: Effects of Additional Early Reflections and Exploratory Head Movements

Many blind people rely on echoes from self-produced sounds to assess their environment. It has been shown that human subjects can use echolocation for directional localization and orientation in a room, but echo-acoustic distance perception - e.g. to determine one''s position in a room - has received little scientific attention, and systematic studies on the influence of additional early reflections and exploratory head movements are lacking. This study investigates echo-acoustic distance discrimination in virtual echo-acoustic space, using the impulse responses of a real corridor. Six blindfolded sighted subjects and a blind echolocation expert had to discriminate between two positions in the virtual corridor, which differed by their distance to the front wall, but not to the lateral walls. To solve this task, participants evaluated echoes that were generated in real time from self-produced vocalizations. Across experimental conditions, we systematically varied the restrictions for head rotations, the subjects'' orientation in virtual space and the reference position. Three key results were observed. First, all participants successfully solved the task with discrimination thresholds below 1 m for all reference distances (0.75–4 m). Performance was best for the smallest reference distance of 0.75 m, with thresholds around 20 cm. Second, distance discrimination performance was relatively robust against additional early reflections, compared to other echolocation tasks like directional localization. Third, free head rotations during echolocation can improve distance discrimination performance in complex environmental settings. However, head movements do not necessarily provide a benefit over static echolocation from an optimal single orientation. These results show that accurate distance discrimination through echolocation is possible over a wide range of reference distances and environmental conditions. This is an important functional benefit of human echolocation, which may also play a major role in the calibration of auditory space representations.     

心理疲劳对情绪图片注意偏向的研究

目的:探讨在实验室中诱发心理疲劳后是否会对带有负性情绪的图片产生注意偏向。方法:选取34名某军医大学本科生为被试者,记录被试者完成认知负荷任务后的主观情绪体验成绩及疲劳量表得分,并采集被试者对情绪性图片进行的点探测任务的反应时及错误率。结果:在主观报告上,被试者的负性情绪和躯体疲劳感得分显著高于完成认知任务前,在点探测任务的行为数据上,被试者对于带有负性情绪的图片的反应时长于带有正性情绪的图片,反应错误率也高于正性图片。结论:认知负荷任务能够有效诱发被试者的躯体疲劳感和负性情绪。具有疲劳感和负性情绪的被试者并未对带有负性情绪的图片产生注意偏向。     

Correlating Behavioral Responses to fMRI Signals from Human Prefrontal Cortex: Examining Cognitive Processes Using Task Analysis

The aim of this methods paper is to describe how to implement a neuroimaging technique to examine complementary brain processes engaged by two similar tasks. Participants'' behavior during task performance in an fMRI scanner can then be correlated to the brain activity using the blood-oxygen-level-dependent signal. We measure behavior to be able to sort correct trials, where the subject performed the task correctly and then be able to examine the brain signals related to correct performance. Conversely, if subjects do not perform the task correctly, and these trials are included in the same analysis with the correct trials we would introduce trials that were not only for correct performance. Thus, in many cases these errors can be used themselves to then correlate brain activity to them. We describe two complementary tasks that are used in our lab to examine the brain during suppression of an automatic responses: the stroop¹ and anti-saccade tasks. The emotional stroop paradigm instructs participants to either report the superimposed emotional ''word'' across the affective faces or the facial ''expressions'' of the face stimuli^1,2. When the word and the facial expression refer to different emotions, a conflict between what must be said and what is automatically read occurs. The participant has to resolve the conflict between two simultaneously competing processes of word reading and facial expression. Our urge to read out a word leads to strong ''stimulus-response (SR)'' associations; hence inhibiting these strong SR''s is difficult and participants are prone to making errors. Overcoming this conflict and directing attention away from the face or the word requires the subject to inhibit bottom up processes which typically directs attention to the more salient stimulus. Similarly, in the anti-saccade task^3,4,5,6, where an instruction cue is used to direct only attention to a peripheral stimulus location but then the eye movement is made to the mirror opposite position. Yet again we measure behavior by recording the eye movements of participants which allows for the sorting of the behavioral responses into correct and error trials⁷ which then can be correlated to brain activity. Neuroimaging now allows researchers to measure different behaviors of correct and error trials that are indicative of different cognitive processes and pinpoint the different neural networks involved.     

Methods to Test Visual Attention Online

Online data collection methods have particular appeal to behavioral scientists because they offer the promise of much larger and much more representative data samples than can typically be collected on college campuses. However, before such methods can be widely adopted, a number of technological challenges must be overcome – in particular in experiments where tight control over stimulus properties is necessary. Here we present methods for collecting performance data on two tests of visual attention. Both tests require control over the visual angle of the stimuli (which in turn requires knowledge of the viewing distance, monitor size, screen resolution, etc.) and the timing of the stimuli (as the tests involve either briefly flashed stimuli or stimuli that move at specific rates). Data collected on these tests from over 1,700 online participants were consistent with data collected in laboratory-based versions of the exact same tests. These results suggest that with proper care, timing/stimulus size dependent tasks can be deployed in web-based settings.     

STEAM教育理念下“线上+线下”混合教学模式初探：以微生物学实验为例

随着信息化手段的不断丰富,新型教育理念结合线上学习平台的新信息化教学模式成为高校课堂的改革新趋势。本次教学改革利用科学(Science)、技术(Technology)、工程(Engineering)、艺术(Art)和数学(Mathematics)多学科融合的超学科教育理念(简称STEAM教育)对教师教学过程进行了整体设计,同时借助“线上+线下”教学平台对学生学习过程进行了全面优化。将原本分散的验证型、操作型实验重新整合串联成以多角度“项目式”任务为主线、以Blackboard线上平台为辅线的自主研究型实验项目。新型教学模式以学生为主体,给学生提供更多自我展示和讨论互动的平台。从学生的课堂表现、知识测验、课后反馈、实验操作及实验报告4个方面对新型模式下的教学效果进行了分析和评价。结果表明,此模式不仅提高了学生在微生物学实验中的学习质量,增强了其学习主观能动性,而且有利于培养和提升学生的问题探究及实践创新能力。这一新型教学模式对其他生物学科实验课程的教学具有一定的借鉴意义。     

The application of generalizability theory to surface electromyographic measurements during psychophysiological stress testing: How many measurements are needed?

Generalizability theory is an extension of classical reliability theory that allows multiple sources of measurement error in an experimental design to be investigated simultaneously. In the present study, generalizability theory was used to evaluate measurement error in psychophysiological test procedures used to differentiate tension headache patients from normal controls based upon measures of electromyographic (EMG) responding. Thirty-three subjects who met diagnostic criteria for tension-type headache and 40 normal control subjects who rarely or never experienced headache participated in two laboratory sessions. EMG activity of head and neck muscles was recorded while subjects performed baseline, relaxation, choice reaction time, psychomotor tracking, and cold pressor tasks. Variance components were computed for an experimental design having subjects nested within experimenters and crossed with sessions and replications. Generalizability coefficients were computed for combinations of various numbers of sessions and replications. The generalizability of EMG measures was highly variable, depending on the experimental conditions in force. The largest sources of measurement error were attributed to the unique responsiveness of individual subjects under a particular set of treatment conditions. For some stress tests currently in use, data from several testing sessions may need to be averaged in order to achieve acceptable levels of generalizability. Generalizability greater than 0.80 can be expected only rarely when data are collected during a single session. In the research setting, low generalizability may account for the failure of EMG-based stress tests to differentiate tension headache patients from controls during stressful task performance. In the clinical setting, the generalizability of information derived from stress profiling or muscle scanning techniques, which depend on results obtained during a single testing session, is doubtful.This research was supported by NIH under grant No. R01-NS-25114.     

SUBJECTIVE PERCEPTIONS OF THE EFFECTS OF SUSTAINED PERFORMANCE UNDER SLEEP-DEPRIVATION CONDITIONS

In today's society, numerous situations arise in which sleep deprivation is a common occurrence. Subjective perceptions are a vital component to understanding the effects of sustained performance during sleep deprivation, as they may be the first indication of the effects of sustained performance or sleep deprivation on the individual. Using the theoretical framework of the Controlled Attention Model, this study examined the effects of 16?h of sustained performance under 28?h of acute sleep deprivation on perceived effort, motivation, and stress of 24 participants while completing a complex cognitive and a simple vigilance task. Perceived effort increased for both tasks, with higher effort reported on the cognitive than the vigilance task at the beginning of the experimental period, but with higher effort reported on the vigilance than the cognitive task at the end. Subjective motivation decreased for both tasks, with significantly higher levels of motivation on the cognitive than the vigilance task. Perceived stress did not change for either task. Results suggest that functioning under sustained performance and sleep-deprivation conditions affects subjective perceptions differently for cognitive versus vigilance tasks. The controlled attention model offers one means of understanding how different tasks could affect a person's subjective perceptions and ability to perform, in that different levels of controlled attention are required for the two tasks. (Author correspondence: hodle@gettysburg.edu)     

师生共建实验室模式的探索

目的:探索新的人体解剖学实验教学模式和促进解剖学实验室标本建设,在提高解剖学实验教学质量同时,使实验室具有数量可观、质量上乘,能立体、全面反映人体形态构造的标本,实现解剖学教学质量和实验室建设的共同发展。方法:组建具备良好的职业道德、崇高的敬业精神和扎实的解剖学理论知识,又能激励和启迪学生的教学团队,遴选具有强烈的事业心,高度的责任感和吃苦耐劳精神学生,师生互动,学生自主动手解剖尸体,制作标本,进行师生共建实验室模式的探索。结果:学生观察能力、动手能力、学生综合素质和知识结构得到提高和强化,教师业务水平得到提高,实验室教学标本得到大量有效补充。结论:师生共建实验室活动取得了良好效果。在推广过程中,加大对学生的人文关怀,使教师不仅是"课程的组织者、情感的支持者、学习的参与者、信息的咨询者"更是学生"生活的关怀者",将使本探索将更具推广意义。     

Chiral analysis of ammuxetine enantiomers in dog plasma using online SPE/liquid chromatography with tandem mass spectrometric detection after precolumn chiral derivatization

Ammuxetine (AMT), a novel chiral antidepressant candidate compound, exhibits better antidepression effects than duloxetine in different animal models. In this article, a chiral derivatization method, combined with online solid phase extraction (online SPE) and liquid chromatography–tandem mass spectrometry (LC–MS/MS), was developed for the chiral separation of AMT enantiomers after administration of racemic AMT to dogs. The derivatization reaction employed 2,3,4,6‐tetra‐O‐acetyl‐b‐glucopyr‐anosyl isothiocyanate (GITC) as a precolumn chiral derivatization reagent. A SPE column Retain PEP Javelin (10 × 2.1 mm) was used to remove proteins and other impurities in plasma samples. The enantiomeric derivatives were separated on a ZORBAX SB‐C₁₈ column (50 × 2.1 mm × 3.5 μm) with an isocratic elution procedure. The selected multiple reaction monitoring mode of the positive ion was performed and the parent to the product transitions m/z 681.0/543.1 and m/z 687.4/543.1 were used to measure the derivatives of AMT and duloxetine (internal standard) with electrospray ionization. The method was validated in terms of specificity, linearity, sensitivity, precision, accuracy, matrix effect, and stability. The method was applied to a pharmacokinetics study of AMT racemate in dogs. The results suggested that the pharmacokinetic of AMT enantiomers might be stereoselective in dogs.     

Venom,speed, and caution: effects on performance in a visual search task

Previous reports of faster responses to threatening compared to benign stimuli in visual search tasks have argued that threatening targets are faster to engage and slower to disengage attention than benign targets. This study reinterprets previous findings and resolves inconsistencies in the literature by replacing the theory of differential disengagement of attention with one of differential caution. It also examines whether visual attentional mechanisms are sensitive to more than just the threatening versus benign categorical status of the targets and introduces a novel measure (a caution score) that appears to be sensitive to the level of threat implied by the target image, but immune to other stimulus features (target-distracter similarity and threat status of distracters) known to affect reaction time. As well as locating threatening targets faster than benign targets, participants were also faster, more accurate, and more cautious to detect lethal spiders compared to nonlethal spiders and even more cautious again if the spiders were presented on a person's hand. These results suggest that mechanisms of attention and threat evaluation interact during visual search tasks, producing behaviour that is sensitive to the target's implied threat level and the context in which that target is presented.     

Seven common mistakes in population genetics and how to avoid them

As the data resulting from modern genotyping tools are astoundingly complex, genotyping studies require great care in the sampling design, genotyping, data analysis and interpretation. Such care is necessary because, with data sets containing thousands of loci, small biases can easily become strongly significant patterns. Such biases may already be present in routine tasks that are present in almost every genotyping study. Here, I discuss seven common mistakes that can be frequently encountered in the genotyping literature: (i) giving more attention to genotyping than to sampling, (ii) failing to perform or report experimental randomization in the laboratory, (iii) equating geopolitical borders with biological borders, (iv) testing significance of clustering output, (v) misinterpreting Mantel's r statistic, (vi) only interpreting a single value of k and (vii) forgetting that only a small portion of the genome will be associated with climate. For every of those issues, I give some suggestions how to avoid the mistake. Overall, I argue that genotyping studies would benefit from establishing a more rigorous experimental design, involving proper sampling design, randomization and better distinction of a priori hypotheses and exploratory analyses.     

Molecular and cellular characterisation of LjAMT2;1, an ammonium transporter from the model legume Lotus japonicus

Two related families of ammonium transporters have been identified and partially characterised in plants in the past; the AMT1 and AMT2 families. Most attention has focused on the larger of the two families, the AMT1 family, which contains members that are likely to fulfil different, possibly overlapping physiological roles in plants, including uptake of ammonium from the soil. The possible physiological functions of AMT2 proteins are less clear. Lack of data on cellular and tissue location of gene expression, and the intracellular location of proteins limit our understanding of the physiological role of all AMT proteins. We have cloned the first AMT2 family member from a legume, LjAMT2;1 of Lotus japonicus, and demonstrated that it functions as an ammonium transporter by complementing a yeast mutant defective in ammonium uptake. However, like AtAMT2 from Arabidopsis, and unlike AMT1 transporters from several plant species, LjAMT2;1 was unable to transport methylammonium. The LjAMT2;1 gene was found to be expressed constitutively throughout Lotus plants. In situ RNA hybridisation revealed LjAMT2;1 expression in all major tissues of nodules. Transient expression of LjAMT2;1-GFP fusion protein in plant cells indicated that the transporter is located on the plasma membrane. In view of the fact that nodules derive ammonium internally, rather than from the soil, the results implicate LjAMT2;1 in the recovery of ammonium lost from nodule cells by efflux. A similar role may be fulfilled in other organs, especially leaves, which liberate ammonium during normal metabolism.     

Using Crowdsourcing to Evaluate Published Scientific Literature: Methods and Example

Systematically evaluating scientific literature is a time consuming endeavor that requires hours of coding and rating. Here, we describe a method to distribute these tasks across a large group through online crowdsourcing. Using Amazon''s Mechanical Turk, crowdsourced workers (microworkers) completed four groups of tasks to evaluate the question, “Do nutrition-obesity studies with conclusions concordant with popular opinion receive more attention in the scientific community than do those that are discordant?” 1) Microworkers who passed a qualification test (19% passed) evaluated abstracts to determine if they were about human studies investigating nutrition and obesity. Agreement between the first two raters'' conclusions was moderate (κ = 0.586), with consensus being reached in 96% of abstracts. 2) Microworkers iteratively synthesized free-text answers describing the studied foods into one coherent term. Approximately 84% of foods were agreed upon, with only 4 and 8% of ratings failing manual review in different steps. 3) Microworkers were asked to rate the perceived obesogenicity of the synthesized food terms. Over 99% of responses were complete and usable, and opinions of the microworkers qualitatively matched the authors'' expert expectations (e.g., sugar-sweetened beverages were thought to cause obesity and fruits and vegetables were thought to prevent obesity). 4) Microworkers extracted citation counts for each paper through Google Scholar. Microworkers reached consensus or unanimous agreement for all successful searches. To answer the example question, data were aggregated and analyzed, and showed no significant association between popular opinion and attention the paper received as measured by Scimago Journal Rank and citation counts. Direct microworker costs totaled $221.75, (estimated cost at minimum wage: $312.61). We discuss important points to consider to ensure good quality control and appropriate pay for microworkers. With good reliability and low cost, crowdsourcing has potential to evaluate published literature in a cost-effective, quick, and reliable manner using existing, easily accessible resources.     

Attention,self-regulation and consciousness.

Consciousness has many aspects. These include awareness of the world, feelings of control over one''s behaviour and mental state (volition), and the notion of continuing self. Focal (executive) attention is used to control details of our awareness and is thus closely related to volition. Experiments suggest an integrated network of neural areas involved in executive attention. This network is associated with our voluntary ability to select among competing items, to correct error and to regulate our emotions. Recent neuroimaging studies suggest that these various functions involve separate areas of the anterior cingulate. We have adopted a strategy of using marker tasks, shown to activate the brain area by imaging studies, as a means of tracing the development of attentional networks. Executive attention appears to develop first to regulate distress during the first year of life. During later childhood the ability to regulate conflict among competing stimuli builds upon the earlier cingulate anatomy to provide a means of cognitive control. During childhood the activation of cingulate structures relates both to the child''s success on laboratory tasks involving conflict and to parental reports of self-regulation and emotional control. These studies indicate a start in understanding the anatomy, circuitry and development of executive attention networks that serve to regulate both cognition and emotion.