The evolution of peer review as a basis for scientific publication: directional selection towards a robust discipline?

Peer review is pivotal to science and academia, as it represents a widely accepted strategy for ensuring quality control in scientific research. Yet, the peer‐review system is poorly adapted to recent changes in the discipline and current societal needs. We provide historical context for the cultural lag that governs peer review that has eventually led to the system's current structural weaknesses (voluntary review, unstandardized review criteria, decentralized process). We argue that some current attempts to upgrade or otherwise modify the peer‐review system are merely sticking‐plaster solutions to these fundamental flaws, and therefore are unlikely to resolve them in the long term. We claim that for peer review to be relevant, effective, and contemporary with today's publishing demands across scientific disciplines, its main components need to be redesigned. We propose directional changes that are likely to improve the quality, rigour, and timeliness of peer review, and thereby ensure that this critical process serves the community it was created for.     

Teleconference versus Face-to-Face Scientific Peer Review of Grant Application: Effects on Review Outcomes

Teleconferencing as a setting for scientific peer review is an attractive option for funding agencies, given the substantial environmental and cost savings. Despite this, there is a paucity of published data validating teleconference-based peer review compared to the face-to-face process.Our aim was to conduct a retrospective analysis of scientific peer review data to investigate whether review setting has an effect on review process and outcome measures.We analyzed reviewer scoring data from a research program that had recently modified the review setting from face-to-face to a teleconference format with minimal changes to the overall review procedures. This analysis included approximately 1600 applications over a 4-year period: two years of face-to-face panel meetings compared to two years of teleconference meetings. The average overall scientific merit scores, score distribution, standard deviations and reviewer inter-rater reliability statistics were measured, as well as reviewer demographics and length of time discussing applications.The data indicate that few differences are evident between face-to-face and teleconference settings with regard to average overall scientific merit score, scoring distribution, standard deviation, reviewer demographics or inter-rater reliability. However, some difference was found in the discussion time.These findings suggest that most review outcome measures are unaffected by review setting, which would support the trend of using teleconference reviews rather than face-to-face meetings. However, further studies are needed to assess any correlations among discussion time, application funding and the productivity of funded research projects.     

Dr. Arthur Conan Doyle and the case of congenital syphilis

In 1894, Dr. Arthur Conan Doyle wrote "The Third Generation," a short story involving the transmission of congenital syphilis from generation to generation. Analysts of his writings have interpreted the pathogenetic mechanism involved in modern terms: infection of mother by father and then transplacental infection of the fetus. However, a review of the contemporary literature and the history of the concepts of congenital and "hereditary" syphilis demonstrates that the late 19th-century understanding of the process involved a Lamarckian transmission of paternal infection, via the sperm at the moment of conception. It was undoubtedly this concept that Doyle learned in medical school in the late 1870s and that provided the background to his story.     

Fraud and misconduct in science: the stem cell seduction

Scientific misconduct and fraud occur in science. The (anonymous) peer review process serves as goalkeeper of scientific quality rather than scientific integrity. In this brief paper we describe some limitations of the peer-review process. We describe the catastrophic facts of the ‘Woo-Suk Hwang fraud case’ and raise some ethical concerns about the issue. Finally, we pay attention to plagiarism, autoplagiarism and double publications. (Neth Heart J 2009;17:25-9.)     

It Is Time to Re‐Evaluate the Peer Review Process for Preclinical Research

Problems in peer review, the backbone of maintaining high standards in scientific publishing, have led to wide spread discontent within the scientific community. Training in the peer review process and a simpler format to assist in decision making are possible courses to improve and expedite the process of peer review and scientific publishing.     

Proposed Standards for Variable Harmonization Documentation and Referencing: A Case Study Using QuickCharmStats 1.1

Comparative statistical analyses often require data harmonization, yet the social sciences do not have clear operationalization frameworks that guide and homogenize variable coding decisions across disciplines. When faced with a need to harmonize variables researchers often look for guidance from various international studies that employ output harmonization, such as the Comparative Survey of Election Studies, which offer recoding structures for the same variable (e.g. marital status). More problematically there are no agreed documentation standards or journal requirements for reporting variable harmonization to facilitate a transparent replication process. We propose a conceptual and data-driven digital solution that creates harmonization documentation standards for publication and scholarly citation: QuickCharmStats 1.1. It is free and open-source software that allows for the organizing, documenting and publishing of data harmonization projects. QuickCharmStats starts at the conceptual level and its workflow ends with a variable recording syntax. It is therefore flexible enough to reflect a variety of theoretical justifications for variable harmonization. Using the socio-demographic variable ‘marital status’, we demonstrate how the CharmStats workflow collates metadata while being guided by the scientific standards of transparency and replication. It encourages researchers to publish their harmonization work by providing researchers who complete the peer review process a permanent identifier. Those who contribute original data harmonization work to their discipline can now be credited through citations. Finally, we propose peer-review standards for harmonization documentation, describe a route to online publishing, and provide a referencing format to cite harmonization projects. Although CharmStats products are designed for social scientists our adherence to the scientific method ensures our products can be used by researchers across the sciences.     

On the lack of consensus over the meaning of openness: an empirical study

This study set out to explore the views and motivations of those involved in a number of recent and current advocacy efforts (such as open science, computational provenance, and reproducible research) aimed at making science and scientific artifacts accessible to a wider audience. Using a exploratory approach, the study tested whether a consensus exists among advocates of these initiatives about the key concepts, exploring the meanings that scientists attach to the various mechanisms for sharing their work, and the social context in which this takes place. The study used a purposive sampling strategy to target scientists who have been active participants in these advocacy efforts, and an open-ended questionnaire to collect detailed opinions on the topics of reproducibility, credibility, scooping, data sharing, results sharing, and the effectiveness of the peer review process. We found evidence of a lack of agreement on the meaning of key terminology, and a lack of consensus on some of the broader goals of these advocacy efforts. These results can be explained through a closer examination of the divergent goals and approaches adopted by different advocacy efforts. We suggest that the scientific community could benefit from a broader discussion of what it means to make scientific research more accessible and how this might best be achieved.     

Aspects of audit. 1. The background.

Methods of reviewing health care already exist in Britain, but the debate continues about how practical and acceptable such a review is. The many different terms used to describe review only confuse the issue. "Audit" is a useful term for describing the review of medical work by medical people. This can be divided into "internal audit," or peer review, and "external audit"--that is, review by organisations outside hospital and general practice. The concepts of internal and external audit have a great impact upon the attitudes held by the medical profession about audit. The shortcomings of audit by the professional standards review organisations in the United States are not inevitable in Britian.     

Review of The Mind at Night: The New Science of How and Why We Dream.

In this article I review "The Mind at Night: The New Science of How and Why We Dream," written by Andrea Rock. To begin with this book is an exciting journey through modern dream research. Scientific facts, which are skillfully explained, are complemented by personal accounts of well-known researchers in the field obtained through interviews. The diversity of the themes addressed in the book (e.g., sleep and memory, animal research, imaging studies, dream content analysis, consciousness research, creativity, and lucid dreaming) clearly shows the extensive "detective work" the author has accomplished. The major problem I had--as a researcher in this field--was the structure, or the lack of structure, within the book. Because of the way the book is organized, I decided to structure this review along the following themes: REM sleep, REM sleep and dreaming, biology of dreaming, dream content findings, and the integration of dream research into cognitive neuroscience in general. Despite the lack of structure of the book, Andrea Rock has written a wonderful book about modern dream research that is stimulating for researchers as well as for interested lay persons. I recommend it to everyone who is interested in dream research, the old question of the mind-body relationship, or understanding consciousness in general. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The exponential growth of invasive species denialism

Since the 1990s, there have appeared numerous articles in scholarly journals and the popular press that deny the risks posed by non-native species and claim that the field of invasion biology is biased, uninformative and pseudoscientific. Unlike normal scientific debates, which are evidence based, this discourse typically uses rhetorical arguments to disregard, misrepresent or reject evidence in attempt to cast doubt on the scientific consensus that species introductions pose significant risks to biodiversity and ecosystems; thus, it is similar to the denialism that has affected climate science and medical science. Invasive species denialism, like science denialism in general, is typically expressed in forums where it avoids expert peer review. Denialist articles have increased exponentially over the past three decades, most notably in the mainstream popular press. This burgeoning phenomenon could impede development and implementation of policies designed to safeguard against invasive species spread and impact.     

Biohumanities: rethinking the relationship between biosciences, philosophy and history of science, and society

We argue that philosophical and historical research can constitute a "Biohumanities" that deepens our understanding of biology itself engages in constructive "science criticism," helps formulate new "visions of biology," and facilitates "critical science communication." We illustrate these ideas with two recent "experimental philosophy" studies of the concept of the gene and of the concept of innateness conducted by ourselves and collaborators. We conclude that the complex and often troubled relations between science and society are critical to both parties, and argue that the philosophy and history of science can help to make this relationship work.     

The Northern River Basins Study: context and design

Large river ecosystems worldwide are under increasing pressuresfrom environmentally degrading processes related to population increase,land use changes, climate change and the release of chemical substancesinto the environment. The large spatial scale and a lack of empiricaland theoretical understanding of the systems complicate environmentalassessment in these systems. Recently, the Northern River Basins Study(NRBS) undertook a multi-disciplinary approach to assess environmentaland socio-economic impacts of development in three large river basins,the Peace, Athabasca and Slave river basins in northern Canada. Thispaper provides the background and context for the NRBS and describes therationale used in developing an assessment framework for the Study. Anindependent science advisory committee comprised of scientific expertsfrom academia, First Nations and government oversaw quality assurance interms of science planning, implementation and reporting. All technicalstudies were subject to both internal and external peer review prior toapproval by a multi-stakeholder Study Board. The NRBS produced over 150technical and 12 synthesis reports that detailed these findings andprovided scientific recommendations. The Study Board used thisinformation to prepare a final report and recommendations. The NRBSprovides a clear example of a participatory, stakeholder approach thatwas successfully implemented in the design and management of anintegrated watershed environmental assessment program.     

Does the Committee Peer Review Select the Best Applicants for Funding? An Investigation of the Selection Process for Two European Molecular Biology Organization Programmes

Does peer review fulfill its declared objective of identifying the best science and the best scientists? In order to answer this question we analyzed the Long-Term Fellowship and the Young Investigator programmes of the European Molecular Biology Organization. Both programmes aim to identify and support the best post doctoral fellows and young group leaders in the life sciences. We checked the association between the selection decisions and the scientific performance of the applicants. Our study involved publication and citation data for 668 applicants to the Long-Term Fellowship programme from the year 1998 (130 approved, 538 rejected) and 297 applicants to the Young Investigator programme (39 approved and 258 rejected applicants) from the years 2001 and 2002. If quantity and impact of research publications are used as a criterion for scientific achievement, the results of (zero-truncated) negative binomial models show that the peer review process indeed selects scientists who perform on a higher level than the rejected ones subsequent to application. We determined the extent of errors due to over-estimation (type I errors) and under-estimation (type 2 errors) of future scientific performance. Our statistical analyses point out that between 26% and 48% of the decisions made to award or reject an application show one of both error types. Even though for a part of the applicants, the selection committee did not correctly estimate the applicant''s future performance, the results show a statistically significant association between selection decisions and the applicants'' scientific achievements, if quantity and impact of research publications are used as a criterion for scientific achievement.     

An Analysis of Citizen Science Based Research: Usage and Publication Patterns

The use of citizen science for scientific discovery relies on the acceptance of this method by the scientific community. Using the Web of Science and Scopus as the source of peer reviewed articles, an analysis of all published articles on “citizen science” confirmed its growth, and found that significant research on methodology and validation techniques preceded the rapid rise of the publications on research outcomes based on citizen science methods. Of considerable interest is the growing number of studies relying on the re-use of collected datasets from past citizen science research projects, which used data from either individual or multiple citizen science projects for new discoveries, such as for climate change research. The extent to which citizen science has been used in scientific discovery demonstrates its importance as a research approach. This broad analysis of peer reviewed papers on citizen science, that included not only citizen science projects, but the theory and methods developed to underpin the research, highlights the breadth and depth of the citizen science approach and encourages cross-fertilization between the different disciplines.     

The use of multiple tools for teaching medical biochemistry

In this work, we describe the use of several strategies employing the philosophies of active learning and problem-based learning (PBL) that may be used to improve the teaching of metabolic biochemistry to medical and nutritional undergraduate students. The main activities are as follows: 1) a seminar/poster system in a mini-congress format (using topics of applied biochemistry); 2) a true/false applied biochemistry exam (written by peer tutors); 3) a 9-h exam on metabolism (based in real publications); 4) the Advanced Biochemistry course (directed to peer tutors, where students learn how to read and criticize real medical papers); 5) experiments about nutrition and metabolism, using students as volunteers, and about free radicals (real science for students); 6) the BioBio blog (taking advantage of the "web age," this enhances out of class exchanges of information between the professor, students, and peer tutors); 7) student lectures on public health issues and metabolic disorders directed to the community and lay people; and 8) the BioBio quiz show. The main objective of these activities is to provide students with a more practical and interesting approach to biochemistry, such as the application of theoretical knowledge to real situations (diseases, experiments, media information, and scientific discoveries). In addition, we emphasize the importance of peer tutor activities for optimized learning of both students and peer tutors, the importance of a closer interaction between students and teaching staff, and the necessity to initiate students precociously in two broad fields of medical activity: "real" basic science and contact with the public (also helping students--future doctors and nutritionists--to be able to communicate with lay people). Most activities were evaluated by the students through written questionnaires and informal conversations, along various semesters, indicating good acceptance and approval of these methods. Good student scores in the biochemistry exams and seminars indicated that these activities are also working as valid educational tools.     

Broadening the voice of science: Promoting scientific communication in the undergraduate classroom

Effective and accurate communication of scientific findings is essential. Unfortunately, scientists are not always well trained in how to best communicate their results with other scientists nor do all appreciate the importance of speaking with the public. Here, we provide an example of how the development of oral communication skills can be integrated with research experiences at the undergraduate level. We describe our experiences developing, running, and evaluating a course for undergraduates that complemented their existing undergraduate research experiences with instruction on the nature of science and intensive training on the development of science communication skills. Students delivered science talks, research monologues, and poster presentations about the ecological and evolutionary research in which they were involved. We evaluated the effectiveness of our approach using the CURE survey and a focus group. As expected, undergraduates reported strong benefits to communication skills and confidence. We provide guidance for college researchers, instructors, and administrators interested in motivating and equipping the next generation of scientists to be excellent science communicators.