Examining the Predictive Validity of NIH Peer Review Scores

The predictive validity of peer review at the National Institutes of Health (NIH) has not yet been demonstrated empirically. It might be assumed that the most efficient and expedient test of the predictive validity of NIH peer review would be an examination of the correlation between percentile scores from peer review and bibliometric indices of the publications produced from funded projects. The present study used a large dataset to examine the rationale for such a study, to determine if it would satisfy the requirements for a test of predictive validity. The results show significant restriction of range in the applications selected for funding. Furthermore, those few applications that are funded with slightly worse peer review scores are not selected at random or representative of other applications in the same range. The funding institutes also negotiate with applicants to address issues identified during peer review. Therefore, the peer review scores assigned to the submitted applications, especially for those few funded applications with slightly worse peer review scores, do not reflect the changed and improved projects that are eventually funded. In addition, citation metrics by themselves are not valid or appropriate measures of scientific impact. The use of bibliometric indices on their own to measure scientific impact would likely increase the inefficiencies and problems with replicability already largely attributed to the current over-emphasis on bibliometric indices. Therefore, retrospective analyses of the correlation between percentile scores from peer review and bibliometric indices of the publications resulting from funded grant applications are not valid tests of the predictive validity of peer review at the NIH.     

The "study" role of past National Institutes of Health study sections

The original National Institutes of Health (NIH) study sections had two missions. The review of grant applications was the enduring one that we all recognize. The second original function, less remembered today, was to stand ready to advise the NIH, and in fact the entire community in a given biomedical field, on the current state of that discipline, as well as to opine on what new vistas were arising and to suggest (or occasionally even launch) appropriate courses of action. The present contribution is intended to remind us of this lesser-known original function of NIH study sections. We might ponder whether today's study sections, although more overworked than Sisyphus, should again take up this second function.     

Assessing Communication Skills of Medical Students in Objective Structured Clinical Examinations (OSCE) - A Systematic Review of Rating Scales

Background

Teaching and assessment of communication skills have become essential in medical education. The Objective Structured Clinical Examination (OSCE) has been found as an appropriate means to assess communication skills within medical education. Studies have demonstrated the importance of a valid assessment of medical students’ communication skills. Yet, the validity of the performance scores depends fundamentally on the quality of the rating scales used in an OSCE. Thus, this systematic review aimed at providing an overview of existing rating scales, describing their underlying definition of communication skills, determining the methodological quality of psychometric studies and the quality of psychometric properties of the identified rating scales.

Methods

We conducted a systematic review to identify psychometrically tested rating scales, which have been applied in OSCE settings to assess communication skills of medical students. Our search strategy comprised three databases (EMBASE, PsycINFO, and PubMed), reference tracking and consultation of experts. We included studies that reported psychometric properties of communication skills assessment rating scales used in OSCEs by examiners only. The methodological quality of included studies was assessed using the COnsensus based Standards for the selection of health status Measurement INstruments (COSMIN) checklist. The quality of psychometric properties was evaluated using the quality criteria of Terwee and colleagues.

Results

Data of twelve studies reporting on eight rating scales on communication skills assessment in OSCEs were included. Five of eight rating scales were explicitly developed based on a specific definition of communication skills. The methodological quality of studies was mainly poor. The psychometric quality of the eight rating scales was mainly intermediate.

Discussion

Our results reveal that future psychometric evaluation studies focusing on improving the methodological quality are needed in order to yield psychometrically sound results of the OSCEs assessing communication skills. This is especially important given that most OSCE rating scales are used for summative assessment, and thus have an impact on medical students’ academic success.     

An analysis of preliminary and post-discussion priority scores for grant applications peer reviewed by the Center for Scientific Review at the NIH

There has been the impression amongst many observers that discussion of a grant application has little practical impact on the final priority scores. Rather the final score is largely dictated by the range of preliminary scores given by the assigned reviewers. The implication is that the preliminary and final scores are the same and the discussion has little impact. The purpose of this examination of the peer review process at the National Institutes of Health is to describe the relationship between preliminary priority scores of the assigned reviewers and the final priority score given by the scientific review group. This study also describes the practical importance of any differences in priority scores. Priority scores for a sample of standard (R01) research grant applications were used in this assessment. The results indicate that the preliminary meeting evaluation is positively correlated with the final meeting outcome but that they are on average significantly different. The results demonstrate that discussion at the meeting has an important practical impact on over 13% of the applications.     

Acute and Chronic Mood and Apathy Outcomes from a Randomized Study of Unilateral STN and GPi DBS

Objective

To study mood and behavioral effects of unilateral and staged bilateral subthalamic nucleus (STN) and globus pallidus internus (GPi) deep brain stimulation (DBS) for Parkinson''s disease (PD).

Background

There are numerous reports of mood changes following DBS, however, most have focused on bilateral simultaneous STN implants with rapid and aggressive post-operative medication reduction.

Methods

A standardized evaluation was applied to a subset of patients undergoing STN and GPi DBS and who were also enrolled in the NIH COMPARE study. The Unified Parkinson Disease Rating Scale (UPDRS III), the Hamilton depression (HAM-D) and anxiety rating scales (HAM-A), the Yale-Brown obsessive-compulsive rating scale (YBOCS), the Apathy Scale (AS), and the Young mania rating scale (YMRS) were used. The scales were repeated at acute and chronic intervals. A post-operative strategy of non-aggressive medication reduction was employed.

Results

Thirty patients were randomized and underwent unilateral DBS (16 STN, 14 GPi). There were no baseline differences. The GPi group had a higher mean dopaminergic dosage at 1-year, however the between group difference in changes from baseline to 1-year was not significant. There were no differences between groups in mood and motor outcomes. When combining STN and GPi groups, the HAM-A scores worsened at 2-months, 4-months, 6-months and 1-year when compared with baseline; the HAM-D and YMRS scores worsened at 4-months, 6-months and 1-year; and the UPDRS Motor scores improved at 4-months and 1-year. Psychiatric diagnoses (DSM-IV) did not change. No between group differences were observed in the cohort of bilateral cases.

Conclusions

There were few changes in mood and behavior with STN or GPi DBS. The approach of staging STN or GPi DBS without aggressive medication reduction could be a viable option for managing PD surgical candidates. A study of bilateral DBS and of medication reduction will be required to better understand risks and benefits of a bilateral approach.     

Application of the immunoperoxidase technique to cell block preparations from fine needle aspirates

Immunocytochemistry on fine needle aspiration (FNA) material has been mainly performed on cytologic preparations; there have been few reports on the use of FNA cell blocks. This study compared the intensity scores of immunoperoxidase staining on FNA cell block preparations from 21 breast, 12 thyroid and 10 lymph node aspirates with the scores on the corresponding surgically excised specimens. FNA materials for cell blocks were fixed in formalin and embedded in agar. Ten commercially available antibodies forming three panels were studied using standard peroxidase-antiperoxidase and avidin-biotin complex techniques. In general, the staining results on the FNA cell block sections agreed with those on the surgical specimens; in addition, there were fewer aberrant positive staining results and much less background staining in the cell block sections. These phenomena were most striking with the cytokeratin antibodies. It is concluded that immunoperoxidase staining on FNA cell block preparations is reliable; the advantages of the use of cell block sections as opposed to smears are discussed.     

Peer review of consultations in primary care: the use of audiovisual recordings.

A rating scale was designed to measure performance in interviewing techniques in primary care. Peer review of audiovisual recordings of their consultations showed that a group of experienced general practitioners achieved significantly higher scores on the rating scale compared with a group of similarly experienced general practitioners who did not observe their recordings. The higher scores were obtained at the expense of longer consultations. The traditional five-minute appointment system in general practice needs to be reconsidered if an improved interviewing technique results in a more favourable outcome.     

Properties of one-zero scores in observational studies of primate social behavior: The effect of assumptions on empirical analyses

Empirical analyses of techniques for observing spontaneous social behavior have been guided by implicit assumptions about the techniques’ validity, reliability, and comparability. The actual duration and actual frequency of a behavior have been used as intuitive standards of validity. Noncorrespondence among absolute values obtained from these measures and from one-zero scores has been taken to mean that one-zero scores are invalid. An analysis of macaque and gelada grooming relationships demonstrated how unjustified assumptions about absolute values can affect conclusions. A statistical comparison of four types of observation scores, and a multiple-regression analysis of one-zero scores, contradicted assumptions made previously about reliability and validity. The four methods were comparable and all were reasonably accurate.     

Mechanisms of oncogene-mediated alterations in metastatic ability.

Transfected ras oncogenes have been shown to induce metastatic properties in some cells. This altered behavior is likely due to changes in ras-mediated signal transduction pathways, resulting in altered expression of genes important to metastasis. Clarification of the mechanisms by which ras is able to induce metastatic ability in model systems will improve our understanding of tumor progression, even in those cells in which ras activation has not been implicated. Many of the consequences of ras expression also have been detected in cells that have become metastatic in the absence of altered ras, suggesting that there is a set of common changes that can lead to metastasis, with multiple signals capable of eliciting these changes. We have identified several changes in metastatic, ras-transformed NIH 3T3 cells that may contribute to their increased malignancy, including expression of proteolytic enzymes and their inhibitors, and adhesive and calcium-binding proteins. Not all cells, however, respond in this way to expression of oncogenic ras. We have found that murine LTA cells, which are tumorigenic but nonmetastatic, are ras resistant and remain nonmetastatic when expressing high levels of transfected ras, in contrast to NIH 3T3 cells, which are ras sensitive and become both tumorigenic and metastatic in response to comparable levels of ras. LTA cells differ in their patterns of gene expression in response to ras when compared with NIH 3T3 cells, suggesting that the two cell lines process the ras signal differently. Here we review our results with ras-transfected NIH 3T3 and LTA cells.(ABSTRACT TRUNCATED AT 250 WORDS)     

Canine behavioral genetics — A review

A chronological review of the history of research in canine behavioral genetics is presented. Prior to the work of Scott and Fuller, many studies attempted to describe observed results in simple Mendelian terms. More recently, a quantitative mode of inheritance has been ascribed to many behavioral traits, and heritabilities have been calculated in different populations. Studies of behavioral traits are complicated by the effects of learning, which may well explain the reports of low heritability of behavioral traits measured on adults. Both genotype and environment have been shown to play major roles in the expression of behavioral traits. Maternal effects and the sex of an individual have important effects on behavior scores. Breeders usually select for both physical and behavioral traits in the same individuals, and this review includes a summary of reported research on the relationships between physical and mental traits.     

Rat c-raf oncogene activation by a rearrangement that produces a fused protein.

In a previous study, activated rat c-raf was detected by an NIH 3T3 cell transfection assay, and a rearrangement was demonstrated in the 5' half of the sequence of the gene. In the present study, the cDNAs of normal and activated rat c-raf were analyzed. Results showed that the activated c-raf gene is transcribed to produce a fused mRNA, in which the 5' half of the sequence is replaced by an unknown rat sequence. This mRNA codes a fused c-raf protein. The normal and activated c-raf cDNAs were each connected to the long terminal repeat of Rous sarcoma virus and transfected into NIH 3T3 cells. Only the activated form had transforming activity. We conclude that the rearrangement is responsible for the activation of c-raf.     

I undervalue you but I need you: the dissociation of attitude and memory toward in-group members

In the present study, the in-group bias or in-group derogation among Mainland Chinese was investigated through a rating task and a recognition test. In two experiments,participants from two universities with similar ranks rated novel faces or names and then had a recognition test. Half of the faces or names were labeled as participants' own university and the other half were labeled as their counterpart. Results showed that, for either faces or names, rating scores for out-group members were consistently higher than those for in-group members, whereas the recognition accuracy showed just the opposite. These results indicated that the attitude and memory for group-relevant information might be dissociated among Mainland Chinese.     

Intraoperative tissue expansion in rhytidectomy revisited

Intraoperative tissue expansion is an adjunct that has been used during rhytidectomy to rejuvenate the face and neck. This technique has been thought to allow for additional skin resection and, thus, increased skin tightening during rhytidectomy. The stretch of the skin by expansion should allow for additional skin resection before closure. Also, when the force of the underlying expander is removed, the expanded skin would recoil and the advancement of the flap should become tighter, with improved results. The technique achieved some popularity a few years ago but has received little recent attention. In this study, the authors attempted to compare face-lift results of adjunctive intraoperative tissue expansion during rhytidectomy with similar techniques without intraoperative expansion. The results of 50 female patients who underwent rhytidectomy for midface rejuvenation by a single operating surgeon composed the study group. Twenty-five of the patients had undergone rhytidectomy that addressed the cheek, chin, and neck areas without expansion (nonexpanded rhytidectomy group). The other 25 patients (expanded rhytidectomy group) had adjunctive intraoperative tissue expansion performed with the rhytidectomy. A tissue expander was temporarily placed beneath the rhytidectomy flaps on each side and expanded in a standard manner before final skin resection and closure. Frontal and lateral photographs were evaluated by 54 examiners. Preoperative and postoperative photographs of the 50 patients were viewed side-by-side by the examiners. The patients were presented in blind fashion and random order. The examiners graded the results of each patient on a scale of improvement from 1 to 10, with 10 being the maximum level of improvement. The scores were recorded and statistically evaluated by using the two-sample test. Evaluation of the examiners' scores showed that the mean rating given to patients in the expanded rhytidectomy group was 5.07 (SD = 1.12). The mean rating for the nonexpanded rhytidectomy group was 5.27 (SD = 1.57). When the two groups were compared using the two-sample test, the difference between the two was not statistically significant (p = 0.6127). Intraoperative tissue expansion as an adjunct to rhytidectomy did not result in improved facial rejuvenation in this patient series. The authors' impression is that the benefits of tissue expansion do not justify the added expense, time, and risks associated with using tissue expansion during rhytidectomy.     

A Modified Turchini Technic for the Differential Staining of Nucleic Acids

The preparation of the 9-methyl-2,3,7-trihydroxy-6-fluorone reagent for the selective staining of both desoxyribose and ribose nucleic acids is described. With slight variations this method follows the Duckert (1937) modification of the Liebermann and Lindenbaum (1904) reaction.

The present modification of the Turchini et al. (1944) staining procedure has been used on human autonomic ganglia fixed in Bouin's fluid, rat tissues, fixed in Bouin's, Zenker's, formol and for-mol-saline fixatives and mouse liver frozen and dried. The modified Turchini method has been examined primarily for its qualitative reliability by means of the following procedure. Ribonuclease treated sections were compared with adjacent sections immersed in distilled water. In succeeding steps half of these sections were stained by the modified Turchini process and the other half by Einarsson's gallocyanin chrome alum. Evidence gleaned from this and other tests indicates that 9-methyl-2,3,7-trihydroxy-6-fluorone may be used for the selective staining of desoxyribose and ribose nucleic acids.     

Understanding muscle markers: lower limbs

Musculoskeletal markers are frequently used to reconstruct past lifestyles and activity patterns. Yet the reliability of muscle marker measurements has been called into question because they may be confounded by body size. In this study, an aggregate muscle marker variable was calculated using 20 insertion sites (14 femoral, 6 tibial), and I examined their effects on lower limb size (as a proxy for body size), age, and sex. Analyses were made of a sample of 77 (57 males, 20 females) Native British Columbians (3,500-1,500 years BP) and 18th century Quebec prisoners. Muscle markers were measured using two-point observer rating scales; size was measured by standard methods; and age and sex were determined through pelvic, cranial, and dental morphology. Lower limb muscle markers correlated with: age, r=0.61; lower limb size, r=0.52; and sex, r=0.49; P <0.001. Older individuals had higher muscle marker scores, as did larger individuals and males. Based on partial correlations and regression analyses, age was the best overall predictor of lower limb muscle markers.     

Pregnancy following spinal cord injury.

Each year about 2,000 women of childbearing age in the United States have a spinal cord injury. Only a few mostly anecdotal reports describe pregnancy after such an injury. In a retrospective study of 16 women with a spinal cord injury, half of whom have a complete injury and about half quadriplegia, 25 pregnancies occurred, with 21 carried to full term. The women delayed pregnancy an average of 6.5 years after their injury, with an average age at first pregnancy of 26.8 years. Cesarean section was necessary in 4 patients because of inadequate progress of labor. In 5 deliveries an episiotomy and local anesthesia were required, 7 required epidural anesthesia, including all cesarean sections, and 10 did not require anesthesia. Several complications have been identified in the antepartum, intrapartum, and postpartum periods including autonomic hyperreflexia, premature labor, pressure sores, urinary tract infections, abnormal presentation, and failure to progress. Ultrasonography and amniocentesis were used selectively. Women with spinal cord injuries can have healthy children, although there are significant risks and these women have special needs.     

Human thrombin: enzymatic properties,stability and standardization of preparation

The work deals with estimation of thrombin preparation having such features as: sedimentation activity 3000-3200 NIH un. per 1 mg of protein and 97% of active centres. The enzyme isolated has been estimated according to the amidolytic activity on synthetic substrates S-2160 and BAPNA being equal 5200 and 185 milli un/mg of protein, respectively. According to the electrophoresis in PAAG in the presence of Ds-Na the preparation is homogenous, its molecular mass is 36000. The fibrinogen sedimentation time dependence on the isolated thrombin concentration has been estimated as well as the comparative analysis with the thrombin of the firm "Sigma" with the previously calibrated activity using the international standartion (coded P4) has been conducted. The absence of proportionality between the substrate sedimentation time and the preparation concentration has been determined. It has been revealed, that if the experimental findings are presented in the units 1/t against the thrombin units NIH the right lines are received within the limits used. The defreezing and secondary freezing of the preparation preserved under -20 degrees C have been showed as rendering an essential effect on thrombin activity. In order of the enzyme stabilizing at preserving the thrombin isolated has been concentrated applying the amycon membranes (MWCo: 30,000). While applying the thrombin water-saline solution in the conditions selected the preparation has showed itself practically stable during a year without utilizing any admixtures. The essential effect on thrombin has been found from the side of 1% glycin, 0.5% PEG, 1% saccharose and so on. The thrombin isolated high functional homogeneity, its stability permit to recommend the preparation as an operative standard.