IMB-ASYNC: a revised method and benchmark to estimate MPI-3 asynchronous progress efficiency

The article presents design and methodology of a novel benchmark suite named IMB-ASYNC. The presented suite and method are aimed at measuring and comparing practical communication-computation overlap levels for Message Passing Interface standard (MPI) implementations with a special accent to some applicable use cases. Some typical MPI communication patterns implying communication-computation overlap are analyzed, and their reflection on a benchmark structure is proposed. We also analyze the previous works on overlap benchmarking and their best practices. We present a new benchmarking approach for non-blocking neighborhood collectives overlap and clarify the overlap estimation methodology. After a short overview of some technical details of currently available MPI asynchronous progress implementations, two benchmarking case studies are presented to illustrate the relevance of the methodology.

     

FROST: a filter-based fold recognition method

To assess the reliability of fold assignments to protein sequences, we developed a fold recognition method called FROST (Fold Recognition-Oriented Search Tool) based on a series of filters and a database specifically designed as a benchmark for this new method under realistic conditions. This benchmark database consists of proteins for which there exists, at least, another protein with an extensively similar 3D structure in a database of representative 3D structures (i.e., more than 65% of the residues in both proteins can be structurally aligned). Because the testing of our method must be carried out under conditions similar to those of real fold recognition experiments, no protein pair with sequence similarity detectable using standard sequence comparison methods such as FASTA is included in the benchmark database. While using FROST, we achieved a coverage of 60% for a rate of error of 1%. To obtain a baseline for our method, we used PSI-BLAST and 3D-PSSM. Under the same conditions, for a 1% error rate, coverages for PSI-BLAST and 3D-PSSM were 33 and 56%, respectively.     

Benchmarking the Environmental Performances of Farms (4 pp)

Background, Aim and Scope The usual route for improvement of agricultural practice towards sustainability runs via labelling schemes for products or farm practices. In most approaches requirements are set in absolute terms, disregarding the variation in environmental performance of farms. Another approach for promoting sustainable farming concerns the concept of benchmarking, which takes into account competition among farmers. The individual agricultural performance is characterized by quantitative criteria and compared with scores of other relevant farms. Methods Therefore, a pilot study has been conducted in the Netherlands concerning benchmarking among arable farmers in the Internet involving crop protection. A voluntary Dutch benchmark initiative in the Internet is described including farmers' perception regarding the tool. Results The results show that the benchmark tool in the Internet allows farmers to compare their environmental and economic performance anonymously and securely in a large-scale open-access environment. The pilot group of farmers responded positively to the instrument. An important factor in success is the ease and speed with which data can be entered into the benchmark tool. Conclusions A benchmark tool for comparing the environmental performance among farmers can form the basis for agreements between farmers and their costumers. An application involving food industry and retailers is discussed.     

DOCKGROUND system of databases for protein recognition studies: unbound structures for docking

Computational docking approaches are important as a source of protein-protein complexes structures and as a means to understand the principles of protein association. A key element in designing better docking approaches, including search procedures, potentials, and scoring functions is their validation on experimentally determined structures. Thus, the databases of such structures (benchmark sets) are important. The previous, first release of the DOCKGROUND resource (Douguet et al., Bioinformatics 2006; 22:2612-2618) implemented a comprehensive database of cocrystallized (bound) protein-protein complexes in a relational database of annotated structures. The current release adds important features to the set of bound structures, such as regularly updated downloadable datasets: automatically generated nonredundant set, built according to most common criteria, and a manually curated set that includes only biological nonobligate complexes along with a number of additional useful characteristics. The main focus of the current release is unbound (experimental and simulated) protein-protein complexes. Complexes from the bound dataset are used to identify crystallized unbound analogs. If such analogs do not exist, the unbound structures are simulated by rotamer library optimization. Thus, the database contains comprehensive sets of complexes suitable for large scale benchmarking of docking algorithms. Advanced methodologies for simulating unbound conformations are being explored for the next release. The future releases will include datasets of modeled protein-protein complexes, and systematic sets of docking decoys obtained by different docking algorithms. The growing DOCKGROUND resource is designed to become a comprehensive public environment for developing and validating new docking methodologies.     

Benchmarking spliced alignment programs including Spaln2, an extended version of Spaln that incorporates additional species-specific features

Spliced alignment plays a central role in the precise identification of eukaryotic gene structures. Even though many spliced alignment programs have been developed, recent rapid progress in DNA sequencing technologies demands further improvements in software tools. Benchmarking algorithms under various conditions is an indispensable task for the development of better software; however, there is a dire lack of appropriate datasets usable for benchmarking spliced alignment programs. In this study, we have constructed two types of datasets: simulated sequence datasets and actual cross-species datasets. The datasets are designed to correspond to various real situations, i.e. divergent eukaryotic species, different types of reference sequences, and the wide divergence between query and target sequences. In addition, we have developed an extended version of our program Spaln, which incorporates two additional features to the scoring scheme of the original version, and examined this extended version, Spaln2, together with the original Spaln and other representative aligners based on our benchmark datasets. Although the effects of the modifications are not individually striking, Spaln2 is consistently most accurate and reasonably fast in most practical cases, especially for plants and fungi and for increasingly divergent pairs of target and query sequences.     

Healthcare systems and motivation

Despite the fact that most American physicians, at least until around the 1970s, stood in the way of developing a universal healthcare system, most are generally not happy with the current state of healthcare--or its lack thereof--today. The primary reasons for this general unhappiness are that insurance companies and managed care have successfully conspired to remove much of the physician's autonomy (via imposed time constraints, burdensome paperwork, the time-consuming chore of having to defend going against stringent treatment algorithms that are often inappropriate for some patients) and the satisfaction of knowing their patients. Few physicians in managed care organizations (MCOs) are able to practice without constant and blindly algorithmic interference concerning the diagnostic tests and therapeutic interventions they order. As copayments have increased, they often find that patients, even though "covered," cannot afford the therapy they deem necessary. While physicians expect to earn sufficient to pay back their not insignificant educational debts, provide their children with help through college, and assure retirements sufficient for themselves and their spouses, these should not be considered unreasonable expectations. Most physicians today do favor universal healthcare -- to the point of having included such language in their various professional codes of ethics (which, perversely enough, bioethicists as a group have failed to do). Contrary to the claims of our colleagues, Altom and Churchill, physicians seem to be genuinely frustrated as to what else they can do to change the current inequitable system.     

iBench: A ground truth approach for advanced validation of mass spectrometry identification method

The discovery of many noncanonical peptides detectable with sensitive mass spectrometry inside, outside, and on cells shepherded the development of novel methods for their identification, often not supported by a systematic benchmarking with other methods. We here propose iBench, a bioinformatic tool that can construct ground truth proteomics datasets and cognate databases, thereby generating a training court wherein methods, search engines, and proteomics strategies can be tested, and their performances estimated by the same tool. iBench can be coupled to the main database search engines, allows the selection of customized features of mass spectrometry spectra and peptides, provides standard benchmarking outputs, and is open source. The proof-of-concept application to tryptic proteome digestions, immunopeptidomes, and synthetic peptide libraries dissected the impact that noncanonical peptides could have on the identification of canonical peptides by Mascot search with rescoring via Percolator (Mascot+Percolator).     

Task shifting and integration of HIV care into primary care in South Africa: The development and content of the streamlining tasks and roles to expand treatment and care for HIV (STRETCH) intervention

Background

The Veterans Health Administration (VHA) oversees the largest integrated healthcare system in the United States. The feasibility of a large-scale, nationwide, group-randomized implementation trial of VHA outpatient practices has not been reported. We describe the recruitment and enrollment of such a trial testing a clinician-directed, Internet-delivered intervention for improving the care of postmyocardial infarction patients with multiple comorbidities.

Methods

With a recruitment goal of 200 eligible community-based outpatient clinics, parent VHA facilities (medical centers) were recruited because they oversee their affiliated clinics and the research conducted there. Eligible facilities had at least four VHA-owned and -operated primary care clinics, an affiliated Institutional Review Board (IRB), and no ongoing, potentially overlapping, quality-improvement study. Between December 2003 and December 2005, in two consecutive phases, we used initial and then intensified recruitment strategies.

Results

Overall, 48 of 66 (73%) eligible facilities were recruited. Of the 219 clinics and 957 clinicians associated with the 48 facilities, 168 (78%) clinics and 401 (42%) clinicians participated. The median time from initial facility contact to clinic enrollment was 222 days, which decreased by over one-third from the first to the second recruitment phase (medians: 323 and 195 days, respectively; p < .001), when more structured recruitment with physician recruiters was implemented and a dedicated IRB manager was added to the coordinating center staff.

Conclusions

Large group-randomized trials benefit from having dedicated physician investigators and IRB personnel involved in recruitment. A large-scale, nationally representative, group-randomized trial of community-based clinics is feasible within the VHA or a similar national healthcare system.     

Environmental benchmarks for buildings: a critical literature review

Purpose

To reduce the environmental impact of the building sector, environmental targets considering the full life cycle of buildings can be supportive. In recent years, various benchmarks based on Life Cycle Assessment (LCA) have been developed as part of regulations, labelling systems, sustainability rating tools and research studies. The objective of this paper is to critically analyse 23 existing benchmarking systems focusing on the benchmark methodology but also on the benchmark applications and communication.

Methods

The critical literature review consists of two parts. In a first part, the choices related to the assessment method, functional equivalent, definition of benchmark values, benchmark scope, benchmark applications and benchmark communication are compared. In the second part, benchmark values are compiled from literature and statistically analysed.

Results and discussion

The comparative analysis allows to identify the main approaches and methods used in benchmarking systems. For each evaluation aspect, the strengths and weaknesses of the various approaches are highlighted. The statistical analysis provides insight in the spread of benchmark values. Important variations are found between the literature sources which can be explained by differences in benchmark approach, scope, system boundaries and applications.

Conclusions

Based on the comparative analysis, recommendations are formulated for the development of LCA benchmarks for the building sector. The results of the statistical analysis furthermore provide reference values which can be used for the validation of future benchmarks. For global warming, the statistical values for the full life cycle impacts (i.e. embodied and operational impacts) range from about 15 up to 35 kg CO₂ eq/m².a.

     

Automated methods of predicting the function of biological sequences using GO and BLAST

Background  

With the exponential increase in genomic sequence data there is a need to develop automated approaches to deducing the biological functions of novel sequences with high accuracy. Our aim is to demonstrate how accuracy benchmarking can be used in a decision-making process evaluating competing designs of biological function predictors. We utilise the Gene Ontology, GO, a directed acyclic graph of functional terms, to annotate sequences with functional information describing their biological context. Initially we examine the effect on accuracy scores of increasing the allowed distance between predicted and a test set of curator assigned terms. Next we evaluate several annotator methods using accuracy benchmarking. Given an unannotated sequence we use the Basic Local Alignment Search Tool, BLAST, to find similar sequences that have already been assigned GO terms by curators. A number of methods were developed that utilise terms associated with the best five matching sequences. These methods were compared against a benchmark method of simply using terms associated with the best BLAST-matched sequence (best BLAST approach).     

Structure-based identification of MHC binding peptides: Benchmarking of prediction accuracy

Identification of MHC binding peptides is essential for understanding the molecular mechanism of immune response. However, most of the prediction methods use motifs/profiles derived from experimental peptide binding data for specific MHC alleles, thus limiting their applicability only to those alleles for which such data is available. In this work we have developed a structure-based method which does not require experimental peptide binding data for training. Our method models MHC-peptide complexes using crystal structures of 170 MHC-peptide complexes and evaluates the binding energies using two well known residue based statistical pair potentials, namely Betancourt-Thirumalai (BT) and Miyazawa-Jernigan (MJ) matrices. Extensive benchmarking of prediction accuracy on a data set of 1654 epitopes from class I and class II alleles available in the SYFPEITHI database indicate that BT pair-potential can predict more than 60% of the known binders in case of 14 MHC alleles with AUC values for ROC curves ranging from 0.6 to 0.9. Similar benchmarking on 29,522 class I and class II MHC binding peptides with known IC(50) values in the IEDB database showed AUC values higher than 0.6 for 10 class I alleles and 9 class II alleles in predictions involving classification of a peptide to be binder or non-binder. Comparison with recently available benchmarking studies indicated that, the prediction accuracy of our method for many of the class I and class II MHC alleles was comparable to the sequence based methods, even if it does not use any experimental data for training. It is also encouraging to note that the ranks of true binding peptides could further be improved, when high scoring peptides obtained from pair potential were re-ranked using all atom forcefield and MM/PBSA method.     

Human subject protections in genetic research

The Certificate of Confidentiality (COC) is a voluntary tool to protect researchers from being compelled to release identifying information about their subjects. Institutional review board (IRB) review and informed consent (IC) procedures are mandatory tools to protect human subjects. Although many studies reveal poor documentation of IRB and IC procurement, most published research undergoes IRB review and has appropriate IC procedures. There are no empirical data about the use of COCs. We examined the procurement and documentation of all these human subject protections in the genetics literature. A total of 112 (55%) articles documented IRB review, 108 (53%) document IC, and 82 (41%) documented both. None documented the procurement of a COC. Returned surveys provided additional information that confirmed that at least 74% (n = 150) of research had received appropriate IRB review, at least 71% (n = 143) had procured IC, and at least 10% (n = 21) had obtained a COC. An additional 22 respondents had procured COCs for other research, whereas 17 respondents were unaware of them and their purpose. In this era of public scrutiny of medical research, we recommend greater familiarity with and documentation of all human subject protections.     

mangal – making ecological network analysis simple

The study of ecological networks is severely limited by 1) the difficulty to access data, 2) the lack of a standardized way to link meta‐data with interactions, and 3) the disparity of formats in which ecological networks themselves are stored and represented. To overcome these limitations, we have designed a data specification for ecological networks. We implemented a database respecting this standard, and released an R package (rmangal) allowing users to programmatically access, curate, and deposit data on ecological interactions. In this article, we show how these tools, in conjunction with other frameworks for the programmatic manipulation of open ecological data, streamlines the analysis process and improves replicability and reproducibility of ecological network studies.     

BALB/c-3T3 fibroblasts resistant to growth inhibition by beta interferon exhibit aberrant platelet-derived growth factor, epidermal growth factor, and fibroblast growth factor signal transduction.

Several lines of evidence now exist to suggest an interaction between the platelet-derived growth factor (PDGF) growth-stimulatory signal transduction pathway and the beta interferon (IFN-beta) growth-inhibitory signal transduction pathway. The most direct examples are inhibition of PDGF-mediated gene induction and mitogenesis by IFN-beta and the effects of activators and inhibitors of the IFN-inducible double-stranded RNA-dependent eIF2 kinase on expression of PDGF-inducible genes. To further investigate the nature of this PDGF/IFN-beta interaction, we selected BALB/c-3T3 cells for resistance to growth inhibition by IFN-beta and analyzed the phenotypes of resulting clonal lines (called IRB cells) with respect to PDGF signal transduction. Although selected only for IFN resistance, the IRB cells were found to be defective for induction of growth-related genes c-fos, c-myc and JE in response to PDGF. This block to signal transduction was not due to loss or inactivation of PDGF receptors, as immunoprecipitation of PDGF receptors with antiphosphotyrosine antibodies showed them to be present at equal levels in the BALB/c-3T3 and IRB cells and to be autophosphorylated normally in response to PDGF. Furthermore, treatment with other peptide growth factors (PDGF-AA, fibroblast growth factor, and epidermal growth factor) also failed to induce c-fos, c-myc, or JE expression in IRB cells. All of these growth factors, however, were able to induce another early growth-related gene, Egr-1. The block to signaling was not due to a defect in inositol phosphate metabolism, as PDGF treatment induced normal calcium mobilization and phosphotidylinositol-3-kinase activation in these cells. Activation of protein kinase C by phorbol esters did induce c-fos, c-myc, and JE in IRB cells, indicating that signalling pathways distal to this enzyme remained intact. We have previously shown that IFN-inducible enzyme activities, including double-stranded RNA-dependent eIF2 kinase and 2',5'-oligoadenylate synthetase, are normal in IRB cells. The finding that the induction of multiple growth-related genes by several independent growth factors is inhibited in these IFN-resistant cells suggests that there is a second messenger common to both growth factor and IFN signaling pathways and that this messenger is defective in these cells.     

Analysis of European mtDNAs for recombination

The standard paradigm postulates that the human mitochondrial genome (mtDNA) is strictly maternally inherited and that, consequently, mtDNA lineages are clonal. As a result of mtDNA clonality, phylogenetic and population genetic analyses should therefore be free of the complexities imposed by biparental recombination. The use of mtDNA in analyses of human molecular evolution is contingent, in fact, on clonality, which is also a condition that is critical both for forensic studies and for understanding the transmission of pathogenic mtDNA mutations within families. This paradigm, however, has been challenged recently by Eyre-Walker and colleagues. Using two different tests, they have concluded that recombination has contributed to the distribution of mtDNA polymorphisms within the human population. We have assembled a database that comprises the complete sequences of 64 European and 2 African mtDNAs. When this set of sequences was analyzed using any of three measures of linkage disequilibrium, one of the tests of Eyre-Walker and colleagues, there was no evidence for mtDNA recombination. When their test for excess homoplasies was applied to our set of sequences, only a slight excess of homoplasies was observed. We discuss possible reasons that our results differ from those of Eyre-Walker and colleagues. When we take the various results together, our conclusion is that mtDNA recombination has not been sufficiently frequent during human evolution to overturn the standard paradigm.     

Coding for pathology tests - strengths and weaknesses

Laboratory professionals will increasingly find themselves called upon to assist with the coding of pathology test requests and reported results in the era of the e-Health Record (EHR). EHR users from outside pathology, including patients and clinicians, will expect seamless integration of pathology services, and question variations in test nomenclature, units, reference intervals, and interpretive comments. Scientists and pathologists will need to be ready to work with colleagues outside their traditional scientific disciplines, along with IT and terminology experts, to resolve illogical historical variations, highlight differences that might endanger patient safety, and help lay the foundations for evolving e-health systems that enhance healthcare without eroding the interpretive value of pathology reports from different laboratories. An overview of medical terminologies and Australasian eHealth harmonisation programs is also provided.     

"Walking along beside the researcher": how Canadian REBs/IRBs are responding to the needs of community-based participatory research

Research ethics boards and institutional review boards (REBs/IRBs) have been criticized for relying on conceptions of research that privilege biomedical, clinical, and experimental designs, and for penalizing research that deviates from this model. Studies that use a community-based participatory research (CBPR) design have been identified as particularly challenging to navigate through existing ethics review frameworks. However, the voices of REB/IRB members and staff have been largely absent in this debate. The objective of this article is to explore the perspectives of members of Canadian university-based REBs/IRBs regarding their capacity to review CBPR protocols. We present findings from interviews with 24 Canadian REB/IRB members, staff, and other key informants. Participants were asked to describe and contrast their experiences reviewing studies using CBPR and mainstream approaches. Contrary to the perception that REBs/IRBs are inflexible and unresponsive, participants described their attempts to dialogue and negotiate with researchers and to provide guidance. Overall, these Canadian REBs/IRBs demonstrated a more complex understanding of CBPR than is typically characterized in the literature. Finally, we situate our findings within literature on relational ethics and explore the possibility of researchers and REBs/IRBs working collaboratively to find solutions to unique ethical tensions in CBPR.     

The Ethics Police?: IRBs' Views Concerning Their Power

Background

In recent years, tensions between IRBs and principal investigators (PIs) have risen, posing the needs to understand these conflicts, their underlying causes, and possible solutions. Researchers frequently complain about IRBs, but how IRBs perceive and respond to these criticisms is unclear.

Methods

I conducted in-depth, semi-structured interviews of two hours each with 46 chairs, administrators, and members. I contacted the leadership of 60 IRBs around the country (every fourth one in the list of the top 240 institutions by NIH funding) and interviewed IRB leaders from 34 of these institutions (response rate = 55%).

Results

Interviewees suggest that IRBs and PIs may view the nature and causes of these conflicts very differently and misunderstand each other, exacerbating tensions. Interviewees often recognized that they were seen by PIs as having power, but many IRBs saw themselves as not having it (e.g., because they are “merely following the regulations,” and their process is “open,” impersonal and unbiased, and they are themselves subject to higher administrative agencies), or as having it, but feeling it is small, and/or justified (e.g., because it is based on overriding goals and “the community values,” and IRBs are trying to help PIs). Questions emerge as to whether IRBs do or should have power, and if so, what kind, how much, and when. Several factors may affect these tensions.

Conclusions

This study, the first to explore how IRBs perceive and understand conflicts and power relationships with PIs, suggests how IRBs and PIs may differ in viewing their respective roles and relationships, exacerbating tensions. These issues have critical implications for IRBs and PIs—to enhance their awareness and understanding of these conflicts (e.g., that IRBs may have discretionary power) and the underlying causes involved, and for increasing attention to research, practice, and policy concerning these areas of IRB functioning and interactions with PIs.     

APDB: a novel measure for benchmarking sequence alignment methods without reference alignments

MOTIVATION: We describe APDB, a novel measure for evaluating the quality of a protein sequence alignment, given two or more PDB structures. This evaluation does not require a reference alignment or a structure superposition. APDB is designed to efficiently and objectively benchmark multiple sequence alignment methods. RESULTS: Using existing collections of reference multiple sequence alignments and existing alignment methods, we show that APDB gives results that are consistent with those obtained using conventional evaluations. We also show that APDB is suitable for evaluating sequence alignments that are structurally equivalent. We conclude that APDB provides an alternative to more conventional methods used for benchmarking sequence alignment packages.     

Gene Function Prediction from Functional Association Networks Using Kernel Partial Least Squares Regression

With the growing availability of large-scale biological datasets, automated methods of extracting functionally meaningful information from this data are becoming increasingly important. Data relating to functional association between genes or proteins, such as co-expression or functional association, is often represented in terms of gene or protein networks. Several methods of predicting gene function from these networks have been proposed. However, evaluating the relative performance of these algorithms may not be trivial: concerns have been raised over biases in different benchmarking methods and datasets, particularly relating to non-independence of functional association data and test data. In this paper we propose a new network-based gene function prediction algorithm using a commute-time kernel and partial least squares regression (Compass). We compare Compass to GeneMANIA, a leading network-based prediction algorithm, using a number of different benchmarks, and find that Compass outperforms GeneMANIA on these benchmarks. We also explicitly explore problems associated with the non-independence of functional association data and test data. We find that a benchmark based on the Gene Ontology database, which, directly or indirectly, incorporates information from other databases, may considerably overestimate the performance of algorithms exploiting functional association data for prediction.