Simulative performance analysis of gossip failure detection for scalable distributed systems

Three protocols for gossip-based failure detection services in large-scale heterogeneous clusters are analyzed and compared. The basic gossip protocol provides a means by which failures can be detected in large distributed systems in an asynchronous manner without the limits associated with reliable multicasting for group communications. The hierarchical protocol leverages the underlying network topology to achieve faster failure detection. In addition to studying the effectiveness and efficiency of these two agreement protocols, we propose a third protocol that extends the hierarchical approach by piggybacking gossip information on application-generated messages. The protocols are simulated and evaluated with a fault-injection model for scalable distributed systems comprised of clusters of workstations connected by high-performance networks, such as the CPlant system at Sandia National Laboratories. The model supports permanent and transient node and link failures, with rates specified at simulation time, for processors functioning in a fail-silent fashion. Through high-fidelity, CAD-based modeling and simulation, we demonstrate the strengths and weaknesses of each approach in terms of agreement time, number of gossips, and overall scalability. This revised version was published online in July 2006 with corrections to the Cover Date.     

Experimental Analysis of a Gossip-Based Service for Scalable, Distributed Failure Detection and Consensus

Gossip protocols and services provide a means by which failures can be detected in large, distributed systems in an asynchronous manner without the limits associated with reliable multicasting for group communications. Extending the gossip protocol such that a system reaches consensus on detected faults can be performed via a flat structure, or it can be hierarchically distributed across cooperating layers of nodes. In this paper, the performance of gossip services employing flat and hierarchical schemes is analyzed on an experimental testbed in terms of consensus time, resource utilization and scalability. Performance associated with a hierarchically arranged gossip scheme is analyzed with varying group sizes and is shown to scale well. Resource utilization of the gossip-style failure detection and consensus service is measured in terms of network bandwidth utilization and CPU utilization. Analytical models are developed for resource utilization and performance projections are made for large system sizes.     

GEMS: Gossip-Enabled Monitoring Service for Scalable Heterogeneous Distributed Systems

Gossip protocols have proven to be effective means by which failures can be detected in large, distributed systems in an asynchronous manner without the limitations associated with reliable multicasting for group communications. In this paper, we discuss the development and features of a Gossip-Enabled Monitoring Service (GEMS), a highly responsive and scalable resource monitoring service, to monitor health and performance information in heterogeneous distributed systems. GEMS has many novel and essential features such as detection of network partitions and dynamic insertion of new nodes into the service. Easily extensible, GEMS also incorporates facilities for distributing arbitrary system and application-specific data. We present experiments and analytical projections demonstrating scalability, fast response times and low resource utilization requirements, making GEMS a potent solution for resource monitoring in distributed computing.     

Multiple gossip statements and their effect on reputation and trustworthiness

Empirical and theoretical evidence from various disciplines indicates that reputation, reputation building and trust are important for human cooperation, social behaviour and economic progress. Recently, it has been shown that reputation gained in games of indirect reciprocity can be transmitted by gossip. But it has also been shown that gossiping has a strong manipulative potential. We propose that this manipulative potential is alleviated by the abundance of gossip. Multiple gossip statements give a better picture of the actual behaviour of a person, and thus inaccurate or fake gossip has little power as long as it is in the minority. In addition, we investigate the supposedly strong connection between reciprocity, reputation and trust. The results of this experimental study (with 11 groups of 12 students each) document that gossip quantity helps to direct cooperation towards cooperators. Moreover, reciprocity, trust and reputations transferred via gossip are positively correlated. This interrelation might have helped to reach the high levels of cooperation that can be observed in humans.     

FaTLease: scalable fault-tolerant lease negotiation with Paxos

A lease is a token which grants its owner exclusive access to a resource for a defined span of time. In order to be able to tolerate failures, leases need to be coordinated by distributed processes. We present FaTLease, an algorithm for fault-tolerant lease negotiation in distributed systems. It is built on the Paxos algorithm for distributed consensus, but avoids Paxos’ main performance bottleneck of requiring persistent state. This property makes our algorithm particularly useful for applications that can not dispense any disk bandwidth. Our experiments show that FaTLease scales up to tens of thousands of concurrent leases and can negotiate thousands of leases per second in both LAN and WAN environments.     

Gossip-Based Solutions for Discrete Rendezvous in Populations of Communicating Agents

The objective of the rendezvous problem is to construct a method that enables a population of agents to agree on a spatial (and possibly temporal) meeting location. We introduce the buffered gossip algorithm as a general solution to the rendezvous problem in a discrete domain with direct communication between decentralized agents. We compare the performance of the buffered gossip algorithm against the well known uniform gossip algorithm. We believe that a buffered solution is preferable to an unbuffered solution, such as the uniform gossip algorithm, because the use of a buffer allows an agent to use multiple information sources when determining its desired rendezvous point, and that access to multiple information sources may improve agent decision making by reinforcing or contradicting an initial choice. To show that the buffered gossip algorithm is an actual solution for the rendezvous problem, we construct a theoretical proof of convergence and derive the conditions under which the buffered gossip algorithm is guaranteed to produce a consensus on rendezvous location. We use these results to verify that the uniform gossip algorithm also solves the rendezvous problem. We then use a multi-agent simulation to conduct a series of simulation experiments to compare the performance between the buffered and uniform gossip algorithms. Our results suggest that the buffered gossip algorithm can solve the rendezvous problem faster than the uniform gossip algorithm; however, the relative performance between these two solutions depends on the specific constraints of the problem and the parameters of the buffered gossip algorithm.     

FRASystem: fault tolerant system using agents in distributed computing systems

In this paper, we present a fault tolerant and recovery system called FRASystem (Fault Tolerant & Recovery Agent System) using multi-agent in distributed computing systems. Previous rollback-recovery protocols were dependent on an inherent communication and an underlying operating system, which caused a decline of computing performance. We propose a rollback-recovery protocol that works independently on an operating system and leads to an increasing portability and extensibility. We define four types of agents: (1) a recovery agent performs a rollback-recovery protocol after a failure, (2) an information agent constructs domain knowledge as a rule of fault tolerance and information during a failure-free operation, (3) a facilitator agent controls the communication between agents, (4) a garbage collection agent performs garbage collection of the useless fault tolerance information. Since agent failures may lead to inconsistent states of a system and a domino effect, we propose an agent recovery algorithm. A garbage collection protocol addresses the performance degradation caused by the increment of saved fault tolerance information in a stable storage. We implemented a prototype of FRASystem using Java and CORBA and experimented the proposed rollback-recovery protocol. The simulations results indicate that the performance of our protocol is better than previous rollback-recovery protocols which use independent checkpointing and pessimistic message logging without using agents. Our contributions are as follows: (1) this is the first rollback-recovery protocol using agents, (2) FRASystem is not dependent on an operating system, and (3) FRASystem provides a portability and extensibility.     

microDuMIP: target-enrichment technique for microarray-based duplex molecular inversion probes

Molecular inversion probe (MIP)-based capture is a scalable and effective target-enrichment technology that can use synthetic single-stranded oligonucleotides as probes. Unlike the straightforward use of synthetic oligonucleotides for low-throughput target capture, high-throughput MIP capture has required laborious protocols to generate thousands of single-stranded probes from DNA microarray because of multiple enzymatic steps, gel purifications and extensive PCR amplifications. Here, we developed a simple and efficient microarray-based MIP preparation protocol using only one enzyme with double-stranded probes and improved target capture yields by designing probes with overlapping targets and unique barcodes. To test our strategy, we produced 11 510 microarray-based duplex MIPs (microDuMIPs) and captured 3554 exons of 228 genes in a HapMap genomic DNA sample (NA12878). Under our protocol, capture performance and precision of calling were compatible to conventional MIP capture methods, yet overlapping targets and unique barcodes allowed us to precisely genotype with as little as 50 ng of input genomic DNA without library preparation. microDuMIP method is simpler and cheaper, allowing broader applications and accurate target sequencing with a scalable number of targets.     

An Efficient Two-Tier Causal Protocol for Mobile Distributed Systems

Causal ordering is a useful tool for mobile distributed systems (MDS) to reduce the non-determinism induced by three main aspects: host mobility, asynchronous execution, and unpredictable communication delays. Several causal protocols for MDS exist. Most of them, in order to reduce the overhead and the computational cost over wireless channels and mobile hosts (MH), ensure causal ordering at and according to the causal view of the Base Stations. Nevertheless, these protocols introduce certain disadvantage, such as unnecessary inhibition at the delivery of messages. In this paper, we present an efficient causal protocol for groupware that satisfies the MDS''s constraints, avoiding unnecessary inhibitions and ensuring the causal delivery based on the view of the MHs. One interesting aspect of our protocol is that it dynamically adapts the causal information attached to each message based on the number of messages with immediate dependency relation, and this is not directly proportional to the number of MHs.     

Error Rates and Polymorphism Frequencies for Three RAPD Protocols

Three amplification protocols were analyzed for error rate and generation of polymorphisms during RAPD analysis. Using a set of 240 primers, the protocols detected similar frequencies of polymorphisms in two inbred sugar beet lines. The error rate was investigated by including a 1:1 mixture of DNA from the two lines in all analyses. Similar error rates, approximately 18%, were detected by the three protocols. Thus, altered amplification conditions did not substantially affect the error rate during RAPD analysis. For each of the three possible pairs of protocols, a positive correlation was obtained for primer and number of polymorphisms. Thus, a set of highly polymorphic RAPD primers can be used effectively, without prior screening, to detect polymorphisms for each protocol.     

Dynamic Quality of Service Model for Improving Performance of Multimedia Real-Time Transmission in Industrial Networks

Nowadays, quality of service (QoS) is very popular in various research areas like distributed systems, multimedia real-time applications and networking. The requirements of these systems are to satisfy reliability, uptime, security constraints and throughput as well as application specific requirements. The real-time multimedia applications are commonly distributed over the network and meet various time constraints across networks without creating any intervention over control flows. In particular, video compressors make variable bit-rate streams that mismatch the constant-bit-rate channels typically provided by classical real-time protocols, severely reducing the efficiency of network utilization. Thus, it is necessary to enlarge the communication bandwidth to transfer the compressed multimedia streams using Flexible Time Triggered- Enhanced Switched Ethernet (FTT-ESE) protocol. FTT-ESE provides automation to calculate the compression level and change the bandwidth of the stream. This paper focuses on low-latency multimedia transmission over Ethernet with dynamic quality-of-service (QoS) management. This proposed framework deals with a dynamic QoS for multimedia transmission over Ethernet with FTT-ESE protocol. This paper also presents distinct QoS metrics based both on the image quality and network features. Some experiments with recorded and live video streams show the advantages of the proposed framework. To validate the solution we have designed and implemented a simulator based on the Matlab/Simulink, which is a tool to evaluate different network architecture using Simulink blocks.     

Extending the scope of the controlled logical clock

Event traces are helpful in understanding the performance behavior of parallel applications since they allow the in-depth analysis of communication and synchronization patterns. However, the absence of synchronized clocks on most cluster systems may render the analysis ineffective because inaccurate relative event timings may misrepresent the logical event order and lead to errors when quantifying the impact of certain behaviors or confuse the users of time-line visualization tools by showing messages flowing backward in time. In our earlier work, we have developed a scalable algorithm called the controlled logical clock that eliminates inconsistent inter-process timings postmortem in traces of pure MPI applications, potentially running on large processor configurations. In this paper, we first demonstrate that our algorithm also proves beneficial in computational grids, where a single application is executed using the combined computational power of several geographically dispersed clusters. Second, we present an extended version of the algorithm that—in addition to message-passing event semantics—also preserves and restores shared-memory event semantics, enabling the correction of traces from hybrid applications.     

Amplitudes and time scales of picosecond-to-microsecond motion in proteins studied by solid-state NMR: a critical evaluation of experimental approaches and application to crystalline ubiquitin

Solid-state NMR provides insight into protein motion over time scales ranging from picoseconds to seconds. While in solution state the methodology to measure protein dynamics is well established, there is currently no such consensus protocol for measuring dynamics in solids. In this article, we perform a detailed investigation of measurement protocols for fast motions, i.e. motions ranging from picoseconds to a few microseconds, which is the range covered by dipolar coupling and relaxation experiments. We perform a detailed theoretical investigation how dipolar couplings and relaxation data can provide information about amplitudes and time scales of local motion. We show that the measurement of dipolar couplings is crucial for obtaining accurate motional parameters, while systematic errors are found when only relaxation data are used. Based on this realization, we investigate how the REDOR experiment can provide such data in a very accurate manner. We identify that with accurate rf calibration, and explicit consideration of rf field inhomogeneities, one can obtain highly accurate absolute order parameters. We then perform joint model-free analyses of 6 relaxation data sets and dipolar couplings, based on previously existing, as well as new data sets on microcrystalline ubiquitin. We show that nanosecond motion can be detected primarily in loop regions, and compare solid-state data to solution-state relaxation and RDC analyses. The protocols investigated here will serve as a useful basis towards the establishment of a routine protocol for the characterization of ps–μs motions in proteins by solid-state NMR.     

Why sampling scheme matters: the effect of sampling scheme on landscape genetic results

There has been a recent trend in genetic studies of wild populations where researchers have changed their sampling schemes from sampling pre-defined populations to sampling individuals uniformly across landscapes. This reflects the fact that many species under study are continuously distributed rather than clumped into obvious “populations”. Once individual samples are collected, many landscape genetic studies use clustering algorithms and multilocus genetic data to group samples into subpopulations. After clusters are derived, landscape features that may be acting as barriers are examined and described. In theory, if populations were evenly sampled, this course of action should reliably identify population structure. However, genetic gradients and irregularly collected samples may impact the composition and location of clusters. We built genetic models where individual genotypes were either randomly distributed across a landscape or contained gradients created by neighbor mating for multiple generations. We investigated the influence of six different sampling protocols on population clustering using program STRUCTURE, the most commonly used model-based clustering method for multilocus genotype data. For models where individuals (and their alleles) were randomly distributed across a landscape, STRUCTURE correctly predicted that only one population was being sampled. However, when gradients created by neighbor mating existed, STRUCTURE detected multiple, but different numbers of clusters, depending on sampling protocols. We recommend testing for fine scale autocorrelation patterns prior to sample clustering, as the scale of the autocorrelation appears to influence the results. Further, we recommend that researchers pay attention to the impacts that sampling may have on subsequent population and landscape genetic results. The U.S. Government's right to retain a non-exclusive, royalty-free license in and to any copyright is acknowledged.     

Resilient sensor authentication in SCADA by integrating physical unclonable function and blockchain

Industrial Control Systems and Supervisory Control and Data Acquisition (ICS/SCADA) systems are profound backbones of the national critical infrastructures and are essential to the sustainability of society since they help monitoring and controlling the cyber-enable services, such as energy, transportation, healthcare, etc. Modern SCADA systems continue to use the legacy communication protocols that lack adequate security mechanisms to provide trusted device authentication and ensure data flow integrity. Furthermore, advent of state-of-the-art network-capable sensor technology exposes many unique vulnerabilities to the adversaries. Thus, integrity of the data originated from field sensors along with their identity must be managed carefully in order to enhance reliability of ICS/SCADA ecosystems. In this paper, we present a blockchain-based SRAM PUF Authentication and Integrity (BloSPAI) protocol that aims to ensure a continuous authentication of field sensors and provide a robust data flow integrity process by leveraging distributed ledger and hardware security primitives. The prototype of the protocol has been implemented in a sensor-integrated Raspberry PI testbed that is interfaced with a permissioned blockchain network. We discuss the performance and overhead aspects of the proposed BloSPAI protocol and compare with state-of-art cybersecurity solutions. Through experimental evaluation demonstrates the relationship between the size of the blockchain network impacts the throughput in terms of time to commit transactions and overall systems setup time.

     

Entropy-based cluster validation and estimation of the number of clusters in gene expression data

Many external and internal validity measures have been proposed in order to estimate the number of clusters in gene expression data but as a rule they do not consider the analysis of the stability of the groupings produced by a clustering algorithm. Based on the approach assessing the predictive power or stability of a partitioning, we propose the new measure of cluster validation and the selection procedure to determine the suitable number of clusters. The validity measure is based on the estimation of the "clearness" of the consensus matrix, which is the result of a resampling clustering scheme or consensus clustering. According to the proposed selection procedure the stable clustering result is determined with the reference to the validity measure for the null hypothesis encoding for the absence of clusters. The final number of clusters is selected by analyzing the distance between the validity plots for initial and permutated data sets. We applied the selection procedure to estimate the clustering results on several datasets. As a result the proposed procedure produced an accurate and robust estimate of the number of clusters, which are in agreement with the biological knowledge and gold standards of cluster quality.     

Tissue disruption and extraction methods for metabolic profiling of an invertebrate sentinel species

Metabolic profiling of tissues needs special attention, because the compartmentalization of cellular constituents will be abolished by sample homogenization. This loss of partitioning leads to protein and metabolite instability in extracts, and therefore metabolite extraction protocols need to ensure very rapid inactivation of macromolecules as well as solubilization of metabolites. There are many published methods for tissue metabolome analysis, but no universally accepted standard, and a lack of measurable quality benchmarks. We developed a protocol for efficient tissue disruption and metabolite extraction of the earthworm Lumbricus rubellus guided by prior biological knowledge as well as metrics based on the data. In particular, we identified an unusual degree of instability of L. rubellus tissue extracts, and evaluated different approaches such as heating and filtration to counteract this. Finally, we evaluated four different solvent systems for comprehensive metabolite extraction using three analytical platforms (¹H NMR spectroscopy, GC?CMS, and direct-infusion FT-ICR-MS), and also compared bead-beating and cryogenic milling for tissue disruption. Initially we ranked methods by common analytical criteria (e.g. numbers and total intensity of detected peaks) in order to compare protocols. These approaches to assess protocol suitability proved to be inadequate to judge earthworm tissue extraction methods because of sample instability. Existing tissue extraction protocols should not be assumed to be automatically applicable to novel species.     

Bacterial binding protein-dependent permeases: characterization of distinctive signatures for functionally related integral cytoplasmic membrane proteins

Bacterial binding protein-dependent transport systems belong to the superfamily of ABC transporters, which is widely distributed among living organisms. Their hydrophobic membrane proteins are the least characterized components. The primary structures of 61 integral membrane proteins from 35 uptake systems were compared in order to characterize a short conserved hydrophilic segment, with a consensus EAA … G ………-I - LP, located approximately 100 residues from the C-terminus. Secondary structure predictions indicated that this conserved region might be formed by two amphipathic α-helices connected by a loop containing the invariant G residue. We classified the conserved motifs and found that membrane proteins from systems transporting structurally related substrates specifically display a greater number of identical residues in the conserved region. We determined a consensus for each class of membrane protein and showed that these can be considered as signatures.     

Protein complex prediction via cost-based clustering

MOTIVATION: Understanding principles of cellular organization and function can be enhanced if we detect known and predict still undiscovered protein complexes within the cell's protein-protein interaction (PPI) network. Such predictions may be used as an inexpensive tool to direct biological experiments. The increasing amount of available PPI data necessitates an accurate and scalable approach to protein complex identification. RESULTS: We have developed the Restricted Neighborhood Search Clustering Algorithm (RNSC) to efficiently partition networks into clusters using a cost function. We applied this cost-based clustering algorithm to PPI networks of Saccharomyces cerevisiae, Drosophila melanogaster and Caenorhabditis elegans to identify and predict protein complexes. We have determined functional and graph-theoretic properties of true protein complexes from the MIPS database. Based on these properties, we defined filters to distinguish between identified network clusters and true protein complexes. Conclusions: Our application of the cost-based clustering algorithm provides an accurate and scalable method of detecting and predicting protein complexes within a PPI network.     

Self Organized Terminode Routing

We consider the problem of routing in a wide area mobile ad hoc network called Terminode Network. Routing in this network is designed with the following objectives. First, it should scale well in terms of the number of nodes and geographical coverage; second, routing should have scalable mechanisms that cope with the dynamicity in the network due to mobility; and third, nodes need to be highly collaborative and redundant, but, most of all, cannot use complex algorithms or protocols. Our routing scheme is a combination of two protocols called Terminode Local Routing (TLR) and Terminode Remote Routing (TRR). TLR is used to route packets to close destinations. TRR is used to route to remote destinations. The combination of TLR and TRR has the following features: (1) it is highly scalable because every node relies only on itself and a small number of other nodes for packet forwarding; (2) it acts and reacts well to the dynamicity of the network because as a rule multipath routing is considered; and (3) it can be implemented and run in very simple devices because the algorithms and protocols are very simple and based on high collaboration. We performed simulations of the TLR and TRR protocols using the GloMoSim simulator. The simulation results for a large, highly mobile ad hoc environment demonstrate benefits of the combination of TLR and TRR over an existing protocol that uses geographical information for packet forwarding.