An information-theoretic approach to curiosity-driven reinforcement learning

We provide a fresh look at the problem of exploration in reinforcement learning, drawing on ideas from information theory. First, we show that Boltzmann-style exploration, one of the main exploration methods used in reinforcement learning, is optimal from an information-theoretic point of view, in that it optimally trades expected return for the coding cost of the policy. Second, we address the problem of curiosity-driven learning. We propose that, in addition to maximizing the expected return, a learner should choose a policy that also maximizes the learner's predictive power. This makes the world both interesting and exploitable. Optimal policies then have the form of Boltzmann-style exploration with a bonus, containing a novel exploration-exploitation trade-off which emerges naturally from the proposed optimization principle. Importantly, this exploration-exploitation trade-off persists in the optimal deterministic policy, i.e., when there is no exploration due to randomness. As a result, exploration is understood as an emerging behavior that optimizes information gain, rather than being modeled as pure randomization of action choices.     

An information-theoretic approach to interactions in images.

In this paper it will be argued that the notion of interactions in images is closely related to that of entropy associated with an image, and it will be shown that interactions make processing of the information coming from the retina computationally less expensive. A procedure will be presented, based on the evolution of joint entropy across different scales, to gauge the contributions of different types of interactions to the structure of the images.     

An algorithmic approach to upper arm contouring

There has been a renewed interest in upper arm contouring given the recent advances and subsequent patient interest in weight loss. Patients undergoing bariatric surgery are often left with a significant amount of redundant skin and laxity of their upper extremity. Some patients within this group have excess fat in their upper arms with relatively good skin tone, while others have a paucity of excess fat with a significant amount of redundant skin. The optimal treatment for each patient can vary. A clinical algorithm is presented that is designed to select the best method for upper arm contouring based on the aesthetic analysis of the upper arm. Case examples are provided demonstrating results that were obtained by following this algorithm.     

An algorithmic approach to multiple complete digest mapping.

Multiple Complete Digest (MCD) mapping is a method of determining the locations of restriction sites along a target DNA molecule. The resulting restriction map has many potential applications in DNA sequencing and genetics. In this work, we present a heuristic algorithm for fragment identification, a key step in the process of constructing an MCD map. Given measurements of the restriction fragment sizes from one or more complete digestions of each clone in a clone library covering the molecule to be mapped, the algorithm identifies groups of restriction fragments on different clones that correspond to the same region of the target DNA. Once these groups are correctly determined the desired map can be constructed by solving a system of simple linear inequalities. We demonstrate the effectiveness of our algorithm on real data provided by the Genome Center at the University of Washington.     

An algorithmic approach to modelling the retinal receptor topography

Based on published research results on the structure of the human retina and the initial assumption of tight hexagonal packing of cones, the mean cone-distance function is derived. Disorder in the cone lattice is explained as the superposition of increasing topological distortion in the hexagonal lattice (providing a possible explanation for observed systematic lattice distortions) and local jitter of neighbor-toneighbor distances, for which a simple statistical model is provided. These individual results are incorporated into a proposed algorithm for simulating the cone receptors' topography in 3D-space. Finally, possible software and hardware applications of the algorithmically defined retina model are briefly touched.     

An information-theoretic approach to evaluating the size and temperature dependence of metabolic rate

The effects of body mass and temperature on metabolic rate (MR) are among the most widely examined physiological relationships. Recently, these relationships have been incorporated into the metabolic theory of ecology (MTE) that links the ecology of populations, communities and ecosystems to the MR of individual organisms. The fundamental equation of MTE derives the relation between mass and MR using first principles and predicts the temperature dependence of MR based on biochemical kinetics. It is a deliberately simple, zeroth-order approximation that represents a baseline against which variation in real biological systems can be examined. In the present study, we evaluate the fundamental equation of MTE against other more parameter-rich models for MR using an information-theoretic approach to penalize the inclusion of additional parameters. Using a comparative database of MR measurements for 1359 species, from 11 groups ranging from prokaryotes to mammals, and spanning 16 orders of magnitude in mass and a 59°C range in body temperature, we show that differences between taxa in the mass and temperature dependence of MR are sufficiently large as to be retained in the best model for MR despite the requirement for estimation of 22 more parameters than the fundamental equation of MTE.     

Reconciling emergences: An information-theoretic approach to identify causal emergence in multivariate data

The broad concept of emergence is instrumental in various of the most challenging open scientific questions—yet, few quantitative theories of what constitutes emergent phenomena have been proposed. This article introduces a formal theory of causal emergence in multivariate systems, which studies the relationship between the dynamics of parts of a system and macroscopic features of interest. Our theory provides a quantitative definition of downward causation, and introduces a complementary modality of emergent behaviour—which we refer to as causal decoupling. Moreover, the theory allows practical criteria that can be efficiently calculated in large systems, making our framework applicable in a range of scenarios of practical interest. We illustrate our findings in a number of case studies, including Conway’s Game of Life, Reynolds’ flocking model, and neural activity as measured by electrocorticography.     

Rigid domains in proteins: An algorithmic approach to their identification

A rigid domain, defined here as a tertiary structure common to two or more different protein conformations, can be identified numerically from atomic coordinates by finding sets of residues, one in each conformation, such that the distance between any two residues within the set belonging to one conformation is the same as the distance between the two structurally equivalent residues within the set belonging to any other conformation. The distance between two residues is taken to be the distance between their respective α carbon atoms. With the methods of this paper we have found in the deoxy and oxy conformations of the human hemoglobin α₁β₁ dimer a rigid domain closely related to that previously identified by Baldwin and Chothia (J. Mol. Biol. 129:175–220,1979). We provide two algorithms, both using the difference-distance matrix, with which to search for rigid domains directly from atomic coordinates. The first finds all rigid domains in a protein but has storage and processing demands that become prohibitively large with increasing protein size. The second, although not necessarily finding every rigid domain, is computationally tractable for proteins of any size. Because of its efficiency we are able to search protein conformations recursively for groups of non-intersecting domains. Different protein conformations, when aligned by superimposing their respective domain structures; can be examined for structural differences in regions complementing a rigid domain. © 1995 Wiley-Liss, Inc.     

A fish-based multimetric index for Brazilian savanna streams

Multimetric indices (MMI) have been widely used to assess ecosystem conditions because they are low-cost, employ a rapid field method, and can incorporate various biological metrics at different levels of biological organization. Our objective was to create a fish-based multimetric index applicable to all streams of the Brazilian savanna biome (Cerrado), the second largest biome in Brazil and deemed a global biodiversity hotspot. We evaluated 156 sites in two river basins (Paraná and São Francisco) and selected metrics capable of distinguishing stream-sites across a gradient of anthropogenic disturbances. We employed two different MMI approaches to determine if an MMI based on natural variation-adjusted metrics performed better than one based on unadjusted metrics. In addition, we assessed the performance of the two final MMIs and their sensitivity to anthropogenic pressures at local (LDI), catchment (CDI) and both scales integrated (IDI). Finally, we employed the power of a probability sample survey design to infer headwater stream conditions across a hydrologic region of approximately 47,000 km². Our final MMI for Brazilian savanna streams included six metrics: % common species; % characiform individuals; % loricariid individuals; % trichomycterid individuals; % invertivore species, and % Poecilia reticulata individuals. MMI₁ (unadjusted metrics) performed better than MMI₂ (natural variation-adjusted metrics) in discriminating least- and most-disturbed sites, but MMI₂ distinguished intermediate from most-disturbed sites better than MMI₁. Both indices were negatively correlated with the CDI scores; however, only MMI₂ was negatively correlated with the IDI scores. We inferred that 709 km (9.35%) of streams in the studied hydrologic region were in good condition, 8115 km (82.73%) were intermediate, and 641 km (7.91%) were in poor condition. We conclude that the MMIs proposed in this study have great potential for widespread application because they integrate data from two of the most important Brazilian river basins included in a biome that represents more than 20% of the country. Furthermore, the metrics retained in the indexes are easy to access with a rapid low-cost field method. However, their feasibility in areas influenced by mining, as well as in different biomes, should be tested.     

An improved macroinvertebrate multimetric index for the assessment of wadeable streams in the neotropical savanna

Multimetric indices (MMIs) have been successfully used to assess ecological conditions in freshwater ecosystems worldwide, and provide an important management tool especially in countries where biological indicators are fostered by environmental regulations. Nonetheless, for the neotropics, the few published papers are limited to small local scales and lack standardized sampling protocols. To fill the gaps left by previous studies, we propose a stream MMI that reflects anthropogenic impacts by using macroinvertebrate assemblage metrics from a data set of 190 sites collected from four hydrologic units in the Paraná and São Francisco River Basins, southeastern Brazil. Sites were selected through use of a probabilistic survey design allowing us to infer ecological condition to the total of 9432 kilometers of wadeable streams in the target population in the four hydrologic units. We used a filtering process to determine the least- and most-disturbed sites based on their water quality, physical habitat structure, and land use. To develop the MMI, we followed a stepwise procedure to screen our initial set of biological metrics for influence of natural variation, responsiveness and discriminance to disturbances, sampling variability, and redundancy. The final MMI is the sum of 7 scaled assemblage metrics describing different aspects of macroinvertebrate assemblage characteristics: Ephemeroptera richness, % Gastropoda individuals, Shannon-Wiener diversity index, % sensitive taxa richness, % scraper individuals, temporarily attached taxa richness, and gill respiration taxa richness. The MMI clearly distinguished the least-disturbed sites from the most-disturbed sites and showed a significant negative response to anthropogenic stressors. Of the total length of wadeable streams in the study area, 38%, 35%, and 27% were classified by the MMI as being in good, fair, and poor condition, respectively. By reducing the subjectivity of site selection, rigorously selecting the set of reference sites, and following a standardized metric screening method, we developed a robust MMI to assess and monitor ecological condition in neotropical savanna streams. This improved MMI provides an effective ecological tool to guide decision makers and managers in developing and implementing improved, cost-effective environmental policies, regulations, and monitoring of those systems.     

An algorithmic approach to constructing the on-line estimation system for the specific growth rate

The objective of this article is to propose an algorithm for the on-line estimation of the specific growth rate in a batch or a fed-batch fermentation process. The algorithm shows the practical procedure for the estimation method utilizing the macroscopic balance and the extended Kalman filter. A number of studies of the on line estimation have been presented. However, there are few studies discussing about the selection of the observed variables and for the tuning of some parameters of the extended Kalman filter, such as covariance matrix and initial values of the state.The beginning of this article is devoted to explain the selection of the observed variable. This information is very important in terms of the practical know-how for using technique. It is discovered that the condition number is a practically useful and valid criterion for number is a practically useful and valid criterion for choosing the variable to be observed.Next, when the extended Kalman filter in applied to the online estimation of the specific growth rate, which is directly unmeasurable, criteria for judging the validity of the estimated value from the observed data are proposed. Based on the proposed criterial, the system equation of the specific growth rate is selected and initial value of the state variable and covariance matrix of the system noises are adjusted. From many experiments, it is certified that the specific growth rate in the batch or fed -batch fermentation can be estimated accurately by means of the algorithm proposed here. In these experiments, that is, when the cell concentration is measured directly, the extended Kalman filter using the convariance matrix with a constant element can estimate more accurately values of the specific growth rate than the adaptive extended Kalman filter does.     

An index approach to assess nitrogen losses to the environment

Nitrogen (N) losses from agriculture are negatively impacting groundwater, air, and surface water quality. New tools are needed to quickly assess these losses and provide nutrient managers and conservationists with effective tools to assess the effects of current and alternative management practices on N loss pathways. A new N-Index tool was developed in spreadsheet format, allowing prompt assessments of management practices on agricultural N losses. The N-Index tool was compared with experimental field data and shown to estimate the effects of management practices on N loss pathways (probability, P < 0.001). The N-Index correctly assessed the nitrate nitrogen (NO₃-N) leaching losses when tested against measured NO₃-N leaching data and atmospheric N losses collected over multiple years (annual basis) and locations. The N-Index tool was developed with international cooperation from several countries and there is potential to use this tool at the international level.     

An algorithmic approach to diagnose haematolymphoid neoplasms in effusion by combining morphology,immunohistochemistry and molecular cytogenetics

Objective

There are limited studies of cytology diagnosis of haematopoietic and lymphoid tumours in serosal effusion except for occasional case reports. We would like to demonstrate an algorithmic approach for accurate diagnosis, especially in patients without previous history.

Methods

We reviewed 36 cases of lymphoma diagnosed in serosal effusion following an algorithmic approach. Suspected tumour cells were classified into small, intermediate and large sizes and two characteristic forms of plasmacytoid and Reed Sternberg‐like on smears (step 1), followed by utilising panels of immunohistochemical markers and Epstein‐Barr encoding region in situ hybridisation on cell blocks (step 2). A panel of CD3, CD20 and Ki‐67 formed the basic workup, followed by pertinent batteries of immunostaining. Molecular tests were applied in 22 selected cases by fluorescence in situ hybridisation (step 3).

Results

There were 15 diffuse large B‐cell lymphomas; 12 plasma cell myelomas; two mantle cell lymphomas; one anaplastic large cell lymphoma ALK +; one small lymphocytic lymphoma; one plasmablastic lymphoma; one peripheral T‐cell lymphoma, not otherwise specified, one extranodal NK/T‐cell lymphoma, nasal type and two T‐cell lymphoblastic lymphomas. 14 cases with previous history had complete concordance in immunophenotype between cytology and histology. Another 14 cases were primarily diagnosed in patients with initial symptom of effusion based on immunophenotyping and cytogenetic test in selected cases. Eight cases were diagnosed based on morphology alone.

Conclusion

An algorithmic approach based on morphology and immunohistochemistry is the key to making an accurate diagnosis of haematopoietic and lymphoid tumours in effusion. A molecular test is also important for confirmation and prognostic prediction. We reviewed 36 haematolymphoid neoplasms diagnosed in effusion including 14 cases primarily diagnosed in patients without previous history following an algorithmic approach by combining morphology, immunohistochemistry and molecular cytogenetics.     

可持续林业管理指标的探讨

可持续林业管理指标的探讨曹志平（中国农业大学资环学院生态系,北京１０００９４）钟晓东（国家环保局国际合作司,北京１０００９４）目前,林业管理的方向正在发生变化,即由过去单一的木材生产向可持续的森林生态系统管理转化,从而更好地满足人类的需求。国际社会认...     

基于底栖动物预测模型构建生物完整性指数评价河流健康

提高生物完整性指数(integrity of biotic index,IBI; 又称多参数指数multi-metric index, MMI)在时间和空间尺度的稳定性是水生态完整性评价和水环境管理实践的重要内容.本研究利用2004—2011年在浙江省多个河流采集的227个点位的底栖动物和水质理化数据,利用地理信息系统(GIS)提取样点及其对应流域的自然预测因子(如地理形态学、气候学)和土地利用数据,通过随机森林模型方法定量时间和空间尺度自然变量对生物群落的影响,构建基于预测模型控制自然因子影响方法和常规方法的MMI,并比较它们的表现力.结果表明: 基于预测模型法和基于常规方法构建的MMI的核心组成参数存在差异,随机森林模型中自然预测因子对预测模型MMI构成生物参数的解释量介于11.4%～61.2%.预测模型方法提高了MMI的精确度和准确度,但其敏感性和响应性低于常规方法的MMI.最近距离方法表明,9个评价点位和1个严重受损点位的自然属性与参照点位的自然属性存在差异性.在计算参照点位自然属性代表性范围的基础上,采用预测模型方法控制自然变量可以提高MMI的精确度和准确度,同时降低评价结果出现Ⅰ型(将健康水体误判为受损水体)或者Ⅱ型(将受损水体误判为健康水体)错误的可能性.研究结果可以为提高完整性指数评价稳定性和表现力提供方法支持.     

An efficient algorithmic approach for mass spectrometry-based disulfide connectivity determination using multi-ion analysis

Background

Determining the disulfide (S-S) bond pattern in a protein is often crucial for understanding its structure and function. In recent research, mass spectrometry (MS) based analysis has been applied to this problem following protein digestion under both partial reduction and non-reduction conditions. However, this paradigm still awaits solutions to certain algorithmic problems fundamental amongst which is the efficient matching of an exponentially growing set of putative S-S bonded structural alternatives to the large amounts of experimental spectrometric data. Current methods circumvent this challenge primarily through simplifications, such as by assuming only the occurrence of certain ion-types (b-ions and y-ions) that predominate in the more popular dissociation methods, such as collision-induced dissociation (CID). Unfortunately, this can adversely impact the quality of results.

Method

We present an algorithmic approach to this problem that can, with high computational efficiency, analyze multiple ions types (a, b, b ^o , b ^* , c, x, y, y ^o , y ^* , and z) and deal with complex bonding topologies, such as inter/intra bonding involving more than two peptides. The proposed approach combines an approximation algorithm-based search formulation with data driven parameter estimation. This formulation considers only those regions of the search space where the correct solution resides with a high likelihood. Putative disulfide bonds thus obtained are finally combined in a globally consistent pattern to yield the overall disulfide bonding topology of the molecule. Additionally, each bond is associated with a confidence score, which aids in interpretation and assimilation of the results.

Results

The method was tested on nine different eukaryotic Glycosyltransferases possessing disulfide bonding topologies of varying complexity. Its performance was found to be characterized by high efficiency (in terms of time and the fraction of search space considered), sensitivity, specificity, and accuracy. The method was also compared with other techniques at the state-of-the-art. It was found to perform as well or better than the competing techniques. An implementation is available at: http://tintin.sfsu.edu/~whemurad/disulfidebond.

Conclusions

This research addresses some of the significant challenges in MS-based disulfide bond determination. To the best of our knowledge, this is the first algorithmic work that can consider multiple ion types in this problem setting while simultaneously ensuring polynomial time complexity and high accuracy of results.

     

An efficient algorithmic approach for mass spectrometry-based disulfide connectivity determination using multi-ion analysis

Background

The identification of drug characteristics is a clinically important task, but it requires much expert knowledge and consumes substantial resources. We have developed a statistical text-mining approach (BInary Characteristics Extractor and biomedical Properties Predictor: BICEPP) to help experts screen drugs that may have important clinical characteristics of interest.

Results

BICEPP first retrieves MEDLINE abstracts containing drug names, then selects tokens that best predict the list of drugs which represents the characteristic of interest. Machine learning is then used to classify drugs using a document frequency-based measure. Evaluation experiments were performed to validate BICEPP's performance on 484 characteristics of 857 drugs, identified from the Australian Medicines Handbook (AMH) and the PharmacoKinetic Interaction Screening (PKIS) database. Stratified cross-validations revealed that BICEPP was able to classify drugs into all 20 major therapeutic classes (100%) and 157 (of 197) minor drug classes (80%) with areas under the receiver operating characteristic curve (AUC) > 0.80. Similarly, AUC > 0.80 could be obtained in the classification of 173 (of 238) adverse events (73%), up to 12 (of 15) groups of clinically significant cytochrome P450 enzyme (CYP) inducers or inhibitors (80%), and up to 11 (of 14) groups of narrow therapeutic index drugs (79%). Interestingly, it was observed that the keywords used to describe a drug characteristic were not necessarily the most predictive ones for the classification task.

Conclusions

BICEPP has sufficient classification power to automatically distinguish a wide range of clinical properties of drugs. This may be used in pharmacovigilance applications to assist with rapid screening of large drug databases to identify important characteristics for further evaluation.     

Use of macroinvertebrate-based multimetric indices for water quality evaluation in Spanish Mediterranean rivers: an intercalibration approach with the IBMWP index

For the European Parliament and Commission to implement the Water Framework Directive (WFD), the water-quality indices that are currently used in Europe need to be compared and calibrated. This will facilitate the comparative assessment of ecological status throughout the European Union. According to the WFD, biologic indices should respond consistently to human impacts, using multimetric approaches and water-quality classification boundaries adjusted to a common set of normative definitions. The European Commission has started an intercalibration exercise to review biologic indices and harmonize class boundaries. We used data from rivers in Spain to compare the IBMWP (Iberian Biological Monitoring Working Party) index, which is commonly used by water authorities in Spain and by several research centers, with the Intercalibration Common Multimetric Index (ICM-Star), which was used as a standard in the intercalibration exercise. We also used data from Spanish rivers to compare the multimetric indices ICM-7 (based on quantitative data) and ICM-9 (based on qualitative data) with the IBMWP. ICM-7 and ICM-9 were proposed by the Mediterranean Geographical Intercalibration Group (Med-GIG). Additionally, we evaluated two new multimetric indices, developed specifically for macroinvertebrate communities inhabiting Mediterranean river systems. One of these is based on quantitative data (ICM-10), while the other is based on qualitative data (ICM-11a). The results show that the IBMWP index responds well to the stressor gradient present in our data, and correlates well with ICM-Star. Moreover, the IBMWP quality class boundaries were consistent with the intercalibration requirements of the WFD. However, multimetric indices showed a more linear relation with the stressor gradient in our data, and less variation in reference values. In addition, they may provide more statistical power for detecting potential environmental impacts. Multimetric indices produced similar results for quantitative and qualitative data. Thus, ICM-10 (also named IMMi-T) and ICM-11a (also named IMMi-L) indices could be used to meet European Commission requirements for assessing the water quality in Spanish Mediterranean rivers. Electronic supplementary material  The online version of this article (doi:) contains supplementary material, which is available to authorized users. Handling editor: Joel Trexler