An information-theoretic approach to curiosity-driven reinforcement learning

We provide a fresh look at the problem of exploration in reinforcement learning, drawing on ideas from information theory. First, we show that Boltzmann-style exploration, one of the main exploration methods used in reinforcement learning, is optimal from an information-theoretic point of view, in that it optimally trades expected return for the coding cost of the policy. Second, we address the problem of curiosity-driven learning. We propose that, in addition to maximizing the expected return, a learner should choose a policy that also maximizes the learner's predictive power. This makes the world both interesting and exploitable. Optimal policies then have the form of Boltzmann-style exploration with a bonus, containing a novel exploration-exploitation trade-off which emerges naturally from the proposed optimization principle. Importantly, this exploration-exploitation trade-off persists in the optimal deterministic policy, i.e., when there is no exploration due to randomness. As a result, exploration is understood as an emerging behavior that optimizes information gain, rather than being modeled as pure randomization of action choices.     

Climate, phylogeny and the ecological distribution of C4 grasses

'C4 photosynthesis' refers to a suite of traits that increase photosynthesis in high light and high temperature environments. Most C4 plants are grasses, which dominate tropical and subtropical grasslands and savannas but are conspicuously absent from cold growing season climates. Physiological attributes of C4 photosynthesis have been invoked to explain C4 grass biogeography; however, the pathway evolved exclusively in grass lineages of tropical origin, suggesting that the prevalence of C4 grasses in warm climates could be due to other traits inherited from their non-C4 ancestors. Here we investigate the relative influences of phylogeny and photosynthetic pathway in determining the ecological distributions of C4 grasses in Hawaii. We find that the restriction of C4 grasses to warmer areas is due largely to their evolutionary history as members of a warm-climate grass clade, but that the pathway does appear to confer a competitive advantage to grasses in more arid environments.     

Body Mass Index and Physical Function in Older Women

Objective: We modified existing standardized measurement tools in the Physical Performance Test and tasks from the Frailty and Injuries: Cooperative Studies of Intervention Technique Study to evaluate physical function in older women. Our objectives were (1) to characterize physical function themes based on combinations of tasks (deriving factors or components) and (2) to quantify the correlation between derived factors and body mass index (BMI). Research Methods and Procedures: Nutrition risk screens from enrollees in a Medicare‐managed risk program served as the sampling frame. To obtain adequate representation for a range of BMI, a random sample was obtained of 90 women from the following BMI strata: BMI, 22 to <27 kg/m²; BMI, 27 to <30 kg/m²; and BMI, ≥30 kg/m². Subjects were asked to perform a series of 18 functional tasks during a home visit. Results: The mean age was similar in the three BMI groups with an overall mean age of 71 ± 4.9 years (SD). Factors characterized by lower‐body function, upper‐body function, coordination, and strength were responsible for 30%, 11%, 9%, and 9% of the variance in task scores, respectively. BMI, controlling for age, explained 5%, 14%, 3%, and 0% of the variation in these factors, respectively. Higher BMI is associated significantly with poorer upper‐ and lower‐body function but is not associated significantly to strength or coordination. Discussion: Higher BMI seems to differentially impede specific aspects of physical function, especially upper‐body function, and to a lesser extent, lower‐body function. BMI does not seem to be associated with levels of coordination or strength. Better understanding of how BMI impacts physical function will aid in the design of interventions to promote independent living in elderly, obese women.     

The early adaptive evolution of calmodulin

Interaction between gene duplication and natural selection in molecular evolution was investigated utilizing a phylogenetic tree constructed by the parsimony procedure from amino acid sequences of 50 calmodulin- family protein members. The 50 sequences, belonging to seven protein lineages related by gene duplication (calmodulin itself, troponin-C, alkali and regulatory light chains of myosin, parvalbumin, intestinal calcium-binding protein, and glial S-100 phenylalanine-rich protein), came from a wide range of eukaryotic taxa and yielded a denser tree (more branch points within each lineage) than in earlier studies. Evidence obtained from the reconstructed pattern of base substitutions and deletions in these ancestral loci suggests that, during the early history of the family, selection acted as a transforming force on expressed genes among the duplicates to encode molecular sites with new or modified functions. In later stages of descent, however, selection was a conserving force that preserved the structures of many coadapted functional sites. Each branch of the family was found to have a unique average tempo of evolutionary change, apparently regulated through functional constraints. Proteins whose functions dictate multiple interaction with several other macromolecules evolved more slowly than those which display fewer protein-protein and protein-ion interactions, e.g., calmodulin and next troponin-C evolved at the slowest average rates, whereas parvalbumin evolved at the fastest. The history of all lineages, however, appears to be characterized by rapid rates of evolutionary change in earlier periods, followed by slower rates in more recent periods. A particularly sharp contrast between such fast and slow rates is found in the evolution of calmodulin, whose rate of change in earlier eukaryotes was manyfold faster than the average rate over the past 1 billion years. In fact, the amino acid replacements in the nascent calmodulin lineage occurred at residue positions that in extant metazoans are largely invariable, lending further support to the Darwinian hypothesis that natural selection is both a creative and a conserving force in molecular evolution.      

Release of growth hormone from ox pituitary slices after pronase treatment

Proteolytic enzymes have been used both to modify properties of the cell membrane and to dissociate cells from many tissues including pituitary (4, 5, 12). Exposure of secretory tissues to pronase can alter their secretory response. Thus incubation of pancreatic islets of Langerhans in the presence of low concentrations of pronase increased the subsequent release of insulin in the presence of stimulatory and nonstimulatory glucose concentrations (7). The purpose of the present investigation was to determine whether low concentrations of pronase have the same stimulatory effect on the release of a pituitary hormone, growth hormone. Such an effect on hormone release could be of some importance in view of the development of dissociated cell systems as models for the study of the control of hormone release (4, 5).     

The presence of polyadenylate sequences in the ribonucleic acid of a higher plant

Rice callus tissue in suspension culture was labeled $\text{in}$$\text{vivo}$ with ³²PO₄^?3. Approximately 4% of the total RNA extracted from the tissue could be bound to nitrocellulose filters in the presence of 0.5 M KCl. The base composition of this material was: A, 30.6; U, 32.7; G, 18.0; and C, 18.6%. If the RNA was digested with both T₁ and pancreatic ribonucleases prior to binding, the base composition of the bound material was: A, 97.8; U, 0.0; G, 1.6; and C, 0.5%. About 14% of the RNA isolated from purified polyribosomes could be bound to filters in 0.5 M KCl. It is concluded that higher plant RNA contains polyadenylate sequences.     

Isolation and Partial Characterization of a Mycotoxin from Penicillium roqueforti

Extracts of pure cultures of Penicillium roqueforti isolated from toxic feed samples and of P. roqueforti NRRL 849 were lethal to rats by either intraperitoneal or oral administration. Purification studies guided by this test led to the isolation of a major toxin which showed intraperitoneal and oral median lethal dose values in weanling rats of 11 and 115 mg/kg, respectively. Partial characterization of the crystalline compound, C(17)H(20)O(6), by infrared, ultra violet, PMR, and mass spectroscopy, and by several chemical transformations indicated the presence of three C-methyl substituents plus one acetoxy, one aldehyde, and one alpha,beta-unsaturated ketone group. Two oxygen atoms are present either in epoxide or ether form.