Structure learning in human sequential decision-making

Studies of sequential decision-making in humans frequently find suboptimal performance relative to an ideal actor that has perfect knowledge of the model of how rewards and events are generated in the environment. Rather than being suboptimal, we argue that the learning problem humans face is more complex, in that it also involves learning the structure of reward generation in the environment. We formulate the problem of structure learning in sequential decision tasks using Bayesian reinforcement learning, and show that learning the generative model for rewards qualitatively changes the behavior of an optimal learning agent. To test whether people exhibit structure learning, we performed experiments involving a mixture of one-armed and two-armed bandit reward models, where structure learning produces many of the qualitative behaviors deemed suboptimal in previous studies. Our results demonstrate humans can perform structure learning in a near-optimal manner.     

Moral and nonmoral innate constraints

Charles J. Lumsden and E.O. Wilson, in their writings together and individually, have proposed that human behaviors, whether moral or nonmoral, are governed by innate constraints (which they have termed epigenetic rules). I propose that if a genetic component of moral behavior is to be discovered, some sorting out of specifically moral from nonmoral innate constraints will be necessary. That some specifically moral innate constraits exist is evidenced by virtuous behaviors exhibited in nonhuman mammals, whose behavior is usually granted to be importantly governed by genetic factors. Propensities for such virtuous behaviors may have been passed to humans as highly conserved mammalian genes and continue to influence us. I propose that these constitute at least a rudimentary morality and may account in part for the moral intuitions. But other innate constraints which are nonmoral in nature interact with the specifically moral innate constraints and with culture to yield human moral decisions and actions. Any model which aims to identify the genetic component of moral behaviors or behaviors with moral import must provide not only a delineation of cultural causes but must also distinguish between those genetic causes which may have their origin in innate moral constraints from others which are fundamentally nonmoral because the critical faculty necessary to higher level human morality itself arises in part from innate constraints of a nonmoral type; i.e., the processes of inductive reasoning common to both ethics and science. Finally, humans who could bring the nonmoral evaluative capacities to bear upon whatever moral intuitions might be genetically conserved in mammalian heritage would have an advantage over similar beings who could not.     

Rhythm Patterns Interaction - Synchronization Behavior for Human-Robot Joint Action

Interactive behavior among humans is governed by the dynamics of movement synchronization in a variety of repetitive tasks. This requires the interaction partners to perform for example rhythmic limb swinging or even goal-directed arm movements. Inspired by that essential feature of human interaction, we present a novel concept and design methodology to synthesize goal-directed synchronization behavior for robotic agents in repetitive joint action tasks. The agents’ tasks are described by closed movement trajectories and interpreted as limit cycles, for which instantaneous phase variables are derived based on oscillator theory. Events segmenting the trajectories into multiple primitives are introduced as anchoring points for enhanced synchronization modes. Utilizing both continuous phases and discrete events in a unifying view, we design a continuous dynamical process synchronizing the derived modes. Inverse to the derivation of phases, we also address the generation of goal-directed movements from the behavioral dynamics. The developed concept is implemented to an anthropomorphic robot. For evaluation of the concept an experiment is designed and conducted in which the robot performs a prototypical pick-and-place task jointly with human partners. The effectiveness of the designed behavior is successfully evidenced by objective measures of phase and event synchronization. Feedback gathered from the participants of our exploratory study suggests a subjectively pleasant sense of interaction created by the interactive behavior. The results highlight potential applications of the synchronization concept both in motor coordination among robotic agents and in enhanced social interaction between humanoid agents and humans.     

Developmental and Genetic Constraints on Neurocranial Globularity: Insights from Analyses of Deformed Skulls and Quantitative Genetics

Neurocranial globularity is one of the few derived traits defining anatomically modern humans. Variations in this trait derive from multiple and complex interactions between portions of the brain and the size and shape of the cranial base, among other factors. Given their evolutionary and functional importance, neurocranial globularity is expected to present high genetic and developmental constraints on their phenotypic expression. Here we applied two independent approaches to investigate both types of constraints. First, we assessed if patterns of morphological integration are conserved or else disrupted on a series of artificially deformed skulls in comparison to non-deformed (ND) ones. Second, after the estimation of the genetic covariance matrix for human skull shape, we explored how neurocranial globularity would respond to putative selective events disrupting the normal morphological patterns. Simulations on these deviations were explicitly set to replicate the artificial deformation patterns in order to compare developmental and genetic constraints under the same biomechanical conditions. In general terms, our results indicate that putative developmental constraints help to preserve some aspects of normal morphological integration even in the deformed skulls. Moreover, we find that the response to selection in neurocranial globularity is pervasive. In other words, induced changes in the vault generate a global response, indicating that departures from normal patterns of neurocranial globularity are genetically constrained. In summary, our combined results suggest that neurocranial globularity behaves as a highly genetic and developmental constrained trait. Electronic supplementary material  The online version of this article (doi:) contains supplementary material, which is available to authorized users. Neus Martínez-Abadías and Rolando González-José contributed equally to this work.     

Baboons' Response Speed Is Biased by Their Moods

The affect-as-information hypothesis (e.g., Schwarz & Clore, 2003), predicts that the positive or negative valence of our mood differentially affects our processing of the details of the environment. However, this hypothesis has only been tested with mood induction procedures and fairly complex cognitive tasks in humans. Here, six baboons (Papio papio) living in a social group had free access to a computerized visual search task on which they were over-trained. Trials that immediately followed a spontaneously expressed emotional behavior were analyzed, ruling out possible biases due to induction procedures. RTs following negatively valenced behaviors are slower than those following neutral and positively valenced behaviors, respectively. Thus, moods affect the performance of nonhuman primates tested in highly automatized tasks, as it does in humans during tasks with much higher cognitive demands. These findings reveal a presumably universal and adaptive mechanism by which moods influence performance in various ecological contexts.     

Operant Procedures for Assessing Behavioral Flexibility in Rats

Executive functions consist of multiple high-level cognitive processes that drive rule generation and behavioral selection. An emergent property of these processes is the ability to adjust behavior in response to changes in one’s environment (i.e., behavioral flexibility). These processes are essential to normal human behavior, and may be disrupted in diverse neuropsychiatric conditions, including schizophrenia, alcoholism, depression, stroke, and Alzheimer’s disease. Understanding of the neurobiology of executive functions has been greatly advanced by the availability of animal tasks for assessing discrete components of behavioral flexibility, particularly strategy shifting and reversal learning. While several types of tasks have been developed, most are non-automated, labor intensive, and allow testing of only one animal at a time. The recent development of automated, operant-based tasks for assessing behavioral flexibility streamlines testing, standardizes stimulus presentation and data recording, and dramatically improves throughput. Here, we describe automated strategy shifting and reversal tasks, using operant chambers controlled by custom written software programs. Using these tasks, we have shown that the medial prefrontal cortex governs strategy shifting but not reversal learning in the rat, similar to the dissociation observed in humans. Moreover, animals with a neonatal hippocampal lesion, a neurodevelopmental model of schizophrenia, are selectively impaired on the strategy shifting task but not the reversal task. The strategy shifting task also allows the identification of separate types of performance errors, each of which is attributable to distinct neural substrates. The availability of these automated tasks, and the evidence supporting the dissociable contributions of separate prefrontal areas, makes them particularly well-suited assays for the investigation of basic neurobiological processes as well as drug discovery and screening in disease models.     

Optimal trajectory formation of constrained human arm reaching movements

Opening a door, turning a steering wheel, and rotating a coffee mill are typical examples of human movements that are constrained by the physical environment. The constraints decrease the mobility of the human arm and lead to redundancy in the distribution of actuator forces (either joint torques or muscle forces). Due to this actuator redundancy, there is an infinite number of ways to form a specific arm trajectory. However, humans form trajectories in a unique way. How do humans resolve the redundancy of the constrained motions and specify the hand trajectory? To investigate this problem, we examine human arm movements in a crank-rotation task. To explain the trajectory formation in constrained point-to-point motions, we propose a combined criterion minimizing the hand contact force change and the actuating force change over the course of movement. Our experiments show a close matching between predicted and experimental data.     

Olfactory conditioning of positive performance in humans

Olfactory conditioning effects have been widely demonstrated in the animal literature but more seldom in human populations and rarely of consciously controlled human behaviors. Building upon previous work on negative performance, we report the first experimental evidence that odors can be used effectively in a classical conditioning paradigm to positively influence human behavior. In the present study, underachieving schoolchildren experienced unexpected success at a paper-and-pencil task in the presence of an ambient odor. When they later experienced the same odor again, performance on other tasks was superior to that of relevant control groups. These data substantially extend previous results on human olfactory classical conditioning and show that odors potentially can be used to exert positive influences on human behavior.     

Formal Models of the Network Co-occurrence Underlying Mental Operations

Systems neuroscience has identified a set of canonical large-scale networks in humans. These have predominantly been characterized by resting-state analyses of the task-unconstrained, mind-wandering brain. Their explicit relationship to defined task performance is largely unknown and remains challenging. The present work contributes a multivariate statistical learning approach that can extract the major brain networks and quantify their configuration during various psychological tasks. The method is validated in two extensive datasets (n = 500 and n = 81) by model-based generation of synthetic activity maps from recombination of shared network topographies. To study a use case, we formally revisited the poorly understood difference between neural activity underlying idling versus goal-directed behavior. We demonstrate that task-specific neural activity patterns can be explained by plausible combinations of resting-state networks. The possibility of decomposing a mental task into the relative contributions of major brain networks, the "network co-occurrence architecture" of a given task, opens an alternative access to the neural substrates of human cognition.     

An Examination of the Generalizability of Motor Costs

Most approaches to understanding human motor control assume that people maximize their rewards while minimizing their motor efforts. This tradeoff between potential rewards and a sense of effort is quantified with a cost function. While the rewards can change across tasks, our sense of effort is assumed to remain constant and characterize how the nervous system organizes motor control. As such, when a proposed cost function compares well with data it is argued to be the underlying cause of a motor behavior, and not simply a fit to the data. Implicit in this proposition is the assumption that this cost function can then predict new motor behaviors. Here we examined this idea and asked whether an inferred cost function in one setting could explain subject’s behavior in settings that differed dynamically but had identical rewards. We found that the pattern of behavior observed across settings was similar to our predictions of optimal behavior. However, we could not conclude that this behavior was consistent with a conserved sense of effort. These results suggest that the standard forms for quantifying cost may not be sufficient to accurately examine whether or not human motor behavior abides by optimality principles.     

Mixing pleasures: Review of the effects of drugs on sex behavior in humans and animal models

Drugs of abuse act on the brain circuits mediating motivation and reward associated with natural behaviors. There is ample evidence that drugs of abuse impact male and female sexual behavior. First, the current review discusses the effect of drugs of abuse on sexual motivation and performance in male and female humans. In particular, we discuss the effects of commonly abused drugs including psychostimulants, opiates, marijuana/THC, and alcohol. In general, drug use affects sexual motivation, arousal, and performance and is commonly associated with increased sexual risk behaviors. Second, studies on effects of systemic administration of drugs of abuse on sexual behavior in animals are reviewed. These studies analyze the effects on sexual performance and motivation but do not investigate the effects of drugs on risk-taking behavior, creating a disconnect between human and animal studies. For this reason, we discuss two studies that focus on the effects of alcohol and methamphetamine on inhibition of maladaptive sex-seeking behaviors in rodents. Third, this review discusses potential brain areas where drugs of abuse may be exerting their effect on sexual behavior with a focus on the mesolimbic system as the site of action. Finally, we discuss recent studies that have brought to light that sexual experience in turn can affect drug responsiveness, including a sensitized locomotor response to amphetamine in female and male rodents as well as enhanced drug reward in male rats.     

Prospective Optimization with Limited Resources

The future is uncertain because some forthcoming events are unpredictable and also because our ability to foresee the myriad consequences of our own actions is limited. Here we studied how humans select actions under such extrinsic and intrinsic uncertainty, in view of an exponentially expanding number of prospects on a branching multivalued visual stimulus. A triangular grid of disks of different sizes scrolled down a touchscreen at a variable speed. The larger disks represented larger rewards. The task was to maximize the cumulative reward by touching one disk at a time in a rapid sequence, forming an upward path across the grid, while every step along the path constrained the part of the grid accessible in the future. This task captured some of the complexity of natural behavior in the risky and dynamic world, where ongoing decisions alter the landscape of future rewards. By comparing human behavior with behavior of ideal actors, we identified the strategies used by humans in terms of how far into the future they looked (their “depth of computation”) and how often they attempted to incorporate new information about the future rewards (their “recalculation period”). We found that, for a given task difficulty, humans traded off their depth of computation for the recalculation period. The form of this tradeoff was consistent with a complete, brute-force exploration of all possible paths up to a resource-limited finite depth. A step-by-step analysis of the human behavior revealed that participants took into account very fine distinctions between the future rewards and that they abstained from some simple heuristics in assessment of the alternative paths, such as seeking only the largest disks or avoiding the smaller disks. The participants preferred to reduce their depth of computation or increase the recalculation period rather than sacrifice the precision of computation.     

A bird's eye view of neural circuit formation

Neural circuits underlying complex learned behaviors, such as speech in humans, develop under genetic constraints and in response to environmental influences. Little is known about the rules and mechanisms through which such circuits form. We argue that songbirds, with their discrete and well studied neural pathways underlying a complex and naturally learned behavior, provide a powerful model for addressing these questions. We briefly review current knowledge of how the song circuit develops during learning and discuss new possibilities for advancing the field given recent technological advances.     

Standardization and application of metrics to quantify human‐interaction behaviors by the bottlenose dolphin (Tursiops spp.)

The conditioning of dolphins to human‐interaction behaviors has been documented in several areas worldwide. However, the metrics used to report human‐interaction behaviors vary among studies, making comparison across study areas difficult. The purpose of this study was to develop standard metrics for reporting human‐interaction behaviors and utilize these metrics to quantify the prevalence of human‐interaction behaviors by common bottlenose dolphins (Tursiops truncatus) near Savannah, Georgia. The four metrics used were percentage of days with human‐interaction behaviors, percentage of sightings with human‐interaction behaviors, percentage of the catalog that interacted with humans, and spatial extent of human‐interaction behaviors. Human‐interaction behaviors were observed on 69.6% of days and 23.5% of sightings near Savannah. In addition, 20.1% of the animals in the catalog were observed interacting with humans. These rates are much higher than those found in other areas with known issues with human‐interaction behaviors. These behaviors were observed across an area of 272.6 km², which is larger than other reported areas. The four metrics used in this study proved to be a valuable way to report human‐interaction behaviors, and their use is recommended for future studies to allow for comparison among areas.     

Effects of a Foot Placement Constraint on Use of Motor Equivalence during Human Hopping

Humans can robustly locomote over complex terrains even while simultaneously attending to other tasks such as accurate foot placement on the ground. We investigated whether subjects would exploit motor redundancy across the joints of the leg to stabilize overall limb kinematics when presented with a hopping task that constrained foot placement position. Subjects hopped in place on one leg (2.2 Hz) while having to place their foot into one of three target sizes upon landing (0.250, 0.063, 0.010 m²). As takeoff and landing angles are critical to this task performance, we hypothesized smaller target sizes would increase the need to stabilize (i.e., make more consistent) the leg orientation through motor equivalent combinations of segment angles. As it was not critical to the targeting task, we hypothesized no changes for leg length stabilization across target size. With smaller target sizes, we saw total segment angle variance increase due to greater signal-dependent noise associated with an increased activation of leg extensor muscles (medial and lateral gastrocnemius, vastus medialis, vastus lateralis and rectus femoris). At smaller target sizes, more segment angle variance was aligned to kinematic deviations with the goal of maintaining leg orientation trajectory. We also observed a decrease in the variance structure for stabilizing leg length at the smallest target conditions. This trade-off effect is explained by the nearly orthogonal relationship between the two goal-equivalent manifolds for leg length vs. leg orientation stabilization. Our results suggest humans increasingly rely on kinematic redundancy in their legs to achieve robust, consistent locomotion when faced with novel conditions that constrain performance requirements. These principles may generalize to other human locomotor gaits and provide important insights into the control of the legs during human walking and running.     

Feature blindness: A challenge for understanding and modelling visual object recognition

Humans rely heavily on the shape of objects to recognise them. Recently, it has been argued that Convolutional Neural Networks (CNNs) can also show a shape-bias, provided their learning environment contains this bias. This has led to the proposal that CNNs provide good mechanistic models of shape-bias and, more generally, human visual processing. However, it is also possible that humans and CNNs show a shape-bias for very different reasons, namely, shape-bias in humans may be a consequence of architectural and cognitive constraints whereas CNNs show a shape-bias as a consequence of learning the statistics of the environment. We investigated this question by exploring shape-bias in humans and CNNs when they learn in a novel environment. We observed that, in this new environment, humans (i) focused on shape and overlooked many non-shape features, even when non-shape features were more diagnostic, (ii) learned based on only one out of multiple predictive features, and (iii) failed to learn when global features, such as shape, were absent. This behaviour contrasted with the predictions of a statistical inference model with no priors, showing the strong role that shape-bias plays in human feature selection. It also contrasted with CNNs that (i) preferred to categorise objects based on non-shape features, and (ii) increased reliance on these non-shape features as they became more predictive. This was the case even when the CNN was pre-trained to have a shape-bias and the convolutional backbone was frozen. These results suggest that shape-bias has a different source in humans and CNNs: while learning in CNNs is driven by the statistical properties of the environment, humans are highly constrained by their previous biases, which suggests that cognitive constraints play a key role in how humans learn to recognise novel objects.     

Oxytocin Receptor Gene Polymorphisms Are Associated with Human Directed Social Behavior in Dogs (Canis familiaris)

The oxytocin system has a crucial role in human sociality; several results prove that polymorphisms of the oxytocin receptor gene are related to complex social behaviors in humans. Dogs'' parallel evolution with humans and their adaptation to the human environment has made them a useful species to model human social interactions. Previous research indicates that dogs are eligible models for behavioral genetic research, as well. Based on these previous findings, our research investigated associations between human directed social behaviors and two newly described (−212AG, 19131AG) and one known (rs8679684) single nucleotide polymorphisms (SNPs) in the regulatory regions (5′ and 3′ UTR) of the oxytocin receptor gene in German Shepherd (N = 104) and Border Collie (N = 103) dogs. Dogs'' behavior traits have been estimated in a newly developed test series consisting of five episodes: Greeting by a stranger, Separation from the owner, Problem solving, Threatening approach, Hiding of the owner. Buccal samples were collected and DNA was isolated using standard protocols. SNPs in the 3′ and 5′ UTR regions were analyzed by polymerase chain reaction based techniques followed by subsequent electrophoresis analysis. The gene–behavior association analysis suggests that oxytocin receptor gene polymorphisms have an impact in both breeds on (i) proximity seeking towards an unfamiliar person, as well as their owner, and on (ii) how friendly dogs behave towards strangers, although the mediating molecular regulatory mechanisms are yet unknown. Based on these results, we conclude that similarly to humans, the social behavior of dogs towards humans is influenced by the oxytocin system.     

A weighted constraint satisfaction approach to human goal-directed decision making

When we plan for long-range goals, proximal information cannot be exploited in a blindly myopic way, as relevant future information must also be considered. But when a subgoal must be resolved first, irrelevant future information should not interfere with the processing of more proximal, subgoal-relevant information. We explore the idea that decision making in both situations relies on the flexible modulation of the degree to which different pieces of information under consideration are weighted, rather than explicitly decomposing a problem into smaller parts and solving each part independently. We asked participants to find the shortest goal-reaching paths in mazes and modeled their initial path choices as a noisy, weighted information integration process. In a base task where choosing the optimal initial path required weighting starting-point and goal-proximal factors equally, participants did take both constraints into account, with participants who made more accurate choices tending to exhibit more balanced weighting. The base task was then embedded as an initial subtask in a larger maze, where the same two factors constrained the optimal path to a subgoal, and the final goal position was irrelevant to the initial path choice. In this more complex task, participants’ choices reflected predominant consideration of the subgoal-relevant constraints, but also some influence of the initially-irrelevant final goal. More accurate participants placed much less weight on the optimality-irrelevant goal and again tended to weight the two initially-relevant constraints more equally. These findings suggest that humans may rely on a graded, task-sensitive weighting of multiple constraints to generate approximately optimal decision outcomes in both hierarchical and non-hierarchical goal-directed tasks.     

Do Brain Networks Evolve by Maximizing Their Information Flow Capacity?

We propose a working hypothesis supported by numerical simulations that brain networks evolve based on the principle of the maximization of their internal information flow capacity. We find that synchronous behavior and capacity of information flow of the evolved networks reproduce well the same behaviors observed in the brain dynamical networks of Caenorhabditis elegans and humans, networks of Hindmarsh-Rose neurons with graphs given by these brain networks. We make a strong case to verify our hypothesis by showing that the neural networks with the closest graph distance to the brain networks of Caenorhabditis elegans and humans are the Hindmarsh-Rose neural networks evolved with coupling strengths that maximize information flow capacity. Surprisingly, we find that global neural synchronization levels decrease during brain evolution, reflecting on an underlying global no Hebbian-like evolution process, which is driven by no Hebbian-like learning behaviors for some of the clusters during evolution, and Hebbian-like learning rules for clusters where neurons increase their synchronization.     

Learning Universal Computations with Spikes

Providing the neurobiological basis of information processing in higher animals, spiking neural networks must be able to learn a variety of complicated computations, including the generation of appropriate, possibly delayed reactions to inputs and the self-sustained generation of complex activity patterns, e.g. for locomotion. Many such computations require previous building of intrinsic world models. Here we show how spiking neural networks may solve these different tasks. Firstly, we derive constraints under which classes of spiking neural networks lend themselves to substrates of powerful general purpose computing. The networks contain dendritic or synaptic nonlinearities and have a constrained connectivity. We then combine such networks with learning rules for outputs or recurrent connections. We show that this allows to learn even difficult benchmark tasks such as the self-sustained generation of desired low-dimensional chaotic dynamics or memory-dependent computations. Furthermore, we show how spiking networks can build models of external world systems and use the acquired knowledge to control them.