Model-based hierarchical reinforcement learning and human action control

Recent work has reawakened interest in goal-directed or ‘model-based’ choice, where decisions are based on prospective evaluation of potential action outcomes. Concurrently, there has been growing attention to the role of hierarchy in decision-making and action control. We focus here on the intersection between these two areas of interest, considering the topic of hierarchical model-based control. To characterize this form of action control, we draw on the computational framework of hierarchical reinforcement learning, using this to interpret recent empirical findings. The resulting picture reveals how hierarchical model-based mechanisms might play a special and pivotal role in human decision-making, dramatically extending the scope and complexity of human behaviour.     

The psychophysiology of the emotions and personality

According to the need-informational approach the need is specific (essential) force of living organisms, procuring their connections with external environment for self-preservation and self-development, a source of living systems' activity in the surrounding world. Emotion-reflection in human and higher animals' brain of an urgent need and of probability (possibility) of its satisfaction. Emotion in its neurophysiological sense is an active state of a system of specialized brain structures, prompting the subject to change his behaviour so that to maximize or to minimize this state which determines the role of emotions in organization of goal-directed behaviour. Personality is individual unique composition and internal hierarchy of basic (vital, social, ideal) needs of a given human being, including their varieties of preservation and development, "for me" and "for others". The most important personality characteristic is which of those needs and for how long are dominant in the hierarchy of coexisting motives, which need is "supplied" by creative intuition (superconsciousness, according to K. S. Stanislavski?'s terminology).     

Spiking neural networks with different reinforcement learning (RL) schemes in a multiagent setting

This paper investigates the effectiveness of spiking agents when trained with reinforcement learning (RL) in a challenging multiagent task. In particular, it explores learning through reward-modulated spike-timing dependent plasticity (STDP) and compares it to reinforcement of stochastic synaptic transmission in the general-sum game of the Iterated Prisoner's Dilemma (IPD). More specifically, a computational model is developed where we implement two spiking neural networks as two "selfish" agents learning simultaneously but independently, competing in the IPD game. The purpose of our system (or collective) is to maximise its accumulated reward in the presence of reward-driven competing agents within the collective. This can only be achieved when the agents engage in a behaviour of mutual cooperation during the IPD. Previously, we successfully applied reinforcement of stochastic synaptic transmission to the IPD game. The current study utilises reward-modulated STDP with eligibility trace and results show that the system managed to exhibit the desired behaviour by establishing mutual cooperation between the agents. It is noted that the cooperative outcome was attained after a relatively short learning period which enhanced the accumulation of reward by the system. As in our previous implementation, the successful application of the learning algorithm to the IPD becomes possible only after we extended it with additional global reinforcement signals in order to enhance competition at the neuronal level. Moreover it is also shown that learning is enhanced (as indicated by an increased IPD cooperative outcome) through: (i) strong memory for each agent (regulated by a high eligibility trace time constant) and (ii) firing irregularity produced by equipping the agents' LIF neurons with a partial somatic reset mechanism.     

Changes in the emotionally conditioned behavior of rats under the influence of the hexapeptide fragment GLLDLK of the protein inhibitor of diazepam binding]

It is shown that suboccipital injection of 100 micrograms of the gexapeptide GLLDLK (the fragment of endogenous peptide--the inhibitor of diazepam binding) modified (for 1-3 days) the emotionally conditioned behaviour of the rats (the test of "emotional resonance"). This modification was realized in some reinforcement of different behavioural patterns and had signs of anxiety and depression. In the test "social hierarchy" the injection of GLLDLK didn't change significantly the hierarchy in the whole rat society, but in the recipient behaviour the exploratory activity has been changed, the time of grooming increased and the quantity of social contacts decreased.     

A syndecan-4 binding peptide derived from laminin 5 uses a novel PKCε pathway to induce cross-linked actin network (CLAN) formation in human trabecular meshwork (HTM) cells

In this study, we examined the role(s) of syndecan-4 in regulating the formation of an actin geodesic dome structure called a cross-linked actin network (CLAN) in which syndecan-4 has previously been localized. CLANs have been described in several different cell types, but they have been most widely studied in human trabecular meshwork (HTM) cells where they may play a key role in controlling intraocular pressure by regulating aqueous humor outflow from the eye. In this study we show that a loss of cell surface synedcan-4 significantly reduces CLAN formation in HTM cells. Analysis of HTM cultures treated with or without dexamethasone shows that laminin 5 deposition within the extracellular matrix is increased by glucocorticoid treatment and that a laminin 5-derived, syndecan-4-binding peptide (PEP75), induces CLAN formation in TM cells. This PEP75-induced CLAN formation was inhibited by heparin and the broad spectrum PKC inhibitor Ro-31–7549. In contrast, the more specific PKCα inhibitor Gö 6976 had no effect, thus excluding PKCα as a downstream effector of syndecan-4 signaling. Analysis of PKC isozyme expression showed that HTM cells also expressed both PKCγ and PKCε. Cells treated with a PKCε agonist formed CLANs while a PKCα?γ agonist had no effect. These data suggest that syndecan-4 is essential for CLAN formation in HTM cells and that a novel PKCε-mediated signaling pathway can regulate formation of this unique actin structure.     

MPF and cyclin: modelling of the cell cycle minimum oscillator

The cell cycle appears to be controlled by the interplay between two protein complexes, MPF and cyclin. Their interactions play an essential role in the structure of the oscillator governing the cell cycle. There seems to be no general agreement on this latter crucial question. Two different mechanisms are proposed: (i) cyclin and p34 kinase combine to form an oligomer with MPF activity; (ii) cyclin enzymatically activates the passage from inactive pre-MPF to active MPF, with the postulate that MPF initiates cyclin degradation. We have modelled these two hypotheses to see whether both actually lead to oscillatory behaviour. The p34-cyclin oligomerization does so without any difficulty. With the second mechanism, however, the strict hypothesis that cyclin degradation is activated by MPF must be re-examined: the system only oscillates if, in disappearing, the MPF and the cyclin react with each other stoichiometrically. The model also demonstrates that it is useless to seek cyclic control of cyclin proteolysis.     

Intelligence is universal in life.

Behaviorists assume that living things memorize random atoms of information (engrams), "reinforced" by success, just as in the neo-Darwinian mutation-selection process. On the contrary we have to recognize the existence of organized and systematic responses in the learning process (Krechevsky). The animals seek desperately to "understand the meaning" of the world around them, by widening its context. Intelligence is not an exclusive prerogative of human mind. The minds of insects operate in the same way as that of man. Even a cell has a sort of intelligence (Cuenot). Consciousness is a state of awareness associated with enhanced mental activity. It occurs also in other "higher" animals (Thorpe). However human themselves are non conscious of their basic underlying motivations. Unconscious or ineffable knowledge plays a great role in shaping our world-view and in determining our influence on the Gaian hierarchy.     

Sequence learning,prediction, and replay in networks of spiking neurons

Sequence learning, prediction and replay have been proposed to constitute the universal computations performed by the neocortex. The Hierarchical Temporal Memory (HTM) algorithm realizes these forms of computation. It learns sequences in an unsupervised and continuous manner using local learning rules, permits a context specific prediction of future sequence elements, and generates mismatch signals in case the predictions are not met. While the HTM algorithm accounts for a number of biological features such as topographic receptive fields, nonlinear dendritic processing, and sparse connectivity, it is based on abstract discrete-time neuron and synapse dynamics, as well as on plasticity mechanisms that can only partly be related to known biological mechanisms. Here, we devise a continuous-time implementation of the temporal-memory (TM) component of the HTM algorithm, which is based on a recurrent network of spiking neurons with biophysically interpretable variables and parameters. The model learns high-order sequences by means of a structural Hebbian synaptic plasticity mechanism supplemented with a rate-based homeostatic control. In combination with nonlinear dendritic input integration and local inhibitory feedback, this type of plasticity leads to the dynamic self-organization of narrow sequence-specific subnetworks. These subnetworks provide the substrate for a faithful propagation of sparse, synchronous activity, and, thereby, for a robust, context specific prediction of future sequence elements as well as for the autonomous replay of previously learned sequences. By strengthening the link to biology, our implementation facilitates the evaluation of the TM hypothesis based on experimentally accessible quantities. The continuous-time implementation of the TM algorithm permits, in particular, an investigation of the role of sequence timing for sequence learning, prediction and replay. We demonstrate this aspect by studying the effect of the sequence speed on the sequence learning performance and on the speed of autonomous sequence replay.     

A neural signature of hierarchical reinforcement learning

Human behavior displays hierarchical structure: simple actions cohere into subtask sequences, which work together to accomplish overall task goals. Although the neural substrates of such hierarchy have been the target of increasing research, they remain poorly understood. We propose that the computations supporting hierarchical behavior may relate to those in hierarchical reinforcement learning (HRL), a machine-learning framework that extends reinforcement-learning mechanisms into hierarchical domains. To test this, we leveraged a distinctive prediction arising from HRL. In ordinary reinforcement learning, reward prediction errors are computed when there is an unanticipated change in the prospects for accomplishing overall task goals. HRL entails that prediction errors should also occur in relation to task subgoals. In three neuroimaging studies we observed neural responses consistent with such subgoal-related reward prediction errors, within structures previously implicated in reinforcement learning. The results reported support the relevance of HRL to the neural processes underlying hierarchical behavior.     

Mother-infant bonding and the evolution of mammalian social relationships

A wide variety of maternal, social and sexual bonding strategies have been described across mammalian species, including humans. Many of the neural and hormonal mechanisms that underpin the formation and maintenance of these bonds demonstrate a considerable degree of evolutionary conservation across a representative range of these species. However, there is also a considerable degree of diversity in both the way these mechanisms are activated and in the behavioural responses that result. In the majority of small-brained mammals (including rodents), the formation of a maternal or partner preference bond requires individual recognition by olfactory cues, activation of neural mechanisms concerned with social reward by these cues and gender-specific hormonal priming for behavioural output. With the evolutionary increase of neocortex seen in monkeys and apes, there has been a corresponding increase in the complexity of social relationships and bonding strategies together with a significant redundancy in hormonal priming for motivated behaviour. Olfactory recognition and olfactory inputs to areas of the brain concerned with social reward are downregulated and recognition is based on integration of multimodal sensory cues requiring an expanded neocortex, particularly the association cortex. This emancipation from olfactory and hormonal determinants of bonding has been succeeded by the increased importance of social learning that is necessitated by living in a complex social world and, especially in humans, a world that is dominated by cultural inheritance.     

Effects of endothelin-1 and flunarizine on human trabecular meshwork cell contraction

Trabecular meshwork (TM) cells are now considered to play an active role in the aqueous outflow mechanism because they exhibit smooth muscle-like contractile properties. Endothelin-1 (ET-1), a potent vasoconstrictor peptide, has been proposed to play a role in the local regulation of aqueous outflow and intraocular pressure (IOP) control. We propose an in vitro culture model as a method for the study of ET-1-induced human TM (HTM) cell contractility and for the study of whether pre-incubation with flunarizine, a calcium-channel blocker, can inhibit the action of ET-1. Experiments were performed on semiconfluent HTM cells (primary cultures established from normotensive human donor eyes) at the second passage, with phosphate-buffered saline (PBS) as a control. The contractile status of the cells was evaluated by a morphometric analysis of cell area, assuming that HTM cells in culture are able to reduce their area as a consequence of cytoskeletal contraction, rather than regulatory volume decrease. After incubation with 10 microM ET-1 for 5 mins, we observed a reduction of HTM cell area with respect to PBS-treated cells: 2425 +/- 876 microm2 versus 3125 +/- 987 microm2 (P < 0.001); and cells exhibited a retraction in shape and a reduction in number of indented profiles. Administration of ET-1 at progressively lower doses produced a corresponding lower reduction of HTM cell area, suggesting a dose-response effect of ET-1. Pre-incubation with 10 microM flunarizine strongly inhibited the ET-1 effect on HTM cell contraction: 2806 +/- 865 microm2 versus 2910 +/- 846 microm2 (P = not significant). Our data indicate that ET-1 induced a statistically significant reduction in the area of HTM cells versus controls, and that ET-1 can directly influence the aqueous outflow. Moreover, we observed that flunarizine inhibited the effect of ET-1 on the HTM cells.     

A characterization of cytostatic factor activity from Xenopus eggs and c-mos-transformed cells

In Xenopus oocytes, the mos proto-oncogene product is required during meiosis I for the activation of maturation promoting factor (MPF) and the subsequent breakdown of the germinal vesicle (GVBD). In addition, the mos product has been shown to be a candidate "initiator" of meiotic maturation and is an active component of cytostatic factor (CSF), an activity responsible for metaphase II arrest. Here we demonstrate that pp39mos is required throughout oocyte maturation. We found that in progesterone stimulated oocytes, depletion of mos RNA immediately before GVBD terminally decreased MPF. Likewise, oocytes depleted of mos RNA and induced to mature with crude MPF proceeded through GVBD but lacked the MPF activity required to arrest mature oocytes at metaphase II. Thus, during maturation the mos product is required, directly or indirectly, to sustain MPF activity. On the other hand, mouse NIH/3T3 cells transformed by the constitutive expression of pp39mosxc possessed CSF activity but lacked constitutive levels of MPF or its associated histone H1 kinase activity. Moreover, cytosols prepared from transformed NIH/3T3 cells or Xenopus eggs had similar levels of CSF activity, but pp39mos levels were greater than 40-fold higher in the transformed cell extract. These analyses show that maintenance of CSF during interphase does not result in the maintenance of MPF.     

Increased regularity of activity of cortical neurons in learning due to disinhibitory effect of reinforcement

This paper reviews the author's studies on neurophysiologic mechanisms of conditioned reflex learning. Electroencephalograms, evoked potentials, activity of neocortical and hippocampal neurons and the rabbits' behavior in the course of elaboration of defensive and inhibitory conditioned reflexes to light flashes have been recorded. Electric shock (ECS) applied to the paw served as reinforcement. The study demonstrated three types of reinforcement effect on the activity of cortical neurons: activating, disinhibitory, and inhibitory. EEG activation due to reinforcement is accompanied by a change in phasic cortical neuronal activity from chaotic or irregular, typical of rest or inhibition, to regular tonic discharges (in neocortex and hippocampus) and group discharges in the stress rhythm, 5-7 Hz in the hippocampus. Following a number of conditioning trials, the effect of reinforcement is simulated by the effect of a conditioned stimulus. With EEG activation and increased regularity in impulses, facilitation of motor reactions is observed.     

Towards a general theory of neural computation based on prediction by single neurons

Although there has been tremendous progress in understanding the mechanics of the nervous system, there has not been a general theory of its computational function. Here I present a theory that relates the established biophysical properties of single generic neurons to principles of Bayesian probability theory, reinforcement learning and efficient coding. I suggest that this theory addresses the general computational problem facing the nervous system. Each neuron is proposed to mirror the function of the whole system in learning to predict aspects of the world related to future reward. According to the model, a typical neuron receives current information about the state of the world from a subset of its excitatory synaptic inputs, and prior information from its other inputs. Prior information would be contributed by synaptic inputs representing distinct regions of space, and by different types of non-synaptic, voltage-regulated channels representing distinct periods of the past. The neuron's membrane voltage is proposed to signal the difference between current and prior information ("prediction error" or "surprise"). A neuron would apply a Hebbian plasticity rule to select those excitatory inputs that are the most closely correlated with reward but are the least predictable, since unpredictable inputs provide the neuron with the most "new" information about future reward. To minimize the error in its predictions and to respond only when excitation is "new and surprising," the neuron selects amongst its prior information sources through an anti-Hebbian rule. The unique inputs of a mature neuron would therefore result from learning about spatial and temporal patterns in its local environment, and by extension, the external world. Thus the theory describes how the structure of the mature nervous system could reflect the structure of the external world, and how the complexity and intelligence of the system might develop from a population of undifferentiated neurons, each implementing similar learning algorithms.     

Divergences of MPF2-like MADS-domain proteins have an association with the evolution of the inflated calyx syndrome within Solanaceae

The inflated calyx syndrome (ICS) is a post-floral novelty within Solanaceae. Previous work has shown that MPF2-like MADS-box genes have been recruited for the development and evolution of ICS through heterotopic expression from vegetative to floral organs. ICS seems to be a plesiomorphic trait in Physaleae, but it has been secondarily lost in some lineages during evolution. We hypothesized that molecular and functional divergences of MPF2-like proteins might play a role in the loss of ICS. In this study we analyzed the phylogeny, selection and various functions of MPF2-like proteins with respect to the evolution of ICS. Directional selection of MPF2-like orthologs toward evolution of ICS was detected. While auto-activation capacity between proteins varies in yeast, MPF2-like interaction with floral MADS-domain proteins is robustly detected, hence substantiating their integration into the floral developmental programs. Dimerization with A- (MPF3) and E-function (PFSEP1/3) proteins seems to be essential for ICS development within Solanaceae. Moreover, the occurrence of the enlarged sepals, reminiscent of ICS, and MPF2-like interactions with these specific partners were observed in transgenic Arabidopsis. The interaction spectrum relevant to ICS seems to be plesiomorphic, reinforcing the plesiomorphy of this trait. The inability of some MPF2-like to interact with either the A-function or any of the E-function partners characterized is correlated with the loss of ICS in the lineages that showed a MPF2-like expression in the calyx. Our findings suggest that, after recruitment of MPF2-like genes for floral development, diversification in their coding region due to directional selection leads to a modification of the MADS-domain protein interacting spectrum, which might serve as a constraint for the evolution of ICS within Solanaceae.     

The ethological analysis of imitation

Theorists and experimental researchers have long debated whether animals are able to imitate. A variety of definitions of imitation have been proposed to describe this complex form of social learning. Experimental research on imitation has often been hampered by either a too loose 'anthropomorphic' approach or by too narrow 'behaviourist' definitions. At present neither associative nor cognitive theories are able to offer an exhaustive explanation of imitation in animals. An ethological approach to imitation offers a different perspective. By integrating questions on function, mechanism, development and evolution one can identify possible directions for future research. At present, however, we are still far from developing a comprehensive theory of imitation. A functional approach to imitation shows that, despite some evidence for imitative learning in food processing in apes, such learning has not been shown to be involved in the social transmission of either tool-use skills or communicative signals. Recently developed procedures offer possible ways of clarifying the role of imitation in tool use and visual communication. The role of imitation in explorative play in apes is also investigated and the available data suggest that copying during play might represent a behavioural homologue of human imitation. It is proposed that the ability to copy the behaviour of a companion is under a strong genetic influence in many social species. Many important factors have not been examined experimentally, e.g. the effect of the demonstrator, the influence of attention and memory and the ability to generalize. The potential importance of reinforcement raises the possibility that copying abilities serving divergent functions might be partly under the control of different mechanisms.     

Effect of sodium nitrite on the activity of the neocortex neurons during realization of defense and inhibition conditioned reflexes

A decrease in intensity and duration of short-latency reaction components of the sensorimotor and visual cortical neurons to specific stimuli (pain reinforcement and light flashes, respectively) was observed after the administration of NO-generating sodium nitrite (11 mg/kg, subcutaneously). Activation decrease in the visual cortex took place irrespective of biological significance of the light flashes, i.e., in case when this stimulus was a signal of defensive conditioning and in case when these flashes were applied with continuous light (a conditioned inhibitor). Sodium nitrite almost did not change the late activation of sensorimotor and visual neurons in response to pain reinforcement and disinhibitory action of the latter. The results confirm the viewpoint about different neurotransmitters in "specifically modal" and "non-specific" pathways to the neocortex during learning.     

MPF localization is controlled by nuclear export.

In eukaryotes, mitosis is initiated by M phase promoting factor (MPF), composed of B-type cyclins and their partner protein kinase, CDK1. In animal cells, MPF is cytoplasmic in interphase and is translocated into the nucleus after mitosis has begun, after which it associates with the mitotic apparatus until the cyclins are degraded in anaphase. We have used a fusion protein between human cyclin B1 and green fluorescent protein (GFP) to study this dynamic behaviour in real time, in living cells. We found that when we injected cyclin B1-GFP, or cyclin B1-GFP bound to CDK1 (i.e. MPF), into interphase nuclei it is rapidly exported into the cytoplasm. Cyclin B1 nuclear export is blocked by leptomycin B, an inhibitor of the recently identified export factor, exportin 1 (CRM1). The nuclear export of MPF is mediated by a nuclear export sequence in cyclin B1, and an export-defective cyclin B1 accumulates in interphase nuclei. Therefore, during interphase MPF constantly shuttles between the nucleus and the cytoplasm, but the bulk of MPF is retained in the cytoplasm by rapid nuclear export. We found that a cyclin mutant with a defective nuclear export signal does not enhance the premature mitosis caused by interfering with the regulatory phosphorylation of CDK1, but is more sensitive to inhibition by the Wee1 kinase.     

Brain-like Intelligent Decision-making Based on Basal Ganglia and Its Application in Automatic Car-following

The anthropomorphic intelligence of autonomous driving has been a research hotspot in the world.However,current stud-ies have not been able to reveal the mechanism of drivers'natural driving behaviors.Therefore,this thesis starts from the perspective of cognitive decision-making in the human brain,which is inspired by the regulation of dopamine feedback in the basal ganglia,and a reinforcement learning model is established to solve the brain-like intelligent decision-making problems in the process of interacting with the environment.In this thesis,first,a detailed bionic mechanism architecture based on basal ganglia was proposed by the consideration and analysis of its feedback regulation mechanism;second,the above mechanism was transformed into a reinforcement Q-learning model,so as to implement the learning and adaptation abilities of an intelligent vehicle for brain-like intelligent decision-making during car-following;finally,the feasibility and effectiveness of the proposed method were verified by the simulations and real vehicle tests.