模拟昆虫视觉-行为抉择的强化学习模型

视觉信息用于行为抉择的过程是一个极其复杂的脑信息处理过程,昆虫或动物对外界环境的学习是以价值来控制的,并可影响其行为抉择,研究这一过程对揭示人类自身脑运行机制有重要意义.文章在郭爱克研究小组果蝇实验提供的生物依据基础上,提出了一种模拟果蝇视觉-行为抉择的神经网络模型.该模型引入了价值和基于价值的强化学习算法,应用于输入视觉图像的强化学习,以此建立果蝇脑内多巴胺和蘑菇体对于抉择判断的价值体系.模拟的结果表明,该模型可以模拟果蝇视觉信息的学习和行为抉择过程,其结果与生物实验相符,同时也为机器人视觉信息控制行为抉择的应用提供了基础.     

A generative spiking neural-network model of goal-directed behaviour and one-step planning

In mammals, goal-directed and planning processes support flexible behaviour used to face new situations that cannot be tackled through more efficient but rigid habitual behaviours. Within the Bayesian modelling approach of brain and behaviour, models have been proposed to perform planning as probabilistic inference but this approach encounters a crucial problem: explaining how such inference might be implemented in brain spiking networks. Recently, the literature has proposed some models that face this problem through recurrent spiking neural networks able to internally simulate state trajectories, the core function at the basis of planning. However, the proposed models have relevant limitations that make them biologically implausible, namely their world model is trained ‘off-line’ before solving the target tasks, and they are trained with supervised learning procedures that are biologically and ecologically not plausible. Here we propose two novel hypotheses on how brain might overcome these problems, and operationalise them in a novel architecture pivoting on a spiking recurrent neural network. The first hypothesis allows the architecture to learn the world model in parallel with its use for planning: to this purpose, a new arbitration mechanism decides when to explore, for learning the world model, or when to exploit it, for planning, based on the entropy of the world model itself. The second hypothesis allows the architecture to use an unsupervised learning process to learn the world model by observing the effects of actions. The architecture is validated by reproducing and accounting for the learning profiles and reaction times of human participants learning to solve a visuomotor learning task that is new for them. Overall, the architecture represents the first instance of a model bridging probabilistic planning and spiking-processes that has a degree of autonomy analogous to the one of real organisms.     

A tabular learning structure with an adaptable k-d tree for relaxation of the curse of dimensionality

Memory-based learning schemes are characterized by their memorization of observed events. A memory-based learning scheme either memorizes the collected data directly or reorganizes such information and stores it distributively into a tabular memory. For the tabular type, the system requires a training process to determine the contents of the associative memory. This training process helps filter out zero-mean noise. Since the stored data are associated to pre-assigned input locations, memory management and data retrieval are easier and more efficient. Despite these merits, a drawback of tabular schemes is the difficulty in applying it to high-dimensional problems due to the curse of dimensionality. As the input dimensionality increases, the number of quantized elements in the input space increases at an exponential rate and that causes a huge demand of memory. In this paper, a dynamic tabular structure is proposed for possible relaxation of such a demand. The memorized data are organized as part of a k-d tree. Nodes in the tree, called vertices, correspond to some regularly assigned input points. Memory resource is allocated only at locations where it is needed. Being able to easily compute the vertex positions helps reduce the searching cost in data retrieval. In addition, the learning process is able to expand the tree structure into one covering the problem domain. With memory allocated based on demand, memory consumption becomes closely related to task complexity instead of input dimensionality, and is minimized. Algorithms for structure construction and training are presented in this paper. Simulation results demonstrate that the memory can be efficiently utilized. The developed scheme offers a solution for high-dimensional learning problems with a manageable size of effective input domain.     

Selforganizing memory: active learning of landmarks used for navigation

We propose a memory architecture that is suited to solve a specific task, namely homing, that is finding a not directly visible home place by using visually accessible landmarks. We show that an agent equipped with such a memory structure can autonomously learn the situation and can later use its memory to accomplish homing behaviour. The architecture is based on neuronal structures and grows in a self-organized way depending on experience. The basic architecture consists of three parts, (i) a pre-processor, (ii) a simple, one-layered feed-forward network, called distributor net, and (iii) a full recurrently connected net for representing the situation models to be stored. Apart from Hebbian learning and a local version of the delta-rule, explorative learning is applied that is not based on passive detection of correlations, but is actively searching for interesting hypotheses. Hypotheses are spontaneously introduced and are verified or falsified depending on how well the network representing the hypothesis approaches an internal error of zero. The stability of this approach is successfully tested by removal of one landmark or shifting the position of one or several landmarks showing results comparable to those found in biological experiments. Furthermore, we applied noise in two ways. The trained network was either due to sensory noise or to noise applied to the bias weights describing the memory content. Finally, we tested to what extent learning of the weights is affected by noisy input given to the sensor data. The architecture proposed is discussed to have some at least superficial similarity to the mushroom bodies of insects.     

The effect of learning on the function of monkey extrastriate visual cortex

One of the most remarkable capabilities of the adult brain is its ability to learn and continuously adapt to an ever-changing environment. While many studies have documented how learning improves the perception and identification of visual stimuli, relatively little is known about how it modifies the underlying neural mechanisms. We trained monkeys to identify natural images that were degraded by interpolation with visual noise. We found that learning led to an improvement in monkeys' ability to identify these indeterminate visual stimuli. We link this behavioral improvement to a learning-dependent increase in the amount of information communicated by V4 neurons. This increase was mediated by a specific enhancement in neural activity. Our results reveal a mechanism by which learning increases the amount of information that V4 neurons are able to extract from the visual environment. This suggests that V4 plays a key role in resolving indeterminate visual inputs by coordinated interaction between bottom-up and top-down processing streams.     

Biview Learning for Human Posture Segmentation from 3D Points Cloud

Posture segmentation plays an essential role in human motion analysis. The state-of-the-art method extracts sufficiently high-dimensional features from 3D depth images for each 3D point and learns an efficient body part classifier. However, high-dimensional features are memory-consuming and difficult to handle on large-scale training dataset. In this paper, we propose an efficient two-stage dimension reduction scheme, termed biview learning, to encode two independent views which are depth-difference features (DDF) and relative position features (RPF). Biview learning explores the complementary property of DDF and RPF, and uses two stages to learn a compact yet comprehensive low-dimensional feature space for posture segmentation. In the first stage, discriminative locality alignment (DLA) is applied to the high-dimensional DDF to learn a discriminative low-dimensional representation. In the second stage, canonical correlation analysis (CCA) is used to explore the complementary property of RPF and the dimensionality reduced DDF. Finally, we train a support vector machine (SVM) over the output of CCA. We carefully validate the effectiveness of DLA and CCA utilized in the two-stage scheme on our 3D human points cloud dataset. Experimental results show that the proposed biview learning scheme significantly outperforms the state-of-the-art method for human posture segmentation.     

Looking at the Ventriloquist: Visual Outcome of Eye Movements Calibrates Sound Localization

A general problem in learning is how the brain determines what lesson to learn (and what lessons not to learn). For example, sound localization is a behavior that is partially learned with the aid of vision. This process requires correctly matching a visual location to that of a sound. This is an intrinsically circular problem when sound location is itself uncertain and the visual scene is rife with possible visual matches. Here, we develop a simple paradigm using visual guidance of sound localization to gain insight into how the brain confronts this type of circularity. We tested two competing hypotheses. 1: The brain guides sound location learning based on the synchrony or simultaneity of auditory-visual stimuli, potentially involving a Hebbian associative mechanism. 2: The brain uses a ‘guess and check’ heuristic in which visual feedback that is obtained after an eye movement to a sound alters future performance, perhaps by recruiting the brain’s reward-related circuitry. We assessed the effects of exposure to visual stimuli spatially mismatched from sounds on performance of an interleaved auditory-only saccade task. We found that when humans and monkeys were provided the visual stimulus asynchronously with the sound but as feedback to an auditory-guided saccade, they shifted their subsequent auditory-only performance toward the direction of the visual cue by 1.3–1.7 degrees, or 22–28% of the original 6 degree visual-auditory mismatch. In contrast when the visual stimulus was presented synchronously with the sound but extinguished too quickly to provide this feedback, there was little change in subsequent auditory-only performance. Our results suggest that the outcome of our own actions is vital to localizing sounds correctly. Contrary to previous expectations, visual calibration of auditory space does not appear to require visual-auditory associations based on synchrony/simultaneity.     

Visual discrimination abilities in the gray bamboo shark (Chiloscyllium griseum)

This study assessed visual discrimination abilities in bamboo sharks (Chiloscyllium griseum). In a visual discrimination task using two-dimensional (2D) geometric stimuli, sharks learned to distinguish between a square, being the positive (rewarded) stimulus, and several negative stimuli, such as two differently sized triangles, a circle, a rhomboid and a cross. Although the amount of sessions to reach the learning criterion and the average trial time needed to solve each new task did not vary significantly, the number of correct choices per session increased significantly with on-going experiments. The results indicate that the sharks did not simply remember the positive stimulus throughout the different training phases. Instead, individuals also seemed to learn each negative symbol and possibly had to “relearn” at least some aspects of the positive stimulus during each training phase. The sharks were able to distinguish between the 2D stimulus pairs at a learning rate corresponding to that found in teleosts. As expected, it took the sharks longer to learn a reversal task (with the positive stimulus now being the negative one) than to discriminate between the other stimulus pairs. Nevertheless, the present results suggest that bamboo sharks can learn visual discrimination tasks, succeed in a reversal task and probably retain (some) information about a previously learned task when progressing to a new one.     

Flexible Switching of Feedback Control Mechanisms Allows for Learning of Different Task Dynamics

To produce skilled movements, the brain flexibly adapts to different task requirements and movement contexts. Two core abilities underlie this flexibility. First, depending on the task, the motor system must rapidly switch the way it produces motor commands and how it corrects movements online, i.e. it switches between different (feedback) control policies. Second, it must also adapt to environmental changes for different tasks separately. Here we show these two abilities are related. In a bimanual movement task, we show that participants can switch on a movement-by-movement basis between two feedback control policies, depending only on a static visual cue. When this cue indicates that the hands control separate objects, reactions to force field perturbations of each arm are purely unilateral. In contrast, when the visual cue indicates a commonly controlled object, reactions are shared across hands. Participants are also able to learn different force fields associated with a visual cue. This is however only the case when the visual cue is associated with different feedback control policies. These results indicate that when the motor system can flexibly switch between different control policies, it is also able to adapt separately to the dynamics of different environmental contexts. In contrast, visual cues that are not associated with different control policies are not effective for learning different task dynamics.     

The role of spatial and temporal information in learning interval time-place tasks

In time-place learning (TPL) paradigms animals are thought to form tripartite memory codes consisting of the spatiotemporal characteristics of biologically significant events. In Phase I, rats were trained on a modified TPL task in which either the spatial or temporal component was constant, while the other component varied randomly. If the memory codes are tripartite then when one aspect of the code is random the rats should have difficulty learning the constant aspect of the code. However, rats that were trained with a fixed spatial sequence of food availability and a random duration did in fact learn the task. Rats that were trained with a fixed duration and a random sequence did not learn the task. In Phase II all rats were placed on a TPL task in which food availability was contingent upon both spatial and temporal information. According to the tripartite theory, prior knowledge of either aspect of the code should have little effect on the acquisition of the task. The rats that received fixed spatial training learned the task relatively more quickly. The use of bipartite, rather than tripartite codes, is better able to explain the results of the current study.     

Feedback control strategies for spatial navigation revealed by dynamic modelling of learning in the Morris water maze

The Morris water maze is an experimental procedure in which animals learn to escape swimming in a pool using environmental cues. Despite its success in neuroscience and psychology for studying spatial learning and memory, the exact mnemonic and navigational demands of the task are not well understood. Here, we provide a mathematical model of rat swimming dynamics on a behavioural level. The model consists of a random walk, a heading change and a feedback control component in which learning is reflected in parameter changes of the feedback mechanism. The simplicity of the model renders it accessible and useful for analysis of experiments in which swimming paths are recorded. Here, we used the model to analyse an experiment in which rats were trained to find the platform with either three or one extramaze cue. Results indicate that the 3-cues group employs stronger feedback relying only on the actual visual input, whereas the 1-cue group employs weaker feedback relying to some extent on memory. Because the model parameters are linked to neurological processes, identifying different parameter values suggests the activation of different neuronal pathways.     

Discriminating signal from noise: recognition of a movement-based animal display by artificial neural networks

In this study, we investigated the feasibility of applying neural networks to understanding movement-based visual signals. Networks based on three different models were constructed, varying in their input format and network architecture: a Static Input model, a Dynamic Input model and a Feedback model. The task for all networks was to distinguish a lizard (Amphibolurus muricatus) tail-flick from background plant movement. Networks based on all models were able to distinguish the two types of visual motion, and generalised successfully to unseen exemplars. We used curves defined by the receiver-operating characteristic (ROC) to select a single network from each model to be used in regression analyses of network response and several motion variables. Collectively, the models predicted that tail-flick efficacy would be enhanced by faster speeds, greater acceleration and longer durations.     

A Model of Self-Organizing Head-Centered Visual Responses in Primate Parietal Areas

We present a hypothesis for how head-centered visual representations in primate parietal areas could self-organize through visually-guided learning, and test this hypothesis using a neural network model. The model consists of a competitive output layer of neurons that receives afferent synaptic connections from a population of input neurons with eye position gain modulated retinal receptive fields. The synaptic connections in the model are trained with an associative trace learning rule which has the effect of encouraging output neurons to learn to respond to subsets of input patterns that tend to occur close together in time. This network architecture and synaptic learning rule is hypothesized to promote the development of head-centered output neurons during periods of time when the head remains fixed while the eyes move. This hypothesis is demonstrated to be feasible, and each of the core model components described is tested and found to be individually necessary for successful self-organization.     

A Cognitive Neural Architecture Able to Learn and Communicate through Natural Language

Communicative interactions involve a kind of procedural knowledge that is used by the human brain for processing verbal and nonverbal inputs and for language production. Although considerable work has been done on modeling human language abilities, it has been difficult to bring them together to a comprehensive tabula rasa system compatible with current knowledge of how verbal information is processed in the brain. This work presents a cognitive system, entirely based on a large-scale neural architecture, which was developed to shed light on the procedural knowledge involved in language elaboration. The main component of this system is the central executive, which is a supervising system that coordinates the other components of the working memory. In our model, the central executive is a neural network that takes as input the neural activation states of the short-term memory and yields as output mental actions, which control the flow of information among the working memory components through neural gating mechanisms. The proposed system is capable of learning to communicate through natural language starting from tabula rasa, without any a priori knowledge of the structure of phrases, meaning of words, role of the different classes of words, only by interacting with a human through a text-based interface, using an open-ended incremental learning process. It is able to learn nouns, verbs, adjectives, pronouns and other word classes, and to use them in expressive language. The model was validated on a corpus of 1587 input sentences, based on literature on early language assessment, at the level of about 4-years old child, and produced 521 output sentences, expressing a broad range of language processing functionalities.     

Interval time-place learning in young children

While previous research has investigated the ability of animals to learn the spatial and temporal contingencies of biologically significant events (known as time-place learning), this ability has not been studied in humans. Children ranging from 5 to 10 years old were tested on a modified interval time-place learning task using a touchscreen computer. Results demonstrate the children were able to quickly learn both the timing and the sequence of this task. Despite a lack of anticipation on baseline trials, the children continued to follow the spatio-temporal contingencies in probe sessions where these contingencies were removed. Performance on the probe sessions provide strong evidence that the children had learned the spatio-temporal contingencies. Future research is needed to determine what age-related changes in iTPL occur. Furthermore, it is argued that this procedure can be used to extend interval timing in research in children, including, but not limited to, investigation of scalar timing with longer durations than have previously been investigated.     

The application of the 5-choice serial reaction time task for the assessment of visual attentional processes and impulse control in rats

One popular way of measuring visual attentional processes in the rat is using 5-choice serial reaction time task (5-CSRTT). This paradigm requires subjects to detect brief flashes of light presented in a pseudorandom order in one of five spatial locations over a large number of trials. For this task, the animals are trained for approximately 30-40 daily sessions during which they gradually learn to respond in the appropriate aperture within a certain amount of time. If they fail to respond, respond in the wrong hole or at an inappropriate time, a short period of darkness (time-out) is presented as punishment and no reward is delivered. The 5-CSRTT provides the possibility to test the effects of various neural, pharmacological and behavioral manipulations on discrete and somewhat independent measures of behavioral control, including accuracy of discrimination, impulsivity, perseverative responses and response latencies.     

Influence of habitat complexity on route learning among different populations of climbing perch (Anabas testudineus Bloch, 1792)

Animals use different behavioral strategies to maximize their fitness in the natural environment. Learning and memory are critical in this context, allowing organisms to flexibly and rapidly respond to environmental changes. We studied how the physical characteristics of the native habitat influence the spatial learning capacity of Anabas testudineus belonging to four different populations collected from two streams and two ponds, in a linear maze. Stream fish were able to learn the route faster than pond fish irrespective of the presence or absence of landmarks in the maze. However, climbing perch collected from ponds learned the route faster in the maze provided with landmarks than in Plain maze. The results indicate that fish inhabiting a lotic ecosystem use egocentric cues in route learning rather than visual cues like landmarks. A local landmark may be a more reliable cue in route learning in a relatively stable habitat like a pond. In flowing aquatic systems, water flow may continually disrupt the visual landscape and thus landmarks as visual cues become unreliable. Spatial learning is thus a fine-tuned response to the complexity of the habitat and early rearing conditions may influence the spatial learning ability in fish.     

Structure Learning in a Sensorimotor Association Task

Learning is often understood as an organism''s gradual acquisition of the association between a given sensory stimulus and the correct motor response. Mathematically, this corresponds to regressing a mapping between the set of observations and the set of actions. Recently, however, it has been shown both in cognitive and motor neuroscience that humans are not only able to learn particular stimulus-response mappings, but are also able to extract abstract structural invariants that facilitate generalization to novel tasks. Here we show how such structure learning can enhance facilitation in a sensorimotor association task performed by human subjects. Using regression and reinforcement learning models we show that the observed facilitation cannot be explained by these basic models of learning stimulus-response associations. We show, however, that the observed data can be explained by a hierarchical Bayesian model that performs structure learning. In line with previous results from cognitive tasks, this suggests that hierarchical Bayesian inference might provide a common framework to explain both the learning of specific stimulus-response associations and the learning of abstract structures that are shared by different task environments.     

Recurrent cerebellar architecture solves the motor-error problem

Current views of cerebellar function have been heavily influenced by the models of Marr and Albus, who suggested that the climbing fibre input to the cerebellum acts as a teaching signal for motor learning. It is commonly assumed that this teaching signal must be motor error (the difference between actual and correct motor command), but this approach requires complex neural structures to estimate unobservable motor error from its observed sensory consequences. We have proposed elsewhere a recurrent decorrelation control architecture in which Marr-Albus models learn without requiring motor error. Here, we prove convergence for this architecture and demonstrate important advantages for the modular control of systems with multiple degrees of freedom. These results are illustrated by modelling adaptive plant compensation for the three-dimensional vestibular ocular reflex. This provides a functional role for recurrent cerebellar connectivity, which may be a generic anatomical feature of projections between regions of cerebral and cerebellar cortex.     

An information theoretic exploratory method for learning patterns of conditional gene coexpression from microarray data

In this article, we introduce an exploratory framework for learning patterns of conditional co-expression in gene expression data. The main idea behind the proposed approach consists of estimating how the information content shared by a set of M nodes in a network (where each node is associated to an expression profile) varies upon conditioning on a set of L conditioning variables (in the simplest case represented by a separate set of expression profiles). The method is non-parametric and it is based on the concept of statistical co-information, which, unlike conventional correlation based techniques, is not restricted in scope to linear conditional dependency patterns. Moreover, such conditional co-expression relationships can potentially indicate regulatory interactions that do not manifest themselves when only pair-wise relationships are considered. A moment based approximation of the co-information measure is derived that efficiently gets around the problem of estimating high-dimensional multi-variate probability density functions from the data, a task usually not viable due to the intrinsic sample size limitations that characterize expression level measurements. By applying the proposed exploratory method, we analyzed a whole genome microarray assay of the eukaryote Saccharomices cerevisiae and were able to learn statistically significant patterns of conditional co-expression. A selection of such interactions that carry a meaningful biological interpretation are discussed.