Mechanisms of behavioral switching

No animal performs only one behavior, so nervous systems must have ways to switch between different behaviors. In this issue of Journal of Comparative Physiology A, several papers discuss how nervous systems achieve this ordered switching between behaviors, from short-term motor control problems, to medium-term decision making based on past experience, to long-term modulation and selection of overall behavioral strategies, such as dominance versus subordinance.The papers discussed here were originally given at the symposium Mechanisms of behavioral switching, held at the annual Animal Behavior Society meeting in Boise, Idaho in July 2003.     

Influence of haptic guidance in learning a novel visuomotor task

In (re)learning of movements, haptic guidance can be used to direct the needed adaptations in motor control. Haptic guidance influences the main driving factors of motor adaptation, execution error, and control effort in different ways. Human-control effort is dissipated in the interactions that occur during haptic guidance. Minimizing the control effort would reduce the interaction forces and result in adaptation. However, guidance also decreases the magnitude of the execution errors, which could inhibit motor adaptation. The aim of this study was to assess how different types of haptic guidance affect kinematic adaptation in a novel visuomotor task. Five groups of subjects adapted to a reaching task in which the visual representation of the hand was rotated 30°. Each group was guided by a different force field. The force fields differed in magnitude and direction in order to discern the adaptation based on execution errors and control effort. The results demonstrated that the execution error did indeed play a key role in adaptation. The more the guiding forces restricted the occurrence of execution errors, the smaller the amount and rate of adaptation. However, the force field that enlarged the execution errors did not result in an increased rate of adaptation. The presence of a small amount of adaptation in the groups who did not experience execution errors during training suggested that adaptation could be driven on a much slower rate and on the basis of minimization of control effort as was evidenced by a gradual decrease of the interaction forces during training. Remarkably, also in the group in which the subjects were passive and completely guided, a small but significant adaptation occurred. The conclusion is that both minimization of execution errors and control effort drives kinematic adaptation in a novel visuomotor task, but the latter at a much slower rate.     

Neuromuscular control of the point to point and oscillatory movements of a sagittal arm with the actor–critic reinforcement learning method

In this study, we have used a single link system with a pair of muscles that are excited with alpha and gamma signals to achieve both point to point and oscillatory movements with variable amplitude and frequency.

The system is highly nonlinear in all its physical and physiological attributes. The major physiological characteristics of this system are simultaneous activation of a pair of nonlinear muscle-like-actuators for control purposes, existence of nonlinear spindle-like sensors and Golgi tendon organ-like sensor, actions of gravity and external loading. Transmission delays are included in the afferent and efferent neural paths to account for a more accurate representation of the reflex loops.

A reinforcement learning method with an actor–critic (AC) architecture instead of middle and low level of central nervous system (CNS), is used to track a desired trajectory. The actor in this structure is a two layer feedforward neural network and the critic is a model of the cerebellum. The critic is trained by state-action-reward-state-action (SARSA) method. The critic will train the actor by supervisory learning based on the prior experiences. Simulation studies of oscillatory movements based on the proposed algorithm demonstrate excellent tracking capability and after 280 epochs the RMS error for position and velocity profiles were 0.02, 0.04?rad and rad/s, respectively.     

Imaging motor imagery: methodological issues related to expertise

Mental imagery (MI) is the mental rehearsal of movements without overt execution. Brain imaging techniques have made it possible to identify the brain regions that are activated during MI and, for voluntary motor tasks involving hand and finger movements, to make direct comparison with those areas activated during actual movement. However, the fact that brain activation differs for different types of imagery (visual or kinetic) and depends on the skill level of the individual (e.g., novice or elite athlete) raises a number of important methodological issues for the design of brain imaging protocols to study MI. These include instructing the subject concerning the type of imagery to use, objective measurement of skill level, the design of motor tasks sufficiently difficult to produce a range of skill levels, the effect of different environments on skill level (including the imaging device), and so on. It is suggested that MI is more about the neurobiology of the development of motor skills that have already been learned, but not perfected, than it is about learning motor skills de novo.     

Retention of improvement in gait stability over 14 weeks due to trip-perturbation training is dependent on perturbation dose

Perturbation training is an emerging approach to reduce fall risk in the elderly. This study examined potential differences in retention of improvements in reactive gait stability over 14 weeks resulting from unexpected trip-like gait perturbations. Twenty-four healthy middle-aged adults (41–62 years) were assigned randomly to either a single perturbation group (SINGLE, n = 9) or a group subjected to eight trip-like gait perturbations (MULTIPLE, n = 15). While participants walked on a treadmill a custom-built brake-and-release system was used to unexpectedly apply resistance during swing phase to the lower right limb via an ankle strap. The anteroposterior margin of stability (MoS) was calculated as the difference between the anterior boundary of the base of support and the extrapolated centre of mass at foot touchdown for the perturbed step and the first recovery step during the first and second (MULTIPLE group only) perturbation trials for the initial walking session and retention-test walking 14 weeks later. Group MULTIPLE retained the improvements in reactive gait stability to the perturbations (increased MoS at touchdown for perturbed and first recovery steps; p < 0.01). However, in group SINGLE no differences in MoS were detected after 14 weeks compared to the initial walking session. These findings provide evidence for the requirement of a threshold trip-perturbation dose if adaptive changes in the human neuromotor system over several months, aimed at the improvement in fall-resisting skills, are to occur.     

Visual search and the importance of time in complex decision making by bees

Psychophysicists studying decision making in animals have overwhelmingly focused on choice accuracy, not speed. Results from human visual search, however, show that there might be a tight link between the two. Here we review both visual-sensory and cognitive mechanisms that affect decision speed in flower visiting bees. We show that decision times are affected by contrast of targets and background, by similarity between targets and distractors, numbers of distractors present in a scene, illuminating light intensity, presence or absence of punishment, and complexity of tasks. We explore between-individual and within-individual speed-accuracy tradeoffs, and show that bees resort to highly dynamic strategies when solving visual search tasks. Where possible, we attempt to link the observed search behaviour to the temporal and spatial properties of neuronal circuits underlying visual object detection. We demonstrate that natural foraging speed may not only be limited by factors such as food item density, flight energetics and scramble competition, as often implied. Our results show that understanding the behavioural ecology of foraging can substantially gain from knowledge about mechanisms of visual information processing.     

Operant control of human eye movements

Saccade and smooth pursuit are the eye movements used by primates to shift gaze. In this article we review evidence for the effects of reinforcement on several dimensions of these responses such as their latencies, velocities or amplitudes. We propose that these responses are operant behaviours controlled by their consequences on performance of visually guided tasks. Studying the conditions under which particular eye movement patterns might emerge from the cumulative effects of reinforcement provides critical insights about how motor responses are attuned to environmental exigencies.     

A comparative psychophysical approach to visual perception in primates

Studies on the visual processing of primates, which have well developed visual systems, provide essential information about the perceptual bases of their higher-order cognitive abilities. Although the mechanisms underlying visual processing are largely shared between human and nonhuman primates, differences have also been reported. In this article, we review psychophysical investigations comparing the basic visual processing that operates in human and nonhuman species, and discuss the future contributions potentially deriving from such comparative psychophysical approaches to primate minds.     

A computational model to link psychophysics and cortical cell activation patterns in human texture processing

The human visual system uses texture information to automatically, or pre-attentively, segregate parts of the visual scene. We investigate the neural substrate underlying human texture processing using a computational model that consists of a hierarchy of bi-directionally linked model areas. The model builds upon two key hypotheses, namely that (i) texture segregation is based on boundary detection--rather than clustering of homogeneous items--and (ii) texture boundaries are detected mainly on the basis of a large scenic context that is analyzed by higher cortical areas within the ventral visual pathway, such as area V4. Here, we focus on the interpretation of key results from psychophysical studies on human texture segmentation. In psychophysical studies, texture patterns were varied along several feature dimensions to systematically characterize human performance. We use simulations to demonstrate that the activation patterns of our model directly correlate with the psychophysical results. This allows us to identify the putative neural mechanisms and cortical key areas which underlie human behavior. In particular, we investigate (i) the effects of varying texture density on target saliency, and the impact of (ii) element alignment and (iii) orientation noise on the detectability of a pop-out bar. As a result, we demonstrate that the dependency of target saliency on texture density is linked to a putative receptive field organization of orientation-selective neurons in V4. The effect of texture element alignment is related to grouping mechanisms in early visual areas. Finally, the modulation of cell activity by feedback activation from higher model areas, interacting with mechanisms of intra-areal center-surround competition, is shown to result in the specific suppression of noise-related cell activities and to improve the overall model capabilities in texture segmentation. In particular, feedback interaction is crucial to raise the model performance to the level of human observers.     

Electromechanical delay estimated by using electromyography during cycling at different pedaling frequencies

The purpose of this study was to propose a new method that can be used to calculate electromechanical delay (EMD) without the measurement of forces. A secondary purpose, as an example of the importance of measuring EMD, was to predict muscle force development events based on the EMG activity of selected muscles during cycling at different pedaling frequencies. EMD was estimated using newly derived equations based on activation dynamics hypothesis. Tibialis anterior (TA) and soleus (SL) muscles of 16 male participants were studied while subjects pedaled at targeted cadences of 60, 80, and 100 revolutions per minute. The estimated EMDs of TA and SL were significantly different from each other with means of 68.1 and 88.7 ms, respectively. The average crank angle for the initiation and time to peak TA contraction was estimated at 75±35° and 26±15° before the crank reached top-dead-center (TDC), while the contraction ended at 31±19° after the TDC on average. The projected starting, peak and end angles of SL contraction activity were 45±18°, 123±13°, and 218±35° after the TDC, respectively. There was no difference among different pedaling cadences observed for these mechanical events. The proposed method was proven to be effective in studying EMD and estimate muscle contraction patterns during cycling.     

Mathematical properties of neuronal TD-rules and differential Hebbian learning: a comparison

A confusingly wide variety of temporally asymmetric learning rules exists related to reinforcement learning and/or to spike-timing dependent plasticity, many of which look exceedingly similar, while displaying strongly different behavior. These rules often find their use in control tasks, for example in robotics and for this rigorous convergence and numerical stability is required. The goal of this article is to review these rules and compare them to provide a better overview over their different properties. Two main classes will be discussed: temporal difference (TD) rules and correlation based (differential hebbian) rules and some transition cases. In general we will focus on neuronal implementations with changeable synaptic weights and a time-continuous representation of activity. In a machine learning (non-neuronal) context, for TD-learning a solid mathematical theory has existed since several years. This can partly be transfered to a neuronal framework, too. On the other hand, only now a more complete theory has also emerged for differential Hebb rules. In general rules differ by their convergence conditions and their numerical stability, which can lead to very undesirable behavior, when wanting to apply them. For TD, convergence can be enforced with a certain output condition assuring that the δ-error drops on average to zero (output control). Correlation based rules, on the other hand, converge when one input drops to zero (input control). Temporally asymmetric learning rules treat situations where incoming stimuli follow each other in time. Thus, it is necessary to remember the first stimulus to be able to relate it to the later occurring second one. To this end different types of so-called eligibility traces are being used by these two different types of rules. This aspect leads again to different properties of TD and differential Hebbian learning as discussed here. Thus, this paper, while also presenting several novel mathematical results, is mainly meant to provide a road map through the different neuronally emulated temporal asymmetrical learning rules and their behavior to provide some guidance for possible applications.     

Electromyography responses of pediatric and young adult volunteers in low-speed frontal impacts

No electromyography (EMG) responses data exist of children exposed to dynamic impacts similar to automotive crashes, thereby, limiting active musculature representation in computational occupant biomechanics models. This study measured the surface EMG responses of three neck, one torso and one lower extremity muscles during low-speed frontal impact sled tests (average maximum acceleration: 3.8 g; rise time: 58.2 ms) performed on seated, restrained pediatric (n = 11, 8–14 years) and young adult (n = 9, 18–30 years) male subjects. The timing and magnitude of the EMG responses were compared between the two age groups. Two normalization techniques were separately implemented and evaluated: maximum voluntary EMG (MVE) and neck cross-sectional area (CSA). The MVE-normalized EMG data indicated a positive correlation with age in the rectus femoris for EMG latency; there was no correlation with age for peak EMG amplitudes for the evaluated muscles. The cervical paraspinous exhibited shorter latencies compared with the other muscles (2–143 ms). Overall, the erector spinae and rectus femoris peak amplitudes were relatively small. Neck CSA-normalized peak EMG amplitudes negatively correlated with age for the cervical paraspinous and sternocleidomastoid. These data can be useful to incorporate active musculature in computational models, though it may not need to be age-specific in low-speed loading environments.     

Rethinking of risk communication: lessons from the decision sciences

Risk communication involves three primary elements: process, content and intent. Much has been written about the first two. Much is known, for example, about the guiding principles that should be considered during the design of a risk communication. Likewise, many studies have been conducted about how best to establish the technical and informational content of a risk communication. Very little attention, by contrast, has been devoted to the intent of risk communication, which is to inform decision making for risk management. While appropriate information upon which to base risk management decisions is important, so to is an understanding of how people instinctively approach decision making under conditions of risk. Work in the decision sciences provides this often-missing perspective for many risk communication efforts and is, therefore, the focus of this paper.

On the use of hybrid reinforcement learning for autonomic resource allocation

Reinforcement Learning (RL) provides a promising new approach to systems performance management that differs radically from standard queuing-theoretic approaches making use of explicit system performance models. In principle, RL can automatically learn high-quality management policies without an explicit performance model or traffic model, and with little or no built-in system specific knowledge. In our original work (Das, R., Tesauro, G., Walsh, W.E.: IBM Research, Tech. Rep. RC23802 (2005), Tesauro, G.: In: Proc. of AAAI-05, pp. 886–891 (2005), Tesauro, G., Das, R., Walsh, W.E., Kephart, J.O.: In: Proc. of ICAC-05, pp. 342–343 (2005)) we showed the feasibility of using online RL to learn resource valuation estimates (in lookup table form) which can be used to make high-quality server allocation decisions in a multi-application prototype Data Center scenario. The present work shows how to combine the strengths of both RL and queuing models in a hybrid approach, in which RL trains offline on data collected while a queuing model policy controls the system. By training offline we avoid suffering potentially poor performance in live online training. We also now use RL to train nonlinear function approximators (e.g. multi-layer perceptrons) instead of lookup tables; this enables scaling to substantially larger state spaces. Our results now show that, in both open-loop and closed-loop traffic, hybrid RL training can achieve significant performance improvements over a variety of initial model-based policies. We also find that, as expected, RL can deal effectively with both transients and switching delays, which lie outside the scope of traditional steady-state queuing theory.

Learning to reach by reinforcement learning using a receptive field based function approximation approach with continuous actions

Reinforcement learning methods can be used in robotics applications especially for specific target-oriented problems, for example the reward-based recalibration of goal directed actions. To this end still relatively large and continuous state-action spaces need to be efficiently handled. The goal of this paper is, thus, to develop a novel, rather simple method which uses reinforcement learning with function approximation in conjunction with different reward-strategies for solving such problems. For the testing of our method, we use a four degree-of-freedom reaching problem in 3D-space simulated by a two-joint robot arm system with two DOF each. Function approximation is based on 4D, overlapping kernels (receptive fields) and the state-action space contains about 10,000 of these. Different types of reward structures are being compared, for example, reward-on- touching-only against reward-on-approach. Furthermore, forbidden joint configurations are punished. A continuous action space is used. In spite of a rather large number of states and the continuous action space these reward/punishment strategies allow the system to find a good solution usually within about 20 trials. The efficiency of our method demonstrated in this test scenario suggests that it might be possible to use it on a real robot for problems where mixed rewards can be defined in situations where other types of learning might be difficult. This work was supported by EU-Grant PACO-PLUS.     

从生态学的观点看城乡发展问题

阐述了城乡发展的意义,提出了城乡发展中所面临的人口、环境和资源等三大问题.为使城乡发展建立在科学的基础上,必须合理利用自然资源,保护生态环境,使得城乡生态系统的收支、结构和功能保持平衡,真正把城乡建成功能齐全、环境优美、生产发展和生活舒适的场所.