Adaptive dynamic programming as a theory of sensorimotor control

Many characteristics of sensorimotor control can be explained by models based on optimization and optimal control theories. However, most of the previous models assume that the central nervous system has access to the precise knowledge of the sensorimotor system and its interacting environment. This viewpoint is difficult to be justified theoretically and has not been convincingly validated by experiments. To address this problem, this paper presents a new computational mechanism for sensorimotor control from a perspective of adaptive dynamic programming (ADP), which shares some features of reinforcement learning. The ADP-based model for sensorimotor control suggests that a command signal for the human movement is derived directly from the real-time sensory data, without the need to identify the system dynamics. An iterative learning scheme based on the proposed ADP theory is developed, along with rigorous convergence analysis. Interestingly, the computational model as advocated here is able to reproduce the motor learning behavior observed in experiments where a divergent force field or velocity-dependent force field was present. In addition, this modeling strategy provides a clear way to perform stability analysis of the overall system. Hence, we conjecture that human sensorimotor systems use an ADP-type mechanism to control movements and to achieve successful adaptation to uncertainties present in the environment.     

Optimization of bioreactor using metabolic control analysis approach

A new methodology based on a metabolic control analysis (MCA) approach is developed for the optimization of continuous cascade bioreactor system. A general framework for representation of a cascade bioreactor system consisting of a large number of reactors as a single network is proposed. The kinetic and transport processes occurring in the system are represented as a reaction network with appropriate stoichiometry. Such representation of the bioreactor systems makes it amenable to the direct application of the MCA approach. The process sensitivity information is extracted using MCA methodology in the form of flux and concentration control coefficients. The process sensitivity information is shown to be a useful guide for determining the choice of decision variables for the purpose of optimization. A generalized problem of optimization of the bioreactor is formulated in which the decision variables are the operating conditions and kinetic parameters. The gradient of the objective function to be maximized with respect to all decision variables is obtained in the form of response coefficients. This gradient information can be used in any gradient-based optimization algorithm. The efficiency of the proposed technique is demonstrated with two examples taken from literature: biotransformation of crotonobetaine and alcohol fermentation in cascade bioreactor system.     

A multi-phase optimal control technique for the simulation of a human vertical jump

A multi-phase optimal control technique is presented that can be used to solve dynamic optimization problems involving musculoskeletal systems. The biomechanical model consists of a set of differential equations describing the dynamics of the multi-body system and the generation of the dynamic forces of the human muscles. Within the optimization technique, subintervals can be defined in which the differential equations are continuous. At the boundaries the dimension of the state- and control vector as well as the dimension of the right-hand side may change. The problem is solved by a multiple shooting approach which converts the problem into a non-linear program. The method is applied to simulate a human jump movement.     

Self-organized control of bipedal locomotion by neural oscillators in unpredictable environment

A new principle of sensorimotor control of legged locomotion in an unpredictable environment is proposed on the basis of neurophysiological knowledge and a theory of nonlinear dynamics. Stable and flexible locomotion is realized as a global limit cycle generated by a global entrainment between the rhythmic activities of a nervous system composed of coupled neural oscillators and the rhythmic movements of a musculo-skeletal system including interaction with its environment. Coordinated movements are generated not by slaving to an explicit representation of the precise trajectories of the movement of each part but by dynamic interactions among the nervous system, the musculo-skeletal system and the environment. The performance of a bipedal model based on the above principle was investigated by computer simulation. Walking movements stable to mechanical perturbations and to environmental changes were obtained. Moreover, the model generated not only the walking movement but also the running movement by changing a single parameter nonspecific to the movement. The transitions between the gait patterns occurred with hysteresis.     

Optimization of biochemical systems by linear programming and general mass action model representations

A new method is proposed for the optimization of biochemical systems. The method, based on the separation of the stoichiometric and kinetic aspects of the system, follows the general approach used in the previously presented indirect optimization method (IOM) developed within biochemical systems theory. It is called GMA-IOM because it makes use of the generalized mass action (GMA) as the model system representation form. The GMA representation avoids flux aggregation and thus prevents possible stoichiometric errors. The optimization of a system is used to illustrate and compare the features, advantages and shortcomings of both versions of the IOM method as a general strategy for designing improved microbial strains of biotechnological interest. Special attention has been paid to practical problems for the actual implementation of the new proposed strategy, such as the total protein content of the engineered strain or the deviation from the original steady state and its influence on cell viability.     

Dynamic optimization: inverse analysis for the Yurchenko layout vault in women's artistic gymnastics

The use of dynamic optimization to compute the trajectory of joint torques is not popular due to the large amount of computation required, the choice of initial "guesstimates" of torque values and the mathematical sophistication required to understand the technique. Modern optimal control algorithms circumvent most of these objections to the method. It is our aim to demonstrate that the dynamic optimization technique is feasible for complex movements, using the Yurchenko layout vault as an example. A dynamic optimization method to compute joint torques so that the histories of the angular orientations of the model segments closely approximate the corresponding observed angular coordinate histories is demonstrated with the Yurchenko layout vault using an optimal control package. The objective function used is a measure of distance of fitted segment angles to the data, plus the distance of the fitted whole body centre of mass (CM), from the whole body CM computed from the data. Including the CM into the objective function, facilitates the optimization process so as to obtain a set of torques which reproduced the data. The paper shows that the approach works well for the task examined, that is, where the dynamics of the system change during a movement (impact to postflight).     

A kinematic analysis of the formation of precise instrumental movements in cats

Kinematic parameters of cats local manipulating movements have been studied in the process of formation and stabilization of precise habit of moving and holding the lever in the zone of "working" space signalled by sound. It is shown that change of activity of the motor control system in the course of training is connected with the transfer from current correction of performed reaction to optimization of controlled parameters of pre-paired movements. It has been established that the formed precise coordination is realized owing to rapid movements with monomodal asymmetric profile of speed. During habit stabilization time to peak velocity significantly dropped from 274.6 +/- 84.7 to 211.0 +/- 22.9 ms and its value increased from 119.5 +/- 27.8 to 182.2 +/- 44.4 degrees/s. The stabilized habit is provided by uniform movements of ballistic type and characterized by independence from sound indication of final position, its reaching time becoming a function of amplitude-temporal values of speed maximum. It has been found that in the process of motor learning the relation of the duration of acceleration growth to the beginning of movement inhibition becomes an invariant parameter of the central program of precise reactions.     

A parameter optimization approach for the optimal control of large-scale musculoskeletal systems.

This paper describes a computational method for solving optimal control problems involving large-scale, nonlinear, dynamical systems. Central to the approach is the idea that any optimal control problem can be converted into a standard nonlinear programming problem by parameterizing each control history using a set of nodal points, which then become the variables in the resulting parameter optimization problem. A key feature of the method is that it dispenses with the need to solve the two-point, boundary-value problem derived from the necessary conditions of optimal control theory. Gradient-based methods for solving such problems do not always converge due to computational errors introduced by the highly nonlinear characteristics of the costate variables. Instead, by converting the optimal control problem into a parameter optimization problem, any number of well-developed and proven nonlinear programming algorithms can be used to compute the near-optimal control trajectories. The utility of the parameter optimization approach for solving general optimal control problems for human movement is demonstrated by applying it to a detailed optimal control model for maximum-height human jumping. The validity of the near-optimal control solution is established by comparing it to a solution of the two-point, boundary-value problem derived on the basis of a bang-bang optimal control algorithm. Quantitative comparisons between model and experiment further show that the parameter optimization solution reproduces the major features of a maximum-height, countermovement jump (i.e., trajectories of body-segmental displacements, vertical and fore-aft ground reaction forces, displacement, velocity, and acceleration of the whole-body center of mass, pattern of lower-extremity muscular activity, jump height, and total ground contact time).     

Interrogating the effect of enzyme kinetics on metabolism using differentiable constraint-based models

Metabolic models are typically characterized by a large number of parameters. Traditionally, metabolic control analysis is applied to differential equation-based models to investigate the sensitivity of predictions to parameters. A corresponding theory for constraint-based models is lacking, due to their formulation as optimization problems. Here, we show that optimal solutions of optimization problems can be efficiently differentiated using constrained optimization duality and implicit differentiation. We use this to calculate the sensitivities of predicted reaction fluxes and enzyme concentrations to turnover numbers in an enzyme-constrained metabolic model of Escherichia coli. The sensitivities quantitatively identify rate limiting enzymes and are mathematically precise, unlike current finite difference based approaches used for sensitivity analysis. Further, efficient differentiation of constraint-based models unlocks the ability to use gradient information for parameter estimation. We demonstrate this by improving, genome-wide, the state-of-the-art turnover number estimates for E. coli. Finally, we show that this technique can be generalized to arbitrarily complex models. By differentiating the optimal solution of a model incorporating both thermodynamic and kinetic rate equations, the effect of metabolite concentrations on biomass growth can be elucidated. We benchmark these metabolite sensitivities against a large experimental gene knockdown study, and find good alignment between the predicted sensitivities and in vivo metabolome changes. In sum, we demonstrate several applications of differentiating optimal solutions of constraint-based metabolic models, and show how it connects to classic metabolic control analysis.     

Sensory and intrinsic coordination of movement

A recently generalized theory of perceptual guidance (general tau theory) was used to analyse coordination in skilled movement. The theory posits that (i) guiding movement entails controlling closure of spatial and/or force gaps between effectors and goals, by sensing and regulating the tau s of the gaps (the time-to-closure at current closure rate), (ii) a principal way of coordinating movements is keeping the tau s of different gaps in constant ratio (known as tau-coupling), and (iii) intrinsically paced movements are guided and coordinated by tau-coupling onto a tau-guide, tau g, generated in the nervous system and described by the equation tau g = 0.5 (t-T 2/t) where T is the duration of the body movement and t is the time from the start of the movement. Kinematic analysis of hand to mouth movements by human adults, with eyes open or closed, indicated that hand guidance was achieved by maintaining, during 80 85% of the movement, the tau-couplings tau alpha-tau r and tau r-tau g, where tau r is tau of the hand-mouth gap, tau alpha is tau of the angular gap to be closed by steering the hand and tau g is an intrinsic tau-guide.     

On-line optimization of biotechnological processes: I. Application to open algal pond

A new on-line optimization and control procedure applicable to biotechnological systems for which a precise mathematical model is unavailable has been developed and tested. The proposed approach is based on an online search for optimum operating conditions by an automatic system using a modified simplex algorithm to which several features have been added to permit real time operation. The simplex algorithm is the upper level of a hierarchical software package in which the other levels are cost evaluation, control, data acquisition, and signal processing. The optimization method was tested in a laboratory minipond for the cultivation of Spirulina platensis. The controlled parameters were light intensity, optical density, pH, and temperature. The proposed optimization method can be applied to other biological processes provided that the pertinent variables can be measured and controlled and the cost function can be defined mathematically.     

Simulating Discrete and Rhythmic Multi-joint Human Arm Movements by Optimization of Nonlinear Performance Indices

An optimization approach applied to mechanical linkage models is used to simulate human arm movements. Predicted arm trajectories are the result of minimizing a nonlinear performance index that depends on kinematic or dynamic variables of the movement. A robust optimization algorithm is presented that computes trajectories which satisfy the necessary conditions with high accuracy. It is especially adapted to the analysis of discrete and rhythmic movements. The optimization problem is solved by parameterizing each generalized coordinate (e.g., joint angular displacement) in terms of Jacobi polynomials and Fourier series, depending on whether discrete or rhythmic movements are considered, combined with a multiple shooting algorithm. The parameterization of coordinates has two advantages. First, it provides an initial guess for the multiple shooting algorithm which solves the optimization problem with high accuracy. Second, it leads to a low dimensional representation of discrete and rhythmic movements in terms of expansion coefficients. The selection of a suitable feature space is an important prerequisite for comparison, recognition and classification of movements. In addition, the separate computational analysis of discrete and rhythmic movements is motivated by their distinct neurophysiological realizations in the cortex. By investigating different performance indices subject to different boundary conditions, the approach can be used to examine possible strategies that humans adopt in selecting specific arm motions for the performance of different tasks in a plane and in three-dimensional space.     

Analyzing the spatial dynamics of a prey–predator lattice model with social behavior

A lattice prey–predator model is studied. Transition rules applied sequentially describe processes such as reproduction, predation, and death of predators. The movement of predators is governed by a local particle swarm optimization algorithm, which causes the formation of swarms of predators that propagate through the lattice. Starting with a single predator in a lattice fully covered by preys, we observe a wavefront of predators invading the zones dominated by preys; subsequent fronts arise during the transient phase, where a monotonic approach to a fixed point is present. After the transient phase the system enters an oscillatory regime, where the amplitude of oscillations appears to be bounded but is difficult to predict. We observe qualitative similar behavior even for larger lattices. An empirical approach is used to determine the effects of the movement of predators on the temporal dynamics of the system. Our results show that the algorithm used to model the movement of predators increases the proficiency of predators.     

A theory on the control of arbitrary movements

A theory dealing with the control of human, arbitrary movements is proposed. A schema is set up to suggest how the relevant information flows and what kind of operations affect it. A number of successive steps are distinguished in the production of a movement. It is assumed that the intended movement is carried out in the imagination, and that this imaginary movement is composed of a spatial trajectory and an intensity course, which are considered to be independent features of the intended movement. The spatial trajectory will be encoded in a special coding, which is related to the lengths of the muscles that effect the movement. From this special coding of the intended movement static and dynamic control signals can be derived. Because afferent and efferent signals are encoded in the same way in this schema, the evaluation and correction of the performed movement is quite simple. The higher levels in the control schema may function in an abstract way, i.e. the signals at these levels are barely concerned with details of the peripheral motor system. This abstract functioning of the higher levels is based on the numerous feedback mechanisms involved at all levels of control and in the peripheral motor system. Nevertheless, it is possible to incorporate specific peripheral properties in the generation of the control signals. The assumptions in this theory will be discussed and aspects of the proposed control schema will be compared with general control principles.     

An approximate stochastic optimal control framework to simulate nonlinear neuro-musculoskeletal models in the presence of noise

Optimal control simulations have shown that both musculoskeletal dynamics and physiological noise are important determinants of movement. However, due to the limited efficiency of available computational tools, deterministic simulations of movement focus on accurately modelling the musculoskeletal system while neglecting physiological noise, and stochastic simulations account for noise while simplifying the dynamics. We took advantage of recent approaches where stochastic optimal control problems are approximated using deterministic optimal control problems, which can be solved efficiently using direct collocation. We were thus able to extend predictions of stochastic optimal control as a theory of motor coordination to include muscle coordination and movement patterns emerging from non-linear musculoskeletal dynamics. In stochastic optimal control simulations of human standing balance, we demonstrated that the inclusion of muscle dynamics can predict muscle co-contraction as minimal effort strategy that complements sensorimotor feedback control in the presence of sensory noise. In simulations of reaching, we demonstrated that nonlinear multi-segment musculoskeletal dynamics enables complex perturbed and unperturbed reach trajectories under a variety of task conditions to be predicted. In both behaviors, we demonstrated how interactions between task constraint, sensory noise, and the intrinsic properties of muscle influence optimal muscle coordination patterns, including muscle co-contraction, and the resulting movement trajectories. Our approach enables a true minimum effort solution to be identified as task constraints, such as movement accuracy, can be explicitly imposed, rather than being approximated using penalty terms in the cost function. Our approximate stochastic optimal control framework predicts complex features, not captured by previous simulation approaches, providing a generalizable and valuable tool to study how musculoskeletal dynamics and physiological noise may alter neural control of movement in both healthy and pathological movements.     

Optimal compensation for temporal uncertainty in movement planning

Motor control requires the generation of a precise temporal sequence of control signals sent to the skeletal musculature. We describe an experiment that, for good performance, requires human subjects to plan movements taking into account uncertainty in their movement duration and the increase in that uncertainty with increasing movement duration. We do this by rewarding movements performed within a specified time window, and penalizing slower movements in some conditions and faster movements in others. Our results indicate that subjects compensated for their natural duration-dependent temporal uncertainty as well as an overall increase in temporal uncertainty that was imposed experimentally. Their compensation for temporal uncertainty, both the natural duration-dependent and imposed overall components, was nearly optimal in the sense of maximizing expected gain in the task. The motor system is able to model its temporal uncertainty and compensate for that uncertainty so as to optimize the consequences of movement.     

Optimization and Control of Agent-Based Models in Biology: A Perspective

Agent-based models (ABMs) have become an increasingly important mode of inquiry for the life sciences. They are particularly valuable for systems that are not understood well enough to build an equation-based model. These advantages, however, are counterbalanced by the difficulty of analyzing and using ABMs, due to the lack of the type of mathematical tools available for more traditional models, which leaves simulation as the primary approach. As models become large, simulation becomes challenging. This paper proposes a novel approach to two mathematical aspects of ABMs, optimization and control, and it presents a few first steps outlining how one might carry out this approach. Rather than viewing the ABM as a model, it is to be viewed as a surrogate for the actual system. For a given optimization or control problem (which may change over time), the surrogate system is modeled instead, using data from the ABM and a modeling framework for which ready-made mathematical tools exist, such as differential equations, or for which control strategies can explored more easily. Once the optimization problem is solved for the model of the surrogate, it is then lifted to the surrogate and tested. The final step is to lift the optimization solution from the surrogate system to the actual system. This program is illustrated with published work, using two relatively simple ABMs as a demonstration, Sugarscape and a consumer-resource ABM. Specific techniques discussed include dimension reduction and approximation of an ABM by difference equations as well systems of PDEs, related to certain specific control objectives. This demonstration illustrates the very challenging mathematical problems that need to be solved before this approach can be realistically applied to complex and large ABMs, current and future. The paper outlines a research program to address them.     

Impedance characteristics of a neuromusculoskeletal model of the human arm II. Movement control

The modulation of neuromusculoskeletal impedance during movements is analysed using a motor control model of the human arm. The motor control system combines feedback and feedforward control and both control modes are determined in one optimization process. In the model, the stiffness varies at the double movement frequency for 2-Hz oscillatory elbow movements and has high values at the movement reversals. During goal-directed two-degrees-of-freedom arm movements, the stiffness is decreased during the movement and may be increased in the initial and final phases, depending on the movement velocity. The stiffness has a considerable curl during the movement, as was also observed in experimental data. The dynamic stiffness patterns of the model can be explained basically by the α−γ coactivation scheme where feedback gains covary with motor control signals. In addition to the modulation of the gain factors, it is argued that the variation of the intrinsic stiffness has a considerable effect on movement control, especially during fast movements. Received: 14 October 1997 / Accepted in revised form: 18 May 1999     

小脑控制不同类型眼动的神经编码机制：从小脑皮层到小脑核团

近年来许多研究发现,小脑作为运动控制的主要脑区,除参与运动控制外也与孤独症、精神分裂症、奖励相关的认知功能和社会行为有关,因此小脑相关研究越来越受到重视。研究小脑参与运动学习和运动控制的神经机制是神经科学中最重要的课题之一。眼睛运动的肌肉协调和生物运动特征比其他类型的运动更简单,这使眼动成为研究小脑在运动控制中作用的理想模型。作为收集外界信息的主要方式之一,视觉对日常生活至关重要。为确保清晰视觉,3种主要类型的眼动（眼跳、平滑追随眼动（SPEM）和注视）需受小脑的精确控制,以确保静止或移动的物体保持在视小凹的中心。异常眼动可导致视力障碍,并可作为诊断各种疾病的临床指标。因此,眼动控制研究具有重要的医学和生物学意义。虽然对小脑皮层和顶核在调节眼动中的作用有基本了解,但眼动动力学编码的确切神经机制,尤其是小脑顶核控制追随眼动和注视的神经机制仍不清楚。本综述总结了目前小脑在运动和认知等方面的主要研究问题与小脑相关研究的潜在应用价值,以及近年来有关小脑控制眼动的相关文献,并深入探讨了利用单细胞记录和线性回归模型分析小脑皮层和顶核同一神经元同时参与控制不同类型的眼动,而不同类型眼动的不同动力学参数编码原则不同。此外,基于检测微眼跳的研究结果,我们讨论了小脑顶核参与控制视觉注视的可能神经机制。最后,讨论了最近技术进步给小脑研究带来的新机遇,为今后与小脑相关的研究和脑控义肢的优化控制（例如通过单独改善运动参数优化义肢控制）提供了新思路。