Iterative manual control model of human operator

In this paper, we present an iterative manual control model of a human operator performing some repetitive task. Various aspects of the model are discussed in detail. Experiments have been done to study the human capability to perform the tasks by learning iteratively. Results of the experiments show the ability of the human operator to perform the tracking of a desired trajectory for some unknown non-linear system with quite reasonable accuracy during the iteration process. It is concluded that the human operator performs the repetitive task by modifying his control action using error and error rate in each iteration. During the modification, the human operator assigns different weights to the error and error rate in each iteration. These results can be implemented in designing more efficient iterative learning control algorithms. Received: 29 September 1998 / Accepted in revised form: 12 May 1999     

Coordinates transformation and learning control for visually-guided voluntary movement with iteration: A Newton-like method in a function space

In order to control visually-guided voluntary movements, the central nervous system (CNS) must solve the following three computational problems at different levels: (1) determination of a desired trajectory in the visual coordinates, (2) transformation of the coordinates of the desired trajectory to the body coordinates and (3) generation of motor command. In this paper, the second and the third problems are treated at computational, representational and hardware levels of Marr. We first study the problems at the computational level, and then propose an iterative learning scheme as a possible algorithm. This is a trial and error type learning such as repetitive training of golf swing. The amount of motor command needed to coordinate activities of many muscles is not determined at once, but in a step-wise, trial and error fashion in the course of a set of repetitions. Actually, the motor command in the (n+1)-th iteration is a sum of the motor command in then-th iteration plus two modification terms which are, respectively, proportional to acceleration and speed errors between the desired trajectory and the realized trajectory in then-th iteration. We mathematically formulate this iterative learning control as a Newton-like method in functional spaces and prove its convergence under appropriate mathematical conditions with use of dynamical system theory and functional analysis. Computer simulations of this iterative learning control of a robotic manipulator in the body or visual coordinates are shown. Finally, we propose that areas 2, 5, and 7 of the sensory association cortex are possible sites of this learning control. Further we propose neural network model which acquires transformation matrices from acceleration or velocity to motor command, which are used in these schemes.     

SVM-based learning control of space robots in capturing operation

In this paper, we presents a novel approach for tracking and catching operation of space robots using learning and transferring human control strategies (HCS). We firstly use an efficient support vector machine (SVM) to parametrize the model of HCS. Then we develop a new SVM-based learning structure to better implement human control strategy learning in tracking and capturing control. The approach is fundamentally valuable in dealing with some problems such as small sample data and local minima, and so on. Therefore this approach is efficient in modeling, understanding and transferring its learning process. The simulation results attest that this approach is useful and feasible in generating tracking trajectory and catching objects autonomously.     

Iterative learning control applied to a hybrid rehabilitation exoskeleton system powered by PAM and FES

Currently upper limb exoskeleton rehabilitation robots powered by electric motors used in the hospitals are usually cumbersome, bulky and unmovable. Our developed RUPERT is a low-cost lightweight portable exoskeleton rehabilitation robot that can encourage stroke patients with high stiffness in arm flexor muscles to receive frequent intensive rehabilitation trainings in the community or home, but its joints are unidirectionally actuated by pneumatic artificial muscles (PAMs). RUPERT with one PAM of each joint is not suitable for stroke patients with weak muscles in the flaccid paralysis period. Functional electrical stimulation (FES) uses current with low frequency to activate paralyzed muscles, which can produce muscle torque and compensate the unidirectional drawbacks of RUPERT, so as to realize the two-way motion of its joints for passive reaching trainings. As both the exoskeleton robot driven by PAMs and neuromuscular skeletal system under FES possess the highly nonlinear and time-varying characteristics, which adds control difficulty to the hybrid dynamic system, iterative learning control (ILC) is chosen to control this newly designed hybrid rehabilitation system to realize repetitive task trainings.     

Human learning characteristics in the tracking tasks of iterative nature

In this paper, human learning characteristics in the tracking tasks of iterative nature are investigated. Various linear and nonlinear systems are used as plant, and a human operator has to generate the proper control inputs to force these systems in tracking the desired trajectory. The learning behaviour of the human operator in modifying his control actions is studied and it is observed that the human operator can improve his performance quite efficiently despite the unavailability of any information about the system or the desired trajectories. It is concluded from the experiments that the human operator not only use the information that is directly available to him (error in this case), but also extracts some useful information (e.g. error rate) that he feels is necessary to generate a good control action. The limitation of the human performance is studied in frequency domain, and the performance of the human operator against the frequency bandwidth of error and error rate signals are highlighted. Analysis of the results revealed that a human operator gives more importance to the error rate in generating his control actions and, accordingly, it is observed that his limitation in term of performance is more sensitive to the frequency bandwidth of the error rate as compared to the error. The human operator cannot improve his performance once the frequency components of the error or error rates shift to the higher frequencies, say above 1.0 Hz.     

A model of time estimation and error feedback in predictive timing behavior

Two key features of sensorimotor prediction are preprogramming and adjusting of performance based on previous experience. Oculomotor tracking of alternating visual targets provides a simple paradigm to study this behavior in the motor system; subjects make predictive eye movements (saccades) at fast target pacing rates (>0.5 Hz). In addition, the initiation errors (latencies) during predictive tracking are correlated over a small temporal window (correlation window) suggesting that tracking performance within this time range is used in the feedback process of the timing behavior. In this paper, we propose a closed-loop model of this predictive timing. In this model, the timing between movements is based on an internal estimation of stimulus timing (an internal clock), which is represented by a (noisy) signal integrated to a threshold. The threshold of the integrate-to-fire mechanism is determined by the timing between movements made within the correlation window of previous performance and adjusted by feedback of recent and projected initiation error. The correlation window size increases with repeated tracking and was estimated by two independent experiments. We apply the model to several experimental paradigms and show that it produces data specific to predictive tracking: a gradual shift from reaction to prediction on initial tracking, phase transition and hysteresis as pacing frequency changes, scalar property, continuation of predictive tracking despite perturbations, and intertrial correlations of a specific form. These results suggest that the process underlying repetitive predictive motor timing is adjusted by the performance and the corresponding errors accrued over a limited time range and that this range increases with continued confidence in previous performance.     

An iterative learning strategy for the auto-tuning of the feedforward and feedback controller in type-1 diabetes

This paper proposes a scheme for the control of the blood glucose in subjects with type-1 diabetes mellitus based on the subcutaneous (s.c.) glucose measurement and s.c. insulin administration. The tuning of the controller is based on an iterative learning strategy that exploits the repetitiveness of the daily feeding habit of a patient. The control consists of a mixed feedback and feedforward contribution whose parameters are tuned through an iterative learning process that is based on the day-by-day automated analysis of the glucose response to the infusion of exogenous insulin. The scheme does not require any a priori information on the patient insulin/glucose response, on the meal times and on the amount of ingested carbohydrates (CHOs). Thanks to the learning mechanism the scheme is able to improve its performance over time. A specific logic is also introduced for the detection and prevention of possible hypoglycaemia events. The effectiveness of the methodology has been validated using long-term simulation studies applied to a set of nine in silico patients considering realistic uncertainties on the meal times and on the quantities of ingested CHOs.     

Learning Control for Biomimetic Undulating Fins: An Experimental Study

Learning control should focus on imitating natural fish's adaptability to complex and dynamic environment to some extent, rather than mimicking streamlined shapes or specific actuators to develop more mechanical prototypes. In this paper, an experimental study on a proposed learning control of the robotic undulating fin, RoboGnilos, is suggested and explored. This study takes inspirations from biological world to practical control algorithms. In detail, an iterative learning scheme based control is studied with the cooperation of a filter to reduce the measurement noise, and a curve fitting component to keep the necessary phase difference between neighboring fin rays. Moreover, the iterative learning control algorithm is designed and implemented for practical applications. The experimental results validate that the proposed learning control can effectively improve the propulsion of RoboGnilos. For instance, the steady propulsion velocity may be enhanced by over 40% with some specified parameters.     

Electrophysiological evidence for right frontal lobe dominance in spatial visuomotor learning

Slow negative potential shifts were recorded together with the error made in motor performance when two different groups of 14 students tracked visual stimuli with their right hand. Various visuomotor tasks were compared. A tracking task (T) in which subjects had to track the stimulus directly, showed no decrease of error in motor performance during the experiment. In a distorted tracking task (DT) a continuous horizontal distortion of the visual feedback had to be compensated. The additional demands of this task required visuomotor learning. Another learning condition was a mirrored-tracking task (horizontally inverted tracking, hIT), i.e. an elementary function, such as the concept of changing left and right was interposed between perception and action. In addition, subjects performed a no-tracking control task (NT) in which they started the visual stimulus without tracking it. A slow negative potential shift was associated with the visuomotor performance (TP: tracking potential). In the learning tasks (DT and hIT) this negativity was significantly enhanced over the anterior midline and in hIT frontally and precentrally over both hemispheres. Comparing hIT and T for every subject, the enhancement of the tracking potential in hIT was correlated with the success in motor learning in frontomedial and bilaterally in frontolateral recordings (r = 0.81-0.88). However, comparing DT and T, such a correlation was only found in frontomedial and right frontolateral electrodes (r = 0.5-0.61), but not at the left frontolateral electrode. These experiments are consistent with previous findings and give further neurophysiological evidence for frontal lobe activity in visuomotor learning. The hemispherical asymmetry is discussed in respect to hemispherical specialization (right frontal lobe dominance in spatial visuomotor learning).     

Adaptive dynamic programming as a theory of sensorimotor control

Many characteristics of sensorimotor control can be explained by models based on optimization and optimal control theories. However, most of the previous models assume that the central nervous system has access to the precise knowledge of the sensorimotor system and its interacting environment. This viewpoint is difficult to be justified theoretically and has not been convincingly validated by experiments. To address this problem, this paper presents a new computational mechanism for sensorimotor control from a perspective of adaptive dynamic programming (ADP), which shares some features of reinforcement learning. The ADP-based model for sensorimotor control suggests that a command signal for the human movement is derived directly from the real-time sensory data, without the need to identify the system dynamics. An iterative learning scheme based on the proposed ADP theory is developed, along with rigorous convergence analysis. Interestingly, the computational model as advocated here is able to reproduce the motor learning behavior observed in experiments where a divergent force field or velocity-dependent force field was present. In addition, this modeling strategy provides a clear way to perform stability analysis of the overall system. Hence, we conjecture that human sensorimotor systems use an ADP-type mechanism to control movements and to achieve successful adaptation to uncertainties present in the environment.     

Comparison of ribotyping and repetitive extragenic palindromic-PCR for identification of fecal Escherichia coli from humans and animals

This report compares the performances of two popular genotypic methods used for tracking the sources of fecal pollution in water, ribotyping and repetitive extragenic palindromic-PCR (rep-PCR). The rep-PCR was more accurate, reproducible, and efficient in associating DNA fingerprints of fecal Escherichia coli with human and animal hosts of origin.     

Computer control of pH and DO in a laboratory fermenter using a neural network technique

In this contribution, the advantages of the artificial neural network approach to the identification and control of a laboratory-scale biochemical reactor are demonstrated. It is very important to be able to maintain the levels of two process variables, pH and dissolved oxygen (DO) concentration, over the course of fermentation in biosystems control. A PC-supported, fully automated, multi-task control system has been designed and built by the authors. Forward and inverse neural process models are used to identify and control both the pH and the DO concentration in a fermenter containing a Saccharomyces cerevisiae based-culture. The models are trained off-line, using a modified back-propagation algorithm based on conjugate gradients. The inverse neural controller is augmented by a new adaptive term that results in a system with robust performance. Experimental results have confirmed that the regulatory and tracking performances of the control system proposed are good.     

Neuromuscular control of the point to point and oscillatory movements of a sagittal arm with the actor-critic reinforcement learning method

In this study, we have used a single link system with a pair of muscles that are excited with alpha and gamma signals to achieve both point to point and oscillatory movements with variable amplitude and frequency.The system is highly nonlinear in all its physical and physiological attributes. The major physiological characteristics of this system are simultaneous activation of a pair of nonlinear muscle-like-actuators for control purposes, existence of nonlinear spindle-like sensors and Golgi tendon organ-like sensor, actions of gravity and external loading. Transmission delays are included in the afferent and efferent neural paths to account for a more accurate representation of the reflex loops.A reinforcement learning method with an actor-critic (AC) architecture instead of middle and low level of central nervous system (CNS), is used to track a desired trajectory. The actor in this structure is a two layer feedforward neural network and the critic is a model of the cerebellum. The critic is trained by state-action-reward-state-action (SARSA) method. The critic will train the actor by supervisory learning based on the prior experiences. Simulation studies of oscillatory movements based on the proposed algorithm demonstrate excellent tracking capability and after 280 epochs the RMS error for position and velocity profiles were 0.02, 0.04 rad and rad/s, respectively.     

Comparison of Ribotyping and Repetitive Extragenic Palindromic-PCR for Identification of Fecal Escherichia coli from Humans and Animals

This report compares the performances of two popular genotypic methods used for tracking the sources of fecal pollution in water, ribotyping and repetitive extragenic palindromic-PCR (rep-PCR). The rep-PCR was more accurate, reproducible, and efficient in associating DNA fingerprints of fecal Escherichia coli with human and animal hosts of origin.     

Performance of deep learning restoration methods for the extraction of particle dynamics in noisy microscopy image sequences

Particle tracking in living systems requires low light exposure and short exposure times to avoid phototoxicity and photobleaching and to fully capture particle motion with high-speed imaging. Low-excitation light comes at the expense of tracking accuracy. Image restoration methods based on deep learning dramatically improve the signal-to-noise ratio in low-exposure data sets, qualitatively improving the images. However, it is not clear whether images generated by these methods yield accurate quantitative measurements such as diffusion parameters in (single) particle tracking experiments. Here, we evaluate the performance of two popular deep learning denoising software packages for particle tracking, using synthetic data sets and movies of diffusing chromatin as biological examples. With synthetic data, both supervised and unsupervised deep learning restored particle motions with high accuracy in two-dimensional data sets, whereas artifacts were introduced by the denoisers in three-dimensional data sets. Experimentally, we found that, while both supervised and unsupervised approaches improved tracking results compared with the original noisy images, supervised learning generally outperformed the unsupervised approach. We find that nicer-looking image sequences are not synonymous with more precise tracking results and highlight that deep learning algorithms can produce deceiving artifacts with extremely noisy images. Finally, we address the challenge of selecting parameters to train convolutional neural networks by implementing a frugal Bayesian optimizer that rapidly explores multidimensional parameter spaces, identifying networks yielding optimal particle tracking accuracy. Our study provides quantitative outcome measures of image restoration using deep learning. We anticipate broad application of this approach to critically evaluate artificial intelligence solutions for quantitative microscopy.     

Neuromuscular control of the point to point and oscillatory movements of a sagittal arm with the actor–critic reinforcement learning method

In this study, we have used a single link system with a pair of muscles that are excited with alpha and gamma signals to achieve both point to point and oscillatory movements with variable amplitude and frequency.

The system is highly nonlinear in all its physical and physiological attributes. The major physiological characteristics of this system are simultaneous activation of a pair of nonlinear muscle-like-actuators for control purposes, existence of nonlinear spindle-like sensors and Golgi tendon organ-like sensor, actions of gravity and external loading. Transmission delays are included in the afferent and efferent neural paths to account for a more accurate representation of the reflex loops.

A reinforcement learning method with an actor–critic (AC) architecture instead of middle and low level of central nervous system (CNS), is used to track a desired trajectory. The actor in this structure is a two layer feedforward neural network and the critic is a model of the cerebellum. The critic is trained by state-action-reward-state-action (SARSA) method. The critic will train the actor by supervisory learning based on the prior experiences. Simulation studies of oscillatory movements based on the proposed algorithm demonstrate excellent tracking capability and after 280 epochs the RMS error for position and velocity profiles were 0.02, 0.04?rad and rad/s, respectively.     

Novel intermolecular iterative mechanism for biosynthesis of mycoketide catalyzed by a bimodular polyketide synthase

In recent years, remarkable versatility of polyketide synthases (PKSs) has been recognized; both in terms of their structural and functional organization as well as their ability to produce compounds other than typical secondary metabolites. Multifunctional Type I PKSs catalyze the biosynthesis of polyketide products by either using the same active sites repetitively (iterative) or by using these catalytic domains only once (modular) during the entire biosynthetic process. The largest open reading frame in Mycobacterium tuberculosis, pks12, was recently proposed to be involved in the biosynthesis of mannosyl-beta-1-phosphomycoketide (MPM). The PKS12 protein contains two complete sets of modules and has been suggested to synthesize mycoketide by five alternating condensations of methylmalonyl and malonyl units by using an iterative mode of catalysis. The bimodular iterative catalysis would require transfer of intermediate chains from acyl carrier protein domain of module 2 to ketosynthase domain of module 1. Such bimodular iterations during PKS biosynthesis have not been characterized and appear unlikely based on recent understanding of the three-dimensional organization of these proteins. Moreover, all known examples of iterative PKSs so far characterized involve unimodular iterations. Based on cell-free reconstitution of PKS12 enzymatic machinery, in this study, we provide the first evidence for a novel "modularly iterative" mechanism of biosynthesis. By combination of biochemical, computational, mutagenic, analytical ultracentrifugation and atomic force microscopy studies, we propose that PKS12 protein is organized as a large supramolecular assembly mediated through specific interactions between the C- and N-terminus linkers. PKS12 protein thus forms a modular assembly to perform repetitive condensations analogous to iterative proteins. This novel intermolecular iterative biosynthetic mechanism provides new perspective to our understanding of polyketide biosynthetic machinery and also suggests new ways to engineer polyketide metabolites. The characterization of novel molecular mechanisms involved in biosynthesis of mycobacterial virulent lipids has opened new avenues for drug discovery.     

Learning to simulate others' decisions

A fundamental challenge in social cognition is how humans learn another person's values to predict their decision-making behavior. This form of learning is often assumed to require simulation of the other by direct recruitment of one's own valuation process to model the other's process. However, the cognitive and neural mechanism of simulation learning is not known. Using behavior, modeling, and fMRI, we show that simulation involves two learning signals in a hierarchical arrangement. A simulated-other's reward prediction error processed in ventromedial prefrontal cortex mediated simulation by direct recruitment, being identical for valuation of the self and simulated-other. However, direct recruitment was insufficient for learning, and also required observation of the other's choices to generate?a simulated-other's action prediction error encoded in dorsomedial/dorsolateral prefrontal cortex. These findings show that simulation uses a core prefrontal circuit for modeling the other's valuation to generate prediction and an adjunct circuit for tracking behavioral variation to refine prediction.     

Development of a robotic walking simulator for gait rehabilitation]

Restoration of gait is a major concern of rehabilitation after stroke or spinal cord injury. Modern concepts of motor learning favour a task-specific repetitive approach, i.e. "whoever wants to learn to walk again must walk." However, the physical demands this places on the therapist, is a limiting factor in the clinical routine setting. This article describes a robotic walking simulator for gait training that enables wheelchair-bound subjects to freely carry out repetitive practicing of an individually adapted gait pattern under simulation of the manual guidance of an experienced therapist. The technical principle applied makes use of programmable footplates with permanent foot/machine contact in combination with compliance control. The solution chosen comprises a planar parallel-serial hybrid kinematic system with three degrees of freedom that moves the feet in the sagittal plane. Gait analysis while floor walking and stair climbing, clinical practicability and safety aspects were the basis for the design. A variable compliance control enables man-machine interaction, ranging from purely position controlled movement to full compliance during swing phase above a virtual ground profile. In full compliance mode the robotic walking simulator behaves like a haptic device. The concept presented offers new prospects for individualized gait rehabilitation.     

Modelling of direct motor program learning in fast human arm motions

We propose and simulate a new paradigm for organization of motor control in fast and accurate human arm motions. We call the paradigm direct motor program learning since the control programs are learned directly without knowing or learning the dynamics of a controlled system.The idea is to approximate the dependence of the motor control programs on the vector of the task parameters rather than to use a model of the system dynamics. We apply iterative learning control and scattered data multivariate approximation techniques to achieve the goal. The advantage of the paradigm is that the control complexity depends neither on the order nor on the nonlinearity of the system dynamics.We simulate the direct motor program learning paradigm in the task of point-to-point control of fast planar human arm motions. Simulation takes into account nonlinear arm dynamics, muscle force dynamics, delay in low-level reflex feedback, time dependence of the feedback gains and coactivation of antagonist muscles. Despite highly nonlinear time-variant dynamics of the controlled system, reasonably good motion precision is obtained over a wide range of the task parameters (initial and final positions of the arm). The simulation results demonstrate that the paradigm is indeed viable and could be considered as a possible explanation for the organization of motor control of fast motions.