The production and generalization of large-magnitude heart rate deceleration by contingently faded biofeedback

Eight subjects were taught to decrease their heart rates via biofeedback training. Four of these received contingently faded, beat-by-beat analogue feedback and contingent reinforcement each time their performance met a specified and adjusting criterion. The other four received continuous, beat-by-beat analogue feedback, but not the contingent reinforcement. Subjects in the two groups were yoked to ensure equal densities of reinforcement. Subjects in the first group were asked to decrease heart rates 15% from baseline and were then trained using only 75%, 50% and 25% of beat-by-beat feedback. It was hypothesized that the immediate reinforcement of appropriate behavior and the contingent fading(following mastery) of feedback would aid in the generalization of the response. Following completion of all criterion steps or 10 training sessions, whichever came first, all subjects were tested with no feedback and no contingent reinforcement. The group receiving contingently faded feedback training showed a significantly greater heart rate decrease in the training sessions and also the test session. These results were interpreted as indicating that biofeedback can be conceptualized as an operant conditioning paradigm, and that the use of operant techniques may help subjects produce clinically significant changes.This research was supported in part by a grant to Robert J. Gatchel from the National Heart, Lung, and Blood Institute (Grant No. NIH HL 21426-01).     

Partial reinforcement in human biofeedback learning

This paper reviews the evidence for the efficacy of partial reinforcement in producing resistance to extinction in human biofeedback experiments. The methodological criteria necessary to demonstrate such effects are discussed, as is the status of the analogy of reinforcement and information feedback. It is suggested that the problem of maintaining responding in the absence of feedback should be tackled empirically rather than assuming the validity of findings from other areas of learning theory.The author thanks Jack Rachman and Clare Philips for comments on a previous version of this paper.     

Operant conditioning of skin resistance tonic levels

The purpose of the current investigation was to determine whether increases and decreases in skin resistance tonic level could be controlled by individuals given discrete visual feedback of such activity. Thirty-six male undergraduate students served as subjects. They were assigned randomly in equal numbers to four groups; two of the groups received accurate feedback of skin resistance level changes and two received inaccurate feedback. The two accurate-feedback groups differed with respect to the order in which increases and decreases in skin resistance level were reinforced. Each noncontingent group was matched with one of the contingent groups in terms of reinforcement density. The results indicated that accurate feedback produced skin resistance level changes consistent with the type of reinforcement employed. However, operant control was not clearly sustained subsequent to a reversal in the type of tonic level change reinforced. Some problems related to the clinical application of skin resistance level training are discussed.     

Operant conditioning of skin resistance tonic levels

The purpose of the current investigation was to determine whether increases and decreases in skin resistance tonic level could be controlled by individuals given discrete visual feedback of such activity. Thirty-six male undergraduate students served as subjects. They were assigned randomly in equal numbers to four groups; two of the groups received accurate feedback of skin resistance level changes and two received inaccurate feedback. The two accurate-feedback groups differed with respect to the order in which increases and decreases in skin resistance level were reinforced. Each noncontingent group was matched with one of the contingent groups in terms of reinforcement density. The results indicated that accurate feedback produced skin resistance level changes consistent with the type of reinforcement employed. However, operant control was not clearly sustained subsequent to a reversal in the type of tonic level change reinforced. Some problems related to the clinical application of skin resistance level training are discussed.Portions of this paper were presented at the meeting of the Midwestern Psychological Association, Chicago, 1973.     

Learning to Obtain Reward,but Not Avoid Punishment,Is Affected by Presence of PTSD Symptoms in Male Veterans: Empirical Data and Computational Model

Post-traumatic stress disorder (PTSD) symptoms include behavioral avoidance which is acquired and tends to increase with time. This avoidance may represent a general learning bias; indeed, individuals with PTSD are often faster than controls on acquiring conditioned responses based on physiologically-aversive feedback. However, it is not clear whether this learning bias extends to cognitive feedback, or to learning from both reward and punishment. Here, male veterans with self-reported current, severe PTSD symptoms (PTSS group) or with few or no PTSD symptoms (control group) completed a probabilistic classification task that included both reward-based and punishment-based trials, where feedback could take the form of reward, punishment, or an ambiguous “no-feedback” outcome that could signal either successful avoidance of punishment or failure to obtain reward. The PTSS group outperformed the control group in total points obtained; the PTSS group specifically performed better than the control group on reward-based trials, with no difference on punishment-based trials. To better understand possible mechanisms underlying observed performance, we used a reinforcement learning model of the task, and applied maximum likelihood estimation techniques to derive estimated parameters describing individual participants’ behavior. Estimations of the reinforcement value of the no-feedback outcome were significantly greater in the control group than the PTSS group, suggesting that the control group was more likely to value this outcome as positively reinforcing (i.e., signaling successful avoidance of punishment). This is consistent with the control group’s generally poorer performance on reward trials, where reward feedback was to be obtained in preference to the no-feedback outcome. Differences in the interpretation of ambiguous feedback may contribute to the facilitated reinforcement learning often observed in PTSD patients, and may in turn provide new insight into how pathological behaviors are acquired and maintained in PTSD.     

A device for functional residual capacity controlled biofeedback of respiratory resistance

A computer-aided procedure is presented providing subjects with analogous visual feedback of respiratory resistance, which is continuously measured using the forced oscillation method. Simultaneous pneumotachographical control of the breathing volume curve makes it possible to prevent reinforcement for decreases of respiratory resistance which are due to increases of functional residual capacity (FRC). Lung hyperinflation is an unsuitable way to reduce respiratory resistance; if it occurs, feedback is interrupted until the subject decreases his FRC to its initial level. Analysis of the data of 15 adult asthmatic subjects which underwent a 12-sessions feedback training showed that no substantial changes of FRC appeared within feedback trials. Advantages of this new biofeedback technique compared to other procedures are discussed with regard to volume control and feedback signal.     

Brain-like Intelligent Decision-making Based on Basal Ganglia and Its Application in Automatic Car-following

The anthropomorphic intelligence of autonomous driving has been a research hotspot in the world.However,current stud-ies have not been able to reveal the mechanism of drivers'natural driving behaviors.Therefore,this thesis starts from the perspective of cognitive decision-making in the human brain,which is inspired by the regulation of dopamine feedback in the basal ganglia,and a reinforcement learning model is established to solve the brain-like intelligent decision-making problems in the process of interacting with the environment.In this thesis,first,a detailed bionic mechanism architecture based on basal ganglia was proposed by the consideration and analysis of its feedback regulation mechanism;second,the above mechanism was transformed into a reinforcement Q-learning model,so as to implement the learning and adaptation abilities of an intelligent vehicle for brain-like intelligent decision-making during car-following;finally,the feasibility and effectiveness of the proposed method were verified by the simulations and real vehicle tests.     

The significance of unrecognized feedback stimuli in cognitive activity

In adult emotionally excitable persons the role was studied of unrecognized reinforcement forms in the function of differentiation of time microintervals. As feedback stimuli the word "good" (positive reinforcement) and a word connected with negative emotions of the subject (negative reinforcement) were applied. Experimental confirmation was obtained of the hypothesis that unrecognized phenomena of environment can influence conscious psychic activity, the process of man learning. Unrecognized semantic stimuli can function as a reinforcing factor and in this way participate in the process of learning of cognitive activity realized at conscious level.     

Operant conditioning of brain steady potential shifts in man.

Steady potential shifts (SPS) recorded from the scalp were conditioned operantly by visual and acoustical feedback. Three groups of seven subjects were each tested with a different response-reinforcement contingency: positive reinforcement for a positive SPS after a cue stimulus, positive reinforcement for a negative SPS after a cue stimulus, and noncontingent reinforcement. The steady potential shifts learned under these three conditions differed significantly. Negative shifts were associated with subjective feelings of activation, positive shifts with inactivation. Cortical genesis and possible artifacts are discussed.     

EMG Biofeedback: The Effects of CRF, FR, VR, FI, and VI Schedules of Reinforcement on the Acquisition and Extinction of Increases in Forearm Muscle Tension

Biofeedback was used to increase forearm-muscle tension. Feedback was delivered under continuous reinforcement (CRF), variable interval (VI), fixed interval (FI), variable ratio (VR), and fixed ratio (FR) schedules of reinforcement when college students increased their muscle tension (electromyograph, EMG) above a high threshold. There were three daily sessions of feedback, and Session 3 was immediately followed by a session without feedback (extinction). The CRF schedule resulted in the highest EMG, closely followed by the FR and VR schedules, and the lowest EMG scores were produced by the FI and VI schedules. Similarly, the CRF schedule resulted in the greatest amount of time-above-threshold and the VI and FI schedules produced the lowest time-above-threshold. The highest response rates were generated by the FR schedule, followed by the VR schedule. The CRF schedule produced relatively low response rates, comparable to the rates under the VI and FI schedules. Some of the data are consistent with the partial-reinforcement-extinction effect. The present data suggest that different schedules of feedback should be considered in muscle-strengthening contexts such as during the rehabilitation of muscles following brain damage or peripheral nervous-system injury.     

The Computational Development of Reinforcement Learning during Adolescence

Adolescence is a period of life characterised by changes in learning and decision-making. Learning and decision-making do not rely on a unitary system, but instead require the coordination of different cognitive processes that can be mathematically formalised as dissociable computational modules. Here, we aimed to trace the developmental time-course of the computational modules responsible for learning from reward or punishment, and learning from counterfactual feedback. Adolescents and adults carried out a novel reinforcement learning paradigm in which participants learned the association between cues and probabilistic outcomes, where the outcomes differed in valence (reward versus punishment) and feedback was either partial or complete (either the outcome of the chosen option only, or the outcomes of both the chosen and unchosen option, were displayed). Computational strategies changed during development: whereas adolescents’ behaviour was better explained by a basic reinforcement learning algorithm, adults’ behaviour integrated increasingly complex computational features, namely a counterfactual learning module (enabling enhanced performance in the presence of complete feedback) and a value contextualisation module (enabling symmetrical reward and punishment learning). Unlike adults, adolescent performance did not benefit from counterfactual (complete) feedback. In addition, while adults learned symmetrically from both reward and punishment, adolescents learned from reward but were less likely to learn from punishment. This tendency to rely on rewards and not to consider alternative consequences of actions might contribute to our understanding of decision-making in adolescence.     

Computational models of reinforcement learning: the role of dopamine as a reward signal

Reinforcement learning is ubiquitous. Unlike other forms of learning, it involves the processing of fast yet content-poor feedback information to correct assumptions about the nature of a task or of a set of stimuli. This feedback information is often delivered as generic rewards or punishments, and has little to do with the stimulus features to be learned. How can such low-content feedback lead to such an efficient learning paradigm? Through a review of existing neuro-computational models of reinforcement learning, we suggest that the efficiency of this type of learning resides in the dynamic and synergistic cooperation of brain systems that use different levels of computations. The implementation of reward signals at the synaptic, cellular, network and system levels give the organism the necessary robustness, adaptability and processing speed required for evolutionary and behavioral success.     

Neurofeedback—The Significance of Reinforcement and the Search for an Appropriate Strategy for the Success of Self-regulation

Nine healthy children took part in five sessions of feedback and instrumental conditioning of slow cortical potentials (SCPs). The feedback conditions (the relation between the feedback signal and amplitude of SCP) were inverted after two sessions. Neither the children nor the therapists were aware of this change. The adjustment of the children to the new feedback setting and the self-regulation strategies employed were investigated. The results were as follows: (a) Healthy children achieved control over cortical negativity within two sessions. (b) The change of feedback conditions worsened the regulation abilities, which then improved again within the following three sessions. (c) After the first two sessions, the participants were able to describe strategies that were successful during different phases of self-regulation. (d) Following the change in the feedback conditions, the children re-evaluated the way they influenced their SCPs. However, they did not alter the cognitive or behavioral strategies. The study demonstrated that positive and negative reinforcement and the knowledge of results are more important for successful self-regulation than the search for effective strategies. The relevance of these findings is discussed.     

Operant conditioning of brain steady potential shifts in man

Steady potential shifts (SPS) recorded from the scalp were conditioned operantly by visual and acoustical feedback. Three groups of seven subjects were each tested with a different response-reinforcement contingency: positive reinforcement for a positive SPS after a cue stimulus, positive reinforcement for a negative SPS after a cue stimulus, and noncontingent reinforcement. The steady potential shifts learned under these three conditions differed significantly. Negative shifts were associated with subjective feelings of activation, positive shifts with inactivation. Cortical genesis and possible artifacts are discussed.This research was supported by a grant from theFond zur Förderung der wissenschaftlichen Forschung.     

The effect of frontal EMG biofeedback training on the behavior of children with activity-level problems

N=1 withdrawal designs were employed with three children evidencing activity-level problems. Tutoring sessions occurred daily over a 2 1/2-month period. Each child was reinforced for decreasing frontalis muscle tension during auditory feedback while working arithmetic problems. Feedback was faded while tension reduction reinforcement was maintained. These procedures were repeated with reinforcement for increasing, rather than decreasing, muscle tension. Frontal EMG level, percent time on task, and motoric activity rate were obtained during sessions. Parent ratings of problem behavior in the home were recorded daily. Biofeedback with reinforcement was effective in both raising and lowering muscle tension. Effects were maintained by reinforcement. Results suggest a direct relationship between tension and activity levels. Academic performance and problem behavior improved significantly with reductions in EMG activity, although individual exceptions to these findings were present. Results lend support to the efficacy of frontal EMG biofeedback training in reducing activity, increasing attention to an academic task, and reducing problem behaviors.     

Perceptual learning and decision-making in human medial frontal cortex

The dominant view that perceptual learning is accompanied by changes in early sensory representations has recently been challenged. Here we tested the idea that perceptual learning can be accounted for by reinforcement learning involving changes in higher decision-making areas. We trained subjects on an orientation discrimination task involving feedback over 4 days, acquiring fMRI data on the first and last day. Behavioral improvements were well explained by a reinforcement learning model in which learning leads to enhanced readout of sensory information, thereby establishing noise-robust representations of decision variables. We find stimulus orientation encoded in early visual and higher cortical regions such as lateral parietal cortex and anterior cingulate cortex (ACC). However, only activity patterns in the ACC tracked changes in decision variables during learning. These results provide strong evidence for perceptual learning-related changes in higher order areas and suggest that perceptual and reward learning are based on a common neurobiological mechanism.     

The sampling distribution of kappa

Research on Herrnstein's single-schedule equation contains conflicting findings; some laboratories report variations in the k parameter with reinforcer value, and others report constancy. The reported variation in k typically occurs across very low reinforcer values, and constancy applies across higher values. Here, simulations were conducted assuming a wide range of reinforcer values, and the parameters of Herrnstein's equation were estimated for simulated responding. In the simulations, responses controlled by current reinforcement contingencies were added to other responses ('noise'), controlled by the experimental environment and by contingencies in effect at other times. Expected reinforcer rates were calculated by entering simulated responding into a reinforcement feedback function. These were then fitted using Herrnstein's hyperbola, and the sampling distributions of the two fitted parameters were studied. Both k and Re were underestimated by curve fitting when low-deprivation or reinforcer-quality conditions were simulated. Further simulations showed that k and Re were increasingly underestimated as the assumed noise level was increased, particularly when low-deprivation or reinforcer quality was assumed. It is concluded that reported variations in k from single schedules should not be taken to indicate that the asymptotic rate of responding depends on reinforcement parameters.     

A model of resource partitioning between foraging bees based on learning

Central place foraging pollinators tend to develop multi-destination routes (traplines) to exploit patchily distributed plant resources. While the formation of traplines by individual pollinators has been studied in detail, how populations of foragers use resources in a common area is an open question, difficult to address experimentally. We explored conditions for the emergence of resource partitioning among traplining bees using agent-based models built from experimental data of bumblebees foraging on artificial flowers. In the models, bees learn to develop routes as a consequence of feedback loops that change their probabilities of moving between flowers. While a positive reinforcement of movements leading to rewarding flowers is sufficient for the emergence of resource partitioning when flowers are evenly distributed, the addition of a negative reinforcement of movements leading to unrewarding flowers is necessary when flowers are patchily distributed. In environments with more complex spatial structures, the negative experiences of individual bees on flowers favour spatial segregation and efficient collective foraging. Our study fills a major gap in modelling pollinator behaviour and constitutes a unique tool to guide future experimental programs.     

Better Than I Thought: Positive Evaluation Bias in Hypomania

Background

Mania is characterised by increased impulsivity and risk-taking, and psychological accounts argue that these features may be due to hypersensitivity to reward. The neurobiological mechanisms remain poorly understood. Here we examine reinforcement learning and sensitivity to both reward and punishment outcomes in hypomania-prone individuals not receiving pharmacotherapy.

Method

We recorded EEG from 45 healthy individuals split into three groups by low, intermediate and high self-reported hypomanic traits. Participants played a computerised card game in which they learned the reward contingencies of three cues. Neural responses to monetary gain and loss were measured using the feedback-related negativity (FRN), a component implicated in motivational outcome evaluation and reinforcement learning.

Results

As predicted, rewards elicited a smaller FRN in the hypomania-prone group relative to the low hypomania group, indicative of greater reward responsiveness. The hypomania-prone group also showed smaller FRN to losses, indicating diminished response to negative feedback.

Conclusion

Our findings indicate that proneness to hypomania is associated with both reward hypersensitivity and discounting of punishment. This positive evaluation bias may be driven by aberrant reinforcement learning signals, which fail to update future expectations. This provides a possible neural mechanism explaining risk-taking and impaired reinforcement learning in BD. Further research will be needed to explore the potential value of the FRN as a biological vulnerability marker for mania and pathological risk-taking.     

Contingently faded heart rate biofeedback: Attempted replication of large-magnitude decreases

McKinney et al. (1980) reported large-magnitude reductions in heart rate (HR) from resting baseline levels, employing shaping and fading techniques and a reinforcement program in which a secondary reinforcer was awarded both contingently and immediately during training. The four male subjects in this group showed significantly greater HR decreases than a group of four males receiving beat-by-beat analogue HR feedback. The present study compared decreases in HR in 20 male subjects receiving the contingently faded biofeedback procedure to those shown by 10 male subjects for whom reinforcement was contingent on vigilant observation of a visual display, and independent of HR. The former group showed significantly greater decreases in HR that could not be attributed to elevated baseline levels. However, the decreases in HR were not as large as those reported by McKinney et al. (1980). It is argued that future research should assess variables contributing to individual differences in performance.This research was supported by Ontario Heart Foundation Research Grant 15–37 to R. Pavloski.