Simulation of rat behavior by a reinforcement learning algorithm in consideration of appearance probabilities of reinforcement signals

Brown and Wanger [Brown, R.T., Wanger, A.R., 1964. Resistance to punishment and extinction following training with shock or nonreinforcement. J. Exp. Psychol. 68, 503-507] investigated rat behaviors with the following features: (1) rats were exposed to reward and punishment at the same time, (2) environment changed and rats relearned, and (3) rats were stochastically exposed to reward and punishment. The results are that exposure to nonreinforcement produces resistance to the decremental effects of behavior after stochastic reward schedule and that exposure to both punishment and reinforcement produces resistance to the decremental effects of behavior after stochastic punishment schedule. This paper aims to simulate the rat behaviors by a reinforcement learning algorithm in consideration of appearance probabilities of reinforcement signals. The former algorithms of reinforcement learning were unable to simulate the behavior of the feature (3). We improve the former reinforcement learning algorithms by controlling learning parameters in consideration of the acquisition probabilities of reinforcement signals. The proposed algorithm qualitatively simulates the result of the animal experiment of Brown and Wanger.     

Changes in glutamate release in the rat nucleus accumbens during food and pain reinforcement

In vivo microdialysis combined with HPLC-EC analysis was used to monitor extracellular glutamate in the n. accumbens of Sprague-Dawley rats during footshock and food delivery. The footshock presentation resulted in a delayed increase in extracellular glutamate level, whereas the food intake caused its decrease. The intra-accumbens infusion of glutamate reuptake blocker D,L-threo-beta-hydroxiaspartate (1 mM) completely prevented the food-induced decrease in glutamate level. The intra-accumbens infusion of sodium channel blocker tetrodotoxin (1 microM) led to an increase in glutamate extracellular level in the n. accumbens in response to food intake. The results suggest that the food-induced decrease in glutamate extracellular level in the n. accumbens occurs due to an enhancement of high-affinity glutamate uptake that is probably under the neuronal control during feeding.     

Dynamics of reorganization of instrumental food getting behavior of monkey under conditions of reinforcement delay

Six adult monkeys (Macaca fascicularis) were trained to switch on a lamp by pressing the lever, to hold the lever for not less than 1 s (the lamp being switched on), and get a portion of food. After reaching a learning criterion, the delay between the lever pressing and food reinforcement was increased to 2.5 s. The experimental procedure was repeated in three experimental sessions with 2-month between-session intervals. It was shown that the retraining process after the uneven change in the delay duration developed in three stages: (1) stage of emotional hyperreactivity that reflected a mismatch between the cation and absence of the expected result; (2) stage of intermediate stabilization, when the percent of efficient attempts was the same as under conditions of 1-s delay; (3) stage of purposive instrumental lever holding till the moment of reinforcement presentation.     

The volume of reinforcement and the structure of rat behavior in a radial labyrinth

In rats choice reaction was elaborated in 12 passages radial labyrinth with different volumes of food only in six of twelve passages. Then the conditions were modified: the first group of rats in the beginning of the experiment was placed to another labyrinth sector, and in the second group the food volume was changed in passages. In new situations in animals of both groups the reaction efficiency temporarily decreased, simultaneously the correction of behaviour structure took place. The character of behavioural tactics (tendencies to visit passages with larger food volume at the beginning of the experiment, spatial-motor asymmetry etc.) is described in dynamics of adaptation to new conditions. Factors determining the rats behaviour structure are discussed.     

Suppression of operant responding for food and water reinforcement following intraventricular injections of Bombesin in the rat

The effects of injecting Bombesin (BBS) into the lateral cerebral ventricle on operant responding for food and water in the rat were investigated in two experiments. In the first experiment injections of BBS suppressed both operant responding for water and post-session water consumption. A combined treatment of water preloading and BBS injections produced greater suppression of post-session water consumption than either BBS injections or water preloading. This suggests that the peptide has a primary antidipsic effect. In the second experiment BBS produced a significant suppression of operant responding for food reward as well as lowered body temperature. This suggests that BBS may serve as a true satiety signal for food motivated behavior.     

Role of brain dopamine in food reward and reinforcement

The ability of food to establish and maintain response habits and conditioned preferences depends largely on the function of brain dopamine systems. While dopaminergic transmission in the nucleus accumbens appears sufficient for some forms of reward, the role of dopamine in food reward does not appear to be restricted to this region. Dopamine plays an important role in both the ability to energize feeding and to reinforce food-seeking behaviour; the role in energizing feeding is secondary to the prerequisite role in reinforcement. Dopaminergic activation is triggered by the auditory and visual as well as the tactile, olfactory, and gustatory stimuli of foods. While dopamine plays a central role in the feeding and food-seeking of normal animals, some food rewarded learning can be seen in genetically engineered dopamine-deficient mice.     

Nerve elements of the heart of the rat after right-sided vagotomy

A quantitative electron microscopical investigation of the sinus node of the atrioventricular His' bundle and of the perinodal working myocardium in intact rats and 7, 15 and 30 days after right-sided vagotomy has revealed variable character of changes in the neural elements of the zones mentioned. Certain ultrastructural rearrangements in the neural apparatus are described; they reflect a combination of destructive and regenerative processes in the heart under vagotomy. Thirty days after the operation, regeneration of the neural elements in the sinoauricular area is of restorative and in the atrioventricular area--of excess character. The data are presented on dynamics of changes in the diameter of the amyelinated neural fibers in the cardiac areas investigated. Participation of both nervi vagi in innervation of the main and additional pacemakers of the organ is discussed.     

Fate of degenerating lactotrophs in rat pituitary gland after interruption of lactation: a histochemical and immunocytochemical study

Summary In the pituitary gland of pregnant and lactating rats a striking proliferation of lactotrophs occurs to meet the increased demands for prolactin. Following interruption of lactation the redundant lactotrophs undergo a massive degeneration until pre-pregnant proportions are re-established. Immunocytochemical detection of prolactin allows the recognition of degenerating lactotrophs until advanced stages of degeneration and leads to the conclusion that this process is autolytic in nature. Histochemistry of acid phosphatase reveals a remarkable accumulation of this enzyme in Golgi cisternae and lysosomes. At later stages of degeneration the acid phosphatase spreads throughout the entire cell. The presence of increased numbers of necrotic cells appears to activate phagocytosis of stellate cells and, to a lesser extent, of follicular cells. Stellate cells responsible for the secondary processing of cell residues are isolated cells characterized by a prominent oval nucleus and an electron-lucent cytoplasm with scarce organelles and extensive cytoplasmic processes. They appear as scavenger cells engulfing cell remnants and debris. Immunocytochemistry of S-100 protein discloses differential staining of two types of cell, one forming clusters of 2–4 cells with faint immunoreactivity, while the other type consists of isolated cells with a stellate profile and stronger labelling to S-100 protein.     

Contamination of foods by food handlers: experiments on hepatitis A virus transfer to food and its interruption

Hepatitis A virus (HAV) is an important pathogen which has been responsible for many food-borne outbreaks. HAV-excreting food handlers, especially those with poor hygienic practices, can contaminate the foods which they handle. Consumption of such foods without further processing has been known to result in cases of infectious hepatitis. Since quantitative data on virus transfer during contact of hands with foods is not available, we investigated the transfer of HAV from artificially contaminated fingerpads of adult volunteers to pieces of fresh lettuce. Touching the lettuce with artificially contaminated fingerpads for 10 s at a pressure of 0.2 to 0.4 kg/cm(2) resulted in transfer of 9.2% +/- 0.9% of the infectious virus. The pretreatments tested to interrupt virus transfer from contaminated fingerpads included (i) hard-water rinsing and towel drying, (ii) application of a domestic or commercial topical agent followed by water rinsing and towel drying, and (iii) exposure to a hand gel containing 62% ethanol or 75% liquid ethanol without water rinsing or towel drying. When the fingerpads were treated with the topical agents or alcohol before the lettuce was touched, the amount of infectious virus transferred to lettuce was reduced from 9.2% to between 0.3 and 0.6% (depending on the topical agent used), which was a reduction in virus transfer of up to 30-fold. Surprisingly, no virus transfer to lettuce was detected when the fingerpads were rinsed with water alone before the lettuce was touched. However, additional experiments with water rinsing in which smaller volumes of water were used (1 ml instead of 15 ml) showed that the rate of virus transfer to lettuce was 0.3% +/- 0.1%. The variability in virus transfer rates following water rinsing may indicate that the volume of water at least in part influences virus removal from the fingerpads differently, a possibility which should be investigated further. This study provided novel information concerning the rate of virus transfer to foods and a model for investigating the transfer of viral and other food-borne pathogens from contaminated hands to foods, as well as techniques for interrupting such transfer to improve food safety.     

Sequential depletion of rat testicular lipids with long-chain and very long-chain polyenoic fatty acids after X-ray-induced interruption of spermatogenesis

When a single dose of X-rays is applied to the adult rat testis, stem spermatogonia are damaged, and spermatogenesis is interrupted. Supported by Sertoli cells, spermatogenic cells that endure irradiation complete their differentiation and gradually leave the testis as spermatozoa. In this study, the in vivo changes taking place a number of weeks after irradiation revealed cell-specific features of testicular lipid classes. A linear drop, taking about six weeks, in testis weight, nonlipid materials, free cholesterol, and 22:5n-6-rich glycerophospholipids took place with germ cell depletion. Sphingomyelins and ceramides with nonhydroxy very long-chain polyenoic fatty acids (n-VLCPUFA) disappeared in four weeks, together with the last spermatocytes, whereas species with 2-hydroxy VLCPUFA lasted for six weeks, disappearing with the last spermatids and spermatozoa. The amount per testis of 22:5n-6-rich triacylglycerols, unchanged for four weeks, fell between weeks 4 and 6, associating these lipids with spermatids and their residual bodies, detected as small, bright lipid droplets. In contrast, 22:5n-6-rich species of cholesterol esters and large lipid droplets increased in seminiferous tubules up to week 6, revealing they are Sertoli cell products. At week 30, the lipid and fatty acid profiles reflected the resulting permanent testicular involution. Our data highlight the importance of Sertoli cells in maintaining lipid homeostasis during normal spermatogenesis.     

Factor analysis of rat behavior in the open-field test

Method of major components, a variety of factor analysis, was used for evaluation of rats behaviour motivational structure in the open field test. 40 outbred rats of Wistar line (20 males and females) were examined in which 22 behaviour characteristics were studied. It was shown that a greater part (60%) of rats individual behaviour variability in the open field test was determined by the action of three major components (factors), which were defined as "investigation", "fear" and "shifted activity". Factor structure was studied of main characteristics of behaviour, recorded in the open field test. The use of factor analysis allowed to carry out animals classification on the basis of their disposition in coordinates of the obtained major components.     

Environmental analysis of packaging‐derived changes in food production and consumer behavior

This study analyzed the environmental impacts of packaging‐derived changes in food production and consumer behavior to assist packaging designers in making environmentally conscious decisions. Packaging can be functionalized to prevent food loss and waste (FLW), for example, extending the expiration date and apportioning the package size, but it can generate additional environmental impacts from changes in food and packaging production. Previous studies assessed additional impacts from packaging production; however, the effects of packaging functionalization are yet to be connected with food production and consumer behavior. To examine the effect of functionalization on these aspects, we analyzed packaging‐derived changes in food production for milk and cabbage products. The case study compared products with functionalized packaging that permits a longer expiration date or a smaller portion size to their base‐case products. Our results showed that the packaging‐derived changes increased the global warming potential (GWP) of food production more than other processes did. Thus, changes in food production weakened the effectiveness of the packaging functionalization to decrease the GWP. Moreover, the analysis of consumer behavior scenarios showed that consumers’ perception of the expiration date decisively influences the effectiveness of packaging functionalization. When consumers discarded food after the expiration date, provided they consumed in small quantities, the packaging functionalization reduced FLW. From the scenario analysis, we identified appropriate combinations of packaging functionalization and consumer behaviors to effectively decrease total GWP. With our expanded analysis, packaging designers can understand the effectiveness of their decisions on the product life cycle in reducing FLW and environmental impacts.     

Viscoelastic behavior of lung and chest wall in dogs determined by flow interruption

Pulmonary and chest wall mechanics were studied in six anesthetized paralyzed dogs, by use of the technique of rapid airway occlusion during constant flow inflation. Analysis of the pressure changes after flow interruption allowed us to partition the overall resistance of the lung (Rl) and chest wall (Rw) and total respiratory system (Rrs) into two components, one (Rinit) reflecting in the lung airway resistance (Raw), the other (delta R) reflecting primarily the viscoelastic properties of the pulmonary and chest wall tissues. The effects of varying inspiratory flow and inflation volume were interpreted in terms of frequency dependence of resistance, by using a spring-and-dashpot model previously proposed and substantiated by Bates et al. (Proc. 9th Annu. Conf. IEEE Med. Biol. Soc., 1987, vol. 3, p. 1802-1803). We observed that 1) Raw and Rw,init were nearly equal and small relative to Rl and Rw (both were unaffected by flow); 2) Rrs,init decreased slightly with increasing volume; 3) both delta Rl and delta Rw decreased with increasing flow and increased with increasing lung volume. These changes were manifestations of frequency dependence of delta R, as it is predicted by the model; 4) Rrs, Rl, and Rw followed the same trends as delta R. These results corroborate data previously reported in the literature with the use of different techniques to measure airways and pulmonary tissue resistances and confirm that the use of Rl to assess bronchial reactivity is problematic. The interrupter techniques provides a convenient way to obtain Raw values, as well as analogs of lung and chest wall tissue resistances in intact dogs.     

Modification of instinctive herding dog behavior using reinforcement and punishment

Abstract

“Instinctive” behavior may be modified using operant techniques. We report here on a field study of training herding dogs in which reinforcers and punishers were used by owners, who were themselves being trained to control their dogs. Access to sheep was assumed to be a primary reinforcer for herding dogs. While blocking their access was aversive to them. Over several months, the number of blocking and access actions by the human were scored during the training of seven naïve herding dogs. We found that rates of punishment by blocking the dog's access to sheep or by stopping the dog occurred at higher levels than positive reinforcement from access or verbal praise. While positive reinforcement can be used exclusively for the training of certain behaviors, it is suggested that in the context of instinctive motor patterns, negative reinforcement and punishment may be desirable and necessary additions to positive reinforcement techniques.