A method of analysis of the models of neural nets

Taking into account Caianiello's work of 1961 a model of a neuron quite similar to his is proposed and studied. For this model, where a temporal summation and a period of refractoriness are assumed, a mathematical approach and a simulation on computer were realized. Particular types of nets were used, namely: nets with topological structures, and fully random nets. The difference between the two types is that the first type has a two-dimensional square structure and depends on the rules of the formation of connection between the neurons, while the second type is realized by means of the probability distribution function governing the formation of the structure of the net.These types of neural nets are analysed by means of a method which permits to obtain various parameters which characterize their behaviour in time and space in terms of the trajectory of the system. Many experiments are also reported; the statistical analyses, made on them, show the great importance and influence of refractoriness on the behaviour of neural networks.In the last part of the work an interesting case is reported, in which the reaction of the net to a disturbance shows that a kind of adaptation takes place, although the structure of the net stays unchanged.On leave of absence from the Lithuanian Academy of Sciences, Vilnius, Lithuanian S.S.R.     

Pattern orthogonalization via channel decorrelation by adaptive networks

The early processing of sensory information by neuronal circuits often includes a reshaping of activity patterns that may facilitate further processing in the brain. For instance, in the olfactory system the activity patterns that related odors evoke at the input of the olfactory bulb can be highly similar. Nevertheless, the corresponding activity patterns of the mitral cells, which represent the output of the olfactory bulb, can differ significantly from each other due to strong inhibition by granule cells and peri-glomerular cells. Motivated by these results we study simple adaptive inhibitory networks that aim to separate or even orthogonalize activity patterns representing similar stimuli. Since the animal experiences the different stimuli at different times it is difficult for the network to learn the connectivity based on their similarity; biologically it is more plausible that learning is driven by simultaneous correlations between the input channels. We investigate the connection between pattern orthogonalization and channel decorrelation and demonstrate that networks can achieve effective pattern orthogonalization through channel decorrelation if they simultaneously equalize their output levels. In feedforward networks biophysically plausible learning mechanisms fail, however, for even moderately similar input patterns. Recurrent networks do not have that limitation; they can orthogonalize the representations of highly similar input patterns. Even when they are optimized for linear neuronal dynamics they perform very well when the dynamics are nonlinear. These results provide insights into fundamental features of simplified inhibitory networks that may be relevant for pattern orthogonalization by neuronal circuits in general.     

Spatial firing patterns of auditory neuron network modelling by computer simulation

This communication examines, in digital computer simulated network, input signals and response patterns established at excitatory neurons' level i.e. the membrane potential of neuron soma. It is restricted to spatial patterns of the auditory neuron networks and time factor for nervous conduction and transmission is neglected compared with long maintained membrane potentials of neuron somas. The model analyzes the change in the spatial patterns of the membrane potential in the two dimensional networks of the auditory system. In order to evaluate the contribution of the various parameters, it is started that the simplest model has only one parameter, lateral inhibition. The other parameters are then added, one at a time, to successive models. The lateral inhibition is a necessary condition in the auditory nervous system if any sharpening of the response areas in the single neurons is to occur. A necessary condition for the validity of the model is that it should be applicable to the other senses such as vision and chemical patterns, taste. The threshold feature of auditory neurons aids in producing a sharpening in the neuron of the auditory relay nuclei. It does this clipping the spatial response patterns in one dimensional arrays of excitatory neurons. Recurrent inhibition seems a necessary condition in the sensory nervous system that any kinds of input signals are to be preserved over a wide range of stimulus intensity. In other words, this network has a wide dynamic range against any kinds of input signals. A simple self-recurrent negative feedback does not contribute to the sharpening, but more complex socalled averaged type does. A neuron network is capable of responding stably to stimuli with a wide range of intensity and with any kind of spatial patterns if there is a simple negative feedback mechanism. When there is no negative feedback, input signals soon disappear or saturate in the neuron network. Therefore, recurrent inhibition is the most important mechanism. Spontaneous activity appears to aid in the sharpening by providing a kind of contrast, that is by reducting the amount of activity in neurons adjacent to the excitatory area. Moreover, the effect of spontaneous activity in the model seems to make repples around the excitatory area and suggests that an introduction of activity at any stage of the networks, from whatever source for example reticulum formation and thalamus, might appreciably alter the response patterns at subsequent neuron network. This suggests that the mechanism of the consciousness that might be controlled by the thalamus and or reticular formation. These two dimensional neuron networks may be expanded to three dimensional neuron networks. The former might simulate the auditory nervous system while the latter might simulate the visual system.     

Neural cross-correlation and signal decorrelation: insights into coding of auditory space

The auditory systems of humans and many other species use the difference in the time of arrival of acoustic signals at the two ears to compute the lateral position of sound sources. This computation is assumed to initially occur in an assembly of neurons organized along a frequency-by-delay surface. Mathematically, the computations are equivalent to a two-dimensional cross-correlation of the input signals at the two ears, with the position of the peak activity along this surface designating the position of the source in space. In this study, partially correlated signals to the two ears are used to probe the mechanisms for encoding spatial cues in stationary or dynamic (moving) signals. It is demonstrated that a cross-correlation model of the auditory periphery coupled with statistical decision theory can predict the patterns of performance by human subjects for both stationary and motion stimuli as a function of stimulus decorrelation. Implications of these findings for the existence of a unique cortical motion system are discussed.     

Complementary molecular models of learning and memory.

The functional capabilities of the brain are formally characterizable interms of a finite system along with a memory space which it can manipulate. Two types of learning are possible: (1) modification-based learning, associated with alternate realizations of the finite system; (2) memory-based learning, associated with the assimilation, manipulation, and retrieval of memories. Constructive models which fulfill these conditions and which at the same time operate on the basis of molecular information processing principles have certain general features. We describe these features in terms of two interfaced submodels, the first for the finite system and the second for the memory space. The finite system may be realized by networks of neurons in which the specificity of enzyme molecules controls the nerve impulse. Such a realization is amenable to modification-based learning mediated by processes analogous to those of natural evolution and selective theories of antibody synthesis. The memory space is realizable by networks of neurons in which the conformation of dendritic receptor molecules controls the nerve impulse. In this case certain neurons firing in response to an external input undergo sensitization at the dendrites and in such a way that they are loadable and later callable by reference neurons, thereby allowing for reconstruction of manipulation of the firing pattern associated with this input. The overall construction makes a large number of biochemical, anatomical, physiological, and psychological predictions which are either testable or in good agreement with fact.     

Responses of recurrent nets of asymmetric ON and OFF cells

A neural field model of ON and OFF cells with all-to-all inhibitory feedback is investigated. External spatiotemporal stimuli drive the ON and OFF cells with, respectively, direct and inverted polarity. The dynamic differences between networks built of ON and OFF cells (“ON/OFF”) and those having only ON cells (“ON/ON”) are described for the general case where ON and OFF cells can have different spontaneous firing rates; this asymmetric case is generic. Neural responses to nonhomogeneous static and time-periodic inputs are analyzed in regimes close to and away from self-oscillation. Static stimuli can cause oscillatory behavior for certain asymmetry levels. Time-periodic stimuli expose dynamical differences between ON/OFF and ON/ON nets. Outside the stimulated region, we show that ON/OFF nets exhibit frequency doubling, while ON/ON nets cannot. On the other hand, ON/ON networks show antiphase responses between stimulated and unstimulated regions, an effect that does not rely on specific receptive field circuitry. An analysis of the resonance properties of both net types reveals that ON/OFF nets exhibit larger response amplitude. Numerical simulations of the neural field models agree with theoretical predictions for localized static and time-periodic forcing. This is also the case for simulations of a network of noisy integrate-and-fire neurons. We finally discuss the application of the model to the electrosensory system and to frequency-doubling effects in retina.     

Neurobiologie und System-Theorie eines visuellen Muster-Erkennungsmechanismus bei Kröten

Quantitative behavioral experiments have shown that the toad uses mainly two types of gestalt information in prey/enemy discrimination: pattern extension in the direction of movement promotes, in general, the signal value prey, while extension perpendicular to the direction of movement promotes that of enemy. Registrations from single fibers and single cells at different stages on the visual path showed that the object extension perpendicular to the direction of movement is chiefly analysed by means of the retinal and thalamus pretectal nerve nets, whereas the extension in the direction of movement is analysed mostly by certain tectal nerve nets. Further neurobiological findings indicated that the prey/enemy discrimination is the result of subtractive interaction between the tectal and thalamus pretectal nerve nets. The system answers given by the retina, the retina-pretectum and the retinatectum to the input patterns used in the neurobiological experiments were determined for relevant space and time parameters on the basis of two dimensional neuron network models. The experimental results agree well with the theoretical ones. If the subtractive interaction between the model networks hypothesized from the neurophysiological results is applied, the resulting system answer describes the behavioral findings very well. So it is shown that the networks investigated would suffice in principle for the behavioral interpretations of the key stimulus prey/enemy — so far as these are known.

---

---

Mit Unterstützung der Deutschen Forschungsgemeinschaft Ew 7/6 u. Forsch.-Gr. Az. 741,29.     

The nervous system in planarians: Peripheral and gastrodermal plexuses,pharynx innervation,and the relationship between central nervous system structure and the acoelomate organization

The Champy-Maillet osmium tetroxide-zinc iodide technique and a new method using azur B-sodium thioglycolate were used to study the general nervous tissue structure in planarians. A subepidermal and a submuscular nerve plexus, partially reported by earlier authors, are described, and a gastrodermal plexus is reported for the first time in triclads. The possible functions for each one of these plexuses are discussed. By the Champy-Maillet method, the innervation within the parenchyma appears as an array of numerous single nerve fibers that course between the parenchyma cells making apparent synaptic contacts. The pharynx has outer and inner nerve nets similar in structure to the submuscular nerve plexus. Both nerve nets are connected to each other by radial nerves. The central nervous system has a sponge-like structure with many lacunae filled with cell bodies, dorso-ventral muscle fibers, parenchymal cell processes and excretory ducts. The existence of this sponge-like nervous tissue structure is discussed in relation to the still incomplete centralization of the nervous tissue in these organisms, to the lack of a true vascular system and to the acoelomate level of organization. A comparison with the nervous tissue structure of more advanced groups like polyclads and nemertines is suggested.     

Zur Informationsverarbeitung im visuellen System der Wirbeltiere. I

Summary In this paper it is tried to find a mathematical model for a number of mainly electrophysiological results concerning pattern recognition of mammals. The interpretations are essentially based on the experiments of Hubel and Wiesel in the visual system of the cat and the monkey.After a short introduction to the applied theory of linear nervous nets the investigations in the retina are interpreted. This part of the visual system can be considered as a bandpass-filter for space dependent oscillations. At the level of the geniculate body, a further filtering takes place which especially attenuates the low and the very high frequencies.The processes in the cortex regions 17, 18 and 19, where the further preprocessing of the pattern recognition takes place, can be interpreted by the theory of matched filters. In Area 17 the input pattern is reduced to the contour lines. In the two other areas the extraction of simple characteristic features such as line ends and corners takes place. By means of the present results it is not possible to draw complete conclusions on the structure of the recognition process.     

The Spatial Structure of Stimuli Shapes the Timescale of Correlations in Population Spiking Activity

Throughout the central nervous system, the timescale over which pairs of neural spike trains are correlated is shaped by stimulus structure and behavioral context. Such shaping is thought to underlie important changes in the neural code, but the neural circuitry responsible is largely unknown. In this study, we investigate a stimulus-induced shaping of pairwise spike train correlations in the electrosensory system of weakly electric fish. Simultaneous single unit recordings of principal electrosensory cells show that an increase in the spatial extent of stimuli increases correlations at short () timescales while simultaneously reducing correlations at long () timescales. A spiking network model of the first two stages of electrosensory processing replicates this correlation shaping, under the assumptions that spatially broad stimuli both saturate feedforward afferent input and recruit an open-loop inhibitory feedback pathway. Our model predictions are experimentally verified using both the natural heterogeneity of the electrosensory system and pharmacological blockade of descending feedback projections. For weak stimuli, linear response analysis of the spiking network shows that the reduction of long timescale correlation for spatially broad stimuli is similar to correlation cancellation mechanisms previously suggested to be operative in mammalian cortex. The mechanism for correlation shaping supports population-level filtering of irrelevant distractor stimuli, thereby enhancing the population response to relevant prey and conspecific communication inputs.     

Experience-dependent plasticity and modulation of growth regulatory molecules at central synapses

Structural remodeling or repair of neural circuits depends on the balance between intrinsic neuronal properties and regulatory cues present in the surrounding microenvironment. These processes are also influenced by experience, but it is still unclear how external stimuli modulate growth-regulatory mechanisms in the central nervous system. We asked whether environmental stimulation promotes neuronal plasticity by modifying the expression of growth-inhibitory molecules, specifically those of the extracellular matrix. We examined the effects of an enriched environment on neuritic remodeling and modulation of perineuronal nets in the deep cerebellar nuclei of adult mice. Perineuronal nets are meshworks of extracellular matrix that enwrap the neuronal perikaryon and restrict plasticity in the adult CNS. We found that exposure to an enriched environment induces significant morphological changes of Purkinje and precerebellar axon terminals in the cerebellar nuclei, accompanied by a conspicuous reduction of perineuronal nets. In the animals reared in an enriched environment, cerebellar nuclear neurons show decreased expression of mRNAs coding for key matrix components (as shown by real time PCR experiments), and enhanced activity of matrix degrading enzymes (matrix metalloproteinases 2 and 9), which was assessed by in situ zymography. Accordingly, we found that in mutant mice lacking a crucial perineuronal net component, cartilage link protein 1, perineuronal nets around cerebellar neurons are disrupted and plasticity of Purkinje cell terminal is enhanced. Moreover, all the effects of environmental stimulation are amplified if the afferent Purkinje axons are endowed with enhanced intrinsic growth capabilities, induced by overexpression of GAP-43. Our observations show that the maintenance and growth-inhibitory function of perineuronal nets are regulated by a dynamic interplay between pre- and postsynaptic neurons. External stimuli act on this interaction and shift the balance between synthesis and removal of matrix components in order to facilitate neuritic growth by locally dampening the activity of inhibitory cues.     

Short-term depression and transient memory in sensory cortex

Persistent neuronal activity is usually studied in the context of short-term memory localized in central cortical areas. Recent studies show that early sensory areas also can have persistent representations of stimuli which emerge quickly (over tens of milliseconds) and decay slowly (over seconds). Traditional positive feedback models cannot explain sensory persistence for at least two reasons: (i) They show attractor dynamics, with transient perturbations resulting in a quasi-permanent change of system state, whereas sensory systems return to the original state after a transient. (ii) As we show, those positive feedback models which decay to baseline lose their persistence when their recurrent connections are subject to short-term depression, a common property of excitatory connections in early sensory areas. Dual time constant network behavior has also been implemented by nonlinear afferents producing a large transient input followed by much smaller steady state input. We show that such networks require unphysiologically large onset transients to produce the rise and decay observed in sensory areas. Our study explores how memory and persistence can be implemented in another model class, derivative feedback networks. We show that these networks can operate with two vastly different time courses, changing their state quickly when new information is coming in but retaining it for a long time, and that these capabilities are robust to short-term depression. Specifically, derivative feedback networks with short-term depression that acts differentially on positive and negative feedback projections are capable of dynamically changing their time constant, thus allowing fast onset and slow decay of responses without requiring unrealistically large input transients.     

Efficient encoding of spectrotemporal information for bat echolocation

In most animals, natural stimuli are characterized by a high degree of redundancy, limiting the ensemble of ecologically valid stimuli to a significantly reduced subspace of the representation space. Neural encodings can exploit this redundancy and increase sensing efficiency by generating low-dimensional representations that retain all information essential to support behavior. In this study, we investigate whether such an efficient encoding can be found to support a broad range of echolocation tasks in bats. Starting from an ensemble of echo signals collected with a biomimetic sonar system in natural indoor and outdoor environments, we use independent component analysis to derive a low-dimensional encoding of the output of a cochlear model. We show that this compressive encoding retains all essential information. To this end, we simulate a range of psycho-acoustic experiments with bats. In these simulations, we train a set of neural networks to use the encoded echoes as input while performing the experiments. The results show that the neural networks’ performance is at least as good as that of the bats. We conclude that our results indicate that efficient encoding of echo information is feasible and, given its many advantages, very likely to be employed by bats. Previous studies have demonstrated that low-dimensional encodings allow for task resolution at a relatively high level. In contrast to previous work in this area, we show that high performance can also be achieved when low-dimensional filters are derived from a data set of realistic echo signals, not tailored to specific experimental conditions.     

On the interpretation of optical illusions

If excited by stimuli adjacent in space and time, the optical system frequently perceives illusions in the form of apparent movements. These effects may be attributed to the dynamic properties of the retinal nerve nets. On the basis of a specific psychophysical experiment the mechanism underlying the generation of optical illusions is interpreted by the methods of systems theory and its use in systems analysis is discussed. It is shown that for the perception of apparent movements the transit times of the signals in the dendrites are particularly important.     

Redundancy reducing processes in single neurons

Interaction mechanisms between excitatory and inhibitory impulse sequences operating on neurons play an important role for the processing of information by the nervous system. For instance, the convergence of excitatory and inhibitory influences on retinal ganglion cells to form their receptive fields has been taken as an example for the process of neuronal sharpening by lateral inhibition. In order to analyze quantitatively the functional behavior of such a system, Shannon's entropy method for multiple access channels has been applied to biological two-inputs-one-output systems using the theoretical model developed by Tsukada et al. (1979). Here we give an extension of this procedure from the point of view to reduce redundancy of information in the input signal space of single neurons and attempt to obtain a new interpretation for the information processing of the system. The concept for the redundancy reducing mechanism in single neurons is examined and discussed for the following two processes. The first process is concerned with a signal space formed by superposing two random sequences on the input of a neuron. In this process, we introduce a coding technique to encode the inhibitory sequence by using the timing of the excitatory sequence, which is closely related to an encoding technique of multiple access channels with a correlated source (Marko, 1966, 1970, 1973; Slepian and Wolf, 1973) and which is an invariant transformation in the input signal space without changing the information contents of the input. The second process is concerned with a procedure of reducing redundant signals in the signal space mentioned before. In this connection, it is an important point to see how single neurons reduce the dimensionality of the signal space via transformation with a minimum loss of effective information. For this purpose we introduce the criterion that average transmission of information from signal space to the output does not change when redundant signals are added. This assumption is based on the fact that two signals are equivalent if and only if they have identical input-output behavior. The mechanism is examined and estimated by using a computer-simulated model. As the result of such a simulation we can estimate the minimal segmentation in the signal space which is necessary and sufficient for temporal pattern sensitivity in neurons.     

Increasing sensor flexibility through neuromodulation

Both biological and man-made motor control networks require input from sensors to allow for modification of the motor program. Real sensory neurons are more flexible than typical robotic sensors because they are dynamic rather than static. The membrane properties of neurons and hence their excitability can be modified by the presence of neuromodulatory substances. In the case of a sensory neuron, this can change, in a functionally significant way, the code used to describe a stimulus. For instance, extension of the neuron's dynamic range or modification of its filtering characteristics can result. This flexibility has an apparent cost. The code used may be situation-dependent and hence difficult to interpret. To address this issue and to understand how neuromodulation is used effectively in a motor control network, I am studying the GPR2 stretch receptor in the crustacean stomatogastric nervous system. Several different neuromodulatory substances can modify its encoding properties. Comparisons of physiological and anatomical evidence suggest that neuromodulation can be effected both by GPR2 itself and by other neurons in the network. These results suggest that the analog of neuromodulation might be useful for improving sensor performance in an artificial motor control system.     

A Neural Network Model of Chemotaxis Predicts Functions of Synaptic Connections in the Nematode Caenorhabditis elegans

The anatomical connectivity of the nervous system of the nematode Caenorhabditis elegans has been almost completely described, but determination of the neurophysiological basis of behavior in this system is just beginning. Here we used an optimization algorithm to search for patterns of connectivity sufficient to compute the sensorimotor transformation underlying C. elegans chemotaxis, a simple form of spatial orientation behavior in which turning probability is modulated by the rate of change of chemical concentration. Optimization produced differentiator networks capable of simulating chemotaxis. A surprising feature of these networks was inhibitory feedback connections on all neurons. Further analysis showed that feedback regulates the latency between sensory input and behavior. Common patterns of connectivity between the model and biological networks suggest new functions for previously identified connections in the C. elegans nervous system.     

Analytical and Simulation Results for Stochastic Fitzhugh-Nagumo Neurons and Neural Networks

An analytical approach is presented for determining the response of a neuron or of the activity in a network of connected neurons, represented by systems of nonlinear ordinary stochastic differential equations—the Fitzhugh-Nagumo system with Gaussian white noise current. For a single neuron, five equations hold for the first- and second-order central moments of the voltage and recovery variables. From this system we obtain, under certain assumptions, five differential equations for the means, variances, and covariance of the two components. One may use these quantities to estimate the probability that a neuron is emitting an action potential at any given time. The differential equations are solved by numerical methods. We also perform simulations on the stochastic Fitzugh-Nagumo system and compare the results with those obtained from the differential equations for both sustained and intermittent deterministic current inputs withsuperimposed noise. For intermittent currents, which mimic synaptic input, the agreement between the analytical and simulation results for the moments is excellent. For sustained input, the analytical approximations perform well for small noise as there is excellent agreement for the moments. In addition, the probability that a neuron is spiking as obtained from the empirical distribution of the potential in the simulations gives a result almost identical to that obtained using the analytical approach. However, when there is sustained large-amplitude noise, the analytical method is only accurate for short time intervals. Using the simulation method, we study the distribution of the interspike interval directly from simulated sample paths. We confirm that noise extends the range of input currents over which (nonperiodic) spike trains may exist and investigate the dependence of such firing on the magnitude of the mean input current and the noise amplitude. For networks we find the differential equations for the means, variances, and covariances of the voltage and recovery variables and show how solving them leads to an expression for the probability that a given neuron, or given set of neurons, is firing at time t. Using such expressions one may implement dynamical rules for changing synaptic strengths directly without sampling. The present analytical method applies equally well to temporally nonhomogeneous input currents and is expected to be useful for computational studies of information processing in various nervous system centers.     

Informationsverarbeitung in homogenen Netzen von Neuronenmodellen

Summary The interconnection structures of the peripheral part of the nervous system, which are considered here, are two-dimensional homogeneous networks with time and space dependent inputs and outputs. The principles of connection under consideration comprise lateral inhibition and facilitation. The transfer functions of those linear networks as well as the stability problem are investigated on a digital computer using different system parameters. A closed form solution is given for an infinitely large element density which describes the network properties. In this case an inhibition system acts as high pass filter on the spatial frequencies of the input, whereas a facilitation network acts as low pass filter. The properties of the networks and the transformations in case of moving patterns are analysed using the methods of systems theory.

---

---

Auszug aus einer Dissertation an der Fakultät für Maschinenwesen der Technischen Universität Hannover, Institut für Theoretische Elektrotechnik (Prof. Dr.-Ing. habil. H. Tischner).

---

Herrn Prof. Dr.-Ing. habil. H. Tischner danke ich für die Anregung zu dieser Arbeit und seine ständige Unterstützung. Mein Dank gilt ferner der Firma Brown, Boveri & Cie. AG, Mannheim, für die finanzielle Förderung. Den Mitarbeitern im Institut für Schwingungsforschung, insbesondere Herrn Direktor Dr.-Ing. A. Schief bin ich für wertvolle Diskussionen verpflichtet.