条件联想神经网络研究

在人脑的某些功能和神经系统中的突前抑制机制启发下,本文提出一个新型的神经网络模型——条件联想神经网络.模型是一个有突触前抑制的联想记忆神经网络.通过初步分析和计算机模拟,证明本模型具有一般联想记忆模型所未有的一些新的特性,如可以在不同条件下,对同一输入有不同的反应.对同一输入,在不同的条件下,又可以有相同的反应.这些特点将有助于人们对神经系统中信息处理过程的了解.此外,文中也指出可能实现本模型的神经结构.     

Complexity and non-commutativity of learning operations on graphs

We present results from numerical studies of supervised learning operations in small recurrent networks considered as graphs, leading from a given set of input conditions to predetermined outputs. Graphs that have optimized their output for particular inputs with respect to predetermined outputs are asymptotically stable and can be characterized by attractors, which form a representation space for an associative multiplicative structure of input operations. As the mapping from a series of inputs onto a series of such attractors generally depends on the sequence of inputs, this structure is generally non-commutative. Moreover, the size of the set of attractors, indicating the complexity of learning, is found to behave non-monotonically as learning proceeds. A tentative relation between this complexity and the notion of pragmatic information is indicated.     

Feedforward neural network with adaptive reference pattern layer.

A hybrid neural network architecture is investigated for modeling purposes. The proposed hybrid is based on the multilayer perceptron (MLP) network. In addition to the usual hidden layers, the first hidden layer is selected to be an adaptive reference pattern layer. Each unit in this new layer incorporates a reference pattern that is located somewhere in the space spanned by the input variables. The outputs of these units are the component wise-squared differences between the elements of a reference pattern and the inputs. The reference pattern layer has some resemblance to the hidden layer of the radial basis function (RBF) networks. Therefore the proposed design can be regarded as a sort of hybrid of MLP and RBF networks. The presented benchmark experiments show that the proposed hybrid can provide significant advantages over standard MLPs and RBFs in terms of fast and efficient learning, and compact network structure.     

Saccade control in a simulated robot camera-head system: neural net architectures for efficient learning of inverse kinematics

The high speed of saccades means that they cannot be guided by visual feedback, so that any saccadic control system must know in advance the correct output signals to fixate a particular retinal position. To investigate neural-net architectures for learning this inverse-kinematics problem we simulated a 4 deg-of-freedom robot camera-head system, in which the head could pan and tilt and the cameras pan and verge. The main findings were: (1) Linear nets, multilayer perceptrons (MLPs) trained by backpropagation, and cerebellar model arithmetic computers (CMACs) all learnt rapidly to 5–10% accuracy when given perfect error feedback. (2) For additional accuracy (down to 2%) two-layer nets learnt much faster than a single MLP or CMAC: the best combination tried was to have a CMAC learn the errors of a trained linear net. (3) Imperfect error signals were provided by a crude controller whose output was simply proportional to retinal input in the relevant axis, thereby providing a mechanism for (a) controlling the camera-head system when the feedforward neural net controller was wrong or inoperative, and (b) converting sensory error signals into motor error signals as required in supervised learning. It proved possible to train neural-net controllers using these imperfect error signals over a range of learning rates and crude-controller gains. These results suggest that appropriate neural-net architectures can provide practical, accurate and robust adaptive control for saccadic movements. In addition, the arrangement of a crude controller teaching a sophisticated one may be similar to that used by the primate saccadic system, with brainstem circuitry teaching the cerebellum.     

Color image correction for scanner and printer using B-spline CMAC neural networks.

The process of eliminating the color errors from the gamut mismatch, resolution conversion, and nonlinearity between scanner and printer is usually recognized as an essential issue of color reproduction. This paper presents a new formulation based on the generalized inverse plant control for the color error reduction process. In our formulation, the printer input and scanner output correspond to the input and output of a system plant, respectively. Obviously, if the printer input equals the scanner output, then there are no color errors involved in the entire system. In other words, the plant becomes an identity system. To achieve this goal, a plant generalized inverse should be identified and added to the original system. Since the system of a combination of both scanner and printer is highly nonlinear, CMAC-based neural networks, which have the capability to learn arbitrary nonlinearity, are applied to identify the plant generalized inverse. CMAC network is a perceptron-like feedforward structure with associative memory properties. Its memory requirements can be greatly reduced by the use of hash coding techniques. In order for CMAC networks to construct high-order, smooth, nonlinear plant inverse, more general CMAC addressing schemes have been proposed in conjunction with use of B-spline receptive functions. It is shown that B-spline CMAC networks learn orders of magnitude more rapidly than typical implementations of back propagation in the multilayered neural networks, due to the local nature of its weighting updating and the finite support of B-spline receptive field functions. Finally, a number of test samples are conducted to verify the effectiveness of the proposed method.     

Multi/infinite dimensional neural networks, multi/infinite dimensional logic theory

A mathematical model of an arbitrary multi-dimensional neural network is developed and a convergence theorem for an arbitrary multi-dimensional neural network represented by a fully symmetric tensor is stated and proved. The input and output signal states of a multi-dimensional neural network/logic gate are related through an energy function, defined over the fully symmetric tensor (representing the connection structure of a multi-dimensional neural network). The inputs and outputs are related such that the minimum/maximum energy states correspond to the output states of the logic gate/neural network realizing a logic function. Similarly, a logic circuit consisting of the interconnection of logic gates, represented by a block symmetric tensor, is associated with a quadratic/higher degree energy function. Infinite dimensional logic theory is discussed through the utilization of infinite dimension/order tensors.     

Minimal models of multidimensional computations

The multidimensional computations performed by many biological systems are often characterized with limited information about the correlations between inputs and outputs. Given this limitation, our approach is to construct the maximum noise entropy response function of the system, leading to a closed-form and minimally biased model consistent with a given set of constraints on the input/output moments; the result is equivalent to conditional random field models from machine learning. For systems with binary outputs, such as neurons encoding sensory stimuli, the maximum noise entropy models are logistic functions whose arguments depend on the constraints. A constraint on the average output turns the binary maximum noise entropy models into minimum mutual information models, allowing for the calculation of the information content of the constraints and an information theoretic characterization of the system's computations. We use this approach to analyze the nonlinear input/output functions in macaque retina and thalamus; although these systems have been previously shown to be responsive to two input dimensions, the functional form of the response function in this reduced space had not been unambiguously identified. A second order model based on the logistic function is found to be both necessary and sufficient to accurately describe the neural responses to naturalistic stimuli, accounting for an average of 93% of the mutual information with a small number of parameters. Thus, despite the fact that the stimulus is highly non-Gaussian, the vast majority of the information in the neural responses is related to first and second order correlations. Our results suggest a principled and unbiased way to model multidimensional computations and determine the statistics of the inputs that are being encoded in the outputs.     

Reconstructing muscle activation during normal walking: a comparison of symbolic and connectionist machine learning techniques

One symbolic (rule-based inductive learning) and one connectionist (neural network) machine learning technique were used to reconstruct muscle activation patterns from kinematic data measured during normal human walking at several speeds. The activation patterns (or desired outputs) consisted of surface electromyographic (EMG) signals from the semitendinosus and vastus medialis muscles. The inputs consisted of flexion and extension angles measured at the hip and knee of the ipsilateral leg, their first and second derivatives, and bilateral foot contact information. The training set consisted of data from six trials, at two different speeds. The testing set consisted of data from two additional trials (one at each speed), which were not in the training set. It was possible to reconstruct the muscular activation at both speeds using both techniques. Timing of the reconstructed signals was accurate. The integrated value of the activation bursts was less accurate. The neural network gave a continuous output, whereas the rule-based inductive learning rule tree gave a quantised activation level. The advantage of rule-based inductive learning was that the rules used were both explicit and comprehensible, whilst the rules used by the neural network were implicit within its structure and not easily comprehended. The neural network was able to reconstruct the activation patterns of both muscles from one network, whereas two separate rule sets were needed for the rule-based technique. It is concluded that machine learning techniques, in comparison to explicit inverse muscular skeletal models, show good promise in modelling nearly cyclic movements such as locomotion at varying walking speeds. However, they do not provide insight into the biomechanics of the system, because they are not based on the biomechanical structure of the system.     

Shaping embodied neural networks for adaptive goal-directed behavior

The acts of learning and memory are thought to emerge from the modifications of synaptic connections between neurons, as guided by sensory feedback during behavior. However, much is unknown about how such synaptic processes can sculpt and are sculpted by neuronal population dynamics and an interaction with the environment. Here, we embodied a simulated network, inspired by dissociated cortical neuronal cultures, with an artificial animal (an animat) through a sensory-motor loop consisting of structured stimuli, detailed activity metrics incorporating spatial information, and an adaptive training algorithm that takes advantage of spike timing dependent plasticity. By using our design, we demonstrated that the network was capable of learning associations between multiple sensory inputs and motor outputs, and the animat was able to adapt to a new sensory mapping to restore its goal behavior: move toward and stay within a user-defined area. We further showed that successful learning required proper selections of stimuli to encode sensory inputs and a variety of training stimuli with adaptive selection contingent on the animat's behavior. We also found that an individual network had the flexibility to achieve different multi-task goals, and the same goal behavior could be exhibited with different sets of network synaptic strengths. While lacking the characteristic layered structure of in vivo cortical tissue, the biologically inspired simulated networks could tune their activity in behaviorally relevant manners, demonstrating that leaky integrate-and-fire neural networks have an innate ability to process information. This closed-loop hybrid system is a useful tool to study the network properties intermediating synaptic plasticity and behavioral adaptation. The training algorithm provides a stepping stone towards designing future control systems, whether with artificial neural networks or biological animats themselves.     

Learning shapes spontaneous activity itinerating over memorized states

Learning is a process that helps create neural dynamical systems so that an appropriate output pattern is generated for a given input. Often, such a memory is considered to be included in one of the attractors in neural dynamical systems, depending on the initial neural state specified by an input. Neither neural activities observed in the absence of inputs nor changes caused in the neural activity when an input is provided were studied extensively in the past. However, recent experimental studies have reported existence of structured spontaneous neural activity and its changes when an input is provided. With this background, we propose that memory recall occurs when the spontaneous neural activity changes to an appropriate output activity upon the application of an input, and this phenomenon is known as bifurcation in the dynamical systems theory. We introduce a reinforcement-learning-based layered neural network model with two synaptic time scales; in this network, I/O relations are successively memorized when the difference between the time scales is appropriate. After the learning process is complete, the neural dynamics are shaped so that it changes appropriately with each input. As the number of memorized patterns is increased, the generated spontaneous neural activity after learning shows itineration over the previously learned output patterns. This theoretical finding also shows remarkable agreement with recent experimental reports, where spontaneous neural activity in the visual cortex without stimuli itinerate over evoked patterns by previously applied signals. Our results suggest that itinerant spontaneous activity can be a natural outcome of successive learning of several patterns, and it facilitates bifurcation of the network when an input is provided.     

Neural cytoskeleton capabilities for learning and memory

This paper proposes a physical model involving the key structures within the neural cytoskeleton as major players in molecular-level processing of information required for learning and memory storage. In particular, actin filaments and microtubules are macromolecules having highly charged surfaces that enable them to conduct electric signals. The biophysical properties of these filaments relevant to the conduction of ionic current include a condensation of counterions on the filament surface and a nonlinear complex physical structure conducive to the generation of modulated waves. Cytoskeletal filaments are often directly connected with both ionotropic and metabotropic types of membrane-embedded receptors, thereby linking synaptic inputs to intracellular functions. Possible roles for cable-like, conductive filaments in neurons include intracellular information processing, regulating developmental plasticity, and mediating transport. The cytoskeletal proteins form a complex network capable of emergent information processing, and they stand to intervene between inputs to and outputs from neurons. In this manner, the cytoskeletal matrix is proposed to work with neuronal membrane and its intrinsic components (e.g., ion channels, scaffolding proteins, and adaptor proteins), especially at sites of synaptic contacts and spines. An information processing model based on cytoskeletal networks is proposed that may underlie certain types of learning and memory.     

Nrk2b-mediated NAD+ production regulates cell adhesion and is required for muscle morphogenesis in vivo: Nrk2b and NAD+ in muscle morphogenesis

Cell-matrix adhesion complexes (CMACs) play fundamental roles during morphogenesis. Given the ubiquitous nature of CMACs and their roles in many cellular processes, one question is how specificity of CMAC function is modulated. The clearly defined cell behaviors that generate segmentally reiterated axial skeletal muscle during zebrafish development comprise an ideal system with which to investigate CMAC function during morphogenesis. We found that Nicotinamide riboside kinase 2b (Nrk2b) cell autonomously modulates the molecular composition of CMACs in vivo. Nrk2b is required for normal Laminin polymerization at the myotendinous junction (MTJ). In Nrk2b-deficient embryos, at MTJ loci where Laminin is not properly polymerized, muscle fibers elongate into adjacent myotomes and are abnormally long. In yeast and human cells, Nrk2 phosphorylates Nicotinamide Riboside and generates NAD+ through an alternative salvage pathway. Exogenous NAD+ treatment rescues MTJ development in Nrk2b-deficient embryos, but not in laminin mutant embryos. Both Nrk2b and Laminin are required for localization of Paxillin, but not β-Dystroglycan, to CMACs at the MTJ. Overexpression of Paxillin in Nrk2b-deficient embryos is sufficient to rescue MTJ integrity. Taken together, these data show that Nrk2b plays a specific role in modulating subcellular localization of discrete CMAC components that in turn plays roles in musculoskeletal development. Furthermore, these data suggest that Nrk2b-mediated synthesis of NAD+ is functionally upstream of Laminin adhesion and Paxillin subcellular localization during MTJ development. These results indicate a previously unrecognized complexity to CMAC assembly in vivo and also elucidate a novel role for NAD+ during morphogenesis.     

A hybrid neural network of addressable and content-addressable memory

We investigate the memory structure and retrieval of the brain and propose a hybrid neural network of addressable and content-addressable memory which is a special database model and can memorize and retrieve any piece of information (a binary pattern) both addressably and content-addressably. The architecture of this hybrid neural network is hierarchical and takes the form of a tree of slabs which consist of binary neurons with the same array. Simplex memory neural networks are considered as the slabs of basic memory units, being distributed on the terminal vertexes of the tree. It is shown by theoretical analysis that the hybrid neural network is able to be constructed with Hebbian and competitive learning rules, and some other important characteristics of its learning and memory behavior are also consistent with those of the brain. Moreover, we demonstrate the hybrid neural network on a set of ten binary numeral patters     

A neural network model for the intersensory coordination involved in goal-directed movements

A neural network model for a sensorimotor system, which was developed to simulate oriented movements in man, is presented. It is composed of a formal neural network comprising two layers: a sensory layer receiving and processing sensory inputs, and a motor layer driving a simulated arm. The sensory layer is an extension of the topological network previously proposed by Kohonen (1984). Two kinds of sensory modality, proprioceptive and exteroceptive, are used to define the arm position. Each sensory cell receives proprioceptive inputs provided by each arm-joint together with the exteroceptive inputs. This sensory layer is therefore a kind of associative layer which integrates two separate sensory signals relating to movement coding. It is connected to the motor layer by means of adaptive synapses which provide a physical link between a motor activity and its sensory consequences. After a learning period, the spatial map which emerges in the sensory layer clearly depends on the sensory inputs and an associative map of both the arm and the extra-personal space is built up if proprioceptive and exteroceptive signals are processed together. The sensorimotor transformations occuring in the junctions linking the sensory and motor layers are organized in such a manner that the simulated arm becomes able to reach towards and track a target in extra-personal space. Proprioception serves to determine the final arm posture adopted and to correct the ongoing movement in cases where changes in the target location occur. With a view to developing a sensorimotor control system with more realistic salient features, a robotic model was coupled with the formal neural network. This robotic implementation of our model shows the capacity of formal neural networks to control the displacement of mechanical devices.     

A canonical form for neural nets without circles

By “neural net” will be meant “neural net without circles.” Every neural net effects a transformation from inputs (i.e., firing patterns of the input neurons) to outputs (firing patterns of the output neurons). Two neural nets will be calledequivalent if they effect the same transformation from inputs to outputs. A canonical form is found for neural nets with respect to equivalence; i.e., a class of neural nets is defined, no two of which are equivalent, and which contains a neural net equivalent to any given neural net. This research was supported by the U.S. Air Force under Contract AF 49(638)-414 monitored by the Air Force Office of Scientific Research.     

A Cognitive Neural Architecture Able to Learn and Communicate through Natural Language

Communicative interactions involve a kind of procedural knowledge that is used by the human brain for processing verbal and nonverbal inputs and for language production. Although considerable work has been done on modeling human language abilities, it has been difficult to bring them together to a comprehensive tabula rasa system compatible with current knowledge of how verbal information is processed in the brain. This work presents a cognitive system, entirely based on a large-scale neural architecture, which was developed to shed light on the procedural knowledge involved in language elaboration. The main component of this system is the central executive, which is a supervising system that coordinates the other components of the working memory. In our model, the central executive is a neural network that takes as input the neural activation states of the short-term memory and yields as output mental actions, which control the flow of information among the working memory components through neural gating mechanisms. The proposed system is capable of learning to communicate through natural language starting from tabula rasa, without any a priori knowledge of the structure of phrases, meaning of words, role of the different classes of words, only by interacting with a human through a text-based interface, using an open-ended incremental learning process. It is able to learn nouns, verbs, adjectives, pronouns and other word classes, and to use them in expressive language. The model was validated on a corpus of 1587 input sentences, based on literature on early language assessment, at the level of about 4-years old child, and produced 521 output sentences, expressing a broad range of language processing functionalities.     

Axon shape as a basis for multinode functional units in a hierarchical neural model

The ability of animals to perform fixed action patterns and to access information by categories suggests that there are several types of hierarchical organization in the nervous system. This paper employs data about axon shape and neurotransmitter effect to demonstrate the emergence of hierarchical structure in a neural model. Two dimensions of neural classification, axon shape and neurotransmitter effect, are used to generate a five-node-type neural model. Neurons are classified as interneurons, relay cells, and monoamine transmitters on the basis of axon shape; the transmitter classifications include excitatory, inhibitory, and parameter-changing. The five types of nodes in the model correspond to all the biologically observed combinations: excitatory and inhibitory short-range, excitatory and inhibitory long-range-directional, and long-lasting long-range-diffuse nodes. The emergence of multinode functional units (MFUs) from the five-node-type model is mathematically demonstrated. These units correspond to cortical columns anatomically defined by the axon fields of relay cells, and are called columnar multinode functional units (CMFUs). CMFUs may, in turn, be part of larger functional groups designated coherent populations, which consist of widely distributed CMFUs in retinotopically equivalent locations. The existence of coherent populations imposes a three-level hierarchical structure on the model. To represent this hierarchical structure, a new type of CMFU node, which has a set of vector-valued inputs and outputs, is introduced. Each CMFU node contains a system of short-range nodes which supplies it with vector-valued inputs. Sets of long-range-diffuse nodes are also treated as vector-valued nodes whose outputs control the size and number of coherent populations. The role of coherent populations and hierarchical organization in the nervous system is discussed for such cognitive tasks as visual perception, attention and learning. Physiological and behavioral evidence are cited which support the existence of a similar three-level hierarchy in vertebrate brains.     

瞳孔光反应系统的空间分布式神经网络模型

为模拟刺激光空间分布变化引起瞳孔反应的实验现象,本文建立了空间分布式神经网络瞳孔模型。它是在瞳孔双通道模型基础上,借鉴Ｃａｎｎｏｎ－Ｒｏｂｉｎｓｏｎ的Ｏｃｕｌｏｍｏｔｏｒ模型的双层网络结构和视网膜的镶嵌式特点,经空间延括而成。空间各部位信号经第一层神经元处理得到对应各部位的线性ＤＣ和非线性ＡＣ输出,在第二层神经元进行空间综合,再经第三层神经元复合去控制效应器官虹膜肌的反应。该分布式部位机制模型能解释多种瞳孔实验现象。     

A structural and a functional aspect of stable information processing by the brain

Brain is an expert in producing the same output from a particular set of inputs, even from a very noisy environment. In this article a model of neural circuit in the brain has been proposed which is composed of cyclic sub-circuits. A big loop has been defined to be consisting of a feed forward path from the sensory neurons to the highest processing area of the brain and feed back paths from that region back up to close to the same sensory neurons. It has been mathematically shown how some smaller cycles can amplify signal. A big loop processes information by contrast and amplify principle. How a pair of presynaptic and postsynaptic neurons can be identified by an exact synchronization detection method has also been mentioned. It has been assumed that the spike train coming out of a firing neuron encodes all the information produced by it as output. It is possible to extract this information over a period of time by Fourier transforms. The Fourier coefficients arranged in a vector form will uniquely represent the neural spike train over a period of time. The information emanating out of all the neurons in a given neural circuit over a period of time can be represented by a collection of points in a multidimensional vector space. This cluster of points represents the functional or behavioral form of the neural circuit. It has been proposed that a particular cluster of vectors as the representation of a new behavior is chosen by the brain interactively with respect to the memory stored in that circuit and the amount of emotion involved. It has been proposed that in this situation a Coulomb force like expression governs the dynamics of functioning of the circuit and stability of the system is reached at the minimum of all the minima of a potential function derived from the force like expression. The calculations have been done with respect to a pseudometric defined in a multidimensional vector space.