Musings on Value vs. Pulsed Neurons and Locality of Reference

Neural systems in the style of Convolutional Neural Networks et. al. use matrix math and tensors to simulate the overall activity level of neurons. The reason for "overall activity level" rather than individual pulses from neurons is that the overall activity is simply the integration over time of what would be pulses in real neurons. Overall activity also is amenable to calculus with derivatives for back propagation.
The reason for matrix math is locality of memory reference and simplicity of written equations for what the system is doing. Microprocessors have local CPU memory cache, and main memory. Getting data from main memory to local cache is very slow relative to processor instruction times.
Matrix math tends to make operations happen in address sequential order so is efficient in cache.  Also GPU design favors sequential and matrix operation and is ultra-fast.
All the above makes tensor style processing for AI efficient.

Pulsed neurons are more object oriented, such that each neuron is an object.  The neural connections can be from any neuron to any neuron and may or may not be pulsed vs. overall activity level. In the case of simulations of neural systems such as robot control and subsumptive architectures, pulsed neurons have several advantages and disadvantages.
The major disadvantage of pulsed neural architectures is the complete non-locality of reference. As a slight efficiency gain one can use a N-dimensional map of neurons to simulate a cluster of neurons and or some concept in the system and gain locality of reference to some degree. This locality of reference makes a linear pass over the cluster updating neuron charge, fire state flags, and activation levels efficient. One still has no locality of reference in regards to connections to other areas of the system.
To further complicate things, a cognitive system with learning going on via. self adaptive connections will be constantly rewiring the neural connections and further reduce locality of reference.
One huge gain in subsumptive pulsed neural simulations is that the vast majority of neurons are inactive most of the time.  If a neuron does not pulse in a given time tick, then the target down stream neurons do not even have to be accessed so there is no locality of reference problem.

References
Cache vs. Memory
Locality of reference and matrix math

Comments

Post a Comment

Popular posts from this blog

UE4 Asset Directory Listing Blueprint

Image
So I needed to be able to, in game, get listing of asset directories and assets in a folder style.  The purpose is to let me make on-the-fly menu selections of something like 10000 assets in game and do it fast. Turns out in UE4.25 you can do all of this in blueprints. The low level problem is that assets get packaged into .pak files when you ship the game.  Regular C++ style file and directory operations will not find any assets.  Instead there is the Asset Manager. Assets are bundled as assets and are not in a directory structure in the pak file.  The path to an asset is in effect just a string.  So the over all flow is... Load all the assets under one or more asset directories.  When the game is packaged what was something like Content/DefaultObjects/Trees becomes /Game/DefaultObjects/Trees. When you want to ask "What sub directories and assets are in /Game/DefaultObject/Trees you call my blueprint node.  It returns a result structure with a list of string for subdir
Read more

Subsumption and Neural Hiding - The 'Object Oriented Paradigm' of Artificial Intelligence

In computer programming Object Oriented Programming (OOP) gave programmers a neat way to compartmentalize programs, rather than having a mess of disorganized code and data.  One can neatly make an object, treat it as a black-box and reduce the cognitive load on the programmer.  Nifty. In reactive neural networks where one is taking world inputs and generating response outputs Subsumption vastly simplifies the work of designing the neural processing. From dictionary.com: verb (used with object),   sub·sumed,   sub·sum·ing. to  consider  or  include  (an  idea,   term,   proposition,   etc.)  as  part  of a  more   comprehensive   one. to  bring  (a  case,   instance,   etc.)   under  a  rule. to  take  up  into  a  more   inclusive   classification. Subsumption in the context of neural networks is where some reflex is designed as a response to inputs or other neurons and overrides 'lower level' reflexes.  Then some other reflex is desired so that can then be des
Read more

Self Cycling Asynchronous Neural Systems

Current neural network systems have a general paradigm of being given some input, and then producing a resulting output. A true general AI should act like a biological system and be self-sequencing. The neural system should just free-run, responding to inputs and creating outputs and learning. One method I am experimenting with in Cognate and NuTank is having a hard-wired set of concept maps (a N-dimensional set of neurons used to embody a concept, e.g. foveolar edge angles.) that take inputs from the external world and produce hard-wired reactions as outputs to control a real or simulated robot. Such a system also uses subsumption to aid in much easier design. This is the Concept Map System (CMS) for the purposes of this post. For now lets ignore the idea that the hard-wired concept map system might also be adapting and evolving over time simulating neroplasticity. One then has a large 3D array of neurons that are in acting as a pattern matcher on the entire CMS.  What this 3D a
Read more