VSA Basic Model

• 2022-04-03 Design notes
  • Objectives
  • Design
• References

2022-04-03 Design notes

Source: 2022-04-03-design.Rmd

This entry contains the VBM1 design notes.

As my design thinking evolves over the course of the project the notes here will be modified so that this note always reflects my most recent position. Consequently the head date shouldn’t be taken too seriously.

Objectives

The purpose of the VSA Basic Model project is to develop a basic VSA model (VBM1), as an alternative to HRR, FHRR, BSC, MAP, etc. (Kleyko et al., 2021), which is conceptually as simple as possible, and relatively cheap computationally. The motivation for this is that VBM1 would be the default VSA model for empirical projects, that is, it would be the VSA equivalent of the geneticist’s fruit fly.

Design

Design of a VSA model consists of determining a vector space and defining the operators on that vector space. This section documents the design choices and the reasoning leading to those choices.

Data type of the scalar field of the vector space

This section is probably severely hampered by my lack of mathematical skills, but I can only work with what I’ve got. VSA models are treated as vector spaces and operators on that vector space. A vector space is defined as a set \(V\) of vectors and a field \(F\) of scalars, and two binary operators, vector addition (bundling) and scalar multiplication, that satisfy eight axioms.

The vector spaces of VSA models have extra requirements that make them special cases of vector spaces. Orthogonality of vectors plays a major role in VSA systems, so the concept of angles between vectors must be meaningful - so the vector spaces must be inner product spaces. Additionally, VSA models also have a multiplication operator between vectors (binding), making them algebras over fields. Finally, VSA models also have a family of unary permutation operators (braiding) - I have no idea of the mathematical consequences of that. However, all these extra features are not relevant to this section, which is only concerned with the scalar field of the vector space.

A \(d\)-dimensional vector can be represented as a \(d\)-tuple of scalar values from some field. This is a coordinate representation of the vector. In the standard interpretation of VSA models as neural networks the elements of the tuple correspond to outputs of neural units. This coordinate based representation corresponds to the implementation level view of the vectors - it is how the hardware “sees” the vectors. The computational level view is all about the relationships between the vectors, that is how they are derived from each other by the available operations. At the computational level the specific coordinates are not relevant, the computational level takes a coordinate-free view of the vectors. That is, the set of relationships between vectors can be instantiated by many different coordinate representations.

The implementation view of a vector is a tuple of scalar values from some field, so choosing the field selects the basic properties of the vector space, and sets the types of the inputs and outputs of the vector space operators. A field is defined as a set of values and two binary operators on those values (addition and multiplication), satisfying six axioms.

The (probably) most widely known fields, which are commonly used in VSA models, are the real numbers (\(\mathbb{R}\)) and the complex numbers (\(\mathbb{C}\)). It’s worth emphasizing that the field consists of both the set of values and the mathematical operators. When we implement VSA models we define may define our own VSA operators (typically operating on the computer data types of real or complex numbers). If the VSA operators we define are not identical to the mathematical operators then the combination of those VSA operators and the computer data types they operate on will not be implementations of (\(\mathbb{R}\)) and (\(\mathbb{C}\)) and might not even implement fields.

The point is, that problem of designing a VSA model is essentially the problem of trying to design a mathematical entity (such as, an algebra over a field) that has desirable behavioural properties (i.e. the axioms satisfied). We, don’t yet know the best (or minimum acceptable) mathematical properties for a VSA model appear to mostly be algebras over inner product spaces.

Having said all that, the process of designing a VSA model doesn’t have to be driven from the mathematical end by defining the desired mathematical properties and then designing an implementation. It can also be approached as a programming problem - choosing the data type of the elements of the \(d\)-tuples, then designing operators on those tuples to have the desired mathematical properties. This is the approach I will be taking.

Another point to remember about physical realisations of VSA models is that they are approximations to the mathematical abstractions. A mathematical model may use real numbers (\(\mathbb{R}\)), but the real computer data type has a limited range and a limited precision. Similarly, a neurophysiological implementation may be in terms of neural firing rates or spike timing, which will have limited ranges (firing rates must be non-negative and will have some maximum) and limited precision due to various noise processes. The modeller may move between viewing the implementation as an approximate realisation of the mathematical ideal, or the mathematical abstraction as an approximation to the reality of the physical implementation. Regardless of whether the mathematical model or the physical implementation is viewed as the approximation, the important point is that the desired behaviour should be displayed over a sufficiently wide range of conditions to be useful.

It is important to emphasize that choice of the scalar data type does not uniquely determine the properties of the VSA system. It is the combination of the operators and the scalar type that matters, and apparently different models can be mathematically equivalent. For example, Plate (1994, Chapters 3 & 4) introduced a real valued VSA model (HRR) with circular convolution as the binding operator, and a complex valued VSA model (FHRR) with elementwise multiplication as the binding operator. The complex valued model was introduced as a computational speed-up of the real valued model (1994, sec. 3.6.2)and is mathematically equivalent. Although Plate (1994, p. 103) refers to the real valued model as the “standard system” he concludes that complex valued model is at least slightly superior. In fact, the computational cost of the complex valued model is dominated by the transformation from real valued model. If we accept that the two models are mathematically equivalent, then why not work entirely with the complex valued model and avoid the cost of the transformations between real and complex values (Plate, 1994, p. 101)?

Complex valued neural networks (also called phasor networks) have been discussed since at least as early as the late 1980s (Noest, 1987). These very earliest mentions suggested implementation of complex values by physical (optical) resonators. More recently (Frady & Sommer, 2019) it has been shown that complex values can be naturally implemented in terms of neural spike timing, which is both neurobiologically interesting and advantageous for neuromorphic computing.

Schlegel et al. (2021, Table 1) lists the scalar field type for a range of better known VSA models. Only one model (FHRR) is shown as using complex numbers. The rest of the models are shown as using real numbers or subsets of the real numbers (integers, \(\{-1, 1\}\), \(\{0, 1\}\)). However, many of these can be interpreted as functionally equivalent to complex valued scalars. Complex values, which can be interpreted as two dimensional vectors in rectangular coordinates, can be reinterpreted as an angle and magnitude in polar coordinates.

CONTINUE FROM HERE

Locality

Label and support interpretation

Parallel local values as approximation to latent vector

xx

Normalisation

Global normalisation

Local normalisation

Permutation

xx

Bundling

Additive bundling

Sampling bundling

xx

Binding

xx

References