Read on Twitter
From logical reasoning to mental simulation, recurrent neural networks (RNNs) possess an incredible capacity for computation. To fully harness this capacity, we developed a programming language for RNNs https://arxiv.org/abs/2203.05032 . Here is a game of pong programmed inside of an RNN.
Okay okay, what do I mean by a programming language for RNNs? RNNs are usually trained on data using a cost function. In contrast, computers are programmed to run a sequence of operations in an algorithm that manipulates data in a precise way https://github.com/yxwangcs/MIPS-CPU.
We can program computers because we know precisely how the data are represented in binary, and precisely how to manipulate that representation according to the algorithm. So first we discovered precisely how RNNs represented data: in a natively symbolic and distributed way.
Then we just program weighted sums of the RNN’s symbolic representation to give us any output that we want! For example, we can program the output to perform independently addressable matrix operations such as a sum, difference, element-wise product, and matrix product.
Because of the recurrent ``R’’ in RNN, we can chain individual neural processing units (NPUs) together and feed their outputs back as inputs to define symbolic algebraic equations. The RNN then converges to the solution of problems like the least-squares linear regression,
or the Lyapunov equation to solve for the controllability Gramian.
Beyond symbolic computations, we can extend our conception of ``memory’’ from a static collection of binary 1’s and 0’s to a time evolving and geometric collection of dynamical random access memory (dRAM) that can be programmed into specific and recallable neural addresses.
Now, this poses an fun question. If we can program our RNNs to store dynamical memories, and an RNN is a dynamical system, then can we program a guest RNN into a host RNN? The answer is: yes! This process of virtualizing RNNs is a significant step towards an RNN theory of mind.
So how far can we take this neural programming language (NPL)? Well, as far as we can take traditional digital logic! For example, we can program RNNs to behave as the logic gates AND, NAND, OR, NOR, XOR, and XNOR,
which means that we can program RNNs to evolve along any logic circuit, including a binary adder to form the basis of a neural logic unit (NLU), a set-reset latch to form the basis of digital RAM, and multivibrator oscillators to generate precise clock timers and phases.
But the most exciting prospect is the development of neural software that runs natively on RNN hardware. RNNs represent and modify information in a natively distributed and dynamical way, so operations like trajectory integration are as basic to RNNs as addition is to computers.
Many thanks to @DaniSBassett for the collaboration, and Melody Lim, @kmurmurs, @harangju, @neuro_dz, and @JenniferStiso for comments!
