Read on Twitter
I've been examining the actual weight values of various models. Many layers have clear patterns that seem to be spectral lines.
It seems like neural networks encode information into specific rows or columns. I wonder why.
It seems like neural networks encode information into specific rows or columns. I wonder why.
Some layers have clear lines (left). Others have almost no discernible pattern (right).
OpenAI's position embedding layer (wpe) has very clear sinusoidal patterns (as observed by /u/fpgaminer: https://www.reddit.com/r/MachineLearning/comments/iifw9h/r_gpt2_position_embeddings_visualized/)
1.5B's wpe layer:
The first 2048 vocab embeddings from 1.5B are interesting.
The next 2048 vocab embeddings from 1.5B don't seem to have a clear pattern. I wonder what was special about the vertical stripes from the previous 2048...
Some layers have "hotspots": the model has decided that one specific weight value should be very different from the surrounding values. It's like a spectral line, but a point rather than a line.
(This layer is 'model/h8/mlp/c_fc/w' from OpenAI's 1558M)
(This layer is 'model/h8/mlp/c_fc/w' from OpenAI's 1558M)
More spectral lines, this time in 'model/h8/attn/c_attn/w' from OpenAI's 1558M.
Zooming out, you can see some clear horizontal patterns. Those are the 25 attention heads.
Zooming out, you can see some clear horizontal patterns. Those are the 25 attention heads.
Spectral lines from 'model/h8/attn/c_proj/w' of OpenAI's 1558M. The model seems to encode information into specific rows for some reason.
The spectral lines might correspond to the layernorm values. Here's a visualization of 'model/h8/ln_1/g' and 'model/h8/ln_1/b', which are 1D layers with 1600 values each.
Some values are very different from the others.
Some values are very different from the others.
The bias values from layer 0 are quite interesting.
('model/h0/attn/c_attn/b' from 1558M)
('model/h0/attn/c_attn/b' from 1558M)
Some spectral lines from VGG (layer 'vgg16_perceptual_distance_1/conv9/conv9')
I'm more and more convinced that these "spectral lines" are a general phenomenon, appearing across most or all models.
I'm more and more convinced that these "spectral lines" are a general phenomenon, appearing across most or all models.
'vgg16_perceptual_distance_1/conv8/conv8' has both spectral lines and hotspots.
