Thread by @charles_irl, 1/hellathis by @daniela_witten is a masterclass in both the #SVD and [...]

1/hella

this

https://abs.twimg.com/emoji/v2/... draggable="false" alt="🧵" title="Thread" aria-label="Emoji: Thread"> by @daniela_witten is a masterclass in both the #SVD and in technical communication on Twitter.

i want to hop on this to expand on the "magic" of this decomposition and show folks where the rabbit goes, because i just gave a talk on it this week!

https://abs.twimg.com/emoji/v2/... draggable="false" alt="🧙‍♂️" title="Männlicher Magier" aria-label="Emoji: Männlicher Magier">

https://abs.twimg.com/emoji/v2/... draggable="false" alt="🐇" title="Hase" aria-label="Emoji: Hase">

https://abs.twimg.com/emoji/v2/... draggable="false" alt="💨" title="Staubwolke" aria-label="Emoji: Staubwolke">

https://abs.twimg.com/emoji/v2/... draggable="false" alt="😱" title="Vor Angst schreiendes Gesicht" aria-label="Emoji: Vor Angst schreiendes Gesicht"> https://twitter.com/WomenInStat/status/1285612667839885312">https://twitter.com/WomenInSt...

https://twitter.com/WomenInStat/status/1285612667839885312

tl;dr: the basic idea of the SVD works for _any_ function.

it& #39;s a three step decomposition:

- throw away the useless bits

https://abs.twimg.com/emoji/v2/... draggable="false" alt="⤵" title="Nach rechts zeigender Pfeil mit Krümmung nach unten" aria-label="Emoji: Nach rechts zeigender Pfeil mit Krümmung nach unten">
- rename what remains

https://abs.twimg.com/emoji/v2/... draggable="false" alt="🔀" title="Verbogene nach rechts zeigende Pfeile" aria-label="Emoji: Verbogene nach rechts zeigende Pfeile">
- insert yourself into the right context

https://abs.twimg.com/emoji/v2/... draggable="false" alt="⤴" title="Nach rechts zeigender Pfeil mit Krümmung nach oben" aria-label="Emoji: Nach rechts zeigender Pfeil mit Krümmung nach oben">

tl;dr: the basic idea of the SVD works for _any_ function. it& #39;s a three step decomposition:- throw away the useless bits https://abs.twimg.com/emoji/v2/... draggable=

- rename what remains https://abs.twimg.com/emoji/v2/... draggable="false" alt="🔀" title="Verbogene nach rechts zeigende Pfeile" aria-label="Emoji: Verbogene nach rechts zeigende Pfeile">- insert yourself into the right context https://abs.twimg.com/emoji/v2/... draggable="false" alt="⤴" title="Nach rechts zeigender Pfeil mit Krümmung nach oben" aria-label="Emoji: Nach rechts zeigender Pfeil mit Krümmung nach oben">" title="tl;dr: the basic idea of the SVD works for _any_ function. it& #39;s a three step decomposition:- throw away the useless bits https://abs.twimg.com/emoji/v2/... draggable="false" alt="⤵" title="Nach rechts zeigender Pfeil mit Krümmung nach unten" aria-label="Emoji: Nach rechts zeigender Pfeil mit Krümmung nach unten">- rename what remains https://abs.twimg.com/emoji/v2/... draggable="false" alt="🔀" title="Verbogene nach rechts zeigende Pfeile" aria-label="Emoji: Verbogene nach rechts zeigende Pfeile">- insert yourself into the right context https://abs.twimg.com/emoji/v2/... draggable="false" alt="⤴" title="Nach rechts zeigender Pfeil mit Krümmung nach oben" aria-label="Emoji: Nach rechts zeigender Pfeil mit Krümmung nach oben">" class="img-responsive" style="max-width:100%;"/>

also, if you& #39;re more of a "YouTube talk" than a "tweet wall" kinda person, check out the video version, given as part of the @weights_biases Deep Learning Salon webinar series https://www.youtube.com/watch?v=H1OP0XOREh4">https://www.youtube.com/watch...

first, let& #39;s orient ourselves.

we& #39;re talking about the _singular value decomposition_ of a matrix M

M = U Σ V^T

where U is tall, V^T is skinny, and Σ is diagonal, like in the pic below

instead of thinking of these matrices as "blocks of numbers", though, i find it more illuminating to think of them as "functions that take in arrays and spit out arrays"

so what do these matrices do?

- V^T removes any part of the input that lies in the kernel of M, i.e. any part that goes to 0, and lowers the dimension (r <= n)
- Σ scales each entry of its input
- U takes low-dimensional inputs and "pads" them so they look like higher-dimensional things (r <= m)

from the perspective of "matrices are functions", it makes more sense to write the SVD with this diagram

how is this read? starting at a node (array), we can "travel" to a different node (new array) by following an arrow (applying a matrix).

this diagram "commutes", meaning that if any two paths start + end at the same nodes, the matrices for each path are equal

#followyourarrow

a path with a single arrow corresponds to the matrix labeling that arrow.

but what if i follow a path w more than one arrow, e.g. Σ then U? what& #39;s the matrix?

it& #39;s the matrix product, ΣU!

in fact, this is my preferred way to define the matrix product! much clearer motivation

this approach is motivated by the idea that linear algebra is _not_ like algebra, where "algebra" here means "method for manipulating equations"

instead, it& #39;s more like programming, where we manipulate and compose functions

see this talk for more! https://www.youtube.com/watch?v=F3lG9_SxCXk">https://www.youtube.com/watch...

so, how does this help us understand the SVD?

well, the idea of breaking down one function into three functions is a very general one.

in its broadest definition, it can be applied to _any_ function, as pictured below

the three steps:

- an "onto" function, aka surjection/epimorphism
- a "reversible" function, aka bijection/isomorphism
- a "one-to-one" function, aka injection/monomorphism

the Latin names (-jection) are more common, the Greek names (-morphism) more general. use them to taste!

i like using emojis:

https://abs.twimg.com/emoji/v2/... draggable="false" alt="⤵" title="Nach rechts zeigender Pfeil mit Krümmung nach unten" aria-label="Emoji: Nach rechts zeigender Pfeil mit Krümmung nach unten">,

https://abs.twimg.com/emoji/v2/... draggable="false" alt="🔀" title="Verbogene nach rechts zeigende Pfeile" aria-label="Emoji: Verbogene nach rechts zeigende Pfeile">, and

https://abs.twimg.com/emoji/v2/... draggable="false" alt="⤴" title="Nach rechts zeigender Pfeil mit Krümmung nach oben" aria-label="Emoji: Nach rechts zeigender Pfeil mit Krümmung nach oben">

here& #39;s an example, applied to a Python function that might show up IRL: is_odd takes in an Int-eger and returns a String that identifies whether the input is even or odd

the first step picks out two "representatives" -- one for each possible output of is_odd. many onto one

https://abs.twimg.com/emoji/v2/... draggable="false" alt="⤵" title="Nach rechts zeigender Pfeil mit Krümmung nach unten" aria-label="Emoji: Nach rechts zeigender Pfeil mit Krümmung nach unten">

the second step assigns each representative to the right output _value_. a perfect pairing

https://abs.twimg.com/emoji/v2/... draggable="false" alt="🔀" title="Verbogene nach rechts zeigende Pfeile" aria-label="Emoji: Verbogene nach rechts zeigende Pfeile">

the final step ensures we have the right output _type_. just "inserting"

https://abs.twimg.com/emoji/v2/... draggable="false" alt="⤴" title="Nach rechts zeigender Pfeil mit Krümmung nach oben" aria-label="Emoji: Nach rechts zeigender Pfeil mit Krümmung nach oben">

the first step picks out two "representatives" -- one for each possible output of is_odd. many onto one https://abs.twimg.com/emoji/v2/... draggable=

the second step assigns each representative to the right output _value_. a perfect pairing https://abs.twimg.com/emoji/v2/... draggable="false" alt="🔀" title="Verbogene nach rechts zeigende Pfeile" aria-label="Emoji: Verbogene nach rechts zeigende Pfeile">the final step ensures we have the right output _type_. just "inserting" https://abs.twimg.com/emoji/v2/... draggable="false" alt="⤴" title="Nach rechts zeigender Pfeil mit Krümmung nach oben" aria-label="Emoji: Nach rechts zeigender Pfeil mit Krümmung nach oben">" title="the first step picks out two "representatives" -- one for each possible output of is_odd. many onto one https://abs.twimg.com/emoji/v2/... draggable="false" alt="⤵" title="Nach rechts zeigender Pfeil mit Krümmung nach unten" aria-label="Emoji: Nach rechts zeigender Pfeil mit Krümmung nach unten">the second step assigns each representative to the right output _value_. a perfect pairing https://abs.twimg.com/emoji/v2/... draggable="false" alt="🔀" title="Verbogene nach rechts zeigende Pfeile" aria-label="Emoji: Verbogene nach rechts zeigende Pfeile">the final step ensures we have the right output _type_. just "inserting" https://abs.twimg.com/emoji/v2/... draggable="false" alt="⤴" title="Nach rechts zeigender Pfeil mit Krümmung nach oben" aria-label="Emoji: Nach rechts zeigender Pfeil mit Krümmung nach oben">" class="img-responsive" style="max-width:100%;"/>

let& #39;s look back at the SVD:

- V^T maps the kernel of M to 0. 0 is in the kernel, all others have unique outputs. it& #39;s picking out representatives!

https://abs.twimg.com/emoji/v2/... draggable="false" alt="⤵" title="Nach rechts zeigender Pfeil mit Krümmung nach unten" aria-label="Emoji: Nach rechts zeigender Pfeil mit Krümmung nach unten">

https://abs.twimg.com/emoji/v2/... draggable="false" alt="✅" title="Fettes weißes Häkchen" aria-label="Emoji: Fettes weißes Häkchen">
- Σ just scales things. that& #39;s reversible!

https://abs.twimg.com/emoji/v2/... draggable="false" alt="🔀" title="Verbogene nach rechts zeigende Pfeile" aria-label="Emoji: Verbogene nach rechts zeigende Pfeile">

https://abs.twimg.com/emoji/v2/... draggable="false" alt="✅" title="Fettes weißes Häkchen" aria-label="Emoji: Fettes weißes Häkchen">
- and U takes r-dim arrays and makes them n-dim. that& #39;s type-matching!

https://abs.twimg.com/emoji/v2/... draggable="false" alt="⤴" title="Nach rechts zeigender Pfeil mit Krümmung nach oben" aria-label="Emoji: Nach rechts zeigender Pfeil mit Krümmung nach oben">

https://abs.twimg.com/emoji/v2/... draggable="false" alt="✅" title="Fettes weißes Häkchen" aria-label="Emoji: Fettes weißes Häkchen">

let& #39;s look back at the SVD:- V^T maps the kernel of M to 0. 0 is in the kernel, all others have unique outputs. it& #39;s picking out representatives! https://abs.twimg.com/emoji/v2/... draggable=

this doesn& #39;t explain the centrality of the SVD in applications or why it& #39;s so efficiently computable. for that, see the original thread from @WomenInStat

but it does explain why the SVD exists for all matrices -- matrices are functions, and this decomp exists for all functions!

this decomposition, of which SVD is just the coolest e.g., is known as the "1st Isomorphism Theorem" in its full generality

everywhere it appears, it sparks insight, connects multiple fundamental ideas, and relates seemingly distant concepts

you might even call it ... magic!

https://abs.twimg.com/emoji/v2/... draggable="false" alt="😈" title="Lächelndes Gesicht mit Hörnern" aria-label="Emoji: Lächelndes Gesicht mit Hörnern">

just one example: in group theory, the 1st Isomorphism Theorem leads to kernels, quotient groups, and more, as explained by @math3ma: https://www.math3ma.com/blog/the-first-isomorphism-theorem-intuitively

check">https://www.math3ma.com/blog/the-... out her blog for more really great math explainers. eagerly awaiting her book on topology!

hella/hella, end of

https://abs.twimg.com/emoji/v2/... draggable="false" alt="🧵" title="Thread" aria-label="Emoji: Thread">

just one example: in group theory, the 1st Isomorphism Theorem leads to kernels, quotient groups, and more, as explained by @math3ma: https://www.math3ma.com/blog/the-... out her blog for more really great math explainers. eagerly awaiting her book on topology!hella/hella, end of https://abs.twimg.com/emoji/v2/... draggable=

" title="just one example: in group theory, the 1st Isomorphism Theorem leads to kernels, quotient groups, and more, as explained by @math3ma: https://www.math3ma.com/blog/the-... out her blog for more really great math explainers. eagerly awaiting her book on topology!hella/hella, end of https://abs.twimg.com/emoji/v2/... draggable="false" alt="🧵" title="Thread" aria-label="Emoji: Thread">" class="img-responsive" style="max-width:100%;"/>

Latest Threads Unrolled: