Read on Twitter
These tweets about Twitter photo cropping have been going around:
https://twitter.com/bascule/status/1307440596668182528
So I started looking into how it works. It's interesting and a good example of how AI tech can produce results that look biased, even when the building blocks don't seem to be. Thread! 1/
https://twitter.com/bascule/status/1307440596668182528
So I started looking into how it works. It's interesting and a good example of how AI tech can produce results that look biased, even when the building blocks don't seem to be. Thread! 1/
We know how it works, because Twitter fortunately published the implementation details here:
https://blog.twitter.com/engineering/en_us/topics/infrastructure/2018/Smart-Auto-Cropping-of-Images.html
(It's from 2018, but I'm assuming it's still in operation.)
The algo is actually refreshingly simple - and interesting.
2/
https://blog.twitter.com/engineering/en_us/topics/infrastructure/2018/Smart-Auto-Cropping-of-Images.html
(It's from 2018, but I'm assuming it's still in operation.)
The algo is actually refreshingly simple - and interesting.
2/
https://blog.twitter.com/engineering/en_us/topics/infrastructure/2018/Smart-Auto-Cropping-of-Images.html
The photo cropping algo doesn't look at faces, etc. It has no idea what it's looking at. (Obviously ;) )
Rather, it's trying to predict what are the "interesting" parts of a photo, that a human might want to look at & focus on those.
So - how does it know what's interesting? 3/
Rather, it's trying to predict what are the "interesting" parts of a photo, that a human might want to look at & focus on those.
So - how does it know what's interesting? 3/
In short: it's using gaze prediction.
It starts with data about how human eye wanders when looking at images.
With this data, one can train ML systems to predict what parts of photos are most likely to be looked at.
There's a great paper here:
https://arxiv.org/pdf/1801.05787.pdf
4/
It starts with data about how human eye wanders when looking at images.
With this data, one can train ML systems to predict what parts of photos are most likely to be looked at.
There's a great paper here:
https://arxiv.org/pdf/1801.05787.pdf
4/
And then, the assumption goes, those parts are the ones that are most interesting to viewers. The white paper calls this "saliency prediction".
So when the system needs to crop the image down, it can try to leave as much of the interesting area in the picture as possible.
5/
So when the system needs to crop the image down, it can try to leave as much of the interesting area in the picture as possible.
5/
Generally speaking, the algo is *really good* in finding some "interesting" parts of a picture.
And it does it without any knowledge about what's in the picture.
All it uses is knowledge about prior studies of human gaze, and what image features attracted human attention. 6/
And it does it without any knowledge about what's in the picture.
All it uses is knowledge about prior studies of human gaze, and what image features attracted human attention. 6/
(Speculation time: since image detection DNNs are very sensitive to textures, I'd guess the algo keys in on areas with high contrast, lots of variety ("high frequency signal"), and maybe some specific patterns.
But: the algo does *not* know who or what it's looking at.)
7/
But: the algo does *not* know who or what it's looking at.)
7/
So far, so good. We have an algorithm that looks at uploaded pictures, and finds the "most salient" areas to keep when cropping.
This works great for selfies, travel photos, etc. Things where there's one main thing in focus.
The problem is: what if there are two? :)
8/
This works great for selfies, travel photos, etc. Things where there's one main thing in focus.
The problem is: what if there are two? :)
8/
The tweets that caused all the furor are designed to force the algorithm into a really tough spot.
The images are designed to have two different "interesting" areas, that are so far apart that only one of them will get picked.
So the algo picks one based on "saliency".
9/
The images are designed to have two different "interesting" areas, that are so far apart that only one of them will get picked.
So the algo picks one based on "saliency".
9/
But as we've already seen, this saliency has nothing to do with the actual topic of the picture.
The algo doesn't even know that it's cropping a person.
It's just trying to predict: "if a human is looking at this picture, which pixels are most likely to be looked at"?
10/
The algo doesn't even know that it's cropping a person.
It's just trying to predict: "if a human is looking at this picture, which pixels are most likely to be looked at"?
10/
So by taking a system designed to "pick some interesting pixels" and forcing it to "pick between two famous people in a politically charged context", the system got turned on its head: it was forced to make a decision that carried completely unexpected meaning.
11/
11/
It's a super interesting example of a system that produced racially biased results, and consistently, even though the constituent pieces don't *seem* like they would encode racial biases (i.e. gaze tracking feels like a "neutral" technology).
12/
12/
But of course, it wasn't neutral. Clearly some biases existed although in normal operation they probably wouldn't get noticed.
But they got amplified by setting up this experiment in an adversarial way, and forcing the system to make a binary choice. And choose it did. :)
13/
But they got amplified by setting up this experiment in an adversarial way, and forcing the system to make a binary choice. And choose it did. :)
13/
It will take a while for Twitter (and others) to untangle where the bias is coming from. It could be any number of things: biased selection of photos to train from, biased humans whose gaze was used for training, bias towards specific photo features (e.g. brightness), etc.
14/
14/
And once we know there's bias, there *are* ways to fix it. For example, by fixing training data.
Or in Twitter's case: instead of making the system pick the "best" photo, make it recognize there were N good matches, and pick among them with a good distribution.
15/
Or in Twitter's case: instead of making the system pick the "best" photo, make it recognize there were N good matches, and pick among them with a good distribution.
15/
And just to finish - I'm actually super interested in another issue.
We would never know about this biased behavior of the system, until it got put into a sufficiently adversarial situation that was right at the boundary of its operational range.
16/
We would never know about this biased behavior of the system, until it got put into a sufficiently adversarial situation that was right at the boundary of its operational range.
16/
I wonder if devs would be able to notice this bias from just everyday use monitoring.
Because I imagine "saliency picking" works just fine most of the time! If people post vacation pics or selfies on Twitter, I imagine it works well without showing any bias. 17/
Because I imagine "saliency picking" works just fine most of the time! If people post vacation pics or selfies on Twitter, I imagine it works well without showing any bias. 17/
So can we predict this kind of "unpredictable" behavior at the edge of the operational range, before people try to misuse it?
That's the big question with no clear answers just yet.
But that's why we need to be vigilant of bad results, and share them when we see them.
18/18
That's the big question with no clear answers just yet.
But that's why we need to be vigilant of bad results, and share them when we see them.
18/18
PS. omg this thread https://twitter.com/therourke/status/1307711835836219394
