So. ML models and figuring out where their bad decisions are coming from. At some point we should give a primer on this topic, but for now, this thread will have to do... There's a specific thing going around that prompted this but it comes up often.

We aren't going to give a full overview of how the people who develop these models debug them. That's beyond our expertise, and it's a large subject. What you do need to know is that it is a long process that requires specialized tools to peek inside what the model is doing.

In most cases, for models that are part of social media platforms or advertising systems, the model is constantly being re-trained based on new data, so any conclusion you get through debugging is likely to be obsolete by the time you reach it.

That doesn't mean it's impossible to understand these systems, but it's not easy.

So, the key thing you need to know: Machine learning is unlike any other type of programming in that it isn't about making a decision through a sequence of simple tests. It's about evaluating the total situation, all at once.

People with less technical background than us have written a lot about how ML bias typically reflects bias in the training data. We won't try to rehash that, we'll just say that they're absolutely right.

In some sense, all that ML is doing is comparing things to its training data and finding the closest match.

Its concept of "closest" can be strange. With large training sets, it approximates what a human would do. With small training sets, it gets weird.

In all cases, you give the model hundreds or thousands of inputs, basically just whatever you can find that's plausibly related to the decision you're asking it to make, and it sorts things out.

For inputs that are images or videos, it gets somewhat more complicated because there's different ways to encode the pixels and the time information and stuff.

The reason we're making this thread is that a lot of the time, when people see a bad decision that an ML model made, and they aren't the developer, they tend to try to make a reduced test case.