I appreciate how many people took the time to engage with this. As the thread is a bit unwieldy, I figured I'd sum up the major classes of response as I saw them. In no particular order: https://twitter.com/neurograce/status/1258715184320970753

1. Agreement with the claim. Some people agree that if the tools of systems neuroscience aren't helping to understand ANNs then that doesn't bode well for their use on the brain.

2. Not so much an argument against the claim, but a rejection of the premise. This came in two forms:

2a. Claiming we do in fact understand ANNs. This was either in a sense that I consider trivial, as described here: https://twitter.com/neurograce/status/1258796028209364993 or a deeper claim that neuroscience-y tools have indeed helped us understand ANNs already and will continue to do so.

2b. Claiming "understanding" is too vague/impossible. For me, its clear there is something that a neuroscientist aims to achieve when they start a project/paper & they use certain tools to help them do it. And that's mostly what I mean by understand (& I think it's possible).

In a related vein, there were some claims that ANNs won't be understandable in the simple way we'd like to make sense of them, but then this problem would apply to brains too (so still not an argument against the claim).

3. The claim is false because brains and ANNs are different. Specific examples of differences were given but I didn't get a clear sense of what differences are the relevant ones that make the tools of systems neuroscience not applicable to ANNs or vice versa.

4. Brains may be "simpler" in a way that allows our current analysis methods to work on them but not on ANNs. This simplicity would come from the brain's modularity, need to have a world model & other constraints of evolution/development. This is to me the strongest rebuttal.

5. We already do understand (parts of) brains, therefore whether our tools work on ANNs is irrelevant because we already know we can understand the brain. I'd say there are plenty of things we don't understand & aren't sure how much progress we're making with our current tools