Since I declined to join any of the closed-door off-the-record sessions, today is my first chance to view the outcome of the Facebook Civil Rights Audit, which I was consulted on https://about.fb.com/wp-content/uploads/2020/07/Civil-Rights-Audit-Final-Report.pdf

Lots of "Facebook says..." in the report. Looks like no one took (or could implement) my advice that the audit should include systematic evidence collection beyond taking Facebook's statements at face value. Sigh. At least they call for better data?

Overall, there's an emphasis on more descriptive data, which is valuable but insufficient. Future audits should expect Facebook to report the results of their internal research and not just repeat unsubstantiated claims like "Facebook completed a successful pilot..."

For example, when FB says their systems that once removed 65% of hate speech before it's flagged now remove 89%, I have SO MANY QUESTIONS. Multi-year comparisons are tricky and it's easy to put a thumb on the scale. It's impossible to know if there was any real improvement.

If advocacy orgs and auditors doing similar work ever want advice on what kinds of evidence to ask for—evidence that will actually help understand problems and lead to improvements, *please* reach out. I do a lot of this already, and early conversations can make a big difference.

For a view from the advocacy side, here is @ColorOfChange's statement: https://twitter.com/ColorOfChange/status/1280907259250446337

I remember being surprised to learn that in the corporate accountability world, audits rarely include data. They usually involve investigators/lawyers asking employees lots of questions. Here's @theGNI independent assessment toolkit: https://globalnetworkinitiative.org/wp-content/uploads/2019/11/GNI-Assessment-Toolkit.pdf

In 2020, anyone auditing a tech firm on civil rights needs someone with social science research training, so they can ask the right questions & critically assess empirical claims coming from tech firms. In the ideal world, they would also have access to internal research reports