Read on Twitter
VERY excited to announce new work from our team (one of the safety teams @OpenAI)!! 
We wanted to make training models to optimize human preferences Actually Workâą.
We applied it to English abstractive summarization, and got some pretty good results.
A thread
: (1/n) https://twitter.com/OpenAI/status/1301914879721234432
We wanted to make training models to optimize human preferences Actually Workâą.
We applied it to English abstractive summarization, and got some pretty good results.
A thread
: (1/n) https://twitter.com/OpenAI/status/1301914879721234432
Our basic approach:
1) We collect a dataset of humans comparing two summaries.
2) We train a reward model (RM) to predict the human-preferred summary.
3) We train a summarization policy to maximize the RM 'reward' using RL (PPO specifically). (2/n)
1) We collect a dataset of humans comparing two summaries.
2) We train a reward model (RM) to predict the human-preferred summary.
3) We train a summarization policy to maximize the RM 'reward' using RL (PPO specifically). (2/n)
(All our models are transformers. We take an earlier version of GPT-3, and fine-tune it via supervised learning to predict the human-written TL;DRs from the Reddit TL;DR dataset. We use that to initialize all our models.)
(3/n)
(3/n)
I think our results are pretty convincing. Our labelers prefer summaries from our 6.7B human feedback model ~70% of the time to the human-written reference TL;DRs.
(Note: this doesn't mean we're at 'human level' -- TL;DRs aren't the best summaries that humans can write).
(4/n)
(Note: this doesn't mean we're at 'human level' -- TL;DRs aren't the best summaries that humans can write).
(4/n)
When we transferred our Reddit-trained models to summarize CNN/DM news articles, they 'just worked'.
Summaries from our transfer model are >2x shorter, but they almost match supervised learning on CNN/DM. When controlling for length, we beat the CNN/DM ref summaries. (5/n)
Summaries from our transfer model are >2x shorter, but they almost match supervised learning on CNN/DM. When controlling for length, we beat the CNN/DM ref summaries. (5/n)
Fun fact: our labelers think the lead-3 baseline is better than the CNN/DM reference summaries! We checked this ourselves, and agreed with our labelers' ratings.
Not sure what this means for the usefulness of CNN/DM as a dataset. (This wasn't the case for TL;DR.)
(6/n)
Not sure what this means for the usefulness of CNN/DM as a dataset. (This wasn't the case for TL;DR.)
(6/n)
Why are our results better than the last time we did this? ( https://openai.com/blog/fine-tuning-gpt-2/)
Our 'secret ingredient' is working very closely with our labelers. We created an onboarding process, had a Slack channel where they could ask us questions, gave them lots of feedback, etc. (7/n)
Our 'secret ingredient' is working very closely with our labelers. We created an onboarding process, had a Slack channel where they could ask us questions, gave them lots of feedback, etc. (7/n)
As a result, our researcher-labeler agreement is about the same as researcher-researcher agreement. This wasn't the case last time, when the data was collected online, which made it harder to track.
(We also scaled up model size.
)
(8/n)
(We also scaled up model size.
)(8/n)
IMO one of the key takeaways of our paper is to work really closely with the humans labeling your data. This wasn't my approach at all in academia (which was more of the 'put it on MTurk and pray' variety). But without it our results wouldn't have been nearly as good. (9/n)
Finally, why is this AI safety? Good question!
Our team's goal is to train models that are aligned with what humans really want them to do.
This isn't going to happen by default. Language models trained on the Internet will make up facts. They aren't 'trying' to be honest. 10/n
Our team's goal is to train models that are aligned with what humans really want them to do.
This isn't going to happen by default. Language models trained on the Internet will make up facts. They aren't 'trying' to be honest. 10/n
ML models optimize what you literally tell them to. If this is different from what you want, you'll get behavior you don't want.
This is sometimes called "reward misspecification". There are lots of examples of this happening (eg: https://openai.com/blog/faulty-reward-functions/).
(11/n)
This is sometimes called "reward misspecification". There are lots of examples of this happening (eg: https://openai.com/blog/faulty-reward-functions/).
(11/n)
For summarization, "what humans what" is fairly straightforward.
But this will get trickier when we move to harder problems + more powerful AI systems, where small differences between 'what's good for humans' and 'what we're optimizing' could have big consequences. (12/n)
But this will get trickier when we move to harder problems + more powerful AI systems, where small differences between 'what's good for humans' and 'what we're optimizing' could have big consequences. (12/n)
One encouraging thing to me about this line of work: safety/ alignment-motivated research can also be *practical*.
It can help you train models that do what you want, and avoid harmful side-effects. That makes AI systems way more useful and helpful!! (13/n)
It can help you train models that do what you want, and avoid harmful side-effects. That makes AI systems way more useful and helpful!! (13/n)
This is only part of the story though -- to optimize 'what humans want', we need to figure out what that is! That'll require cross-disciplinary work from a lot of fields, plus participation from people who are actually going to be affected by the technology. (14/n)
The "AI research bubble" isn't really a bubble any more. Our models are being used in the real world. It's only going to increase from here. So, the 'safety' of a model doesn't just depend on the model itself, but on how it's deployed as part of a system that affects humans. 15/n
All of this work is with my incredible collaborators on the Reflection team: @paulfchristiano, Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel Ziegler, and @csvoss, plus @AlecRad and Dario Amodei. 
(fin)
(fin)
Whoops, didn't realize @d_m_ziegler and @longouyang had twitter accounts :).
