Read on Twitter
. @Openai GPT-3 Thoughts and Takeaways
Demos are fun, but let& #39;s discuss the details.
This thread talks about about sentence completion, trade-offs, few shot learning, fine-tuning, technical takeaways, industry impacts, ethics, fun facts, and open questions.
cc @gdb
(1/13)
Demos are fun, but let& #39;s discuss the details.
This thread talks about about sentence completion, trade-offs, few shot learning, fine-tuning, technical takeaways, industry impacts, ethics, fun facts, and open questions.
cc @gdb
(1/13)
There are
There are
(2/13)
Completion Parameters
https://abs.twimg.com/emoji/v2/... draggable="false" alt="🥇" title="Goldmedaille" aria-label="Emoji: Goldmedaille"> Prompt - Input text.
https://abs.twimg.com/emoji/v2/... draggable="false" alt="🥈" title="Silbermedaille" aria-label="Emoji: Silbermedaille"> Max_tokens - Output token length.
https://abs.twimg.com/emoji/v2/... draggable="false" alt="🌡️" title="Thermometer" aria-label="Emoji: Thermometer"> Temperature -
https://abs.twimg.com/emoji/v2/... draggable="false" alt="⬇️" title="Pfeil nach unten" aria-label="Emoji: Pfeil nach unten"> = less random + more deterministic. https://abs.twimg.com/emoji/v2/... draggable="false" alt="⬆️" title="Pfeil nach oben" aria-label="Emoji: Pfeil nach oben"> = more “creative.”
https://abs.twimg.com/emoji/v2/... draggable="false" alt="4⃣" title="Tastenkappe Ziffer 4" aria-label="Emoji: Tastenkappe Ziffer 4"> Top_p - Diversity via nucleus sampling.
https://abs.twimg.com/emoji/v2/... draggable="false" alt="5⃣" title="Tastenkappe Ziffer 5" aria-label="Emoji: Tastenkappe Ziffer 5"> Frequency_Penalty - https://abs.twimg.com/emoji/v2/... draggable="false" alt="⬆️" title="Pfeil nach oben" aria-label="Emoji: Pfeil nach oben"> = https://abs.twimg.com/emoji/v2/... draggable="false" alt="⬇️" title="Pfeil nach unten" aria-label="Emoji: Pfeil nach unten"> repetition.
(3/13)
(3/13)
(4/13)
GPT-3 thinks it& #39;s 10 years old and wants to be a doctor when it grows because it wants to help people.
The playground is a fun toy, but the API makes running GPT-3 easier than running a linear regression using @scikit_learn.
(5/13)
Example using "Playground": Q and AGPT-3 thinks it& #39;s 10 years old and wants to be a doctor when it grows because it wants to help people. The playground is a fun toy, but the API makes running GPT-3 easier than running a linear regression using @scikit_learn. (5/13)" title="https://abs.twimg.com/emoji/v2/... draggable="false" alt="👶" title="Baby" aria-label="Emoji: Baby"> Example using "Playground": Q and AGPT-3 thinks it& #39;s 10 years old and wants to be a doctor when it grows because it wants to help people. The playground is a fun toy, but the API makes running GPT-3 easier than running a linear regression using @scikit_learn. (5/13)" class="img-responsive" style="max-width:100%;"/>
I discovered an interesting trade-off between random creativity and reproducible logic when experimenting on the GRE multiple choice sentence completion task. Increasing the temperature (or creativity) decreased the accuracy.
(6/13)
Example using API: GPT-3 Cracking the GREI discovered an interesting trade-off between random creativity and reproducible logic when experimenting on the GRE multiple choice sentence completion task. Increasing the temperature (or creativity) decreased the accuracy.(6/13)" title="https://abs.twimg.com/emoji/v2/... draggable="false" alt="🤨" title="Gesicht mit hochgezogener Augenbraue" aria-label="Emoji: Gesicht mit hochgezogener Augenbraue"> Example using API: GPT-3 Cracking the GREI discovered an interesting trade-off between random creativity and reproducible logic when experimenting on the GRE multiple choice sentence completion task. Increasing the temperature (or creativity) decreased the accuracy.(6/13)" class="img-responsive" style="max-width:100%;"/>
I wonder if GPT-3 wouldn& #39;t be easily be good at writing a math book because you& #39;d like the text part of the book to be more creative and the mathematical part to be logical and repeatable. You probably wouldn& #39;t want a math book that was creatively written and then 2+2=5.
(7/13)
(7/13)
Back in the day (a few months ago), you needed to fine-tune a pre-trained model on a task-specific supervised dataset.
Today, you get similar results by simply prepending a few task-specific examples to the prompt during inference using GPT-3.
(8/13)
Zero-shot performance improves steadily with model size.
Few-shot performance increases more rapidly.
Larger models are better at in-context learning.
Graph from paper: https://arxiv.org/pdf/2005.14165.pdf
(9/13)">https://arxiv.org/pdf/2005....
Technical TakeawaysZero-shot performance improves steadily with model size. Few-shot performance increases more rapidly. Larger models are better at in-context learning.Graph from paper: https://arxiv.org/pdf/2005...." title="https://abs.twimg.com/emoji/v2/... draggable="false" alt="🤯" title="Explodierender Kopf" aria-label="Emoji: Explodierender Kopf"> Technical TakeawaysZero-shot performance improves steadily with model size. Few-shot performance increases more rapidly. Larger models are better at in-context learning.Graph from paper: https://arxiv.org/pdf/2005...." class="img-responsive" style="max-width:100%;"/>
@OpenAI will be competing with AI-as-an-API startups, like @rev, and big tech companies with ML solutions, like @googlecloud.
Bigger models need better hardware.
Companies will need to upgrade their ML serving infrastructure for bigger models.
(10/13)
The paper talked about social impact and potential misuse. @openai enabled “Flag Toxicity” filter by default and allowed us to send feedback about “unsafe” content. They’re also working on a semantically-deep toxicity filter built on the API.
(11/13)
GPT - June 2018 release date, 150M parameters, 5GB training set.
GPT-2 - February 2019, 1.5B, 50GB.
GPT-3 - June 2020, 175B, 570GB.
GPT-4 - June 2021, 1.5T, 5.7TB.
GPT-4 predicted by GPT-3.
(12/13)
How deep is the model& #39;s understanding?
How do we optimize the parameters? Random search?
How do we evaluate the model generally and specifically to priming?
If anyone has any ideas, please feel free to reply.
(13/13)
