Language Models are Unsupervised Multitask Learners
Paper: Language Models are Unsupervised Multitask Learners
Authors: Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever
Year / Venue: 2019
The Main Idea
Most NLP tasks before 2019 were solved with supervised learning, where a model is trained on a large dataset of labeled examples for a specific task. This paper introduces GPT-2, a large-scale unsupervised language model that demonstrates strong performance on a variety of natural language processing tasks, without task-specific training data.
Their argument: If a language model becomes large enough, then while learning to predict text it will be able to solve most NLP tasks without the need for supervised learning or task specific architectures.
How It Works
The authors arugued that NLP tasks can be solved with building a much larger model trained on a large corpus of text. Their intuition was that ordinary web text naturally contains many kinds of tasks: - question → answer - English → French translation - article → summary - document → question answering - conversations - explanations
The training objective remained simple; predict the next token: \[ P(x)=\prod_t P(x_t \mid x_{<t}) \]
What’s New
Previously, each task required a specialized network architecture. GPT-2 helped shift this paradigm by showing that a single model could perform different tasks without architectural changes—the task could instead be specified through natural-language instructions. For example, rather than building a separate model with a fixed task format:
Task: Translate
Input: Hello
Output: Bonjour
The same model could be prompted directly:
translate to french: Hello
What the Results Show
Summarize the main results and what they tell us. Include a useful comparison and any important limitations.
Comments