ChatGPT

arXiv link
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions – something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.
arXiv链接
最近的工作表明，通过对大量文本语料库进行预训练，然后对特定任务进行微调，在许多NLP任务和基准测试上取得了实质性进展。虽然在架构中通常与任务无关，但这种方法仍然需要特定于任务的数千或数万个示例的微调数据集。相比之下，人类通常只能从几个例子或简单的指令中执行新的语言任务——这是当前NLP系统仍然难以做到的。在这里，我们表明，扩展语言模型可以大大提高与任务无关的少镜头性能，有时甚至可以通过先前最先进的微调方法达到竞争力。具体来说，我们训练 GPT-3，一种具有 1750 亿个参数的自回归语言模型，比之前任何非稀疏语言模型多 10 倍，并在少数镜头设置中测试其性能。对于所有任务，GPT-3 在没有任何梯度更新或微调的情况下应用，任务和少数镜头演示纯粹通过与模型的文本交互来指定。GPT-3 在许多 NLP 数据集上实现了强大的性能，包括翻译、问答和完形填空任务，以及一些需要即时推理或领域适应的任务，例如解密单词、在句子中使用新单词或执行 3 位数算术。同时，我们还确定了 GPT-3 的少数镜头学习仍在挣扎的一些数据集，以及 GPT-3 面临与大型网络语料库训练相关的方法问题的一些数据集。最后，我们发现 GPT-3 可以生成人类评估人员难以区分的新闻文章样本与人类撰写的文章。我们讨论了这一发现和一般 GPT-3 更广泛的社会影响。
Contents
175b_samples.jsonl - Unconditional, unfiltered 2048 token samples from GPT-3 with p=.85, t=1.　 CONTENT WARNING: GPT-3 was trained on arbitrary data from the web, so may contain offensive content and language.
data - Synthetic datasets for word scramble and arithmetic tasks described in the paper.
dataset_statistics - Statistics for all languages included in the training dataset mix.
overlap_frequency.md - Samples of 13-gram overlaps between our training data and benchmarks, selected by frequency in the training set.
model-card.md - GPT-3 Model Card.

内容
175b_samples.jsonl - 来自 GPT-3 的无条件、未过滤的 2048 个令牌样本，p=.85，t=1。内容警告：GPT-3 是根据来自网络的任意数据进行训练的，因此可能包含令人反感的内容和语言。
data - 论文中描述的单词加扰和算术任务的合成数据集。
dataset_statistics - 训练数据集组合中包含的所有语言的统计信息。
overlap_frequency.md - 我们的训练数据和基准之间 13 克重叠的样本，按训练集中的频率选择。

How to cite
@article{brown2020language,

   title={Language Models are Few-Shot Learners},

   author={Tom B. Brown and Benjamin Mann and Nick Ryder and Melanie Subbiah and Jared Kaplan and Prafulla Dhariwal and Arvind Neelakantan and Pranav Shyam and Girish Sastry and Amanda Askell and Sandhini Agarwal and Ariel Herbert-Voss and Gretchen Krueger and Tom Henighan and Rewon Child and Aditya Ramesh and Daniel M. Ziegler and Jeffrey Wu and Clemens Winter and Christopher Hesse and Mark Chen and Eric Sigler and Mateusz Litwin and Scott Gray and Benjamin Chess and Jack Clark and Christopher Berner and Sam McCandlish and Alec Radford and Ilya Sutskever and Dario Amodei},

   year={2020},

   eprint={2005.14165},

   archivePrefix={arXiv},

   primaryClass={cs.CL}

}