In June 2020, OpenAI published a paper that would reshape the trajectory of artificial intelligence: "Language Models are Few-Shot Learners." The paper introduced GPT-3, a language model with 175 billion parameters — over 100 times larger than its predecessor GPT-2 — that demonstrated an astonishing ability to perform tasks it had never been explicitly trained for, simply by being given a few examples in its prompt (Brown et al., 2020).
GPT-3 could write coherent essays, generate functional code, translate between languages, answer trivia questions, create poetry, and even perform basic arithmetic — all without task-specific fine-tuning. This capability, called "few-shot learning" or "in-context learning," meant that a single model could function as a general-purpose language tool rather than a specialist system trained for one narrow task. The implications were immediately recognized as transformative.
The model's capabilities emerged as a consequence of scale. Researchers had observed that increasing the size of language models produced smooth, predictable improvements in performance — a finding known as "scaling laws." But GPT-3 demonstrated something more: certain capabilities appeared to emerge at scale that were absent in smaller models, suggesting that bigger models were not just better but qualitatively different. This finding launched the "scaling hypothesis" that would drive massive investment in ever-larger models.
The reaction was electric. Developers who received access to the GPT-3 API created applications that seemed like science fiction: AI-generated blog posts that fooled human readers, code generators that built websites from natural language descriptions, and creative writing tools that produced publishable short fiction. The Guardian published an op-ed written entirely by GPT-3 in September 2020, sparking global debate about AI authorship and the future of writing.
GPT-3 also crystallized concerns that would define the subsequent AI debate. The model cost an estimated $4.6 million to train and required computational resources available to only a handful of organizations, raising questions about the concentration of AI power. It reproduced biases from its training data, generating sexist, racist, and otherwise harmful content. And its confident but sometimes wildly incorrect outputs previewed the "hallucination" problem that would plague its successors. GPT-3 was thus a perfect encapsulation of the bitter-sweet nature of AI: extraordinary capability intertwined with extraordinary risk.
Key Sources
- Brown T.B. et al. (2020). Language Models are Few-Shot Learners. NeurIPS 2020.
- The Guardian (2020). A robot wrote this entire article. Are you scared yet, human?
- Kaplan J. et al. (2020). Scaling Laws for Neural Language Models. arXiv:2001.08361.