Generativer vortrainierter Transformer

The original GPT model

Generative pre-trained transformers (GPT) are a family of large language models (LLMs) and a prominent framework for generative artificial intelligence. The concept and first such model were introduced in 2018 by the American artificial intelligence organization OpenAI. GPT models are artificial neural networks that are based on the transformer architecture, pre-trained on large datasets of unlabelled text, and able to generate novel human-like text. As of 2023, most LLMs have these characteristics and are sometimes referred to broadly as GPTs.

Between 2018 and 2023, OpenAI released four major numbered GPT foundational models, with each being significantly more capable than the previous due to increased size (number of trainable parameters) and training. The most recent of these is GPT-4 (2023), for which OpenAI declined to publish the size or training details, citing "the competitive landscape and the safety implications of large-scale models". These "GPT-n" models have been the basis for various other products and technologies, including models fine-tuned for instruction followingVorlage:Mdashwhich in turn power the ChatGPT chatbot service.

The term "GPT" is also used in the names of some generative LLMs developed by others, such as a series of GPT-3 inspired models created by EleutherAI, and recently a series of seven models created by Cerebras. Major companies in other industries (e.g. sales, finance) also use the term "GPT" in the names of their services involving or utilizing a GPT technology, like "EinsteinGPT" and "BloombergGPT".

History

Generative pre-training (GP) was a long-established concept in machine learning applications, but the transformer architecture was not available until 2017 when it was invented by Google. That development led to the emergence of large language models like BERT in 2018 and XLNet in 2019, which were pre-trained transformers (PT) but not designed to be generative (they were "encoder-only"). Also around that time, in 2018, OpenAI published its article entitled "Improving Language Understanding by Generative Pre-Training," in which it introduced the first generative pretrained transformer (GPT) system.

Prior to transformer-based architectures, the best-performing neural NLP (natural language processing) models commonly employed supervised learning from large amounts of manually-labeled data. The reliance on supervised learning limited their use on datasets that were not well-annotated, and also made it prohibitively expensive and time-consuming to train extremely large language models.

The semi-supervised approach OpenAI employed to make a large scale generative systemVorlage:Mdashand was first to do with a transformer modelVorlage:Mdashinvolved two stages: an unsupervised generative "pre-training" stage to set initial parameters using a language modeling objective, and a supervised discriminative "fine-tuning" stage to adapt these parameters to a target task.

Foundational models

A foundational model is an AI model trained on broad data at scale such that it can be adapted to a wide range of downstream tasks. Thus far the most notable GPT foundation models are from OpenAI's numbered "GPT-n" series, the most recent of which is GPT-4.

Notable GPT foundation models

Model	Architecture	Parameter count	Training data	Release date
Original GPT (GPT-1)	12-level, 12-headed Transformer decoder (no encoder), followed by linear-softmax.	117 million	BookCorpus: 4.5 GB of text, from 7000 unpublished books of various genres.	Vorlage:Date table sorting
GPT-2	GPT-1, but with modified normalization	1.5 billion	WebText: 40 GB of text, 8 million documents, from 45 million webpages upvoted on Reddit.	Vorlage:Date table sorting (initial/limited version) and Vorlage:Date table sorting (full version)
GPT-3	GPT-2, but with modification to allow larger scaling	175 billion	570 GB plaintext, 0.4 trillion tokens. Mostly CommonCrawl, WebText, English Wikipedia, and two books corpora (Books1 and Books2).	Vorlage:Date table sorting (then March 15, 2022, for a revision ultimately termed GPT-3.5)
GPT-4	Also trained with both text prediction and RLHF; accepts both text and images as input. Further details are not public.	Undisclosed	Undisclosed	Vorlage:Date table sorting

Related models and products

Foundational GPT models can be used to produce other related systems via additional fine-tuning to further target a particular task and/or domain.

In January 2022, OpenAI introduced "InstructGPT"Vorlage:Mdasha series of models which were fine-tuned to follow instructions using a combination of supervised training and reinforcement learning from human feedback (RLHF) on base GPT-3 language models. Advantages this had over the bare foundational models included higher accuracy, less negative/toxic sentiment, and generally better alignment with user needs. Hence, OpenAI began using this as the basis for its API service offerings.

In November 2022, OpenAI launched ChatGPTVorlage:Mdashan online chat interface powered by an instruction-tuned language model trained in a similar fashion to InstructGPT. They trained this model using RLHF, with human AI trainers providing conversations in which they played both the user and the AI. They mixed this new dialogue dataset with the InstructGPT dataset, for a conversational format suitable for a chatbot.