Transformer (deep learning)

This article is about the neural network architecture. For other uses, see Transformer (disambiguation).

A standard transformer architecture. Many modern diagrams show the pre-layer normalization (pre-LN) convention, while the original 2017 paper used post-layer normalization (post-LN).

In deep learning, the transformer is a family of artificial neural network architectures built around the attention mechanism. Transformers were introduced to model sequential data without recurrence and without convolutions, allowing much more parallel computation during training. They are now a dominant architecture for natural language processing, computer vision, speech processing, multimodal learning, robotics, and many other sequence-modelling tasks.

Transformers usually begin by converting text or other discrete inputs into numerical tokens, then into vector representations through an embedding table. The model repeatedly mixes information across positions using multi-head attention, then transforms each position independently using a feed-forward network. Positional information is added so that the model can distinguish one token order from another, since plain self-attention alone is permutation-invariant.

The original transformer architecture was proposed in the 2017 paper Attention Is All You Need by Ashish Vaswani and colleagues at Google. The original design used an encoder-decoder structure with post-layer normalization; later work showed that placing layer normalization inside the residual blocks, usually called pre-layer normalization, often improves optimization stability and can reduce or remove the need for learning-rate warm-up. Modern transformer systems also commonly use rotary positional embeddings, flash-attention style kernels, mixture-of-experts layers, and long-context scaling methods.

History

Before transformers, many sequence tasks were dominated by recurrent neural networks and sequence-to-sequence models with attention. These models could achieve strong results, but they processed tokens in a largely serial way and were difficult to scale efficiently. The transformer replaced recurrence with attention-only blocks, making large-scale training more parallelizable.

The transformer soon became the basis for large pretrained language models such as BERT and GPT-style models, and later spread to vision, audio, and multimodal systems.

Core idea

The main idea is simple: each token should be able to look at other tokens and decide which ones matter most. Instead of processing a sequence step by step like an RNN, the transformer computes relationships between all relevant tokens in parallel. This makes it easier to learn long-range dependencies, copying behavior, alignment, retrieval-style reasoning, and context-sensitive interpretations.

A transformer layer generally has four important pieces:

1. token embeddings
2. positional information
3. attention
4. a feed-forward block

These components are connected by residual paths and layer normalization.

Tokenization and embeddings

Before a transformer can process text, the text is broken into tokens. A token may be a word, subword, character, byte pair, or byte-level unit depending on the tokenizer. Each token id is mapped to a vector using an embedding matrix.

In many language models, the input embedding matrix and the output projection matrix are tied, which reduces parameter count and often helps performance. The exact tokenizer is not part of the transformer architecture itself, but it strongly affects model quality, vocabulary size, multilingual behavior, and efficiency.

Positional information

Self-attention by itself does not know token order. For that reason, transformers add position information in some form.

Common approaches include:

• absolute positional embeddings
• sinusoidal positional encodings
• relative positional encodings
• rotary positional embeddings (RoPE)
• ALiBi-style attention bias
• learned position schemes
• multimodal position schemes for images, audio, and video

RoPE has become one of the most widely used modern approaches for language models because it injects position directly into the query and key vectors and naturally supports relative-position behavior.

Self-attention

For each token, the model creates three vectors: a query $Q$, a key $K$, and a value $V$. Attention compares the query against keys, produces weights, and then blends the values accordingly.

The standard scaled dot-product attention is:

$\mathrm {Attention} (Q,K,V)=\mathrm {softmax} \left({\frac {QK^{\top }}{\sqrt {d_{k}}}}+M\right)V$

where:

• $d_{k}$ is the key dimension
• $M$ is a mask, if needed

The mask may be absent in encoder self-attention, or causal in decoder self-attention so that the model cannot see future tokens.

This attention mechanism lets the model propagate information directly between far-apart positions, which is one of the main reasons transformers work so well on long-context tasks.

Multi-head attention

Instead of performing a single attention operation, transformers usually split representation space into multiple heads. Each head can specialize in a different type of relationship, such as syntax, coreference, entity tracking, position, or semantic similarity.

For head $i$:

$\mathrm {head} _{i}=\mathrm {Attention} (QW_{i}^{Q},KW_{i}^{K},VW_{i}^{V})$

The heads are concatenated and projected back into the model dimension:

$\mathrm {MultiHead} (Q,K,V)=\mathrm {Concat} (\mathrm {head} _{1},\dots ,\mathrm {head} _{h})W^{O}$

Multi-head attention is one of the key reasons transformers can represent rich, overlapping patterns in the same layer.

Feed-forward network

After attention, each token goes through a position-wise feed-forward network, often called an MLP block. This block is applied independently to every position, which means it does not mix token positions directly; that job belongs to attention.

A common form is:

$\mathrm {FFN} (x)=W_{2}\sigma (W_{1}x+b_{1})+b_{2}$

where $\sigma$ is often GELU, SwiGLU, or a similar nonlinearity in modern models.

Residual connections and layer normalization

Transformer blocks use residual connections so gradients can flow through deep stacks more easily. Layer normalization stabilizes training.

The original transformer used post-layer normalization, where normalization is applied after the residual addition. Later work showed that pre-layer normalization often makes optimization more stable and can reduce dependence on warm-up schedules.

In practice, many modern large language models use pre-LN or related variants because they are easier to train at scale.

Encoder, decoder, and encoder-decoder variants

Transformers are commonly grouped into three architectural families.

Encoder-only models process the full input bidirectionally and are usually used for classification, retrieval, tagging, and representation learning. BERT is the best-known example.

Decoder-only models use causal masking so each token can attend only to earlier tokens. These models are used for autoregressive generation and are the dominant design for many large language models.

Encoder-decoder models use one stack for encoding the input and another stack for generating the output. They are well suited for translation, summarization, speech-to-text, and other sequence-to-sequence tasks. The original transformer belonged to this family.

Training objective

The training objective depends on the architecture.

Encoder-only models are often trained with masked-token prediction or related denoising objectives. Decoder-only models are usually trained with next-token prediction. Encoder-decoder systems are commonly trained with teacher forcing, where the decoder predicts the next output token conditioned on the encoded source and previously generated target tokens.

For language modelling, the model is optimized to estimate:

$P(x_{1},x_{2},\dots ,x_{n})=\prod _{t=1}^{n}P(x_{t}\mid x_{<t})$

for causal decoders.

Why transformers replaced many older sequence models

Transformers have several practical advantages.

They allow parallel processing during training, which is much faster on modern accelerators than fully sequential recurrence. They usually handle long-range dependencies better than classic RNNs. They scale well with large datasets and large model sizes. They also fit naturally with large-scale pretraining and transfer learning.

This does not mean transformers are perfect. Standard attention has quadratic complexity in sequence length, which becomes expensive for very long contexts. That limitation motivated many later efficiency improvements.

Efficient attention and long-context variants

A major research direction has been to make attention faster and more memory-efficient.

FlashAttention is an exact attention algorithm designed to reduce memory traffic between GPU high-bandwidth memory and on-chip SRAM, making attention much more efficient in practice. FlashAttention-3 further improves speed and accuracy with asynchrony and low-precision execution.

Other approaches include:

• sparse attention
• block-sparse attention
• sliding-window attention
• linear attention
• hybrid attention
• retrieval-augmented context
• attention scaling methods for very long sequences

Recent long-context work has examined how attention behaves as context windows grow, including the need for careful scaling to avoid collapse or overly identity-like behavior at large lengths.

By 2025–2026, surveys of efficient language-model architectures increasingly treat the original transformer as the central baseline while also covering sparse sequence models, efficient full-attention variants, sparse mixture-of-experts, and hybrid architectures.

Mixture of Experts

In a mixture-of-experts (MoE) transformer, only a subset of experts is activated for each token or each example. This can increase parameter count without increasing compute proportionally. MoE has become important in large-scale language models, especially when model capacity must grow faster than inference cost.

Transformer in vision, audio, video, and multimodal systems

Transformers are not limited to text.

Vision transformers split images into patches and treat the patches as tokens. Audio transformers model speech and music. Multimodal transformers combine text with image, audio, video, or document inputs. Modern multimodal large language models often rely on transformer backbones for cross-modal reasoning and fusion.

In video systems, position encoding often needs to represent both space and time, which has led to variants of rotary embeddings and other multimodal position schemes.

Practical implementation sketch

A simple decoder-only transformer for language modelling usually contains the following steps:

1. Tokenize text into ids.
2. Convert ids into embeddings.
3. Add positional information.
4. Pass the sequence through repeated transformer blocks.
5. Use causal self-attention in every block.
6. Apply an MLP after attention.
7. Project the final hidden states to vocabulary logits.
8. Predict the next token.
9. Train with next-token cross-entropy loss.

A minimal block often looks like this:

x = x + Attention(LayerNorm(x))
x = x + MLP(LayerNorm(x))

This is the pre-LN style often used in modern deep models. Other implementations place normalization in different positions, but the overall logic remains similar.

How to build a small transformer model

To build a small transformer from scratch, it is usually enough to decide:

• vocabulary size
• context length
• model width
• number of layers
• number of attention heads
• feed-forward size
• positional encoding type
• dropout
• learning-rate schedule
• optimizer
• generation method

A useful starting point is a decoder-only model with pre-LN, rotary embeddings, causal attention, and a standard next-token objective. That setup is simple enough to implement and close enough to modern language-model practice to be educational.

Common hyperparameters

Typical design choices include:

• model dimension $d_{model}$
• number of layers $L$
• number of heads $h$
• head dimension $d_{k}$
• feed-forward expansion ratio
• context length
• batch size
• warm-up steps
• learning-rate decay
• weight decay
• gradient clipping

These choices interact strongly. For example, deeper models may need more careful initialization and normalization, while longer context lengths may require different positional schemes or attention implementations.

Limitations

Transformers are powerful, but they have limitations.

Standard self-attention has quadratic compute and memory cost with respect to sequence length. They can still struggle with exact long-range retrieval, formal reasoning, and reliable multi-step planning. They are sensitive to data quality and training recipe. They may also hallucinate outputs in generative settings if not properly grounded.

Because of these limits, current research often combines transformers with retrieval, external tools, memory, sparsity, compression, or hybrid sequence models.

2026 perspective

By 2026, transformers remain the central architecture for frontier language and multimodal models, but the design space around them has become much richer. Recent work has focused on long-context scaling, improved attention kernels, sparse and hybrid architectures, better normalization choices, and theoretical understanding of how attention behaves at very large context lengths.

The practical takeaway is that the classic transformer is still the foundation, but many state-of-the-art systems now use modified attention, specialized positional encodings, or mixture-of-experts layers to improve cost, stability, and context handling.

See also

• Attention mechanism
• BERT
• Generative pre-trained transformer
• Vision transformer
• Long short-term memory
• Recurrent neural network
• Mixture of experts
• Rotary position embedding
• FlashAttention

Notes

The original 2017 transformer paper used post-layer normalization, but many modern architectures use pre-layer normalization. The figure caption should make that distinction explicit to avoid confusion.

Further reading

• Vaswani et al., Attention Is All You Need (2017).
• Xiong et al., On Layer Normalization in the Transformer Architecture (2020).
• Su et al., RoFormer: Enhanced Transformer with Rotary Position Embedding (2021).
• Dao et al., FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022).
• Shah et al., FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision (2024).
• Chen et al., Critical attention scaling in long-context transformers (ICLR 2026).

History

See also: Timeline of machine learning

Predecessors

For many years, sequence modelling and generation was done by using plain recurrent neural networks (RNNs). A well-cited early example was the Elman network (1990). In theory, the information from one token can propagate arbitrarily far down the sequence, but in practice the vanishing-gradient problem leaves the model's state at the end of a long sentence without precise, extractable information about preceding tokens.

A key breakthrough was LSTM (originally described in a 1995 technical report and formally published in 1997), an RNN that introduced gating mechanisms to mitigate the vanishing gradient problem, allowing efficient learning of long-sequence modelling. One key architectural element was the use of multiplicative gating units, in which the outputs of some neurons modulate the outputs of others. These multiplicative units are conceptually distinct from the additive attention mechanism later introduced for sequence-to-sequence models. Neural networks using multiplicative units were later called sigma-pi networks or higher-order networks. LSTM became the standard architecture for long sequence modelling until the 2017 publication of transformers. However, LSTM still used sequential processing, like most other RNNs. Specifically, RNNs operate one token at a time from first to last; they cannot operate in parallel over all tokens in a sequence.

Modern transformers overcome this problem, but unlike RNNs, they require computation time that is quadratic in the size of the context window. The linearly scaling fast weight controller (1992) learns to compute a weight matrix for further processing depending on the input. One of its two networks has "fast weights" or "dynamic links" (1981). A slow neural network learns by gradient descent to generate keys and values for computing the weight changes of the fast neural network which computes answers to queries. This was later shown to be equivalent to the unnormalized linear transformer.

Attention with seq2seq

Main article: Seq2seq § History

The idea of encoder–decoder sequence transduction had been developed in the early 2010s; commonly cited as the originators that produced seq2seq are two concurrently published papers from 2014.

A 380M-parameter model for machine translation uses two long short-term memories (LSTM). Its architecture consists of two parts. The encoder is an LSTM that takes in a sequence of tokens and turns it into a vector. The decoder is another LSTM that converts the vector into a sequence of tokens. Similarly, another 130M-parameter model used gated recurrent units (GRU) instead of LSTM. Later research showed that GRUs are neither better nor worse than LSTMs for seq2seq.

These early seq2seq models had no attention mechanism, and the state vector is accessible only after the last word of the source text was processed. Although in theory such a vector retains the information about the whole original sentence, in practice the information is poorly preserved. This is because the input is processed sequentially by one recurrent network into a fixed-size output vector, which is then processed by another recurrent network into an output. If the input is long, then the output vector would not be able to contain all relevant information, degrading the output. As evidence, reversing the input sentence improved seq2seq translation.

The RNN search model introduced an attention mechanism to seq2seq for machine translation to solve the bottleneck problem (of the fixed-size output vector), allowing the model to process long-distance dependencies more easily. The name is because it "emulates searching through a source sentence during decoding a translation".

The relative performances were compared between global (that of RNN search) and local (sliding window) attention model architectures for machine translation, finding that mixed attention had higher quality than global attention, while local attention reduced translation time.

In 2016, Google Translate was revamped to Google Neural Machine Translation, which replaced the previous model based on statistical machine translation. The new model was a seq2seq model where the encoder and the decoder were both 8 layers of bidirectional LSTM. It took nine months to develop, and it outperformed the statistical approach, which took ten years to develop.

Parallelizing attention

Main article: Attention (machine learning) § History

Seq2seq models with attention (including self-attention) still suffered from the same issue with recurrent networks, which is that they are hard to parallelize, which prevented them from being accelerated on GPUs. In 2016, decomposable attention applied a self-attention mechanism to feedforward networks, which are easy to parallelize, and achieved SOTA result in textual entailment with an order of magnitude fewer parameters than LSTMs. One of its authors, Jakob Uszkoreit, suspected that attention without recurrence would be sufficient for language translation, thus the title "attention is all you need". That hypothesis was against conventional wisdom at the time, and even his father Hans Uszkoreit, a well-known computational linguist, was skeptical. In the same year, self-attention (called intra-attention or intra-sentence attention) was proposed for LSTMs.

On 2017-06-12, the original (100M-parameter) encoder–decoder transformer model was published in the "Attention is all you need" paper. At the time, the focus of the research was on improving seq2seq for machine translation, by removing its recurrence to process all tokens in parallel, but preserving its dot-product attention mechanism to keep its text processing performance. This led to the introduction of a multi-head attention model that was easier to parallelize due to the use of independent heads and the lack of recurrence. Its parallelizability was an important factor to its widespread use in large neural networks.

AI boom era

As early as spring 2017, even before the "Attention is all you need" preprint was published, one of the co-authors applied the "decoder-only" variation of the architecture to generate fictitious Wikipedia articles. Transformer architecture is now used alongside many generative models that contribute to the ongoing AI boom.

The "reference implementation" of the original Transformer was written in a TensorFlow library. In language modelling, ELMo (2018) was a bi-directional LSTM that produces contextualized word embeddings, improving upon the line of research from bag of words and word2vec. It was followed by BERT (2018), an encoder-only transformer model. In October 2019, Google started using BERT to process search queries. In 2020, Google Translate replaced the previous RNN-encoder–RNN-decoder model by a transformer-encoder–RNN-decoder model.

Starting in 2018, the OpenAI GPT series of decoder-only transformers became state of the art in natural language generation. At the end of 2022, ChatGPT, a chatbot based on a fine-tuned variant of GPT-3.5, became unexpectedly popular, triggering a boom around large language models.

Transformers have been applied in modalities beyond text. Four days after the publication of "Attention is All You Need", a multimodal transformer architecture, MultiModel, was published by most authors of that paper. Other examples include the vision transformer, speech recognition, robotics, and multimodal. The vision transformer, in turn, stimulated new developments in convolutional neural networks. Image and video generators like DALL-E (2021), Stable Diffusion 3 (2024), and Sora (2024), use transformers to analyse input data (like text prompts) by breaking it down into "tokens" and then calculating the relevance between each token using self-attention, which helps the model understand the context and relationships within the data.

Training

Methods for stabilizing training

The plain transformer architecture had difficulty in converging. In the original paper, the authors recommended using learning rate warmup. That is, the learning rate should linearly scale up from 0 to maximal value for the first part of the training (usually recommended to be 2% of the total number of training steps), before decaying again.

A 2020 paper found that using layer normalization before (instead of after) multihead attention and feedforward layers stabilizes training, not requiring learning rate warmup. This is the "pre-LN Transformer" and is more commonly used, compared to the original "post-LN Transformer".

Pretrain-finetune

Transformers typically are first pretrained by self-supervised learning on a large generic dataset, followed by supervised fine-tuning on a small task-specific dataset. The pretrain dataset is typically an unlabeled large corpus, such as The Pile. Tasks for pretraining and fine-tuning commonly include:

• language modeling
• next-sentence prediction
• question answering
• reading comprehension
• sentiment analysis
• paraphrasing

The T5 transformer report documents a large number of natural language pretraining tasks. Some examples are:

• restoring or repairing incomplete or corrupted text. For example, the input, "Thank you ~~ me to your party ~~ week", might generate the output, "Thank you for inviting me to your party last week".
• translation between natural languages (machine translation)
• judging the pragmatic acceptability of natural language. For example, the following sentence might be judged "not acceptable", because even though it is syntactically well-formed, it is improbable in ordinary human usage: The course is jumping well.

Note that while each of these tasks is trivial or obvious for human native speakers of the language (or languages), they have typically proved challenging for previous generations of machine learning architecture.

Tasks

See also: Large language model § Evaluation

In general, there are 3 classes of language modelling tasks: "masked", "autoregressive", and "prefixLM". These classes are independent of a specific modeling architecture such as transformer, but they are often discussed in the context of transformer.

In a masked task, one or more of the tokens is masked out, and the model would produce a probability distribution predicting what the masked-out tokens are based on the context. The loss function for the task is typically sum of log-perplexities for the masked-out tokens: $${\text{Loss}}=-\sum _{t\in {\text{masked tokens}}}\ln({\text{probability of }}t{\text{ conditional on its context}})$$and the model is trained to minimize this loss function. The BERT series of models are trained for masked token prediction and another task.

In an autoregressive task, the entire sequence is masked at first, and the model produces a probability distribution for the first token. Then the first token is revealed and the model predicts the second token, and so on. The loss function for the task is still typically the same. The GPT series of models are trained by autoregressive tasks.

In a prefixLM task, the sequence is divided into two parts. The first part is presented as context, and the model predicts the first token of the second part. Then that would be revealed, and the model predicts the second token, and so on. The loss function for the task is still typically the same. The T5 series of models are trained by prefixLM tasks.

Note that "masked" as in "masked language modelling" is not "masked" as in "masked attention", and "prefixLM" as in "prefix language modeling" is not "prefixLM" as in " prefix language model".

Architecture

All transformers have the same primary components:

• Tokenizers, which convert text into tokens.
• Embedding layer, which converts tokens and positions of the tokens into vector representations.
• Transformer layers, which carry out repeated transformations on the vector representations, extracting more and more linguistic information. These consist of alternating attention and feedforward layers. There are two major types of transformer layers: encoder layers and decoder layers, with further variants.
• Un-embedding layer, which converts the final vector representations back to a probability distribution over the tokens.

The following description follows exactly the transformer as described in the original paper. There are variants, described in the following section.

By convention, we write all vectors as row vectors. For example, pushing a vector through a linear layer means multiplying it by a weight matrix on the right, as $xW$.

Tokenization

As the transformer architecture natively consists of operations over numbers (matrix multiplications, dot products, activation functions) rather than over text, there must first be a mapping from any input text to some numerical representation. This happens in three steps.

First, the input text is treated by a preprocessor, which performs both textual transformations and splits the text into coarse-grained segments called pretokens. The latter is referred to as pretokenization. Second, each pretoken is segmented further into tokens by a tokenizer that expects to only see pretokens output by its preprocessor. Each token it produces is a string of one or more characters belonging to a finite set of strings called the vocabulary $V$. Third, because the vocabulary is finite and known beforehand, each token can be assigned an integer identifier, and this mapping is applied to the sequence of tokens to represent any input text as a numerical sequence. Since this mapping is bijective, the output side can produce a sequence of integer identifiers which can then be turned back into tokens. After undoing some of the preprocessing, the result is again legible text.

Training a tokenizer (sometimes referred to as vocabularization) means finding a suitable vocabulary $V$, but also learning how to use it, since any given string $s$ of length $|s|$ has $2^{|s|-1}$ hypothetical segmentations, some of which containing segments that are not in the vocabulary. The most important hyperparameter during vocabularization is the vocabulary size $|V|$: when it is small, the learned vocabulary generally consists of characters and smaller strings, and words will be segmented into many tokens. At larger sizes, it becomes affordable to dedicate tokens to full words, although depending on the preprocessor and tokenizer, it is not necessarily the case that large vocabularies will always use the largest token(s) available to segment a word.

Because tokens are not always full words, they may also be referred to as subwords and tokenization algorithms may be referred to as subword tokenizers. This is also to differentiate these systems from traditional terminology used in older information retrieval and natural language processing systems, where "tokenization" was used to denote what is today called "pretokenization" (very crudely: splitting into words). In tokenizers that produce tokens that are not part of the vocabulary, a special token that does belong to the vocabulary is used as a generic stand-in, written as "[UNK]" for "unknown". In principle, any string could be hidden by such an [UNK]. Indeed, in information retrieval, pretokenizers were themselves used as tokenizers (and also called "tokenizers") with a word-level vocabulary that contained an [UNK].

Commonly used subword tokenization algorithms are byte pair encoding (BPE) and the unigram language model (ULM), which each include a vocabularization algorithm and a dedicated segmentation algorithm. There also exist several segmentation algorithms that require no learning and can be applied given a vocabulary (produced by BPE or ULM, for example), like greedily recognising tokens in a pretoken by moving through it left-to-right. Well-known software implementations of subword tokenizers are Hugging Face's tokenizers Python package implemented in Rust, and the sentencepiece Python package implemented in C++. The latter package is named as such because one of its configuration options allows disabling the built-in pretokenizer, hence effectively making entire sentences a pretoken and thus having the tokenizer see entire sentences, rather than individual words.

Embedding

Further information: Word embedding

Each integer token identifier is converted into an embedding vector via a lookup table. Equivalently stated, it multiplies a one-hot representation of the token identifier by an embedding matrix $M$. For example, if the input token's identifier is $3$, then the one-hot representation is $[0,0,0,1,0,0,\dots ]$, and its embedding vector is$$\mathrm {Embed} (3)=[0,0,0,1,0,0,\dots ]M$$The token embedding vectors are added to their respective positional encoding vectors (see below), producing the sequence of input vectors.

The dimension of an embedding vector is called hidden size or embedding size and written as $d_{\text{emb}}$. This size is written as $d_{\text{model}}$ in the original transformer paper.

Un-embedding

An un-embedding layer is almost the reverse of an embedding layer. Whereas an embedding layer converts a token identifier into a vector, an un-embedding layer converts a vector into a probability distribution over tokens.

An illustration of the top 16 token probabilities at temperature 1, for each output token in the chain-of-thought response, with colour representing how that output differs from the same prompt but at temperature 0.

The un-embedding layer is a linear-softmax layer:$$\mathrm {UnEmbed} (x)=\mathrm {softmax} (xW+b)$$The matrix has shape $(d_{\text{emb}},|V|)$. Some architectures use the transpose of the embedding matrix $M$ as the un-embedding matrix $W$ in order to avoid needing double the amount of embedding-related parameters and to avoid divergence during training. This practice is called weight tying.

Positional encoding

Illustration of (absolute) positional encoding with parameters $N=10000,d=100$

A positional encoding is a fixed-size vector representation of the relative positions of tokens within a sequence: it provides the transformer model with information about where the words are in the input sequence. This induces a bias towards the order of the input sequence, so that, for example, the input sequence "man bites dog" is processed differently from "dog bites man".

The positional encoding is defined as a function of type $f:\mathbb {R} \to \mathbb {R} ^{d}$, where $d$ is a positive even integer. The full positional encoding defined in the original paper is:$$(f(t)_{2k},f(t)_{2k+1})=(\sin(\theta ),\cos(\theta ))\quad \forall k\in \{0,1,\ldots ,d/2-1\}$$where $\theta ={\frac {t}{r^{k}}},r=N^{2/d}$.

Here, $N$ is a free parameter that should be significantly larger than the biggest $k$ that would be input into the positional encoding function. The original paper uses $N=10000$.

The function is in a simpler form when written as a complex function of type $f:\mathbb {R} \to \mathbb {C} ^{d/2}$$$f(t)=\left(e^{it/r^{k}}\right)_{k=0,1,\ldots ,{\frac {d}{2}}-1}$$where $r=N^{2/d}$.

The main reason for using this positional encoding function is that using it, shifts are linear transformations:$$f(t+\Delta t)=\mathrm {diag} (f(\Delta t))f(t)$$where $\Delta t\in \mathbb {R}$ is the distance one wishes to shift. This allows the transformer to take any encoded position, and find the encoding of the position n-steps-ahead or n-steps-behind, by a matrix multiplication.

By taking a linear sum, any convolution can also be implemented as linear transformations:$$\sum _{j}c_{j}f(t+\Delta t_{j})=\left(\sum _{j}c_{j}\,\mathrm {diag} (f(\Delta t_{j}))\right)f(t)$$for any constants $c_{j}$. This allows the transformer to take any encoded position and find a linear sum of the encoded locations of its neighbors. This sum of encoded positions, when fed into the attention mechanism, would create attention weights on its neighbors, much like what happens in a convolutional neural network language model. In the author's words, "we hypothesized it would allow the model to easily learn to attend by relative position."

In typical implementations, all operations are done over the real numbers, not the complex numbers, but since complex multiplication can be implemented as real 2-by-2 matrix multiplication, this is a mere notational difference.

Encoder–decoder (overview)

One encoder–decoder block

A transformer is composed of stacked encoder layers and decoder layers.

Like earlier seq2seq models, the original transformer model used an encoder–decoder architecture. The encoder consists of encoding layers that process all the input tokens together one layer after another, while the decoder consists of decoding layers that iteratively process the encoder's output and the decoder's output tokens so far.

The purpose of each encoder layer is to create contextualized representations of the tokens, where each representation corresponds to a token that "mixes" information from other input tokens via self-attention mechanism. Each decoder layer contains two attention sublayers: (1) cross-attention for incorporating the output of encoder (contextualized input token representations), and (2) self-attention for "mixing" information among the input tokens to the decoder (i.e. the tokens generated so far during inference time).

Both the encoder and decoder layers have a feed-forward neural network for additional processing of their outputs and contain residual connections and layer normalization steps. These feed-forward layers contain most of the parameters in a transformer model.

Feedforward network

The feedforward network module. It is a two-layered network that maps $d_{\text{emb}}$-dimensional vectors into $d_{\text{emb}}$-dimensional vectors.

The feedforward network (FFN) modules in a transformer are 2-layered multilayer perceptrons:$$\mathrm {FFN} (x)=\phi (xW^{(1)}+b^{(1)})W^{(2)}+b^{(2)}$$where $W^{(1)}$ and $W^{(2)}$ are weight matrices and $b^{(1)}$ and $b^{(2)}$ are bias vectors, and $\phi$ is its activation function. The original transformer used ReLU activation.

The number of neurons in the middle layer is called intermediate size (GPT), filter size (BERT), or feedforward size (BERT). It is typically larger than the embedding size. For example, in both GPT-2 series and BERT series, the intermediate size of a model is 4 times its embedding size: $d_{\text{ffn}}=4d_{\text{emb}}$.

Scaled dot-product attention

Main article: Dot-product attention

Attention head

Scaled dot-product attention, block diagram

Exact dimension counts within an attention head module

The attention mechanism used in the transformer architecture are scaled dot-product attention units. For each unit, the transformer model learns three weight matrices: the query weights $W^{Q}$, the key weights $W^{K}$, and the value weights $W^{V}$.

The module takes three sequences, a query sequence, a key sequence, and a value sequence. The query sequence is a sequence of length $\ell _{\text{seq, query}}$, and each entry is a vector of dimension $d_{\text{emb, query}}$. Similarly for the key and value sequences.

For each vector $x_{i,{\text{query}}}$ in the query sequence, it is multiplied by a matrix $W^{Q}$ to produce a query vector $q_{i}=x_{i,{\text{query}}}W^{Q}$. The matrix of all query vectors is the query matrix:$$Q=X_{\text{query}}W^{Q}$$Similarly, we construct the key matrix $K=X_{\text{key}}W^{K}$ and the value matrix $V=X_{\text{value}}W^{V}$.

It is usually the case that all $W^{Q},W^{K},W^{V}$ are square matrices, meaning $d_{\text{emb, query}}=d_{\text{query}}$, etc.

Attention weights are calculated using the query and key vectors: the attention weight $a_{ij}$ from token $i$ to token $j$ is the dot product between $q_{i}$ and $k_{j}$. The attention weights are divided by the square root of the dimension of the key vectors, ${\sqrt {d_{k}}}$, which stabilizes gradients during training, and passed through a softmax which normalizes the weights. The fact that $W^{Q}$ and $W^{K}$ are different matrices allows attention to be non-symmetric: if token $i$ attends to token $j$ (i.e. $q_{i}\cdot k_{j}$ is large), this does not necessarily mean that token $j$ will attend to token $i$ (i.e. $q_{j}\cdot k_{i}$ could be small). The output of the attention unit for token $i$ is the weighted sum of the value vectors of all tokens, weighted by $a_{ij}$, the attention from token $i$ to each token.

The attention calculation for all tokens can be expressed as one large matrix calculation using the softmax function, which is useful for training due to computational matrix operation optimizations that quickly compute matrix operations. The matrices $Q$, $K$ and $V$ are defined as the matrices where the $i$th rows are vectors $q_{i}$, $k_{i}$, and $v_{i}$ respectively. Then we can represent the attention as$${\begin{aligned}{\text{Attention}}(Q,K,V)={\text{softmax}}\left({\frac {QK^{\mathrm {T} }}{\sqrt {d_{k}}}}\right)V\end{aligned}}$$

where the softmax is applied over each of the rows of the matrix.

The number of dimensions in a query vector is query size $d_{\text{query}}$ and similarly for the key size $d_{\text{key}}$ and value size $d_{\text{value}}$. The output dimension of an attention head is its head dimension $d_{\text{head}}$. The attention mechanism requires the following three equalities to hold:$$\ell _{\text{seq, key}}=\ell _{\text{seq, value}},\;d_{\text{query}}=d_{\text{key}},\;d_{\text{value}}=d_{\text{head}}$$but is otherwise unconstrained.

If the attention head is used in a self-attention fashion, then $X_{\text{query}}=X_{\text{key}}=X_{\text{value}}$. If the attention head is used in a cross-attention fashion, then usually $X_{\text{query}}\neq X_{\text{key}}=X_{\text{value}}$. It is theoretically possible for all three to be different, but that is rarely the case in practice.

Multihead attention

Multihead attention, block diagram

Exact dimension counts within a multihead attention module

One set of $\left(W^{Q},W^{K},W^{V}\right)$ matrices is called an attention head, and each layer in a transformer model has multiple attention heads. While each attention head attends to the tokens that are relevant to each token, multiple attention heads allow the model to do this for different definitions of "relevance". Specifically, the query and key projection matrices, $W^{Q}$ and $W^{K}$ , which are involved in the attention score computation, defines the "relevance". Meanwhile, the value projection matrix $W^{V}$, in combination with the part of the output projection matrix $W^{O}$, determines how the attended tokens influence what information is passed to subsequent layers and ultimately the output logits. In addition, the scope of attention, or the range of token relationships captured by each attention head, can expand as tokens pass through successive layers. This allows the model to capture more complex and long-range dependencies in deeper layers. Many transformer attention heads encode relevance relations that are meaningful to humans. For example, some attention heads can attend mostly to the next word, while others mainly attend from verbs to their direct objects. The computations for each attention head can be performed in parallel, which allows for fast processing. The outputs for the attention layer are concatenated to pass into the feedforward neural network layers.

Concretely, let the multiple attention heads be indexed by $i$, then we have$${\text{MultiheadAttention}}(Q,K,V)={\text{Concat}}_{i\in [n_{\text{heads}}]}({\text{Attention}}(XW_{i}^{Q},XW_{i}^{K},XW_{i}^{V}))W^{O}$$ where the matrix $X$ is the concatenation of word embeddings, and the matrices $W_{i}^{Q},W_{i}^{K},W_{i}^{V}$ are "projection matrices" owned by individual attention head $i$, and $W^{O}$ is a final projection matrix owned by the whole multihead attention head.

It is theoretically possible for each attention head to have a different head dimension $d_{\text{head}}$, but that is rarely the case in practice.

As an example, in the smallest GPT-2 model, there are only self-attention mechanisms. It has the following dimensions:$$d_{\text{emb}}=768,n_{\text{head}}=12,d_{\text{head}}=64$$Since $12\times 64=768$, its output projection matrix $W^{O}\in \mathbb {R} ^{(12\times 64)\times 768}$ is a square matrix.

Masked attention

The transformer architecture is constructed to calculate output tokens iteratively. Assuming $t=0$ refers to the calculation of the first output token $i=0$, for step $t>0$, the output token $i=0$ shall remain constant. This ensures properties of the model similar to autoregressive models. Therefore, at every time step $t$, the calculation for all outputs $i$ should not have access to tokens at position $j$ for $j>=i$ (as it naturally is the case for time step $t=i$, when tokens $j>t$ are not yet calculated). This behavior may be accomplished before the softmax stage by adding a mask matrix $M$ that is $-\infty$ at entries where the attention link must be cut, and $0$ at other places:$${\begin{aligned}{\text{MaskedAttention}}(Q,K,V)={\text{softmax}}\left(M+{\frac {QK^{\mathrm {T} }}{\sqrt {d_{k}}}}\right)V\end{aligned}}$$ The following matrix is commonly used in decoder self-attention modules, called "causal masking":$$M_{\text{causal}}={\begin{bmatrix}0&-\infty &-\infty &\dots &-\infty \\0&0&-\infty &\dots &-\infty \\0&0&0&\dots &-\infty \\\vdots &\vdots &\vdots &\ddots &\vdots \\0&0&0&\dots &0\end{bmatrix}}$$

In words, it means that each token can pay attention to itself, and every token before it, but not any after it. A non-masked attention module can be thought of as a masked attention module where the mask has all entries zero. As an example of an uncommon use of mask matrix, the XLNet considers all masks of the form $PM_{\text{causal}}P^{-1}$, where $P$ is a random permutation matrix.

Encoder

One encoder layer

An encoder consists of an embedding layer, followed by multiple encoder layers.

Each encoder layer consists of two major components: a self-attention mechanism and a feed-forward layer. It takes an input as a sequence of input vectors, applies the self-attention mechanism, to produce an intermediate sequence of vectors, then applies the feed-forward layer for each vector individually. Schematically, we have:${\begin{aligned}{\text{given input vectors }}&h_{0},h_{1},\dots \\{\text{combine them into a matrix }}H&={\begin{bmatrix}h_{0}\\h_{1}\\\vdots \end{bmatrix}}\\{\text{EncoderLayer}}(H)&={\begin{bmatrix}{\text{FFN}}({\text{MultiheadAttention}}(H,H,H)_{0})\\{\text{FFN}}({\text{MultiheadAttention}}(H,H,H)_{1})\\\vdots \end{bmatrix}}\\\end{aligned}}$

where ${\text{FFN}}$ stands for "feed-forward network". We can more succinctly write it as$${\text{EncoderLayer}}(H)={\text{FFN}}({\text{MultiheadAttention}}(H,H,H))$$with the implicit convention that the ${\text{FFN}}$ is applied to each row of the matrix individually.

The encoder layers are stacked. The first encoder layer takes the sequence of input vectors from the embedding layer, producing a sequence of vectors. This sequence of vectors is processed by the second encoder, and so on. The output from the final encoder layer is then used by the decoder.

As the encoder processes the entire input all at once, every token can attend to every other token (all-to-all attention), so there is no need for causal masking.

Decoder

One decoder layer

A decoder consists of an embedding layer, followed by multiple decoder layers, followed by an un-embedding layer.

Each decoder consists of three major components: a causally masked self-attention mechanism, a cross-attention mechanism, and a feed-forward neural network. The decoder functions in a similar fashion to the encoder, but an additional attention mechanism is inserted which instead draws relevant information from the encodings generated by the encoders. This mechanism can also be called the encoder–decoder attention.

Like the first encoder, the first decoder takes positional information and embeddings of the output sequence as its input, rather than encodings. The transformer must not use the current or future output to predict an output, so the output sequence must be partially masked to prevent this reverse information flow. This allows for autoregressive text generation. For decoding, all-to-all attention is inappropriate, because a token cannot attend to tokens not yet generated. Thus, the self-attention module in the decoder is causally masked.

In contrast, the cross-attention mechanism attends to the output vectors of the encoder, which is computed before the decoder starts decoding. Consequently, there is no need for masking in the cross-attention mechanism.

Schematically, we have:$${\begin{aligned}H'&={\text{MaskedMultiheadAttention}}(H,H,H)\\{\text{DecoderLayer}}(H)&={\text{FFN}}({\text{MultiheadAttention}}(H',H^{E},H^{E}))\end{aligned}}$$where $H^{E}$ is the matrix with rows being the output vectors from the encoder.

The last decoder is followed by a final un-embedding layer to produce the output probabilities over the vocabulary. Then, one of the tokens is sampled according to the probability, and the decoder can be run again to produce the next token, etc., autoregressively generating output text.

Full transformer architecture

Sublayers

(a) One encoder layer and one decoder layer. (b) Two encoder layers and two decoder layers. The sublayers are labelled as well.

Each encoder layer contains 2 sublayers: the self-attention and the feedforward network. Each decoder layer contains 3 sublayers: the causally masked self-attention, the cross-attention, and the feedforward network.

Transformer encoder with norm-first and norm-last

Transformer decoder with norm-first and norm-last

Block diagram for the full transformer architecture

Schematic object hierarchy for the full transformer architecture, in object-oriented programming style

The final points of detail are the residual connections and layer normalization, (denoted as "LayerNorm", or "LN" in the following), which while conceptually unnecessary, are necessary for numerical stability and convergence.

The residual connection, which is introduced to avoid vanishing gradient issues and stabilize the training process, can be expressed as follows: y = F(x) + x. The expression indicates that an output y is the sum of the transformation of input x (F(x)) and the input itself (x). Adding the input x can preserve the input information and avoid issues when the gradient of F(x) is close to zero.

Similarly to how the feedforward network modules are applied individually to each vector, the LayerNorm is also applied individually to each vector.

There are two common conventions in use: the post-LN and the pre-LN convention. In the post-LN convention, the output of each sublayer is $$\mathrm {LayerNorm} (x+\mathrm {Sublayer} (x))$$where $\mathrm {Sublayer} (x)$ is the function implemented by the sublayer itself.

In the pre-LN convention, the output of each sublayer is$$x+\mathrm {Sublayer} (\mathrm {LayerNorm} (x))$$The original 2017 transformer used the post-LN convention. It was difficult to train and required careful hyperparameter tuning and a "warm-up" in learning rate, where it starts small and gradually increases. The pre-LN convention, proposed several times in 2018, was found to be easier to train, requiring no warm-up, leading to faster convergence.

Pseudocode

The following is the pseudocode for a standard pre-LN encoder–decoder transformer, adapted from Formal Algorithms for Transformers

input: Encoder input t_e
       Decoder input t_d
output: Array of probability distributions, with shape (decoder vocabulary size x length(decoder output sequence))

/* encoder */
z_e ← encoder.tokenizer(t_e)

for each t in 1:length(z_e) do
    z_e[t] ← encoder.embedding(z_e[t]) + encoder.positional_embedding(t)

for each l in 1:length(encoder.layers) do
    layer ← encoder.layers[l]

    /* first sublayer */
    z_e_copy ← copy(z_e)
    for each t in 1:length(z_e) do
        z_e[t] ← layer.layer_norm(z_e[t])
    z_e ← layer.multihead_attention(z_e, z_e, z_e)
    for each t in 1:length(z_e) do
        z_e[t] ← z_e[t] + z_e_copy[t]

    /* second sublayer */
    z_e_copy ← copy(z_e)
    for each t in 1:length(z_e) do
        z_e[t] ← layer.layer_norm(z_e[t])
    z_e ← layer.feedforward(z_e)
    for each t in 1:length(z_e) do
        z_e[t] ← z_e[t] + z_e_copy[t]

for each t in 1:length(z_e) do
    z_e[t] ← encoder.final_layer_norm(z_e[t])

/* decoder */
z_d ← decoder.tokenizer(t_d)

for each t in 1:length(z_d) do
    z_d[t] ← decoder.embedding(z_d[t]) + decoder.positional_embedding(t)

for each l in 1:length(decoder.layers) do
        layer ← decoder.layers[l]

        /* first sublayer */
        z_d_copy ← copy(z_d)
        for each t in 1:length(z_d) do
            z_d[t] ← layer.layer_norm(z_d[t])
        z_d ← layer.masked_multihead_attention(z_d, z_d, z_d)
        for each t in 1:length(z_d) do
            z_d[t] ← z_d[t] + z_d_copy[t]

        /* second sublayer */
        z_d_copy ← copy(z_d)
        for each t in 1:length(z_d) do
            z_d[t] ← layer.layer_norm(z_d[t])
        z_d ← layer.multihead_attention(z_d, z_e, z_e) 
       for each t in 1:length(z_d) do
           z_d[t] ← z_d[t] + z_d_copy[t]

        /* third sublayer */
        z_d_copy ← copy(z_d)
        for each t in 1:length(z_d) do
            z_d[t] ← layer.layer_norm(z_d[t])
        z_d ← layer.feedforward(z_d)
        for each t in 1:length(z_d) do
            z_d[t] ← z_d[t] + z_d_copy[t]

z_d ← decoder.final_layer_norm(z_d)

output_distributions ← []
for each t in 1:length(z_d) do
    output_distributions.append(decoder.unembed(z_d[t]))

return output_distributions

Terminology

The transformer architecture, being modular, allows variations. Several common variations are described here.

An "encoder-only" transformer applies the encoder to map an input text into a sequence of vectors that represent the input text. This is usually used for text embedding and representation learning for downstream applications. BERT is encoder-only. They are less often used currently, as they were found to be not significantly better than training an encoder–decoder transformer, then taking just the encoder. They are also referred to as "all-to-all" or "BERT-like".

A "decoder-only" transformer is not literally decoder-only, since without an encoder, the cross-attention mechanism has nothing to attend to. Thus, the decoder layers in a decoder-only transformer is composed of just two sublayers: the causally masked self-attention, and the feedforward network. This is usually used for text generation and instruction following. The models in the GPT series and Chinchilla series are decoder-only. They are also referred to as "autoregressive" or "causal".

An "encoder–decoder" transformer is generally the same as the original transformer, with 2 sublayers per encoder layer and 3 sublayers per decoder layer, etc. They might have minor architectural improvements, such as alternative activation functions, changing the location of normalization, etc. This is also usually used for text generation and instruction following. The models in the T5 series are encoder–decoder.

A "prefixLM" (prefix language model) is a decoder-only architecture, but with prefix masking, which is different from causal masking. Specifically, it has mask of the form$$M_{\text{prefixLM}}={\begin{bmatrix}\mathbf {0} &-\infty \\\mathbf {0} &M_{\text{causal}}\end{bmatrix}}$$where the first columns correspond to the "prefix", and the subsequent columns correspond to the autoregressively generated text based on the prefix. They resemble encoder–decoder models, but has less "sparsity". Such models are rarely used, though they are cited as theoretical possibilities and benchmarked comparisons.

There are also mixed seq2seq models. For example, in 2020, Google Translate replaced the previous RNN-encoder–RNN-decoder model with a transformer-encoder–RNN-decoder model, as transformer-based decoders did not appear to significantly increase quality unlike the encoder, while the RNN decoder was much faster.

Subsequent work

Alternative activation functions

The original transformer uses ReLU activation function. Other activation functions were developed. The Llama series and PaLM used SwiGLU; both GPT-1 and BERT used GELU.

Alternative activation functions are often used in combination with Gated Linear Units in the feedforward module.

Alternative normalizations

The normalization used in the transformer can be different from LayerNorm. One example is RMSNorm which is used in the Llama series. Other examples include ScaleNorm and FixNorm.

Alternative positional encodings

Transformers may use other positional encoding methods than sinusoidal.

The original transformer paper reported using a learned positional encoding, but finding it not superior to the sinusoidal one. Later, found that causal masking itself provides enough signal to a transformer decoder that it can learn to implicitly perform absolute positional encoding without the positional encoding module.

RoPE

RoPE (rotary positional embedding), is best explained by considering a list of 2-dimensional vectors $[(x_{1}^{(1)},x_{1}^{(2)}),(x_{2}^{(1)},x_{2}^{(2)}),(x_{3}^{(1)},x_{3}^{(2)}),...]$. Now pick some angle $\theta$. Then RoPE encoding is$${\text{RoPE}}{\big (}x_{m}^{(1)},x_{m}^{(2)},m{\big )}={\begin{pmatrix}\cos m\theta &-\sin m\theta \\\sin m\theta &\cos m\theta \end{pmatrix}}{\begin{pmatrix}x_{m}^{(1)}\\x_{m}^{(2)}\\\end{pmatrix}}={\begin{pmatrix}x_{m}^{(1)}\cos m\theta -x_{m}^{(2)}\sin m\theta \\x_{m}^{(2)}\cos m\theta +x_{m}^{(1)}\sin m\theta \\\end{pmatrix}}$$Equivalently, if we write the 2-dimensional vectors as complex numbers $z_{m}:=x_{m}^{(1)}+ix_{m}^{(2)}$, then RoPE encoding is just multiplication by an angle:$${\text{RoPE}}{\big (}z_{m},m{\big )}=e^{im\theta }z_{m}$$For a list of $2n$-dimensional vectors, a RoPE encoder is defined by a sequence of angles $\theta ^{(1)},...,\theta ^{(n)}$. Then the RoPE encoding is applied to each pair of coordinates.

The benefit of RoPE is that the dot-product between two vectors depends on their relative location only:$${\text{RoPE}}{\big (}x,m{\big )}^{T}{\text{RoPE}}{\big (}y,n{\big )}={\text{RoPE}}{\big (}x,m+k{\big )}^{T}{\text{RoPE}}{\big (}y,n+k{\big )}$$ for any integer $k$.

ALiBi

ALiBi (Attention with Linear Biases) is not a replacement for the positional encoder on the original transformer. Instead, it is an additional positional encoder that is directly plugged into the attention mechanism. Specifically, the ALiBi attention mechanism is$${\begin{aligned}{\text{Attention}}(Q,K,V)={\text{softmax}}\left({\frac {QK^{\mathrm {T} }}{\sqrt {d_{k}}}}+sB\right)V\end{aligned}}$$Here, $s$ is a real number ("scalar"), and $B$ is the linear bias matrix defined by$$B={\begin{pmatrix}0&1&2&3&\cdots \\-1&0&1&2&\cdots \\-2&-1&0&1&\cdots \\-3&-2&-1&0&\cdots \\\vdots &\vdots &\vdots &\vdots &\ddots \\\end{pmatrix}}$$in other words, $B_{i,j}=j-i$. The idea being that the linear bias matrix is a softened mask. Just as $0$ represent full attention paid, and $-\infty$ represents no attention paid, the linear bias matrix increases attention paid in one direction and decreases attention paid in the other direction.

ALiBi allows pretraining on short context windows, then fine-tuning on longer context windows. Since it is directly plugged into the attention mechanism, it can be combined with any positional encoder that is plugged into the "bottom" of the entire network (which is where the sinusoidal encoder on the original transformer, as well as RoPE and many others, are located).

Relative Position Encodings

Relative Position Encodings is similar to ALiBi, but more generic:$${\begin{aligned}{\text{Attention}}(Q,K,V)={\text{softmax}}\left({\frac {QK^{\mathrm {T} }}{\sqrt {d_{k}}}}+B\right)V\end{aligned}}$$where $B$ is a Toeplitz matrix, that is, $B_{i,j}=B_{i',j'}$ whenever $i-j=i'-j'$. This is contrasted with the original sinusoidal positional encoding, which is an "absolute positional encoding".

Efficient implementation

The transformer model has been implemented in standard deep learning frameworks such as TensorFlow and PyTorch. Transformers is a library produced by Hugging Face that supplies transformer-based architectures and pretrained models.

KV caching

When an autoregressive transformer is used for inference, such as generating text, the query vector is different at each step, but the already-computed key and value vectors are always the same. The KV caching method saves the computed key and value vectors at each attention block, so that they are not recomputed at each new token. PagedAttention applies memory paging to KV caching.

If a transformer is used with a baked-in prompt, such as ["You are a customer support agent..."], then the key and value vectors can be computed for the prompt, and saved on disk. The saving in compute is significant when the model is used for many short real-time interactions, such as in online chatbots.

In general, when a user uses an autoregressive transformer to generate a continuation to a sequence of tokens, the model would first perform a forward-pass on this sequence, whereby the KV caches over this sequence are computed. This is called prefilling. Hyperscalers serving large Transformer models may use disaggregated inference, wherein prefilling and decoding are performed on separately specialized hardware.

FlashAttention

FlashAttention is an algorithm that implements the transformer attention mechanism efficiently on a GPU. It is a communication-avoiding algorithm that performs matrix multiplications in blocks, such that each block fits within the cache of a GPU, and by careful management of the blocks it minimizes data copying between GPU caches (as data movement is slow). See the page on softmax for details.

An improved version, FlashAttention-2, was developed to cater to the rising demand for language models capable of handling longer context lengths. It offers enhancements in work partitioning and parallelism, enabling it to achieve up to 230 TFLOPs/s on A100 GPUs (FP16/BF16), a 2x speed increase over the original FlashAttention.

Key advancements in FlashAttention-2 include the reduction of non-matmul FLOPs, improved parallelism over the sequence length dimension, better work partitioning between GPU warps, and added support for head dimensions up to 256 and multi-query attention (MQA) and grouped-query attention (GQA).

Benchmarks revealed FlashAttention-2 to be up to 2x faster than FlashAttention and up to 9x faster than a standard attention implementation in PyTorch. Future developments include optimization for new hardware like H100 GPUs and new data types like FP8.

FlashAttention-4 focuses on pipelining to increase instruction throughput, and was developed to perform particularly well on Blackwell GPUs.

Multi-Query Attention

Comparison between several different forms of attention mechanism and the amount of KV caching necessary for each

Multi-Query Attention changes the Multihead Attention mechanism. Whereas normally,

$${\text{MultiheadAttention}}(Q,K,V)={\text{Concat}}_{i\in [n_{\text{heads}}]}\left({\text{Attention}}(XW_{i}^{Q},XW_{i}^{K},XW_{i}^{V})\right)W^{O}$$with Multi-Query Attention, there is just one $W^{K},W^{V}$, thus:

$${\text{MultiQueryAttention}}(Q,K,V)={\text{Concat}}_{i\in [n_{\text{heads}}]}\left({\text{Attention}}(XW_{i}^{Q},XW^{K},XW^{V})\right)W^{O}$$

This has a neutral effect on model quality and training speed, but increases inference speed.

More generally, grouped-query attention (GQA) partitions attention heads into groups, each of which shares the key-value pair. MQA is GQA with one group, while standard Multihead Attention is GQA with the maximal number of groups.

The architecture of V2, showing both MLA and a variant of mixture of experts

Multihead Latent Attention (MLA) is a low-rank approximation to standard MHA. Specifically, each hidden vector, before entering the attention mechanism, is first projected to two low-dimensional spaces ("latent space"), one for query and one for key-value (KV vector). This design minimizes the KV cache, as only the low-dimensional KV vector needs to be cached.

Speculative decoding

Main article: Speculative decoding

Speculative decoding is a method to accelerate token decoding. Similarly to speculative execution in CPUs, future tokens are computed quickly, then verified. If the quickly computed tokens are incorrect, they are discarded and computed slowly.

The key factor in speculative decoding is that a transformer decoder can verify faster than it can decode, in the following sense.

Suppose we have two transformer models like GPT-3 and GPT-3-small, both with a context window size of 512. To generate an entire context window autoregressively with greedy decoding with GPT-3, it must be run for 512 times, each time generating a token $x_{1},x_{2},...,x_{512}$, taking time $512T_{\text{GPT-3}}$. However, if we had some educated guess for the values of these tokens, we could verify all of them in parallel, in one run of the model, by checking that each $x_{t}$ is indeed the token with the largest log-likelihood in the $t$-th output.

In speculative decoding, a smaller model or some other simple heuristic is used to generate a few speculative tokens that are subsequently verified by the larger model. For example, suppose we use GPT-3-small to generate four speculative tokens: ${\tilde {x}}_{1},{\tilde {x}}_{2},{\tilde {x}}_{3},{\tilde {x}}_{4}$. This only takes $4T_{\text{GPT-3-small}}$. These tokens are then run through the larger GPT-3 in one go. Suppose that ${\tilde {x}}_{1}$ and ${\tilde {x}}_{2}$ are verified by GPT-3 as what it would have picked, then those are kept, but ${\tilde {x}}_{3}$ is not, so ${\tilde {x}}_{3},{\tilde {x}}_{4}$ are discarded, and GPT-3 is run on those. This would take $4T_{\text{GPT-3-small}}+3T_{\text{GPT-3}}$, which might be shorter than $4T_{\text{GPT-3}}$.

For non-greedy decoding, similar ideas apply, except the speculative tokens are accepted or rejected stochastically, in a way that guarantees the final output distribution is the same as if speculative decoding was not used.

Multi-token prediction

In Multi-Token Prediction, a single forward pass creates a final embedding vector, which then is un-embedded into a token probability. However, that vector can then be further processed by another transformer block to predict the next token, and so on for arbitrarily many steps into the future. This trades off accuracy for speed, since each new token costs just one more transformer block, rather than the entire stack.

Sub-quadratic transformers

Training transformer-based architectures can be expensive, especially for long inputs. Many methods have been developed to attempt to address the issue. In the image domain, Swin transformer is an efficient architecture that performs attention inside shifting windows. In the audio domain, SepTr decouples the attention in time and frequency domains. Long Range Arena (2020) is a standard benchmark for comparing the behavior of transformer architectures over long inputs.

Alternative attention graphs

The standard attention graph is either all-to-all or causal, both of which scales as $O(N^{2})$ where $N$ is the number of tokens in a sequence.

Reformer (2020) reduces the computational load from $O(N^{2})$ to $O(N\ln N)$ by using locality-sensitive hashing and reversible layers.

Sparse attention uses attention graphs that grows slower than $O(N^{2})$. For example, BigBird (2020) uses random small-world networks which grows as $O(N)$.

Ordinary transformers require a memory size that is quadratic in the size of the context window. Attention-free transformers reduce this to a linear dependence while still retaining the advantages of a transformer by linking the key to the value.

Random Feature Attention

Random Feature Attention (2021) uses Fourier random features:$$\varphi (x)={\frac {1}{\sqrt {D}}}[\cos \langle w_{1},x\rangle ,\sin \langle w_{1},x\rangle ,\cdots \cos \langle w_{D},x\rangle ,\sin \langle w_{D},x\rangle ]^{T}$$where $w_{1},...,w_{D}$ are independent samples from the normal distribution $N(0,\sigma ^{2}I)$. This choice of parameters satisfy $\mathbb {E} [\langle \varphi (x),\varphi (y)\rangle ]=e^{-{\frac {\|x-y\|^{2}}{2\sigma ^{2}}}}$, or $$e^{\langle x,y\rangle /\sigma ^{2}}=\mathbb {E} [\langle e^{\|x\|^{2}/2\sigma ^{2}}\varphi (x),e^{\|y\|^{2}/2\sigma ^{2}}\varphi (y)\rangle ]\approx \langle e^{\|x\|^{2}/2\sigma ^{2}}\varphi (x),e^{\|y\|^{2}/2\sigma ^{2}}\varphi (y)\rangle$$Consequently, the one-headed attention, with one query, can be written as $${\text{Attention}}(q,K,V)={\text{softmax}}\left({\frac {qK^{\mathrm {T} }}{\sqrt {d_{k}}}}\right)V\approx {\frac {\varphi (q)^{T}\sum _{i}e^{\|k_{i}\|^{2}/2\sigma ^{2}}\varphi (k_{i})v_{i}^{T}}{\varphi (q)^{T}\sum _{i}e^{\|k_{i}\|^{2}/2\sigma ^{2}}\varphi (k_{i})}}$$where $\sigma =d_{K}^{1/4}$. Similarly for multiple queries, and for multihead attention.

This approximation can be computed in linear time, as we can compute the matrix $\varphi (k_{i})v_{i}^{T}$ first, then multiply it with the query. In essence, we have managed to obtain a more precise version of $${\text{Attention}}(Q,K,V)={\text{softmax}}\left({\frac {QK^{\mathrm {T} }}{\sqrt {d_{k}}}}\right)V\approx Q(K^{T}V/{\sqrt {d_{k}}})$$Performer (2022) uses the same Random Feature Attention, but $w_{1},...,w_{D}$ are first independently sampled from the normal distribution $N(0,\sigma ^{2}I)$, then they are Gram–Schmidt processed.

Multimodality

Transformers can also be used/adapted for modalities (input or output) beyond just text, usually by finding a way to "tokenize" the modality.

Multimodal models can either be trained from scratch, or by finetuning. A 2022 study found that transformers pretrained only on natural language can be finetuned on only 0.03% of parameters and become competitive with LSTMs on a variety of logical and visual tasks, demonstrating transfer learning. The LLaVA was a vision-language model composed of a language model (Vicuna-13B) and a vision model (ViT-L/14), connected by a linear layer. Only the linear layer is finetuned.

Vision transformers adapt the transformer to computer vision by breaking down input images as a series of patches, turning them into vectors, and treating them like embedding vector of tokens in a standard transformer.

Conformer and later Whisper follow the same pattern for speech recognition, first turning the speech signal into a spectrogram, which is then treated like an image, i.e. broken down into a series of patches, turned into vectors and treated like embedding vector of tokens in a standard transformer.

Perceivers are a variant of transformers designed for multimodality.

For image generation, notable architectures are DALL-E 1 (2021), Parti (2022), Phenaki (2023), and Muse (2023). Unlike later models, DALL-E is not a diffusion model. Instead, it uses a decoder-only transformer that autoregressively generates a text, followed by the token representation of an image, which is then converted by a variational autoencoder to an image. Parti is an encoder–decoder transformer, where the encoder processes a text prompt, and the decoder generates a token representation of an image. Muse is an encoder-only transformer that is trained to predict masked image tokens from unmasked image tokens. During generation, all input tokens are masked, and the highest-confidence predictions are included for the next iteration, until all tokens are predicted. Phenaki is a text-to-video model. It is a bidirectional masked transformer conditioned on pre-computed text tokens. The generated tokens are then decoded to a video.

Applications

The transformer has had great success in natural language processing (NLP). Many large language models such as GPT-2, GPT-3, GPT-4, Gemini, AlbertAGPT, Claude, BERT, Grok, XLNet, RoBERTa and ChatGPT demonstrate the ability of transformers to perform a wide variety of NLP-related subtasks and their related real-world applications, including:

• machine translation
• time series prediction
• document summarization
• document generation
• named entity recognition (NER)
• writing computer code based on requirements expressed in natural language.
• speech-to-text

Beyond traditional NLP, the transformer architecture has had success in other applications, such as:

• Disaster response
• biological sequence analysis
• video understanding
• protein folding (such as AlphaFold)
• evaluating chess board positions. Using static evaluation alone (that is, with no Minimax search) transformer achieved an Elo of 2895, putting it at grandmaster level.

See also

• seq2seq – Family of machine learning approaches
• Circuit (neural network) – Interpretable computational sub-graphs within artificial neural networks
• Perceiver – Variant of Transformer designed for multimodal data
• Vision transformer – Machine learning model for vision processing
• Large language model – Type of machine learning model
• BERT (language model) – Series of language models developed by Google AI
• Generative pre-trained transformer – Type of large language model
• T5 (language model) – Series of large language models developed by Google AI

Further reading

• Alexander Rush, The Annotated transformer Archived 2021-09-22 at the Wayback Machine, Harvard NLP group, 3 April 2018
• Phuong, Mary; Hutter, Marcus (2022). "Formal Algorithms for Transformers". arXiv:2207.09238 [cs.LG].
• Ferrando, Javier; Sarti, Gabriele; Bisazza, Arianna; Costa-jussà, Marta R. (2024-05-01). "A Primer on the Inner Workings of Transformer-based Language Models". arXiv:2405.00208 [cs.CL].
• Leech, Gavin (2024-11-06). "Transformer++". argmin gravitas. Archived from the original on 2025-02-26. Retrieved 2025-05-08.
• US patent 10452978, Noam M. Shazeer; Aidan Nicholas Gomez; Lukasz Mieczyslaw Kaiser; Jakob D. Uszkoreit; Llion Owen Jones; Niki J. Parmar; Illia Polosukhin; Ashish Teku Vaswani, "Attention-based sequence transduction neural networks", issued 2019-10-22, assigned to Google LLC 
• Raschka, Sebastian (2026-03-11). "The Big LLM Architecture Comparison: From DeepSeek V3 to GLM-5: A Look At Modern LLM Architecture Design". Sebastian Raschka’s AI Magazine. Retrieved 2026-03-25.