1. Introduction

Update (2026-10-28): This revision expands the optimizers and learning rate strategies in §5, and updates FP8 / FlashAttention-4 in §9 and reasoning reinforcement learning and hybrid attention architectures in §11. Years generally refer to first public release; conference publication years are noted separately. Performance claims are limited to the experimental settings of the original papers. Primary sources for the additions are listed at the end.

Deep learning is a machine learning approach built on multi-layer neural networks. AlexNet’s breakthrough in the 2012 ImageNet Challenge accelerated its adoption in computer vision, natural language processing, speech recognition and scientific computing. Representative systems such as GPT, BERT, Stable Diffusion and AlphaFold are also built on deep neural networks.

The power of deep learning lies in its ability to learn multi-level abstract feature representations automatically from raw data (pixels, words, signals) without manual feature engineering. However, training a high-performance deep network is not easy: problems such as vanishing gradients, overfitting and learning-rate tuning have long troubled researchers, and they gave rise to a body of systematic training techniques.

Deep learning is represented by network learning data; CNN, RNN and Transformer are optional architectures, and the training goals guide parameter updates.
Figure: deep learning learns representations from data and completes tasks. CNN, RNN, and Transformer are optional and composable architectures; the target and parameter updates below are part of the training process.

To understand deep learning, you need to grasp two levels at once: architecture design (how to build the network) and training methods (how to optimize it effectively). Both are indispensable. If you only know how to stack layers like building blocks without understanding what problem each training technique solves, you will often end up in the dilemma of “adding Dropout only makes things worse” or “swapping in a new optimizer changes nothing”.

This article systematically sorts out the core principles and key techniques of deep learning, providing a searchable map for learning, reproduction and research. The full text is organized as follows:

  • §2–3 Basics: the three-step machine learning framework, neurons and backpropagation
  • §4 Architectures: the main line of evolution from MLP → CNN → RNN/LSTM → Transformer, supplemented by graph neural networks (GNNs)
  • §5 Training optimization: from AdamW to Muon / SOAP / NorMuon; learning-rate schedules and Schedule-Free; initialization, normalization and residual connections
  • §6 Loss and regularization: choosing a loss, Dropout, data augmentation, AdamW
  • §7 Method taxonomy: mapping all techniques onto the “three steps × two goals” coordinate system
  • §8 Benchmarks: a review of classic benchmarks such as MNIST/CIFAR/ImageNet/GLUE/MMLU/HumanEval/long-context
  • §9 Efficient training and inference: BF16 / FP8, gradient clipping and checkpointing, FlashAttention 1–4
  • §10 Model compression and adaptation: MoE, LoRA/PEFT, quantization, knowledge distillation and other lightweight, deployment-oriented methods
  • §11 Frontier architectures and large models: RLHF / GRPO / RLVR, generative models (AE/VAE/GAN), diffusion models, Mamba/SSMs and hybrid attention

Reading suggestions: Beginners can read §2–§6 in order; when training models, prioritize §5–§7 and §9; for large models, focus on §4.3, §10 and §11. The accuracies, parameter counts and years in the article illustrate historical context and depend on model version, data processing and evaluation protocol. When reproducing an experiment, defer to the original paper or official report.


2. Overview of deep learning basics

2.1 What is deep learning?

Deep learning is a branch of machine learning that uses artificial neural networks (ANNs) as its base model. “Deep” usually means the model contains multiple layers of nonlinear transformations; there is no fixed layer threshold that applies to all tasks. Stacking many layers lets the network extract increasingly abstract features layer by layer.

The basic computing unit of a neural network is the neuron:

\[a = f\left(\sum_i w_i x_i + b\right)\]

Here $x_i$ is the input, $w_i$ the weight, $b$ the bias and $f$ the activation function. Neurons are stacked into layers, and layers are chained into a deep network that ultimately forms a complex mapping from input to output.

Schematic diagram of multi-layer neural network
Schematic of a multi-layer neural network: input layer (left) → hidden layers (middle) → output layer (right)

Universal approximation theorem

One of the important theoretical foundations for the expressive power of neural networks is the universal approximation theorem (Cybenko 1989, Hornik 1991):

Under conditions such as continuity and a non-polynomial activation function, a single-hidden-layer feedforward network with sufficient width can approximate any continuous function on a compact set to arbitrary accuracy.

This result answers the fundamental question “why can neural networks work?”, but it comes with two key limitations:

Limitation Meaning
Only guarantees existence The theorem only says that “a set of parameters exists that achieves the approximation”; it does not guarantee that SGD can find them.
Width may be huge A single hidden layer may need a very large number of neurons to reach the target accuracy; for some functions, adding depth is more efficient than simply widening
Guarantees fitting, not generalization A perfect fit on the training set does not mean good performance on the validation set; this is where the generalization problem comes from

The advantage of depth: Theoretically (Telgarsky 2016), there are classes of functions that deep networks can express with far fewer units than shallow ones. Note that the theorem describes expressive power and parameter efficiency; it does not guarantee that optimization will find those parameters, nor does it directly guarantee generalization.

Why deep learning took off

Factor Description Representative event
Algorithmic breakthroughs The ReLU activation alleviates vanishing gradients; residual connections make hundred-layer networks trainable AlexNet 2012, ResNet 2015
Data explosion The Internet generates massive amounts of labeled data; the more data, the better deep learning performs ImageNet: 1.2 million images
Compute revolution Parallel GPU computing cuts training time from weeks to hours NVIDIA GPU + CUDA

2.2 Machine learning three-step framework

Any machine learning method can be broken down into three steps. Understanding this framework is the basis for all the training techniques that follow:

  1. Step 1 - Define the loss function: Measure the difference between the model output and the correct answer, i.e. “how to judge good from bad”
  2. Step 2 - Choose the function search space (model architecture): Select the network structure and delimit the set of candidate functions, i.e. “where to search”
  3. Step 3 - Optimization: Find the function in the search space that minimizes the loss, i.e. “how to search efficiently”

The ultimate goal of training is to find a function whose loss is low on the training set and stays low on the validation set. Whether the training error can be reduced mainly reflects optimization, while the gap between training and validation error reflects generalization; in practice the two interact.

2.3 Two core goals

Deep learning training techniques fall into two categories according to the problem they solve:

Goal Symptom Meaning Typical methods
Optimization Training loss will not go down Problem in the optimization process Adam, skip connections, batch norm
Generalization Training loss is low, validation loss is high Overfitting Dropout, data augmentation, regularization

Rule of thumb: First look at the training loss, then at the gap between the training and validation curves. When the training loss will not go down, check the learning rate, initialization, normalization and the data pipeline first; when the training loss is already low but the validation loss is clearly higher, consider regularization and data augmentation. Real problems may involve both kinds of factors.

2.4 Development timeline

flowchart LR
    subgraph G1990 ["1990s: laying the foundation"]
        A["LeNet 1998<br/>CNN foundations"] --> B["Backpropagation<br/>becomes widespread"]
    end
    subgraph G2012 ["2012-2014: breakthrough"]
        C["AlexNet 2012<br/>ImageNet breakthrough"] --> D["Dropout 2014"]
        D --> E["Adam 2014"]
    end
    subgraph G2015 ["2015-2017: going deeper"]
        F["ResNet 2015<br/>Skip connections"] --> G["BN 2015"]
        G --> H["AdamW 2017"]
    end
    subgraph G2018 ["2018-2022: pre-training"]
        I["BERT/GPT 2018<br/>Pre-training paradigm"] --> J["Transformer<br/>takes over"]
        J --> R["RoPE 2021<br/>Rotary position encoding"]
        R --> K["LoRA 2021<br/>Parameter-efficient fine-tuning"]
    end
    subgraph G2022 ["2022 to date: large models"]
        L["Chinchilla<br/>2022 scaling laws"] --> M["ChatGPT 2022<br/>RLHF alignment"]
        M --> N["FlashAttention 2/3<br/>2023-2024"]
        N --> O["Muon / SOAP<br/>2024 matrix optimizers"]
        O --> P["GRPO / reasoning RL<br/>2024-2025"]
        P --> Q["FlashAttention-4<br/>2026 hardware co-design"]
    end
    B --> C
    E --> F
    H --> I
    K --> L

2.5 Main abbreviations

  • ANN: Artificial Neural Network
  • MLP: Multi-Layer Perceptron
  • CNN: Convolutional Neural Network
  • RNN: Recurrent Neural Network
  • LSTM: Long Short-Term Memory
  • SGD: Stochastic Gradient Descent
  • BN: Batch Normalization
  • LN: Layer Normalization
  • LR: Learning Rate
  • PEFT: Parameter-Efficient Fine-Tuning
  • SFT: Supervised Fine-Tuning
  • RLHF: Reinforcement Learning from Human Feedback

3. Neural network basics

3.1 Multi-layer perceptron

The multi-layer perceptron (MLP) is the most basic form of neural network. It consists of an input layer, several hidden layers and an output layer, with each layer fully connected:

\[h^{(l)} = f\left(W^{(l)} h^{(l-1)} + b^{(l)}\right)\]

Fully connected means each neuron in layer $l$ connects to every neuron in layer $l-1$, so the parameter count is the product of the two layers’ sizes. When the input dimension is extremely large (e.g. a 1000×1000 color image has 3 million dimensions once flattened), the number of fully connected parameters exceeds billions, which makes training hard and overfitting extremely easy. Specialized architectures such as CNNs were proposed to solve this problem.

3.2 Activation function

Activation functions introduce nonlinearity into neural networks so they can fit complex functions. Without nonlinear activations, a multi-layer network is equivalent to a single-layer linear model.

Activation function Formula Output range Advantages Disadvantages Typical use
Sigmoid $\frac{1}{1+e^{-x}}$ (0, 1) Output can be interpreted as a probability Vanishing gradient; not zero-centered Output layer for binary classification
Tanh $\tanh(x)$ (-1, 1) Zero-centered output Still suffers from vanishing gradients RNN hidden layers (earlier)
ReLU $\max(0, x)$ [0, ∞) Cheap to compute; effectively alleviates vanishing gradients Dead neuron problem CNN/MLP hidden layers (the most common choice today)
Leaky ReLU $\max(0.01x, x)$ (-∞, ∞) Fixes dead neurons Slope needs tuning Drop-in improvement over ReLU
GELU $x \cdot \Phi(x)$ (-∞, ∞) Smooth; excellent performance Slightly more expensive to compute Standard in Transformers (BERT, GPT)
SiLU/Swish $x \cdot \sigma(x)$ (-∞, ∞) Self-gated; similar to GELU — Large language models such as LLaMA and Qwen
Sigmoid activation function curve Tanh activation function curve ReLU activation function curve
Left: Sigmoid | Middle: Tanh | Right: ReLU (currently the most commonly used)

ReLU (Rectified Linear Unit) succeeds because its gradient in the positive region is always 1, which effectively alleviates the vanishing-gradient problem of deep networks and allows stable training of networks with dozens or even hundreds of layers.

3.3 Backpropagation algorithm

Backpropagation is the core algorithm for training neural networks. Based on the chain rule, it passes the gradient of the loss with respect to each parameter from the output layer back to the input layer, layer by layer:

\[\frac{\partial \mathcal{L}}{\partial W^{(l)}} = \frac{\partial \mathcal{L}}{\partial h^{(l)}} \cdot \frac{\partial h^{(l)}}{\partial W^{(l)}}\]
flowchart LR
    Loss["Loss<br/>Compute the loss"] -->|"∂L/∂ŷ"| OutLayer["Output layer<br/>Compute gradient"]
    OutLayer -->|"∂L/∂h²"| H2["Hidden layer 2<br/>Compute gradient"]
    H2 -->|"∂L/∂h¹"| H1["Hidden layer 1<br/>Compute gradient"]
    H1 -->|"∂L/∂W¹"| Update["Parameter update<br/>Gradient descent"]
    style Loss fill:#ff9999
    style Update fill:#99ff99

The computation graph automatically records the forward computation path, and during backpropagation the gradient flows back along that path in the opposite direction. PyTorch’s autograd is built on this principle: users only define the forward computation, and the framework computes the gradients automatically.

A tiny backprop example

Consider the simplest single-hidden-layer network $y = w_2 \cdot \sigma(w_1 \cdot x)$ with loss $\mathcal{L} = \frac{1}{2}(y - t)^2$. Let $x=1, w_1=0.5, w_2=0.8, t=0.5$, and let the activation $\sigma$ be the sigmoid.

Forward pass:

\[z_1 = w_1 x = 0.5, \quad a_1 = \sigma(0.5) \approx 0.622, \quad y = w_2 a_1 \approx 0.498, \quad \mathcal{L} \approx 2\times 10^{-6}\]

Backward pass (chain rule, step by step):

\[\frac{\partial \mathcal{L}}{\partial y} = y - t \approx -0.002\] \[\frac{\partial \mathcal{L}}{\partial w_2} = \frac{\partial \mathcal{L}}{\partial y} \cdot a_1 \approx -0.00124\] \[\frac{\partial \mathcal{L}}{\partial a_1} = \frac{\partial \mathcal{L}}{\partial y} \cdot w_2 \approx -0.0016\] \[\frac{\partial \mathcal{L}}{\partial z_1} = \frac{\partial \mathcal{L}}{\partial a_1} \cdot a_1(1-a_1) \approx -0.000376\] \[\frac{\partial \mathcal{L}}{\partial w_1} = \frac{\partial \mathcal{L}}{\partial z_1} \cdot x \approx -0.000376\]

Observation: The gradient $\partial \mathcal{L}/\partial w_2 \approx 10^{-3}$ near the output and $\partial \mathcal{L}/\partial w_1 \approx 10^{-4}$ near the input. Even with only 2 layers, the gradient is attenuated by roughly $\sim 3$ times. Every additional layer in a deep network adds another multiplication by $\sigma’(\cdot) \leq 0.25$. This is the source of vanishing gradients.

Three variations of gradient descent

Gradient descent comes in three forms, depending on the number of samples used per update:

Variant Samples per update Advantages Disadvantages
Batch GD (full batch) The entire training set Accurate gradient and stable convergence path Each iteration is very expensive and cannot exploit big data
SGD (stochastic) 1 Minimal memory use; gradient noise helps escape saddle points Severe convergence oscillation and low hardware utilization
Mini-batch SGD $B$ (usually 32–4096) Balances efficiency and stability; compatible with GPU parallelism Batch size must be tuned as a hyperparameter

Modern deep learning almost always uses mini-batch SGD. When practitioners say “SGD” they usually mean the mini-batch version. Batch size directly affects training stability, hardware utilization and final generalization: large batches converge quickly but generalize slightly worse (Keskar et al., 2017, “On Large-Batch Training”), making it an important hyperparameter to tune.


4. Classic network architecture

4.1 Convolutional Neural Network

Convolutional neural networks (CNNs) are the standard architecture for grid-structured data (images, time-series signals). A CNN imposes two key constraints on an MLP:

Local receptive fields: Each neuron observes only a local region of the input (such as a 3×3 kernel) rather than the whole image. Local patterns in images (edges, textures) can be detected from local information alone, without a global view.

Parameter sharing: Neurons at different positions share the same set of parameters (a filter / convolution kernel). The same pattern (such as a horizontal edge) can appear anywhere in the image and should be handled by the same detector. This constraint gives translation equivariance; only after pooling or global aggregation does the model acquire some translation invariance.

Convolution hyperparameters

Hyperparameter Meaning Typical values Effect
Kernel size $k$ Spatial size of the convolution kernel 3×3 (mainstream), 5×5, 7×7 Determines the receptive field; 3×3 has been standard since VGG
Stride $s$ Step size of the convolution sliding window 1 (regular), 2 (downsampling) A stride above 1 shrinks the spatial size and can replace pooling
Padding $p$ Number of pixels padded at the edges “same” / “valid” “same” preserves the output size; “valid” does no padding
Dilation $d$ Dilation rate 1 (regular), 2, 4 Enlarges the receptive field without adding parameters (DeepLab semantic segmentation)
Channels $C$ Number of output feature maps 64/128/256/512 The number of channels in a layer determines feature diversity

Output size formula:

\[\text{out} = \left\lfloor \frac{\text{in} + 2p - d(k-1) - 1}{s} \right\rfloor + 1\]

Regular convolution ($d=1$) simplifies to $\text{out} = \lfloor (\text{in} + 2p - k)/s \rfloor + 1$.

Special convolution

Type Core idea Representative models Purpose
1×1 convolution Pointwise linear combination across channels NiN, ResNet bottleneck Changes the channel count (reducing or expanding dimensions) and adds nonlinearity
Depthwise separable Depthwise (per-channel) convolution + 1×1 pointwise convolution MobileNet, EfficientNet Reduces parameters and computation to about $\sim 1/k^2$
Dilated / atrous The kernel has “holes” DeepLab Enlarges the receptive field without adding parameters; suited to dense prediction
Transposed conv Inverse of convolution, used for upsampling FCN, GAN generator Upsampling for segmentation and generation tasks
Grouped conv Channels are split into groups that are convolved independently AlexNet, ResNeXt Reduces computation and increases feature diversity

Pooling layer (Pooling)

Pooling downsamples the feature map, reducing its spatial size, enlarging the receptive field and providing some translation invariance. Pooling layers have no learnable parameters.

Method Operation Use case
Max pooling Takes the maximum within the window Extracts the most salient features; the most common choice in the middle of a CNN
Average pooling Takes the mean within the window Smooths features; common in early networks
Global average pooling (GAP) Takes the average over the whole feature map Replaces fully connected layers, greatly reducing parameters (at the end of GoogLeNet and ResNet)
Adaptive pooling Specify the output size and the kernel is computed automatically Adapts to different input resolutions

Why GAP matters: The fully connected layers of VGG-16 hold 123M parameters (89% of the total); this part can simply be dropped after switching to GAP. This is one of the key reasons ResNet and EfficientNet have far fewer parameters than VGG.

A typical CNN alternates convolutional layers (feature extraction) and pooling layers (downsampling), followed by fully connected layers or GAP to produce the prediction. Below is the structure of LeNet-5 (LeCun, 1998), the foundational CNN work:

LeNet-5 network architecture diagram
LeNet-5 architecture (LeCun et al., 1998): convolutional layer → pooling layer → convolutional layer → pooling layer → fully connected layer

CNN architecture evolution comparison

Model Year Parameters ImageNet Top-1 Key innovation
LeNet-5 1998 ~60K — (MNIST) The foundation of CNNs: convolution + pooling structure
AlexNet 2012 60M ~56.5% GPU training, ReLU, Dropout
VGG-16 2014 138M ~71.5% Uniform 3×3 convolutions, deeper network
GoogLeNet 2014 6.8M ~69.8% Inception module, greatly reduced parameters
ResNet-50 2015 25M ~76.0% Residual connections make very deep networks trainable
ResNet-152 2015 60M ~77.8% The classic paper reports a Top-5 error of about 3.57%
SENet-154 2017 ~115M ~82.7% Channel attention (squeeze-and-excitation)
EfficientNet-B0 2019 5.3M 77.1% Compound scaling (NAS finds the optimal ratio)
EfficientNet-B7 2019 66M 84.4% Best parameter efficiency
ViT-B/16 2020 86M ~81.8% Pure Transformer for images
ConvNeXt-XL 2022 350M ~87.8% A CNN that absorbs Transformer design ideas

Key insight: ResNet-50 (25M parameters) outperforms VGG-16 (138M parameters) with only 18% of its parameters; EfficientNet-B0 (5.3M) reaches VGG-level accuracy with only 4% of the parameters. More parameters do not mean higher performance; architectural design is what matters.

ResNet residual block structure

Schematic diagram of ResNet residual block structure
ResNet residual block (He et al., 2016): output = F(x) + x, identity mapping allows the gradient to flow directly back to the shallow layer

4.2 Recurrent Neural Network and LSTM

Recurrent neural networks (RNNs) are designed for sequential data and pass information between time steps through the hidden state $h_t$:

\[h_t = f(W_h h_{t-1} + W_x x_t + b)\]

An RNN uses the same parameters $W_h, W_x$ at every time step, which makes it naturally suited to variable-length sequences. The figure below shows the RNN “unrolled” along the time dimension:

flowchart LR
    x0["x₀"] --> h0["h₀"]
    x1["x₁"] --> h1["h₁"]
    x2["x₂"] --> h2["h₂"]
    x3["x₃"] --> h3["h₃"]
    h0 -->|"Wₕ"| h1
    h1 -->|"Wₕ"| h2
    h2 -->|"Wₕ"| h3
    h0 --> y0["y₀"]
    h1 --> y1["y₁"]
    h2 --> y2["y₂"]
    h3 --> y3["y₃"]
    style h0 fill:#e1f5ff
    style h1 fill:#e1f5ff
    style h2 fill:#e1f5ff
    style h3 fill:#e1f5ff

However, the standard RNN suffers from a serious long-range dependency problem: when the gradient is backpropagated through time, the matrix $W_h$ is multiplied $T$ times. When the largest singular value of $W_h$ is $< 1$, the gradient decays exponentially with sequence length (vanishing gradients); when it is $> 1$, the gradient explodes exponentially (exploding gradients), making it hard for the model to remember distant context.

LSTM (Long Short-Term Memory, Hochreiter & Schmidhuber, 1997) solves this problem by introducing a gating mechanism:

Schematic diagram of LSTM chain structure
LSTM chain structure (Colah, 2015): Cell state (top horizontal line) runs through the entire sequence, carrying long-term memory

An LSTM maintains two states: the hidden state $h_t$ (short-term memory) and the cell state $c_t$ (long-term memory).

Internal data flow of a single LSTM cell:

flowchart LR
    subgraph INPUT ["Input"]
        XT["xₜ"]
        HT1["hₜ₋₁"]
        CT1["cₜ₋₁"]
    end
    CONCAT["Concatenate [hₜ₋₁, xₜ]"]
    XT --> CONCAT
    HT1 --> CONCAT
    CONCAT --> FGATE["Forget gate fₜ<br/>σ(W·)"]
    CONCAT --> IGATE["Input gate iₜ<br/>σ(W·)"]
    CONCAT --> CAND["Candidate c̃ₜ<br/>tanh(W·)"]
    CONCAT --> OGATE["output gate oₜ<br/>σ(W·)"]
    CT1 --> MUL1["× fₜ<br/>Forget old memory"]
    FGATE --> MUL1
    IGATE --> MUL2["× c̃ₜ<br/>Write new memory"]
    CAND --> MUL2
    MUL1 --> ADD["+"]
    MUL2 --> ADD
    ADD --> CT["cₜ<br/>New cell state"]
    CT --> TANH["tanh"]
    TANH --> MUL3["× oₜ"]
    OGATE --> MUL3
    MUL3 --> HT["hₜ<br/>New hidden state"]
    style CT fill:#ffe1c4
    style HT fill:#c4e1ff
    style CT1 fill:#ffe1c4
    style HT1 fill:#c4e1ff

Cell state update:

\[c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_t\]
Component Formula Function
Forget gate $f_t$ $\sigma(W_f [h_{t-1}, x_t] + b_f)$ Decides how much of the historical cell state to forget
Input gate $i_t$ $\sigma(W_i [h_{t-1}, x_t] + b_i)$ Decides how much new information to write
Candidate value \(\tilde{c}_t\) $\tanh(W_c [h_{t-1}, x_t] + b_c)$ Computes the new information to be written
Output gate $o_t$ $\sigma(W_o [h_{t-1}, x_t] + b_o)$ Decides how much of the cell state to expose as output

Why does gating alleviate vanishing gradients? The cell-state update $c_t = f_t \odot c_{t-1} + \ldots$ is an additive path (compare the RNN’s multiplicative recursion $h_t = f(Wh_{t-1} + \ldots)$). When the forget gate $f_t \approx 1$, $c_{t-1}$ passes to $c_t$ almost without attenuation, so the gradient stays stable when backpropagated along the cell state. This resembles ResNet’s skip connection, except that it acts along time rather than depth.

GRU: a simplified version of LSTM

The Gated Recurrent Unit (GRU, Cho et al., 2014) simplifies the LSTM’s 3 gates into 2 and merges the cell state with the hidden state:

\[r_t = \sigma(W_r [h_{t-1}, x_t]) \quad \text{(reset gate)}\] \[z_t = \sigma(W_z [h_{t-1}, x_t]) \quad \text{(update gate)}\] \[\tilde{h}_t = \tanh(W_h [r_t \odot h_{t-1}, x_t])\] \[h_t = (1-z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t\]
Dimension LSTM GRU
Number of gates 3 (f, i, o) 2 (r, z)
Number of states 2 (h, c) 1 (h)
Parameters (same hidden size) 4 weight sets 3 weight sets
Convergence speed Slower Faster
Long-range dependency modeling Stronger (protected cell state) Slightly weaker but sufficient
Typical use Long sequences, complex tasks Short sequences, limited resources

Empirical conclusion: The systematic experiments of Chung et al. (2014) show that GRU and LSTM perform comparably on most tasks. GRU has fewer parameters and trains faster, and is more common on short and medium-length sequence tasks (dialogue, sentiment classification); LSTM retains an advantage on tasks that demand strict long-term memory (machine translation of long sentences, speech recognition).

Bidirectional RNN

A standard RNN can only use past information (left to right). But for many tasks (named entity recognition, sentiment analysis, reading comprehension) future context is just as important. For example, deciding whether “bank” means a financial institution or a river bank requires looking at the words that follow.

A bidirectional RNN (Schuster & Paliwal, 1997) runs two RNNs in opposite directions at once and concatenates their outputs:

\[\overrightarrow{h}_t = \text{RNN}_\text{fwd}(x_t, \overrightarrow{h}_{t-1}), \quad \overleftarrow{h}_t = \text{RNN}_\text{bwd}(x_t, \overleftarrow{h}_{t+1})\] \[h_t = [\overrightarrow{h}_t; \overleftarrow{h}_t]\]
flowchart LR
    x1["x₁"] --> fh1["→h₁"]
    x2["x₂"] --> fh2["→h₂"]
    x3["x₃"] --> fh3["→h₃"]
    fh1 --> fh2 --> fh3
    x1 --> bh1["←h₁"]
    x2 --> bh2["←h₂"]
    x3 --> bh3["←h₃"]
    bh3 --> bh2 --> bh1
    fh1 --> c1["[→;←] h₁"]
    bh1 --> c1
    fh2 --> c2["[→;←] h₂"]
    bh2 --> c2
    fh3 --> c3["[→;←] h₃"]
    bh3 --> c3
    style c1 fill:#c4e1ff
    style c2 fill:#c4e1ff
    style c3 fill:#c4e1ff

Limitations: A bidirectional RNN must see the entire sequence before it can compute, so it cannot be used for autoregressive generation (there is no future information when generating the next word). Therefore:

  • Suitable for: sequence labeling, classification, NER and reading comprehension (BiLSTM-CRF was the mainstream NER solution in 2016–2018)
  • Not suitable for: the decoder side of language models and machine translation

BERT’s MLM (masked language model) can be seen as the Transformer’s generalization of “bidirectional modeling”, using masking to ensure that bidirectional attention does not leak the target.

Teacher forcing

Training a Seq2Seq / RNN decoder involves a subtle choice: what is the input at decoding step $t$?

Strategy Input at step $t$ Advantages Disadvantages
Teacher forcing Ground-truth label $y_{t-1}$ Stable training and fast convergence Mismatch between training and inference distributions (inference uses the model’s own predictions)
Free running Model prediction \(\hat{y}_{t-1}\) Fully consistent with inference Early in training the predictions are poor, errors cascade, and convergence is difficult
Scheduled sampling Sample the ground truth or the prediction with probability $p$ (Bengio et al., 2015) A compromise with a gradual switch Requires tuning the schedule hyperparameters

The problem with teacher forcing is exposure bias: during training the model never “sees” its own wrong predictions. Once a prediction goes astray at some step during inference, it drifts further and further off (the prototype of error accumulation / hallucination).

Modern remedies:

  • Scheduled sampling: Gradually mix in the model’s own predictions later in training
  • Minimum risk training / REINFORCE: Use the overall quality of the generated sequence directly as the training signal
  • Transformer + causal masking: Compute all positions in parallel (see §4.3). This is still essentially teacher forcing, but far more efficient than the RNN’s step-by-step recursion.

Representative work: GRU (2014, a simplified LSTM), Seq2Seq (2014), Attention + LSTM (2015), bidirectional LSTM (proposed in 2005, with BiLSTM-CRF widely used from 2016)


4.3 Transformer

The Transformer (Vaswani et al., 2017) completely changed NLP and the wider deep learning landscape. It abandons recurrence and convolution entirely and models the global dependency between any two positions in a sequence directly through the self-attention mechanism.

Self-Attention architecture diagram
Self-attention architecture: after linear projection of the query, key and value, attention weights are computed with scaled dot-product and used to form a weighted sum as the output

Core: scaled dot-product attention

\[\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V\]
  • Query (Q): represents “what am I looking for” (such as the current word).
  • Key (K): represents “what features do I have” (used to match Q).
  • Value (V): represents “the information I will pass on” (the content delivered after a successful match).
  • $\sqrt{d_k}$ scaling: When the dimension is large, dot products can become extremely large and push the softmax into a region with tiny gradients. Scaling keeps the gradients stable.

A concrete example: the sentence “The cat sat on the mat”

Suppose the model is processing the verb “sat” and needs to find “who sat” (the subject). The weights below are for illustration only and are not the output of a real model:

Token Q (what am I looking for?) K (what am I?) V (what can I offer?)
The Determiner context Determiner No substantive information
cat — Noun, singular, animate Semantic vector for “cat”
sat Looking for a singular noun subject that can perform an action Past-tense verb Semantics of the action “sit”
on Looking for a prepositional object Preposition Positional relation
the Determiner context Determiner No substantive information
mat Noun, singular, object Noun, singular, object Semantic vector for “mat”
  • The Q of “sat” and the K of “cat” have the largest dot product (matching “singular noun subject”), and the attention weight after softmax is $\approx 0.7$
  • The Q of “sat” and the K of “mat” also match to some degree (both are nouns), but the semantic role is wrong, so the weight is only $\approx 0.15$
  • The final output for “sat” is $\approx 0.7 \cdot V_\text{cat} + 0.15 \cdot V_\text{mat} + \ldots$, i.e. it mainly absorbs the semantics of “cat”

This is how attention works: each token broadcasts a “query”, all tokens respond with “how well do I match”, and the information is finally aggregated with weights given by the degree of match. Q/K/V are produced from the same input by three different linear projections $W_Q, W_K, W_V$, and this flexibility lets the model learn a wide variety of information-routing patterns.

Multi-head attention

The input is mapped into $h$ different subspaces, attention is computed in each separately, and the results are concatenated: \(\text{MultiHead}(Q,K,V) = \text{Concat}(head_1, \ldots, head_h) W^O\) Intuition: Different “heads” can attend to different information at the same time (for example, one head to syntactic relations and another to semantic relations).

Schematic diagram of the multi-head attention mechanism (Multi-Head Attention) architecture
Architecture of multi-head attention. Left: scaled dot-product attention; right: multi-head attention

Attention variants

As context lengths grow, the GPU memory used by the KV cache (which stores K and V for all previous tokens during inference) becomes a bottleneck. Multi-head attention has therefore evolved into several variants whose core idea is to reduce the number of K/V heads to shrink the KV cache:

Variant Year K/V heads KV cache vs. MHA Representative models
MHA (Multi-Head Attention) 2017 Each Q head has its own K/V ($h$ groups in total) 100% Original Transformer, GPT-2
MQA (Multi-Query Attention) 2019 All Q heads share 1 group of K/V $1/h$ PaLM, Falcon
GQA (Grouped-Query Attention) 2023 Q heads are grouped, and K/V are shared within each group ($g$ groups in total, $1 < g < h$) $g/h$ LLaMA-2/3, Mixtral, Qwen
MLA (Multi-head Latent Attention) 2024 K/V are compressed into a low-rank latent space $\sim 5-13\%$ DeepSeek-V2/V3
flowchart LR
    subgraph MHA ["MHA: independent KV for every Q head"]
        Q1["Q₁"] --> KV1["K₁,V₁"]
        Q2["Q₂"] --> KV2["K₂,V₂"]
        Q3["Q₃"] --> KV3["K₃,V₃"]
        Q4["Q₄"] --> KV4["K₄,V₄"]
    end
    subgraph GQA ["GQA: groups share KV (g=2)"]
        Q1G["Q₁"] --> KVA["K,V group A"]
        Q2G["Q₂"] --> KVA
        Q3G["Q₃"] --> KVB["K,V group B"]
        Q4G["Q₄"] --> KVB
    end
    subgraph MQA ["MQA: all Q heads share one KV group"]
        Q1M["Q₁"] --> KVS["Shared K,V"]
        Q2M["Q₂"] --> KVS
        Q3M["Q₃"] --> KVS
        Q4M["Q₄"] --> KVS
    end

Why GQA is widely used: MQA compresses K/V into a single group, which may cost quality, while MHA has a large KV cache. GQA is a compromise between quality and GPU memory. For example, LLaMA-2 70B uses 8 K/V groups with 64 Q heads, so its KV cache is about 1/8 that of MHA; actual gains still depend on the model and task.

MLA’s innovation: DeepSeek uses a low-rank matrix to compress K/V into a latent space of about $\sim$128 dimensions. Only the latent vector is cached during inference, and the full K/V is reconstructed when needed. This brings a significant GPU-memory advantage for extremely long contexts.

Key components

1. Positional encoding

Why is positional encoding needed? The core of the Transformer, the self-attention mechanism, is permutation equivariant when there is no positional encoding: if the order of the input tokens changes, the output is rearranged in the same way. When computing $O = \sum \alpha_i V_i$, the model has no extra cue about order, so it cannot reliably distinguish the sentences “you hit me” and “I hit you”; position information must be injected.

Schematic diagram of absolute position encoding
Absolute positional encoding

Absolute positional encoding: the sinusoidal design The original Transformer builds position vectors from predefined sine and cosine functions: \(PE_{(pos, 2i)} = \sin\left(\frac{pos}{10000^{2i/d_{model}}}\right)\) \(PE_{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{2i/d_{model}}}\right)\)

Schematic diagram of position encoding method
Positional encoding methods
Position encoding visualization
Visualization of positional encoding: different dimensions correspond to sine and cosine waves of different periods, forming a unique positional fingerprint
Position coding multi-frequency pointer diagram (1)
Multi-frequency hands (1): low dimensions correspond to high-frequency hands (the "second hand"), which complete one revolution every few tokens
Position coding multi-frequency pointer indication (2)
Multi-frequency hands (2): high dimensions correspond to low-frequency hands (the "hour hand"); together they form a precise positional fingerprint

Intuition: multi-frequency clock hands We can think of each $(\sin, \cos)$ pair as a “hand” in a two-dimensional plane.

  • Low dimensions ($i$ small): high frequency, so the hand rotates extremely fast. Like a clock’s “second hand”, it completes a circle every few tokens.
  • High dimensions ($i$ large): low frequency, so the hand rotates extremely slowly. Like the “hour hand”, it may take tens of thousands of tokens to make one revolution. The Transformer is like reading a dashboard of dozens of hands rotating at different speeds, which lets it pinpoint the absolute position of the current token in the sequence.

Mathematical property: modeling relative position The beauty of choosing sine and cosine is that the model can express relative position through a linear transformation. By the angle-addition formulas of trigonometry, there is a transformation matrix $M_r$ that depends only on the relative distance $r$, such that: \(PE_{pos+r} = M_r \cdot PE_{pos}\) This means that, when computing attention, the model can more easily capture the distance between two words rather than only their absolute coordinates.

RoPE: the design of rotary position embedding

Absolute positional encoding adds position information to the input embedding, which affects attention only indirectly; ALiBi directly subtracts a distance penalty from the attention score. RoPE takes another path: before taking the inner product of Q and K, it encodes position information directly into the vectors themselves by rotation.

Schematic diagram of RoPE rotational position encoding
RoPE positional encoding: every two dimensions form a group and are rotated by the angle mθ according to position m, encoding position as an amount of rotation

Core idea: rotation instead of addition

RoPE treats each pair of dimensions ($d_0, d_1$) of a vector as a vector in a two-dimensional plane and rotates it by an angle determined by the token’s position $m$:

\[\text{K}^{(m)} = R(m\theta) \cdot \text{K}, \quad \text{Q}^{(m)} = R(m\theta) \cdot \text{Q}\]

The rotation matrix is the standard two-dimensional rotation matrix:

\[R(\alpha) = \begin{pmatrix} \cos\alpha & -\sin\alpha \\ \sin\alpha & \cos\alpha \end{pmatrix}\]

For a $d$-dimensional vector, the dimensions are grouped into $d/2$ pairs, and each pair uses a different base angle $\theta_i$:

\[\theta_i = 10000^{-2i/d}, \quad i = 0, 1, \ldots, \frac{d}{2}-1\]

This design is consistent with sinusoidal PE: different dimensions correspond to different frequencies, with low dimensions rotating fast and high dimensions rotating slowly, together forming a unique positional fingerprint.

How does RoPE encode relative position naturally?

When Q is at position $m$ and K is at position $n$, their inner product is:

\[\langle \text{Q}^{(m)}, \text{K}^{(n)} \rangle = \text{Q}^T \cdot R((m-n)\theta) \cdot \text{K}\]

The inner product depends only on the relative distance $m-n$ and not on absolute coordinates. This is the elegance of RoPE: there is no need to model relative distance explicitly as in relative PE, because the geometry of rotation guarantees it automatically.

Schematic diagram of relative position properties of RoPE
The relative-position property of RoPE: Q and K rotate synchronously, so the inner product depends only on the relative distance m-n and not on absolute position

Geometric argument for translation invariance

Suppose “cat” is at position 1 and “fish” is at position 3, and we compute the attention value $A$. Now insert 100 unrelated tokens in front, so “cat” moves to position 101 and “fish” to position 103. The relative distance between them stays the same (still 2), and the rotation angle $(m-n)\theta$ in the inner product is unchanged, so the attention value is still $A$.

The geometry is more intuitive: Q rotates by $N\theta$ and K rotates by $N\theta$, both rotating synchronously, so the inner product (the cosine of the angle between them) is unchanged.

Compatibility with engineering optimizations

RoPE modifies only Q and K themselves. The attention computation is exactly the same as in the original version, so it is naturally compatible with all attention acceleration techniques:

  • FlashAttention: usable directly without modifying the kernel;
  • KV cache: the rotated $\text{K}^{(m)}$ can be cached directly and read back without injecting position again.

This is also one of the key engineering reasons RoPE ultimately won: it is not only effective, but also fully compatible with the whole engineering ecosystem.

Common misconceptions clarified

Many people think RoPE guarantees, as ALiBi does, that “the farther apart Q and K are, the smaller the attention”. RoPE does not actually guarantee this: rotation produces a periodic oscillation pattern, and attention fluctuates in a zigzag as distance grows.

This is actually an advantage: RoPE lets the model learn attention patterns that are “distant but still highly relevant” (such as long-range dependencies), whereas ALiBi forcibly suppresses long-range attention.

Evolution of positional encoding:

Scheme Core idea Features Representative models
Absolute PE Sinusoidal, or a learnable embedding Simple, but extrapolates poorly (cannot handle sequences longer than those seen in training) BERT, original Transformer
Relative PE Models the relative distance between positions $i$ and $j$ Focuses on distance rather than absolute coordinates T5
ALiBi Subtracts a distance bias from the attention score Extrapolates extremely well and is extremely simple to compute MPT, Bloom
RoPE Rotates $Q, K$ by position-dependent angles (rotary position embedding) Captures relative position while staying engineering-friendly, so it is widely adopted LLaMA, Qwen, Gemma

2. Position-wise feed-forward network (FFN) Each attention layer is followed by a fully connected block (usually $d_{model} \to 4d_{model} \to d_{model}$) that introduces nonlinearity: \(\text{FFN}(x) = \text{max}(0, xW_1 + b_1)W_2 + b_2\)

3. Residual connections and normalization (Add & Norm) Each sub-layer uses a residual connection together with layer normalization (LayerNorm). There are two mainstream layouts:

  • Post-LN (the original): compute the sub-layer, add the residual, then apply LN. Performs well but is hard to train at depth.
  • Pre-LN (common in modern large models): apply LN first, then the sub-layer. It is usually easier to train stably; whether warmup is needed still depends on the model, optimizer and training settings.

Encoder vs. decoder

The Transformer uses an encoder-decoder architecture. The core difference between the two is masking:

  • Encoder: bidirectional attention; every token can see all tokens in the sequence.
  • Decoder: uses masked self-attention so that when generating the $t$-th token only the first $t-1$ tokens are visible (to prevent information leakage); it also contains cross-attention, which attends to the encoder’s output.

The concrete form of the causal mask is an $n \times n$ lower-triangular matrix (taking sequence length 4 as an example):

  $k_1$ $k_2$ $k_3$ $k_4$
$q_1$ ✓ −∞ −∞ −∞
$q_2$ ✓ ✓ −∞ −∞
$q_3$ ✓ ✓ ✓ −∞
$q_4$ ✓ ✓ ✓ ✓

Implementation: before the softmax, the attention scores at masked positions are set to $-\infty$ (in practice a large negative number such as -1e9), so after the softmax their weights become 0. This guarantees causality (“token $t$ can only see positions $\leq t$”) and also makes teacher forcing parallelizable during training: instead of recursing step by step like an RNN, the model predicts the next token at all positions in a single forward pass.

Attention visualization

One of the greatest charms of attention is its interpretability: the attention weights of each head in each layer can be visualized directly as heat maps, revealing the model’s “attention patterns”. Typical findings:

Attention pattern Example Explanation
Coreference head In “The cat … it was hungry”, “it” attends to “cat” Some heads specialize in pronoun resolution
Syntax head Verbs attend to their subjects and objects; adjectives attend to the nouns they modify Implicitly learned grammatical dependencies
Positional head Each token attends to the token immediately before or after it Captures local bigram patterns
Delimiter head All tokens attend to [SEP] or punctuation Acts as a “global information pool”
Redundant head Highly similar to other heads Why attention-head pruning works

Tools such as BertViz (Vig, 2019) and attention-flow (Abnar & Zuidema, 2020) can display these patterns interactively. Clark et al. (2019), “What Does BERT Look At?”, systematically analyzed BERT’s 144 heads and found that many heads capture clearly interpretable syntactic or semantic functions, which is important evidence that the Transformer goes beyond the “black box” label.

Transformer architecture diagram
Transformer architecture (Vaswani et al., 2017 "Attention Is All You Need")

RNN vs. Transformer

Dimension RNN/LSTM Transformer
Computation Sequential recursion, cannot be parallelized Fully parallel, very high hardware utilization
Long-range dependencies Decay exponentially with distance, so very long text is hard to capture The distance between any two positions is always 1, with no information loss
Context scope Local context Global context
Inductive bias Strong (temporal correlation) Weak (fully connected), so more training data is needed
Memory complexity $O(n \cdot d^2)$ $O(n^2 \cdot d)$, which puts heavy pressure on GPU memory for long sequences

Representative work: BERT (2018, encoder-only), the GPT series (2018 to present, decoder-only), T5 (2019, encoder-decoder), ViT (2020, treating image patches as a sequence).

Vision Transformer (ViT)

ViT (Vision Transformer, Dosovitskiy et al., 2020) showed for the first time that, given enough data, a pure Transformer can directly beat CNNs on image tasks, ending the nearly ten-year dominance of CNNs in vision.

Core design: treat an image as a “sentence”

flowchart LR
    IMG["Original image<br/>224×224×3"] --> PATCH["Split into patches<br/>16×16 each"]
    PATCH --> FLAT["Flatten into 196<br/>768-dimensional vectors"]
    FLAT --> EMBED["Patch Embedding<br/>(linear projection)"]
    EMBED --> POS["+ Position Embedding<br/>+ [CLS] token"]
    POS --> TRANS["Transformer<br/>Encoder × L"]
    TRANS --> CLS["Take the [CLS] output"]
    CLS --> HEAD["MLP head<br/>Classification"]
    style IMG fill:#e1f5ff
    style HEAD fill:#c4ffc4

Key steps:

  1. Patch splitting: Cut the $224 \times 224$ image into $14 \times 14 = 196$ patches of $16 \times 16$
  2. Patch embedding: Flatten each patch into a $16 \times 16 \times 3 = 768$-dimensional vector and obtain a token through a linear projection
  3. [CLS] token: Prepend a learnable “[CLS]” token to the sequence; its final output is used for classification
  4. Position embedding: Add a learnable positional encoding (ViT uses learnable PE rather than sinusoidal)
  5. Transformer encoder: A standard Transformer stack, usually 12–32 layers
  6. Classification head: Take the [CLS] output and feed it to an MLP

ViT inductive bias and data

Dimension CNN ViT
Inductive bias Strong (locality, translation invariance) Weak (almost no prior assumptions)
Small-data performance Excellent (the bias speeds up convergence) Poor (needs massive data to compensate)
Big-data performance Saturates sooner Keeps improving and eventually surpasses CNNs
Receptive field Small in shallow layers and gradually expanding in deep layers Global from the first layer

Empirical conclusion (original ViT paper):

  • Data < 1M (such as ImageNet-1k): ViT is clearly worse than ResNet
  • Data ~14M (ImageNet-21k): ViT is on par with ResNet
  • Data ~300M (JFT-300M): ViT is clearly better

This confirms a core principle of deep learning: less inductive bias × more data = a more flexible architecture with a higher ceiling.

The evolution of ViT

Model Year Key innovation
ViT 2020 The pioneering pure-Transformer model for images
DeiT 2020 Distillation strategy + data augmentation; reaches SOTA when trained on ImageNet-1k alone
Swin Transformer 2021 Hierarchical window attention that reintroduces local inductive bias, balancing efficiency and generalization
MAE 2021 Masked self-supervised pre-training (randomly masking 75% of patches), a new paradigm for ViT pre-training
DINOv2 2023 Self-supervised ViT reaching the level of a general-purpose vision foundation model
SigLIP / CLIP 2021-2023 ViT serves as the image encoder for image-text contrastive learning, driving multimodal large models

ViT not only replaces CNNs as the visual backbone but, more importantly, lets images and text share the same Transformer architecture. This is the basis on which large multimodal models such as CLIP, LLaVA and GPT-4V unify vision and language in a simple way.


4.4 Graph neural networks (GNNs)

CNNs process grid data and RNNs/Transformers process sequences, but in reality a great deal of data has graph structure: social networks, molecular structures, knowledge graphs, and the user-item bipartite graphs of recommender systems. Such data has a variable number of nodes, no fixed order and an uncertain number of neighbors, so the regular-structure assumptions of CNNs/RNNs no longer hold. Graph neural networks (GNNs) are designed specifically for this kind of non-Euclidean data.

Message passing

The unified paradigm of GNNs is neighborhood aggregation: each node repeatedly “collects neighbor information → aggregates → updates its own representation”. The update at layer $l$ can be written as:

\[h_v^{(l+1)} = \sigma\!\left( W^{(l)} \cdot \text{AGG}\big(\{ h_u^{(l)} : u \in \mathcal{N}(v) \}\big) + B^{(l)} h_v^{(l)} \right)\]

Here \(\mathcal{N}(v)\) is the neighbor set of node $v$, $\text{AGG}$ is a permutation-invariant aggregation function (sum / mean / max / attention-weighted), and $\sigma$ is a nonlinearity. After stacking $L$ layers, each node can perceive the subgraph structure within $L$ hops, much as a CNN expands its receptive field layer by layer.

flowchart LR
    U1["Neighbor u₁<br/>representation h(u₁)"] --> AGG["Aggregation AGG<br/>sum / mean / attention"]
    U2["Neighbor u₂<br/>representation h(u₂)"] --> AGG
    U3["Neighbor u₃<br/>representation h(u₃)"] --> AGG
    V0["Node v<br/>old representation h(v)"] --> UPD["Update<br/>σ(W·agg + B·h(v))"]
    AGG --> UPD
    UPD --> V1["Node v<br/>new representation h(v)′"]
    style V0 fill:#c4e1ff
    style V1 fill:#c4ffc4
    style AGG fill:#ffe1c4

Classic GNN comparison

Model Year Aggregation Key features
GCN (Kipf & Welling) 2017 Degree-normalized weighted sum First-order approximation of spectral graph convolution; the most classic baseline
GraphSAGE (Hamilton et al.) 2017 Sampled neighbors + mean / pooling Inductive: generalizes to unseen nodes and suits large graphs
GAT (Veličković et al.) 2018 Attention-weighted neighbors Introduces attention to assign each neighbor an adaptive weight
GIN (Xu et al.) 2019 Sum + MLP Theoretically reaches the discriminative upper bound of the WL graph isomorphism test, giving the strongest expressive power

Task types and core challenges

GNN tasks come in three types: node-level (node classification, e.g. determining the topic of a paper in a citation network), edge-level (link prediction, e.g. recommender systems) and graph-level (whole-graph classification, e.g. molecular property prediction).

Main challenge: over-smoothing: When there are too many layers, all node representations converge and lose discriminability. GNNs therefore usually use only 2–4 layers, far shallower than CNNs/Transformers. Residual connections, jumping knowledge (JKNet) and PairNorm are common mitigations.

Relationship to the Transformer: A Transformer can be viewed as a GNN defined on a fully connected graph, where each token is a node and self-attention aggregates all other nodes with attention weights. GAT’s neighbor attention shares its origin with the Transformer’s self-attention. In recent years, Graph Transformers (such as Graphormer) have carried positional encoding and global attention over to graphs, achieving SOTA on graph-level tasks such as molecules.

Representative work: GCN (2017), GraphSAGE (2017), GAT (2018), GIN (2019), Graphormer (2021); applications include AlphaFold (protein residue contact graphs), drug discovery, traffic prediction and recommender systems.


5. Training optimization

5.1 Gradient descent and optimizers

The update rule of standard gradient descent is:

\[\theta_{t+1} = \theta_t - \eta \cdot g_t\]

Here $\eta$ is the learning rate and $g_t$ is the gradient of the loss with respect to the parameters. All parameters share the same learning rate, yet on a real loss surface the gradients differ greatly across directions, and a fixed learning rate cannot suit them all. This gave rise to adaptive optimizers.

Momentum

Momentum (Polyak 1964, revived for deep learning by Sutskever et al., 2013) simulates the inertia of a ball rolling down a hillside:

\[v_t = \beta \cdot v_{t-1} + g_t \quad \text{(velocity accumulation)}\] \[\theta_{t+1} = \theta_t - \eta \cdot v_t\]

Here $\beta \in [0, 1)$ is the momentum coefficient (typically $0.9$) and $v_t$ is the exponential moving average of the gradient, which acts like “inertia”.

Physical intuition:

Scenario Plain SGD Momentum SGD
Ravine-shaped loss (steep on one side, flat on the other) Oscillates back and forth between the steep walls and makes extremely slow progress The oscillations accumulate and cancel, leaving only the net motion along the valley floor, so it accelerates forward
Flat region (small gradient) Very small steps, almost stagnant Velocity accumulates many small gradients and keeps moving forward
Saddle point Stalls near zero gradient Momentum helps escape saddle points
Sharp local minimum Easy to get stuck Momentum may carry the optimizer past shallow local minima

Nesterov accelerated gradient (NAG) is an improved version of momentum: it first “takes a step ahead” according to the current momentum and computes the gradient at the look-ahead position, making the inertia more responsive to the slope ahead (“look-ahead momentum”).

\[v_t = \beta \cdot v_{t-1} + \nabla \mathcal{L}(\theta_t - \eta\beta v_{t-1})\]

Enable it in PyTorch with torch.optim.SGD(momentum=0.9, nesterov=True).

Optimizer evolution

flowchart TD
    SGD["SGD<br/>stochastic gradient descent<br/>1951"] -->|"Add momentum"| Momentum["Momentum SGD<br/>Suppresses oscillation"]
    SGD -->|"Adaptive LR"| Adagrad["Adagrad 2011<br/>Accumulates squared gradients"]
    Adagrad -->|"Exponential moving average"| RMSprop["RMSprop 2012<br/>Handles non-stationary settings"]
    Momentum -->|"Combine"| Adam["Adam 2014<br/>First + second moments"]
    Adam -->|"Decouple weight decay"| AdamW["AdamW 2017<br/>Common baseline"]
    AdamW -->|"Curvature preconditioning"| Sophia["Sophia 2023<br/>Diagonal curvature estimate"]
    AdamW -->|"Fast and slow momentum"| AdEMAMix["AdEMAMix 2024"]
    Momentum -->|"Orthogonalize matrix updates"| Muon["Muon 2024"]
    Shampoo["Shampoo 2018<br/>Matrix preconditioning"] -->|"Adam in the eigenbasis"| SOAP["SOAP 2024"]
    Muon -->|"Update scaling and weight decay"| Moonlight["Moonlight 2025<br/>Large-scale training validation"]
    Muon -->|"Neuron-wise adaptive scaling"| NorMuon["NorMuon 2025<br/>ICML 2026"]

The arrows illustrate related ideas. Muon orthogonalizes momentum updates for matrices; it should not be understood as a switch added to AdamW. Methods based on norm geometry, such as Scion, are discussed below.

Classical and recent optimizers

Optimizer First public release Update mechanism Main cost or limitation Application direction
SGD / Momentum SGD Classical methods Raw gradient or momentum direction Requires learning rate and schedule tuning Baselines for supervised learning, including CNNs
Adagrad 2011 Coordinate-wise scaling by accumulated squared gradients Learning rates may decay too quickly Sparse features
RMSprop 2012 Exponential moving average of squared gradients Hyperparameters depend on the task Historically common for RNNs
Adam / AdamW 2014 / 2017 First and second moments; AdamW decouples weight decay Standard implementations store two per-parameter states General training and fine-tuning baselines
Adafactor 2018 Factorizes matrix second moments into row and column statistics Saves state memory, but approximation and configuration affect results Limited optimizer-state memory
Lion 2023 Sign of a combination of momentum and gradient Usually needs a lower learning rate than AdamW and corresponding decay adjustments Candidate for vision and language tasks
Sophia 2023 Diagonal curvature estimates and clipped updates Requires additional curvature estimation; step count alone is insufficient LLM pretraining research
AdEMAMix 2024 Combines fast and slow gradient EMAs with a second moment Adds a slow momentum buffer and its hyperparameters relative to AdamW Long training runs and reduced forgetting
Muon / Moonlight extension 2024 / 2025 Approximately orthogonalizes matrix momentum and scales updates by layer shape Usually paired with AdamW; adds matrix operations and communication costs Validated in large-scale dense / MoE pretraining
SOAP 2024 Runs Adam in the eigenbasis of Shampoo’s preconditioner Stores matrix statistics and periodically updates the eigenbasis Training that can afford preconditioning overhead
Scion 2025 Produces updates through a linear minimization oracle under layer-specific norm constraints Requires norms and scales for different layers Hyperparameter transfer and activation-scale control
NorMuon 2025; ICML 2026 Adds neuron-wise adaptive scaling to Muon updates Usually still needs AdamW for parameters outside the hidden weight matrices Improving Muon for large-batch training

A second moment is not a second derivative: Adam’s $v_t$ is a moving average of squared gradients, not a Hessian. Sophia estimates curvature; Shampoo / SOAP use matrix preconditioning; Muon changes the singular-value structure of matrix updates. These approaches address different aspects of optimization and cannot be adequately described by a single “adaptive LR” label.

Adam’s core formula (Kingma & Ba, 2014):

\[m_t = \beta_1 m_{t-1} + (1-\beta_1)g_t \quad \text{(first moment)}\] \[v_t = \beta_2 v_{t-1} + (1-\beta_2)g_t^2 \quad \text{(second moment)}\] \[\hat{m}_t = \frac{m_t}{1-\beta_1^t}, \qquad \hat{v}_t = \frac{v_t}{1-\beta_2^t}\] \[\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{\hat{v}_t}+\epsilon}\hat{m}_t\]

Standard hyperparameters: $\beta_1=0.9$, $\beta_2=0.999$, $\epsilon=10^{-8}$. Adam’s $m_t$ determines the direction (and can carry the iterate across saddle points), while $v_t$ determines the magnitude (adaptively adjusting each parameter’s learning rate). The two complement each other and together address the two core problems of gradient descent.

Muon: from coordinate-wise scaling to matrix updates

Muon (MomentUm Orthogonalized by Newton-Schulz) treats hidden-layer weights as matrices and approximately orthogonalizes their momentum. Intuitively, AdamW adjusts each coordinate’s step size, while Muon rebalances different singular directions of a matrix update. Omitting Nesterov momentum, shape-dependent scaling, and weight decay, an illustrative formulation is:

\[M_t = \beta M_{t-1} + (1-\beta)G_t, \qquad M_t = U_t\Sigma_t V_t^\top\] \[Q_t = U_t V_t^\top, \qquad W_{t+1} = W_t - \eta_t Q_t\]

The compact SVD expresses the ideal polar factor. Implementations approximate this transformation with a few Newton–Schulz iterations, avoiding an explicit SVD at every step. For a full-rank rectangular matrix, the ideal factor is semi-orthogonal; a finite approximation need not be strictly semi-orthogonal. The update direction is orthogonalized, rather than requiring the weight matrix itself to remain orthogonal. See the author’s explanation and implementation for the original algorithm and parameter grouping conventions.

Scaling to larger models: Moonlight adds weight decay and update-scale calibration to Muon and validates it on a MoE with 3B active / 16B total parameters, trained on 5.7T tokens. The paper reports roughly twice the compute efficiency in its comparisons. This does not imply that arbitrary end-to-end training runs take half the time, but it does establish evidence beyond small and medium-sized models. Original paper

Parameter grouping: Common implementations use Muon for hidden-layer matrices and AdamW for embeddings, output heads, biases, and normalization parameters. Grouping solely by p.ndim == 2 is insufficient: embeddings and output heads are also matrices, and tied weights must be deduplicated. Convolution-kernel flattening, whether Q/K/V are orthogonalized separately, and communication for sharded parameters should follow the conventions of the chosen implementation. Author’s implementation

SOAP: Adam in a better coordinate system

SOAP (Shampoo with Adam in the Preconditioner’s eigenbasis) accumulates left and right statistics of matrix gradients, periodically obtains their eigenbases, transforms gradients into that coordinate system, applies Adam there, and transforms updates back. It combines Shampoo’s matrix structure with Adam’s coordinate-wise adaptation. Original paper

It is inaccurate to state that “SOAP is equivalent to Shampoo.” The theoretical connection relies on particular assumptions, and practical behavior also depends on basis-update frequency, block size, and momentum handling. For an $m\times n$ matrix, the unblocked left and right statistics alone require $O(m^2+n^2)$ storage, with additional eigendecomposition costs. Fewer training steps must therefore be assessed alongside time per step and peak memory.

AdEMAMix, Scion, and NorMuon: three directions for improvement

  • Retaining older gradient information—AdEMAMix (2024): A fast EMA tracks recent directions, while a slow EMA retains a longer history; a mixing coefficient combines them. The paper validates benefits in language modeling and image classification. This generally adds a per-parameter momentum buffer relative to AdamW, so it should not also be presented as saving optimizer-state memory. Original paper
  • Changing the geometry of updates—Scion (2025): Norms are chosen according to layer input and output scales, and a linear minimization oracle produces updates. The framework includes constrained and unconstrained variants. One goal is better hyperparameter transfer across model sizes; this does not mean that every model requires no tuning. ICML 2025 paper
  • Restoring neuron-wise adaptation—NorMuon (2025; ICML 2026): It introduces per-neuron second-moment statistics and row normalization after Muon orthogonalization to address uneven neuron updates, with particular attention to larger batches. It combines matrix updates with neuron-wise adaptation, rather than simply applying coordinate-wise Adam to Muon. ICML 2026 paper

Recent work in 2026: two directions to follow

Work First public release New idea Scope and limitations
Aurora 2026-06-26 Improves row-update balance for tall matrices within Muon’s spectral geometry, focusing on MLP up / gate projections The authors report language-model pretraining and nanoGPT experiments; these do not establish benefits for every layer and model
AF-Muon 2026-10-01 Designs updates for a shared vocabulary matrix and auxiliary parameters in models with tied embeddings / output heads, removing the need for companion AdamW The authors test models including 124M–1B parameters; frontier-scale pretraining remains untested, and this is not a universal configuration for original Muon

As of this update, both are presented as recent preprints. AF-Muon’s reported savings of about 20% concern optimizer-state memory relative to its Hybrid Muon baseline, not a 20% reduction in total training memory. Beyond new optimizer names, useful questions include how vocabulary matrices are handled, how neuron-wise adaptation is retained after orthogonalization, and how orthogonalization and communication costs can be reduced.

Practical selection and fair comparisons

Current problem Consider first What to record
Need a reliable starting point, or perform conventional fine-tuning / LoRA Establish an AdamW baseline Learning rate, effective batch size, and decay groups
Pretrain a Transformer from scratch and seek better compute efficiency Muon + AdamW; then compare NorMuon Parameter grouping, update scaling, distributed implementation, and end-to-end time
Want matrix preconditioning and have sufficient memory and compute SOAP Block size, preconditioner update interval, and peak memory
Run long training and want to retain older gradient information AdEMAMix Slow momentum settings, additional state, and training budget
Optimizer states are the memory bottleneck Adafactor, low-bit states, or state sharding Optimizer-state and total memory separately, plus changes in quality

These are selection recommendations based on the mechanisms and available experimental evidence. Compare optimizers using the same data, model, token / sample budget, and validation protocol, with a reasonable tuning budget for each. Report GPU time to target quality, final quality, peak memory, and stability. Fewer steps, fewer tokens, fewer FLOPs, and less elapsed time are different metrics. Benefits in pretraining do not automatically transfer to LoRA, reinforcement learning post-training, or other modalities.

5.2 Learning rate scheduling

A fixed learning rate can be too large, causing oscillation, or too small, causing slow progress. Warmup + Cosine Decay is a common approach; WSD and Schedule-Free reduce dependence on a predetermined training endpoint in different ways.

training phase Relative LR Description
Starting point (step 0) 0 Linear heating from zero
Warmup ends 1.0 (peak value) Reach target learning rate
mid-training ~0.5 Cosine decay through half way
end of training ~0.05 Close to zero, the parameters slowly “land”
stage Description purpose
Warmup (preheat) In the first few steps, LR increases linearly from 0 to the target value. Let Adam’s $m_t$/$v_t$ accumulate accurate statistics to avoid initial instability
Cosine Decay (cosine decay) LR drops to close to 0 according to the cosine curve Let the parameters “land” slowly to avoid continuous oscillation near the optimal point

Other commonly used schedulers

Scheduler Curve characteristics Applicable scenarios
StepLR Multiply LR by $\gamma$ every $k$ epoch (e.g. 0.1) CV classic training (ResNet original paper 30/60/90 epoch reduced to 10×)
MultiStepLR descend the ladder at the specified epoch list Manual version of StepLR, precise control
ExponentialLR $\eta_t = \eta_0 \cdot \gamma^t$, proportional attenuation in each step Scenes that require smooth attenuation
ReduceLROnPlateau Validation set Loss stagnated and dropped to LR after $p$ epochs Adaptive; no need to know the total epochs in advance
Cosine Annealing Warm Restart The cosine decays to 0 and then “restarts” back to high LR, with period increment (SGDR) Escape from local optimality and integrate multiple checkpoints
1Cycle(Leslie Smith, 2018) First rise and then fall + Momentum reversely changes synchronously CV fast training (recommended by FastAI)
Warmup + Linear Decay Linear decay after warmup BERT original paper, GLUE fine-tuning commonly used
Warmup + Cosine Cosine decay after warmup Pretraining and fine-tuning with a predetermined budget
Warmup–Stable–Decay (WSD) Warmup → constant-LR trunk → final decay Extending pretraining flexibly or comparing different budgets

WSD: separating the training trunk from final convergence

WSD (Warmup–Stable–Decay) maintains a constant learning rate after warmup. Once an endpoint for a particular budget is chosen, training branches from a trunk checkpoint into a decay phase. Unlike a cosine schedule tied to a total step count from the start, this allows the stable-phase trunk to be reused to produce models at different compute budgets. WSD analysis paper

This is still a learning rate schedule: the peak rate, warmup, decay duration, and decay shape must be chosen, and decay consumes compute. A stable-phase checkpoint and one that has completed cooldown should not be treated as having equal budgets and convergence states.

Schedule-Free: iterate averaging instead of a preset decay curve

Schedule-Free (2024) combines momentum with iterate averaging, distinguishes the gradient-evaluation point from the parameters used for evaluation, and avoids a decay curve tied to a predetermined stopping step. Variants include SGD and AdamW. Its mechanism differs from WSD, and learning rate, warmup, and weight decay still require configuration. Original paper

Implementation details: The authors’ non-closure interface requires optimizer.train() during training and optimizer.eval() before validation or saving evaluation weights, alongside model.train() / model.eval(); switching model mode alone is insufficient. Resumable checkpoints should retain optimizer state according to the implementation’s instructions. BatchNorm also requires handling running statistics for the evaluation parameters. The official implementation still recommends warmup. Author’s repository

Selection principles:

  • Training a CNN from scratch: StepLR or cosine annealing (warm restarts for long training)
  • Fine-tuning a pretrained model: warmup + linear/cosine (low peak LR, usually $1-5 \times 10^{-5}$)
  • LLM pretraining: Compare Warmup + Cosine for fixed budgets and WSD when the budget may grow. Calibrate peak learning rate and warmup to model scale, batch size, and optimizer rather than copying them across configurations
  • A validation set is available and the number of epochs is uncertain: ReduceLROnPlateau
  • Rapid prototyping / competitions: 1Cycle (Smith reported it can achieve “super-convergence”)

5.3 Parameter initialization

Different initial parameters $\theta_0$ may lead to different local optima.

Xavier initialization (Glorot & Bengio, 2010; suited to symmetric activations such as sigmoid/tanh):

\[W \sim \mathcal{N}\left(0, \frac{2}{n_{in}+n_{out}}\right)\]

This method keeps the variance of the forward activations and the backward gradients stable, and is the default choice for symmetric activation functions.

Kaiming initialization (He et al., 2015; suited to ReLU activations):

\[W \sim \mathcal{N}\left(0, \frac{2}{n_{in}}\right)\]

ReLU zeroes out half of its inputs, so Xavier’s variance assumption no longer holds; Kaiming therefore doubles the variance to compensate. This keeps the activation variance stable early in the training of ReLU networks dozens of layers deep, preventing it from exploding or vanishing.

Selection principle: Xavier is commonly used for sigmoid/tanh, and Kaiming for ReLU and its variants. In practice, the initialization must also be considered together with the residual structure, normalization and the framework’s default implementation.

Pre-training as initialization: Train on large-scale data first, then transfer the parameters to the target task. This improves both optimization (a better starting point) and generalization (learning common features).

5.4 Normalization methods

Normalization forces the output of each layer into a reasonable range, which makes the loss surface flatter and the learning rate easier to tune.

Comparison of normalization methods

Method Year Normalized dimensions Depends on batch Typical use Representative models
Batch Norm (BN) 2015 Across samples, per feature dimension ✓ CNN image classification ResNet, EfficientNet
Layer Norm (LN) 2016 Single sample, all features ✗ Transformers, sequence models BERT, GPT series
Group Norm 2018 Single sample, groups of features ✗ Small-batch CV (object detection) Mask R-CNN
Instance Norm 2017 Single sample, single channel ✗ Image style transfer StyleGAN
RMSNorm 2019 Single sample (RMS scaling) ✗ Efficient LLM training LLaMA, Qwen, etc.

Batch Normalization(BN, Ioffe & Szegedy, 2015):

\[\hat{x} = \frac{x - \mu_{batch}}{\sqrt{\sigma_{batch}^2 + \epsilon}}, \quad y = \gamma\hat{x} + \beta\]

Here $\gamma, \beta$ are learnable scale and shift parameters, which give the model the ability to “undo the normalization”.

Key differences between BN in training and inference

BN behaves differently during training and inference, which is the most common pitfall in engineering practice:

Stage Source of $\mu, \sigma^2$ Behavior
Training Computed on the fly from the current mini-batch Each batch has a slightly different mean and variance, which introduces noise that acts as regularization; running_mean/running_var (exponential moving averages) are updated at the same time
Inference The running_mean/running_var accumulated during training Independent of the batch, so the same input always gives the same output

Common bug: In PyTorch, model.eval() switches BN to its running statistics for inference; if you forget to call it, inference still uses the current batch’s statistics, which makes the output random when the batch size is 1.

Limitations of BN and why LN is needed

BN has three fatal shortcomings, which correspond to the three reasons the Transformer chose LN:

Problem BN behavior LN behavior
Batch too small Statistics become noisy and training becomes unstable at extremely small batch sizes Independent of the batch; works even with batch size 1
Varying sequence length Samples have different lengths, and padding pollutes the statistics Each sample is normalized independently
Distributed training Batch statistics must be synchronized across devices (SyncBN) Fully local computation, no communication overhead

Layer Normalization(LN, Ba et al., 2016):

\[\hat{x}_i = \frac{x_i - \mu_{layer}}{\sqrt{\sigma_{layer}^2 + \epsilon}}\]

Layer normalization normalizes across the feature dimensions of each sample, independently of batch size and sequence length. This is why it is standard in Transformers.

Normalized dimensions compared intuitively (taking a 4D feature map of shape [N, C, H, W] as an example):

Method Normalization range Across samples? Across channels?
BN Over the (N, H, W) dimensions, separately for each C ✓ ✗
LN Over the (C, H, W) dimensions, separately for each N ✗ ✓
Instance Norm Over (H, W), separately for each (N, C) ✗ ✗
Group Norm Split C into groups and normalize within each group ✗ Partially

RMSNorm (2019): drops the mean-centering and keeps only the variance scaling:

\[\text{RMSNorm}(x) = \frac{x}{\sqrt{\frac{1}{d}\sum_i x_i^2 + \epsilon}} \cdot \gamma\]

It is typically slightly cheaper to compute than LN and performs similarly in many language models; adoption depends on the architecture and training recipe.

5.5 Residual connection

Deep networks (100+ layers) suffer from both vanishing and exploding gradients. The skip connection / residual connection (He et al., ResNet, 2015) is:

\[\text{output} = F(x) + x\]

That is, the input $x$ (an identity mapping) is added directly to the transformation $F(x)$. Even if $F(x)$ has little effect, the gradient can still flow back to the shallow layers directly through the identity path, which greatly alleviates vanishing gradients and enables stable training of extremely deep networks such as ResNet-152.

Variants of residual connections

After ResNet, residual connections developed into several variants, each trading off “degree of identity” against “information density”:

Variant Year Core formula Features
Highway Network 2015 $y = T(x) \odot F(x) + (1-T(x)) \odot x$ A gate $T$ controls the residual scale; the predecessor of ResNet
ResNet v1 2015 $y = \text{ReLU}(F(x) + x)$ The original residual block, with the activation applied after the residual addition
ResNet v2 (pre-activation) 2016 $y = F(x) + x$, with BN/ReLU inside $F$ The activation is placed before the residual branch, so the gradient flows back more cleanly; supports networks of more than 1000 layers
DenseNet 2017 $x_l = H_l([x_0, x_1, \ldots, x_{l-1}])$ Each layer is concatenated with all preceding layers, maximizing feature reuse
ResNeXt 2017 ResNet + grouped conv Multi-path (cardinality) idea: fewer parameters and higher performance

Key insight of pre-activation ResNet (v2): In the original ResNet’s ReLU(F(x)+x), the ReLU truncates part of the gradient; v2 moves BN-ReLU inside $F$, so the residual path is a pure identity mapping and the gradient passes through losslessly. This makes ResNet-1001 (1001 layers!) trainable. The Transformer’s Pre-LN layout is essentially a generalization of the same idea.

DenseNet offers a counterintuitive result: each layer concatenates all preceding layers, which seems likely to cause an explosion in parameter count. Feature reuse allows each layer to remain narrow (often with a growth rate of only 32 channels), so the total parameter count can be smaller than that of a ResNet of the same depth. The trade-off is high GPU memory use because many intermediate activations must be retained.

The skip connection has become a standard component of almost all modern deep architectures (ResNet, Transformer, U-Net, diffusion UNet). What it improves is optimization: it does not address whether deep networks can express complex functions (shallow ones can too), but whether deep networks can be trained by SGD.


6. Loss function and regularization

6.1 Loss function

The loss function is the first step of the three-step machine learning framework and defines “how to measure the quality of a model”. Choosing an appropriate loss function is crucial for training.

Mean Squared Error (MSE)

Mean squared error (MSE) is suited to regression tasks and measures the mean squared difference between predictions and targets:

\[\mathcal{L}_{MSE} = \frac{1}{N}\sum_{i=1}^N (y_i - \hat{y}_i)^2\]
  • Advantages: differentiable everywhere with a simple gradient; larger errors are penalized more heavily (the squaring effect)
  • Disadvantages: extremely sensitive to outliers; gradients may explode when the prediction error is large

Mean Absolute Error (MAE)

\[\mathcal{L}_{MAE} = \frac{1}{N}\sum_{i=1}^N |y_i - \hat{y}_i|\]
  • Advantages: more robust to outliers (linear rather than squared penalty)
  • Disadvantages: not differentiable at \(y_i = \hat{y}_i\) (Huber loss is used as a compromise)

Huber Loss

\[\mathcal{L}_{Huber} = \begin{cases} \frac{1}{2}(y-\hat{y})^2 & |y-\hat{y}| \leq \delta \\ \delta|y-\hat{y}| - \frac{1}{2}\delta^2 & |y-\hat{y}| > \delta \end{cases}\]

Huber loss uses MSE (smooth and differentiable) when the error is small and MAE (robust to outliers) when the error is large, with $\delta$ controlling the switching threshold. It is commonly used for the regression branch in object detection.

Cross-entropy loss

Cross-entropy loss is suited to classification tasks, including image classification and language models (next-token prediction is essentially multi-class classification).

Step 1: Convert the network’s output logits into a probability distribution with softmax:

\[p_i = \frac{e^{z_i}}{\sum_j e^{z_j}}\]

Step 2: Compute the cross-entropy with the true label (take the log-probability of the true class and negate it):

\[\mathcal{L}_{CE} = -\sum_i \hat{p}_i \log p_i = -\log p_{y^*}\]

Here $y^*$ is the true class and \(\hat{p}_i\) is the one-hot label.

Why not use accuracy as the loss? Accuracy is a step function: when the parameters change slightly, the loss barely changes (its gradient is almost always zero), so gradient descent cannot be performed. Cross-entropy is differentiable everywhere, and the smaller it is, the higher the accuracy tends to be.

Numerical stability: In practice, softmax and cross-entropy are fused (LogSoftmax + NLLLoss) to avoid numerical overflow of $e^{z_i}$.

Label smoothing

Label smoothing (Szegedy et al., 2016; widely popularized by Inception-v3 and the original Transformer) is a simple but powerful regularization trick for cross-entropy.

Motivation: A one-hot label requires the model to predict 1.0 for the correct class and 0 for all others. To fit this fully, the model must push the logits towards infinity, which causes overconfidence, poorer generalization and weak distillation.

Approach: soften the hard label \(\hat{p}_i\) to:

\[\hat{p}_i^{\text{LS}} = \begin{cases} 1 - \epsilon & i = y^* \\ \epsilon/(K-1) & i \neq y^* \end{cases}\]

Here $K$ is the number of classes and $\epsilon$ is the smoothing coefficient (typically $0.1$). The target probability of the correct class drops to $0.9$, and the remaining $0.1$ is distributed evenly over the other classes.

Effects:

  • Suppresses overconfidence and gives better calibration: when the model predicts 90%, the actual accuracy is also close to 90%
  • Slightly improves generalization (roughly 0.2–0.5% higher Top-1 accuracy on ImageNet)
  • Improves the quality of the teacher’s soft labels for knowledge distillation (a soft distribution carries more information than a one-hot label)

Note: Label smoothing discards a small amount of confidence information, which must be weighed in tasks that rely on confidence (active learning, rejection).

Binary cross-entropy (BCE)

For binary classification tasks (with a sigmoid output layer):

\[\mathcal{L}_{BCE} = -[y \log \hat{p} + (1-y)\log(1-\hat{p})]\]

Focal Loss

Focal loss (Lin et al., 2017, proposed in RetinaNet) is designed specifically for scenarios with severe class imbalance, such as object detection, where background boxes far outnumber foreground boxes (about 1000:1), or medical images, where positive samples are rare.

Standard cross-entropy still assigns a non-negligible loss to easy samples (predicted probability $p \to 1$), and a large number of easy samples will “swamp” the gradients of the few hard samples. Focal loss multiplies CE by a modulating factor $(1-p)^\gamma$:

\[\mathcal{L}_{Focal} = -\alpha(1-p)^\gamma \log p\]
  • $\gamma$ (focusing parameter, typically 2): When $p \to 1$, $(1-p)^\gamma \to 0$ and almost no gradient is produced; when $p \to 0$ (a hard sample), the factor approaches 1 and the gradient is normal
  • $\alpha$ (class balancing, typically 0.25): A static adjustment of the positive and negative class weights

Effect: Focal loss played an important role in RetinaNet becoming the first single-stage detector to match the accuracy of the two-stage Faster R-CNN. It has since been widely used in long-tail classification, anomaly detection and medical image segmentation.

KL divergence

KL divergence (Kullback-Leibler divergence) measures the difference between two probability distributions and is often used in knowledge distillation and variational autoencoders (VAEs):

\[\mathcal{L}_{KL}(P \| Q) = \sum_i P(i) \log \frac{P(i)}{Q(i)}\]

Mathematical relationship between cross-entropy and KL divergence

Cross-entropy and KL divergence are not two independent losses; they are different views of the same quantity:

\[\text{CE}(P, Q) = H(P) + \text{KL}(P \| Q)\]

where $H(P) = -\sum_i P(i) \log P(i)$ is the entropy of the true distribution $P$.

Corollary: When $P$ is a one-hot hard label, $H(P) = 0$, so CE $\equiv$ KL. In standard classification, “minimizing cross-entropy” and “minimizing KL divergence” are therefore exactly equivalent.

Practical differences:

  • Hard labels (one-hot): use CE, which is simpler to compute
  • Soft labels (distillation, label smoothing, the RLHF reference policy): use KL. When $H(P) \neq 0$, CE contains a constant term that cannot be optimized, whereas KL measures the “distance from the target distribution” more cleanly

Contrastive losses

Metric learning aims to learn an embedding space in which “similar samples” are close together and “dissimilar samples” are far apart. There are two classic losses:

Contrastive loss (Hadsell et al., 2006): given a pair of samples $(x_i, x_j)$, the label $y=1$ means the same class and $y=0$ means different classes:

\[\mathcal{L}_{\text{contrastive}} = y \cdot d(x_i, x_j)^2 + (1-y) \cdot \max(0, m - d(x_i, x_j))^2\]

Here $d$ is the embedding distance (usually Euclidean) and $m$ is the margin hyperparameter.

Triplet loss (Schroff et al., 2015, FaceNet): given a triplet (anchor $a$, positive $p$, negative $n$):

\[\mathcal{L}_{\text{triplet}} = \max(0, d(a, p) - d(a, n) + m)\]

The intuition is that “the distance from the positive to the anchor plus a margin should be smaller than the distance from the negative to the anchor”. FaceNet achieves 99.63% accuracy on LFW with the face embeddings it learns.

InfoNCE (Contrastive Predictive Coding, Oord et al., 2018): the mainstream form of modern contrastive learning, in which one positive sample is contrasted against $N-1$ negative samples with a softmax:

\[\mathcal{L}_{\text{InfoNCE}} = -\log \frac{\exp(\text{sim}(q, k^+)/\tau)}{\sum_{i=0}^{N-1} \exp(\text{sim}(q, k_i)/\tau)}\]

where $\tau$ is the temperature coefficient. SimCLR, MoCo and CLIP are all based on InfoNCE; this type of loss drove the explosion of self-supervised representation learning in 2020–2023.

InfoNCE is essentially the cross-entropy of an $N$-way classification: the positive sample is the only “correct class” and the negatives are the “other classes”. This once again confirms the universality of CE.

Loss function selection cheat sheet

Task type Recommended loss Notes
Regression (no outliers) MSE Smooth gradients and fast convergence
Regression (with outliers) Huber loss Balances smoothness and robustness
Binary classification Binary cross-entropy With a sigmoid output layer
Multi-class classification Cross-entropy With a softmax output layer
Language modeling Cross-entropy Next-token prediction = multi-class classification
Distribution matching KL divergence VAE, knowledge distillation, the RLHF KL constraint
Object detection (box regression) Smooth L1 / IoU loss Matches the characteristics of detection tasks
Severe class imbalance Focal loss Object detection, long-tail classification, medical images
Metric learning / face recognition Triplet loss, ArcFace Learns a discriminative embedding space
Self-supervised learning InfoNCE (contrastive learning) SimCLR, MoCo, CLIP

6.2 Dropout

Dropout (Srivastava et al., 2014): with dropout rate $p$, each neuron’s output is set to zero with probability $p$ during training, and the retained activations are divided by $1-p$ (inverted dropout). Dropout is turned off at inference and the full network is used directly, so no additional scaling is needed.

Dropout diagram
Dropout: neurons are randomly dropped (×) during training, which is equivalent to ensembling a large number of sub-networks

Intuition: forcing the network to predict correctly even when some neurons are missing prevents excessive co-adaptation between neurons. It is equivalent to training a large number of differently structured sub-networks at the same time and obtaining an ensemble effect.

When to use it: When there is a clear gap between training and validation error, Dropout can serve as a regularization option; if the training loss itself will not go down, check the optimization settings first. Too strong a Dropout may slow convergence further.

6.3 Data augmentation

Data augmentation artificially enlarges the dataset by applying semantics-preserving transformations to training samples:

Method Domain Core idea Year
Flip / crop / color jitter Image Classic hand-designed augmentations that preserve semantics —
Mixup General Mixes two samples in proportion and mixes their labels in the same way 2018
CutMix Image Cuts and pastes regions, mixing labels in proportion to area 2019
AutoAugment Image Searches for the best augmentation policy with reinforcement learning 2019
RandAugment Image Randomly samples augmentations with only 2 hyperparameters; a simplified AutoAugment 2020
AugMix Image Mixes multiple augmentation chains with a Jensen-Shannon consistency loss to improve distribution robustness 2020
Time stretch / pitch shift Speech Changes speed and pitch while keeping the content the same —
Synonym replacement / back-translation Text Semantically equivalent rewriting —

Mixup formula: $\tilde{x} = \lambda x_i + (1-\lambda)x_j$, $\tilde{y} = \lambda y_i + (1-\lambda)y_j$

Note: Augmentations must preserve the label semantics. If the task is to determine which way a bird’s head is pointing, left-right flipping is not allowed; if the task is speaker identification, swapping speakers is not allowed.

When to use it: Data augmentation is usually used to improve generalization and distribution robustness, but its strength must match the task. If the training loss will not go down, first confirm that the augmentation does not destroy the label semantics, then check the optimization settings.

6.4 L2 regularization and AdamW

L2 regularization (weight decay) adds an L2-norm penalty on the parameters to the loss, biasing the optimization towards solutions with smaller parameter magnitudes (“simpler” functions):

\[\mathcal{L}' = \mathcal{L}_{data} + \lambda \sum_i \theta_i^2\]

AdamW (Loshchilov & Hutter, 2017) fixed a common mistake in adding L2 regularization to Adam: the traditional approach merges the regularization gradient into the ordinary gradient and then rescales both with Adam’s adaptive terms, so the regularization effect is “diluted” by the adaptive learning rate. AdamW instead applies weight decay directly to the parameters and then performs the Adam update:

θ = θ × (1 - lr × λ)       # Weight decay acts directly on the parameters (no adaptive scaling)
θ = θ - lr × Adam_update    # Normal Adam update

AdamW is currently the standard optimizer for training large language models and is usually used together with gradient clipping.

6.5 Semi-supervised and self-supervised learning

In real-world settings, labeled data is scarce while unlabeled data is cheap. Semi-supervised learning and self-supervised learning are the two mainstream ways to exploit unlabeled data.

Core idea

Method Representative work Core mechanism
Entropy minimization Grandvalet 2005 Requires the model to predict unlabeled samples as confidently as possible (low entropy), implicitly assuming that “class boundaries should lie far from high-density regions”
Pseudo-labeling Lee 2013 Uses the current model’s predictions as pseudo-labels for unlabeled samples and adds the high-confidence ones to training
Consistency regularization Π-Model 2016, Mean Teacher 2017 Applies different perturbations (augmentation, dropout) to the same sample and requires the outputs to be consistent
FixMatch Sohn et al. 2020 Predictions on weakly augmented inputs serve as pseudo-labels to supervise predictions on strongly augmented inputs; the current mainstream SSL paradigm
Contrastive learning SimCLR 2020, MoCo 2020 Pulls together representations of different views of the same image and pushes apart those of different images
Masked modeling BERT 2018, MAE 2021 Randomly masks part of the input and asks the model to predict the masked content

Relationship to modern large models

The pre-training of large language models is essentially the largest self-supervised learning effort: “predicting the next token” (causal LM) or “predicting the masked token” (masked LM) on massive unlabeled text, followed by adaptation to downstream tasks through fine-tuning (SFT, RLHF) with a small amount of labeled data. The success of the pre-training + fine-tuning paradigm means that scarce annotated data is no longer the main bottleneck for deploying deep learning.


7. Method classification summary

flowchart TD
    A["Neural network training techniques"] --> B["Step 3: improve optimization"]
    A --> C["Step 2: improve the network architecture"]
    A --> D["Step 1: improve the loss and the data"]
    B --> B1["AdamW / Muon / SOAP<br/>Coordinate-wise or matrix updates"]
    B --> B2["Cosine / WSD<br/>or Schedule-Free iterate averaging"]
    B --> B3["Kaiming init<br/>Parameter initialization"]
    B --> B4["Pre-training<br/>Improves both optimization and generalization"]
    C --> C1["CNN<br/>Receptive field + parameter sharing<br/>Improves generalization"]
    C --> C2["Skip connection<br/>Mitigates vanishing and exploding gradients<br/>Improves optimization"]
    C --> C3["BN / LN / RMSNorm<br/>Mainly improve optimization"]
    D --> D1["Cross-entropy<br/>Makes optimization feasible"]
    D --> D2["Dropout<br/>Improves generalization"]
    D --> D3["Data augmentation<br/>Improves generalization"]
    D --> D4["AdamW / weight decay<br/>Improves generalization"]

A comparison of the objectives of each method:

Method Step improved Goal Remarks
Adagrad / RMSprop Step 3 Optimization Predecessors of adaptive learning rates
Adam / AdamW Step 3 Optimization Common coordinate-wise adaptive baseline
Muon / NorMuon Step 3 Optimization Orthogonalized matrix updates; parameter grouping must be explicit
SOAP / Scion Step 3 Optimization Matrix preconditioning and layer-specific norm geometry, respectively
LR Scheduling / Schedule-Free Step 3 Optimization Cosine, WSD, or iterate averaging, chosen for the training budget
Kaiming init Step 3 Optimization Standard initialization for ReLU networks
Pre-training Step 3 Opt + Gen Improves both at once; the core of modern large models
CNN Step 2 Opt + Gen Improves sample efficiency through inductive bias
Skip connection Step 2 Optimization Makes deep networks trainable
Batch norm Step 2 Optimization (+Gen) Depends on batch statistics
Layer norm Step 2 Optimization (+Gen) Standard in Transformers
Cross-entropy Step one Makes optimization feasible Standard loss for classification / generation tasks
Dropout Step one Generalization Noise regularization during training
Data augmentation Step one Generalization Must preserve label semantics
L2 reg / AdamW Step 1/3 Generalization Prefers functions with smaller parameter values
Semi-supervised Step one Generalization Exploits unlabeled data

7.1 Training troubleshooting quick check

When a training anomaly appears, first diagnose the problem from the curves and then decide what to change. Changing only a few factors at a time and recording the data, random seeds and evaluation protocol makes results easy to compare.

Symptom Check first Changes to try
Training loss barely decreases Data and labels, learning rate, initialization, and whether gradients are zero Lower or raise the learning rate, check normalization and activation functions, and first test whether a small dataset can be overfit
Loss quickly becomes NaN or spikes violently Learning rate, mixed-precision overflow, exploding gradients Use BF16 / loss scaling, enable gradient clipping, and check for abnormal samples
Training loss is low but validation loss is high Data volume, data splits, label leakage, overfitting Augment the data, add weight decay or dropout, and reduce model capacity
Both training and validation loss are poor Representational capacity, input preprocessing, task definition or annotation quality Increase model size or training steps, improve features and labels, and confirm that the evaluation metrics are reasonable
Training is slow or GPU memory is insufficient Sequence length, batch size, data loading and operator utilization Mixed precision, gradient accumulation, checkpointing, FlashAttention or parameter-efficient fine-tuning

8. Common experimental benchmarks

Benchmarks are tools for measuring progress. This section reviews the classic benchmarks by field: MNIST/ImageNet for CV, GLUE/MMLU for NLP, plus coding, mathematical reasoning and long-context tasks. Models, data, prompts and evaluation scripts differ between papers, so the scales and historical results in the tables are meant to build intuition and should not be used directly as cross-paper rankings.

8.1 Computer vision benchmarks

MNIST

Property Details
Release year 1998
Size 70,000 handwritten digit images (28×28, grayscale)
Classes 10 (digits 0-9)
SOTA accuracy >99.8% (essentially saturated)
Features The most classic entry-level benchmark, suited to verifying basic methods

CIFAR-10 / CIFAR-100

Property CIFAR-10 CIFAR-100
Release year 2009 2009
Size 60,000 color images (32×32) 60,000 color images (32×32)
Classes 10 100
Suited to Regularization, data augmentation and architecture verification Fine-grained classification

The effects of Dropout, batch normalization, ResNet and data augmentation have all been extensively verified on this benchmark.

ImageNet(ILSVRC)

Property Details
Release year 2010
Size 1.2 million training images, 50,000 validation images
Classes 1,000
Features The industrial-grade deep learning benchmark and the main battlefield in the history of CNNs

Evolution of ImageNet Top-1 accuracy (see the table in Section 4): AlexNet (56.5%) → VGG (71.5%) → ResNet (76.0%) → EfficientNet (84.4%). These numbers come from classic models and training recipes of different eras and mainly illustrate the evolution of architectures.

8.2 NLP benchmarks

GLUE / SuperGLUE

Benchmark Released Number of tasks Purpose
GLUE 2018 9 Comprehensive evaluation of NLU tasks such as text classification, inference and similarity
SuperGLUE 2019 8 A harder successor after GLUE saturated, covering more challenging NLU tasks

BERT (2018) achieved leading results on GLUE when it was released; as the benchmark gradually saturated, researchers subsequently introduced the harder SuperGLUE.

MMLU

Property Details
Release year 2021
Size 57 subjects, about 16,000 multiple-choice questions
Coverage Mathematics, law, medicine, history, computer science, etc.
Purpose Evaluating an LLM’s breadth of knowledge and reasoning ability
Human level About 89.8% (experts)

MMLU is suited to observing a model’s knowledge coverage and multi-disciplinary reasoning, but scores are affected by prompts, calibration methods and training-data contamination, so the evaluation protocol must be fixed when comparing.

Language model perplexity

Penn Treebank (PTB) and WikiText are the standard benchmarks for traditional language models. The metric is perplexity (PPL): lower is better and means the model is more certain when predicting the next token. PPL remains suitable for comparing language models under the same corpus, tokenization and modeling objective, but it cannot by itself represent knowledge, coding or reasoning ability.

Code benchmarks

These evaluate LLM coding ability and are required for almost every frontier model release.

Benchmark Released Size Task format Evaluation method
HumanEval 2021 (OpenAI) 164 problems Python function completion Unit-test pass@1
MBPP 2021 (Google) 974 problems Basic Python problems Unit tests
HumanEval+ 2023 Extends HumanEval Adds about 80× more test cases Stricter pass@1
LiveCodeBench 2024 Continuously updated The latest LeetCode problems Prevents training-data contamination
SWE-bench 2023 2,294 problems Real GitHub repository bug fixes Run the test suite
SWE-bench Verified 2024 500 problems Human-validated subset More reliable evaluation

Usage suggestions: HumanEval/MBPP suit function-level code generation, while SWE-bench is closer to real repository repair. When reporting results, also state the number of samples, the test-set version and whether tool calls are allowed.

Mathematical reasoning benchmarks

Benchmark Released Difficulty Features
GSM8K 2021 Grade-school word problems 8.5K problems, testing multi-step arithmetic reasoning
MATH 2021 Competition level (AMC/AIME) 12.5K problems in the style of math olympiads
AIME Updated annually American Invitational Mathematics Examination Avoids training contamination; the main battlefield for frontier models
Putnam — Undergraduate mathematics competition Extremely difficult, testing top-level reasoning

GSM8K focuses on basic multi-step arithmetic, while MATH and AIME demand competition-level mathematical reasoning. As model capabilities improve, training-data contamination, sampling strategies and answer-verification methods significantly affect performance, so a single score is not enough.

Long-context benchmarks

As context windows expand from 2K to 32K to 128K to 1M, simple perplexity can no longer reflect a model’s long-context ability, and dedicated benchmarks have emerged:

Benchmark Released Evaluation format Focus
Needle-in-a-Haystack 2023 Hide a key piece of information in a long document and require the model to recall it Retrieval accuracy as a function of position and length
LongBench 2023 21 tasks in 6 categories (QA, summarization, code, etc.) Multi-dimensional comprehensive evaluation
RULER 2024 13 synthetic tasks with controllable, scalable difficulty Tests multiple long-context capabilities
∞BENCH 2024 Average of 200K tokens Tests realistic ultra-long scenarios
LOFT 2024 Long-document RAG and tool calls Realistic agent scenarios

Usage suggestions: Needle-in-a-Haystack mainly tests retrieval and cannot replace real long-document QA. Real evaluations should also consider position, document length, question type, retrieval interference and the accuracy of answer citations.


9. Efficient training and inference

9.1 Mixed-precision training

Mixed-precision training uses formats such as FP16 / BF16 for suitable operations, while retaining higher precision for sensitive computations, accumulation, or master weights. Memory savings and speedups depend on hardware, operator coverage, activations, and optimizer states. Halving a tensor’s byte count does not imply halving total memory use.

Format Exponent bits Mantissa bits Advantages Disadvantages
FP32 8 23 Most stable Large memory footprint
FP16 5 10 Fast Small numerical range and prone to overflow
BF16 8 7 Same numerical range as FP32, stable Slightly less precise than FP16
FP8 E4M3 / E5M2 4 / 5 3 / 2 Higher matrix-multiplication throughput and less storage and transfer on supported hardware Requires scaling and quantization strategies; more sensitive to errors and overflow

BF16 (Brain Float 16) has a wider exponent field (8 bits versus 5 in FP16), giving it a larger numerical range. It has become the preferred low-precision format for large language model training.

FP8: training precision as part of system design

FP8 training depends on scaling, accumulation precision, and operator selection. The DeepSeek-V3 report uses fine-grained scaling and FP8 inputs for most Linear GEMMs, while retaining higher precision for normalization, routing, embeddings, output heads, and other sensitive components, and increasing precision for parts of the accumulation. Technical report

Long training runs can reveal problems that short experiments miss. Scaling FP8 Training to Trillion-Token LLMs analyzes SwiGLU outlier amplification over trillion-token runs and introduces Smooth-SwiGLU. FP8 should therefore be evaluated using loss, gradient norms, and downstream quality over the full training run, in addition to throughput. Simply casting BF16 tensors to FP8 does not establish equivalent results. Original paper

9.2 Gradient clipping

Occasional exploding gradients (loss spikes) during large-model training can crash the run. Gradient clipping prevents this by capping the L2 norm of the gradient:

\[g \leftarrow g \cdot \min\left(1, \frac{\tau}{\|g\|_2}\right)\]

The usual threshold is $\tau = 1.0$, the standard companion of AdamW.

9.3 Gradient checkpointing

During training, backpropagation needs all the intermediate activations from the forward pass to compute gradients. In deep networks, activation memory can be several times the size of the parameters themselves, becoming the main GPU-memory cost of large-model training.

Gradient checkpointing (Chen et al., 2016) retains activations only at selected checkpoint layers. During backpropagation, it recomputes the missing intermediate activations with another forward pass, trading approximately 33% additional computation for activation memory on the order of $O(\sqrt{N})$.

Item Standard training Gradient checkpointing
Activation memory $O(N)$ $O(\sqrt{N})$
Forward computation 1× 1×
Backward computation 1× ~2× (recomputes the forward pass)

Combining gradient checkpointing with mixed precision, ZeRO and tensor parallelism is the standard memory-optimization recipe for training models with hundreds of billions of parameters (gradient_checkpointing=True in HuggingFace Transformers, torch.utils.checkpoint in PyTorch).

9.4 FlashAttention

The bottleneck of standard attention is that the $N \times N$ attention matrix must be written to GPU HBM. The memory complexity is $O(N^2)$, which is extremely expensive for long sequences.

Version First public release Main optimization target Key contribution
FlashAttention 2022 Attention memory traffic Tiling and online softmax avoid explicitly storing the full attention matrix
FlashAttention-2 2023 GPU parallelism and work partitioning Fewer non-matmul operations and better thread-block and warp partitioning
FlashAttention-3 2024 Hopper / H100 Asynchronous pipelines overlap matrix multiplication and softmax, with an FP8 path
FlashAttention-4 2026 Blackwell / B200 Uses Tensor Memory, asynchronous MMA, and 2-CTA mode, implemented in CuTe-DSL

FlashAttention equivalently rearranges standard dense softmax attention, while still introducing finite-precision numerical error. It reduces IO and intermediate memory use, but does not change dense attention’s computational complexity from $O(N^2)$ to $O(N)$.

FlashAttention-4 results need their comparison conditions: On B200 with BF16 and the tested shapes, the paper reports up to roughly 1.3× speedup over cuDNN 9.13 and 2.7× over Triton. These are kernel results for specific versions and hardware, not end-to-end model-training speedups. Results from different generations should not be combined across hardware into a single speedup ranking. Original paper


10. Model compression and adaptation

10.1 Mixture-of-Experts (MoE)

The core idea of Mixture of Experts (MoE): split a large FFN into $N$ independent “experts”, and let a router / gating network select the top-$k$ experts to compute for each token. Activating only some of the experts means that the parameter count is huge, but only a small part of it is used each time.

flowchart LR
    X["Token<br/>hidden state"] --> R["Router<br/>(small MLP)"]
    R -->|"top-2 scores"| S["Softmax weighting"]
    X --> E1["Expert 1<br/>FFN"]
    X --> E2["Expert 2<br/>FFN"]
    X --> EN["Expert N<br/>FFN"]
    E1 -.->|"selected"| S
    E2 -.->|"selected"| S
    S --> Y["Weighted output"]
    style E1 fill:#c4e1ff
    style E2 fill:#c4e1ff
    style EN fill:#f0f0f0

Core metrics

Metric Meaning Example (Mixtral 8×7B)
Total parameters Sum of all expert parameters 47B
Active parameters Parameters actually computed in each forward pass 13B
Number of experts $N$ 8
Top-k Number of experts activated per token 2

Representative work

Model Year Total parameters Active parameters Expert structure
Switch Transformer 2021 1.6T — Top-1 routing
GShard 2020 600B — Top-2 routing
Mixtral 8×7B 2023 47B 13B 8 experts, top-2
DeepSeek-V3 2024 671B 37B Each MoE layer has 1 shared expert and 256 routed experts; each token selects 8 routed experts

Key challenge: Load balancing prevents tokens from concentrating on a few experts, often through auxiliary losses or capacity constraints. Total and active parameter counts are not reported uniformly across models; routing strategies and FFN sizes also matter. See the technical report for DeepSeek-V3’s configuration above and its load-balancing design.

10.2 Parameter-efficient fine-tuning

As the parameter counts of pretrained models grow to tens or hundreds of billions, the GPU memory and storage costs of full fine-tuning tend to be high. Parameter-efficient fine-tuning (PEFT) trains only a very small number of additional parameters (usually < 1%) and can approach the quality of full fine-tuning.

Comparison of mainstream PEFT methods

Method Year Trainable parameter ratio Core idea Features
Adapter 2019 ~3% Inserts a small bottleneck MLP in each layer Adds extra computation at inference
Prefix Tuning 2021 <1% Prepends learnable “soft prompts” to the input The original model is unchanged; results are sensitive to the task
Prompt Tuning 2021 <0.1% Adds soft prompts only at the embedding layer Lightest weight; works well only on very large models
LoRA 2021 0.1–1% Injects low-rank updates $\Delta W = BA$ into the weight matrices Widely used strong baseline
QLoRA 2023 Same as LoRA 4-bit quantized base model + LoRA fine-tuning Can fine-tune a 65B model on a single 24GB GPU
DoRA 2024 Slightly more than LoRA Decomposes weights into direction and magnitude and updates them separately Better quality than LoRA in low-rank settings

LoRA core formula

For a pretrained weight $W_0 \in \mathbb{R}^{d \times k}$, LoRA freezes $W_0$ and trains only a low-rank decomposition $B \in \mathbb{R}^{d \times r}, A \in \mathbb{R}^{r \times k}$ ($r \ll \min(d,k)$, typically $r=8, 16, 64$):

\[W = W_0 + \Delta W = W_0 + BA\]

Forward computation: $h = W_0 x + BAx$.

Advantages:

  • Training: only $B, A$ need gradients and optimizer state, so GPU memory requirements drop to roughly $\sim 1/100$
  • Storage: each task only needs to save $BA$ (on the order of MB), so one base model can load LoRA adapters for hundreds of tasks
  • Inference: $BA$ can be merged into $W_0$, with zero inference overhead
  • Task switching: load different LoRA weights to switch scenarios within seconds

LoRA/QLoRA are common strong baselines for fine-tuning open-source large models, but the best choice still depends on the task, base model, quantization method and available GPU memory.

10.3 Quantization

Quantization compresses model weights (and activations) from FP16/BF16 to INT8/INT4 or even lower bit widths, trading a small loss of accuracy for a model size reduced to 1/4–1/8 and an inference speedup of 2–4×.

Quantization types

Type When quantization is applied Representative methods Features
PTQ (post-training quantization) After training is complete GPTQ, AWQ, SmoothQuant No retraining required; finishes in a few hours
QAT (quantization-aware training) Quantization is simulated during training LSQ, DoReFa Highest accuracy, but requires retraining
Mixed precision Key layers keep high precision Most deployment scenarios An engineering trade-off

Modern quantization methods

Method Year Bit width Core idea
GPTQ 2022 INT4/INT3 Minimizes quantization error layer by layer (a second-order method)
AWQ 2023 INT4 Activation-aware: protects the weights that matter most for the output from quantization error
SmoothQuant 2023 W8A8 “Migrates” activation outliers into the weights, making quantization easier
bitsandbytes 2022 INT8/NF4 The standard library in the HuggingFace ecosystem; supports QLoRA
FP8 2022 FP8 Natively supported by H100 hardware, usable for both training and inference

Rule of thumb: INT4 can often cut GPU memory use significantly, but the quality loss depends on the model, calibration data, quantization granularity and inference kernel; measure latency, throughput and accuracy on the target task before deployment.

10.4 Knowledge distillation

Knowledge distillation (Hinton et al., 2015) transfers the capabilities of a large “teacher” model to a small “student” model, so that the small model approaches the large model’s performance at deployment.

Core idea: soft labels

A one-hot hard label only tells the student “this picture is a cat”, whereas the soft probability distribution output by the teacher tells it “this picture is 90% cat, 8% dog, 1% fox and 0% car”. The relative relationships between classes (“a dog is closer to a cat than a fox is”) carry the teacher’s knowledge.

Distillation loss:

\[\mathcal{L}_{KD} = \alpha \cdot \mathcal{L}_{CE}(y^*, p_s) + (1-\alpha) \cdot T^2 \cdot \mathcal{L}_{KL}(p_t^{(T)} \| p_s^{(T)})\]

Where:

  • $p_t^{(T)} = \text{softmax}(z_t / T)$ is the teacher’s output softened by temperature $T$ (the distribution is smoother for $T > 1$)
  • $p_s^{(T)}$ is the corresponding softened student output
  • $\alpha$ balances hard-label supervision and soft-label distillation

Three distillation paradigms

Paradigm What is transferred Representative work
Response-based (output distillation) The final logit distribution Hinton 2015, DistilBERT
Feature-based (intermediate-layer distillation) Hidden-layer activations, attention maps FitNets, TinyBERT
Relation-based (relational distillation) Relations between samples, relations between layers RKD, CRD

Representative work

Model Year Teacher → student Effect
DistilBERT 2019 BERT-base → 6 layers 40% fewer parameters, retaining 97% of performance
TinyBERT 2019 BERT → 4 layers 87% fewer parameters, 9× faster
MobileBERT 2020 BERT-large → compact model On-device inference
MiniLM 2020 BERT → small Transformer Distills attention and hidden representations

For LLMs, distillation can act on logits, hidden representations, preference data or generated trajectories. It is often combined with quantization, pruning or parameter sharing, but the capability gap between teacher and student, the data distribution and the generation temperature all affect the final result.


11. Frontier architectures and large models

11.1 RLHF

RLHF (Reinforcement Learning from Human Feedback) was key to ChatGPT’s success: it aligns a pretrained large model, moving it from merely “speaking” to “following instructions and matching human preferences”.

flowchart LR
    PT["Pre-trained LLM<br/>(GPT-3)"] --> SFT["Step 1: SFT<br/>Supervised fine-tuning"]
    SFT --> RM["Step 2: Train a reward model<br/>Humans rank multiple answers"]
    RM --> PPO["Step 3: PPO reinforcement learning<br/>Maximize rewards"]
    PPO --> ALIGNED["Aligned model<br/>(ChatGPT)"]
    style PT fill:#e1f5ff
    style ALIGNED fill:#c4ffc4

The three-stage process

Stage Data Goal
SFT (supervised fine-tuning) Human-written (prompt, high-quality answer) pairs Teaches the model basic instruction-following
Reward model training Human rankings of multiple responses to the same prompt Learns a model that scores answers
PPO reinforcement learning The SFT model generates answers, the RM scores them, and PPO optimizes the policy Makes the model’s outputs maximize the RM reward

Alternatives to RLHF

method Year Relative to RLHF
DPO (Direct Preference Optimization) 2023 Expresses preference learning as a directly optimizable loss, without a separately trained RM or online PPO; parameters are still trained through gradients
IPO / KTO 2023 / 2024 Modify the preference objective or use desirable / undesirable feedback, respectively; their data requirements differ from DPO
RLAIF 2023 Replaces human preference labeling with AI (such as Claude)
Constitutional AI 2022 Proposed by Anthropic: a set of “constitutional” principles guides the model’s self-critique

DPO has become a common preference optimization method because the training process is simple and there is no need to separately train the reward model. PPO, DPO, rejection sampling, and other methods can also be used in combination; the specific training process of closed-source models is often not fully disclosed.

From preference alignment to reasoning training: GRPO and RLVR

GRPO (Group Relative Policy Optimization) was introduced in DeepSeekMath (2024). It samples a group of responses to the same question and uses within-group reward statistics to construct relative advantages, retaining a PPO-style clipped objective while removing the separately trained value / critic network. For the case of one outcome reward per response, the advantage can be illustrated as:

\[A_i = \frac{r_i-\operatorname{mean}(r_1,\ldots,r_G)}{\operatorname{std}(r_1,\ldots,r_G)+\epsilon}\]

Here, $G$ is the number of samples per question, $r_i$ is the reward for response $i$, and $\epsilon$ provides numerical stability. Removing the critic does not remove sampling costs: generating multiple responses per question can dominate compute, and groups with identical rewards provide little relative discrimination. DeepSeekMath

RLVR (Reinforcement Learning with Verifiable Rewards) uses verifiable outcomes as rewards, such as checked mathematical answers or code tests. It describes the reward source, while GRPO / PPO describe policy-training algorithms; they can be combined. DeepSeek-R1 (2025) trains reasoning with rule-based rewards and combines cold-start data, reinforcement learning, and other training stages. R1-Zero and the full R1 pipeline should be distinguished. Technical report

Approach Data and signal Main role Practical considerations
Offline preference learning such as DPO Fixed preferred responses or feedback Learn preferences and response style Data coverage, reference policy, and distribution shift
GRPO + verifiable rewards Multiple responses from the current policy, scored by rules / tests Improve reasoning on verifiable tasks Generation costs, reward design, and within-group reward variance
DAPO (2025) Large-scale online reasoning RL Improves clipping, dynamic sampling, token-level loss, and reward handling for overlong responses Reproduction requires the full sampling and training system

This also completes the three-step framework: RLVR changes the training signal; GRPO / DAPO define policy objectives and the sampling and update procedure; underlying parameter updates still require optimizers such as AdamW. These methods operate at a different level from the gradient optimizers in §5. Gains on verifiable-task rewards should also be evaluated on independent test sets and other tasks to assess generalization.

11.2 Generative model: from AE to GAN

A discriminative model learns \(p(y\mid x)\) (predicting a label from the input), whereas a generative model learns the data distribution \(p(x)\) itself, so it can sample new examples “out of nothing”. The four major families of deep generative models evolved along the trade-off between “sample quality, training stability and latent-space controllability”: autoencoder (AE) → variational autoencoder (VAE) → generative adversarial network (GAN) → diffusion model (covered in the next section).

Autoencoder (AE)

The encoder $f$ compresses the input into a low-dimensional latent vector $z$, the decoder $g$ reconstructs $\hat{x}$, and the training objective is the reconstruction error \(\mathcal{L} = \|x - g(f(x))\|^2\). An AE is good at dimensionality reduction, denoising and representation learning, but its latent space has no probabilistic constraint and is irregularly distributed, so you cannot sample from it to generate new examples. It is essentially a compressor, not a true generative model.

Variational autoencoder (VAE)

A VAE adds probabilistic structure to the latent space: the encoder outputs a distribution \(q_\phi(z\mid x)\) (rather than a single point) and constrains it to be close to the prior \(p(z)=\mathcal{N}(0,I)\). Training maximizes the evidence lower bound (ELBO):

\[\mathcal{L}_{\text{ELBO}} = \mathbb{E}_{q_\phi(z|x)}[\log p_\theta(x|z)] - D_{KL}\big(q_\phi(z|x)\,\|\,p(z)\big)\]

The first term is the reconstruction term, and the second is the regularizer that pulls the latent distribution towards the prior. The reparameterization trick \(z = \mu + \sigma \odot \epsilon\) (\(\epsilon \sim \mathcal{N}(0,I)\)) makes the sampling differentiable so gradients can flow back. After training, samples are generated by sampling directly from the prior \(p(z)\). VAE sampling is stable and the latent space is continuous and interpolable, but the generated images are blurry.

Generative adversarial network (GAN)

A GAN uses an adversarial game to bypass the explicit likelihood: the generator $G$ maps noise to fake samples, the discriminator $D$ learns to tell real from fake, and the two alternately optimize a minimax objective:

\[\min_G \max_D \ \mathbb{E}_{x \sim p_{\text{data}}}[\log D(x)] + \mathbb{E}_{z \sim p_z}[\log(1 - D(G(z)))]\]

GAN images are sharp and high-quality (StyleGAN’s faces are almost lifelike), but training is notoriously unstable: mode collapse, games that are hard to converge, and the need for many tricks (WGAN, spectral normalization, etc.) to stabilize them.

The four families at a glance

Family Latent variable Training objective Sampling Stability Typical shortcoming
AE Deterministic Reconstruction error ✗ Cannot generate High Irregular latent space
VAE Probabilistic (Gaussian) ELBO (reconstruction + KL) One step, fast High Somewhat blurry outputs
GAN Noise → sample Adversarial minimax One step, fast Low Mode collapse
Diffusion Gradually added noise Denoising regression Multi-step iteration, slow High Slow inference
flowchart LR
    AE["AE<br/>Compressed reconstruction"] --> VAE["VAE 2013<br/>Probabilistic latent space"]
    VAE --> GAN["GAN 2014<br/>Adversarial game"]
    GAN --> DIFF["Diffusion 2020<br/>Denoising generation"]
    style AE fill:#e1f5ff
    style DIFF fill:#c4ffc4

The diffusion model combines the training stability of the VAE with the high quality of the GAN and has become the mainstream of generative modeling; the next section expands on it.

11.3 Diffusion models

The diffusion model is the most successful generative paradigm in deep learning in recent years, powering landmark systems such as Stable Diffusion, DALL-E 3, Midjourney and Sora. Alongside the Transformer, it is another pillar of deep generative modeling.

Core idea: adding and removing noise

flowchart LR
    X0["x₀<br/>Clean image"] -->|"Forward: gradually add noise"| XT["xₜ<br/>Pure noise"]
    XT -->|"Reverse: learn to denoise"| XM["x̂₀<br/>Generated image"]
    style X0 fill:#c4ffc4
    style XT fill:#ffcccc
    style XM fill:#c4e1ff

Forward (diffusion) process: Gaussian noise is gradually added to real data, and after $T$ steps (usually 1000) it becomes approximately a standard normal distribution:

\[q(x_t | x_{t-1}) = \mathcal{N}(x_t; \sqrt{1-\beta_t} \cdot x_{t-1}, \beta_t I)\]

Because of its closed-form properties, $x_t$ can be obtained from $x_0$ in a single step: \(x_t = \sqrt{\bar{\alpha}_t} \cdot x_0 + \sqrt{1-\bar{\alpha}_t} \cdot \epsilon\), where \(\bar{\alpha}_t = \prod_{s=1}^t (1-\beta_s)\).

Reverse (denoising) process: A neural network \(\epsilon_\theta(x_t, t)\) is trained to predict the noise added at each step. The training loss is:

\[\mathcal{L} = \mathbb{E}_{t, x_0, \epsilon}\left[\|\epsilon - \epsilon_\theta(x_t, t)\|^2\right]\]

Inference: sample $x_T$ from the standard normal distribution, then use \(\epsilon_\theta\) to remove the noise step by step; after $T$ steps you obtain the generated sample.

Why diffusion beat GANs

Dimension GAN Diffusion
Training stability Two-network game; mode collapse is easy Single-network regression; very stable
Sample diversity Mode collapse (only a few kinds of samples are generated) Covers the data distribution thoroughly
Generation quality High single-sample quality but unstable training Typically achieves higher quality and broader coverage
Inference speed Single forward pass (fast) Multi-step iteration (slow)
Controllable generation Conditional control is more complex Naturally fits classifier-free guidance

Representative work

Model Year Key innovation
DDPM 2020 The modern revival of diffusion models, with a concise training objective
DDIM 2020 Deterministic sampling, cutting the number of inference steps from 1000 to 50
Classifier-Free Guidance 2022 Enables conditional control without a classifier
Latent Diffusion / Stable Diffusion 2022 Runs diffusion in the VAE latent space, cutting compute by 10×
DALL-E 2 2022 CLIP + diffusion, a breakthrough in text-to-image generation
Imagen 2022 Large text encoder + cascaded diffusion
ControlNet 2023 Precise control by pose, edges or depth
SDXL 2023 Open-source SOTA image generation
Sora 2024 Video diffusion model with a DiT architecture
Stable Diffusion 3 / Flux 2024 Rectified Flow + MMDiT

From UNet to DiT

Early diffusion models used a UNet (an encoder-decoder CNN with skip connections) as the denoising network. In 2022, DiT (Diffusion Transformer, Peebles & Xie, 2022) replaced the UNet with a Transformer; Sora and Stable Diffusion 3 both use DiT. This once again confirms the trend of the Transformer unifying multiple fields.

11.4 State space model and Mamba

The Transformer’s attention mechanism has $O(N^2)$ computational and memory complexity in sequence length $N$, which is extremely expensive for long sequences. State space models (SSMs) offer an alternative architectural route with linear complexity.

SSM basics

SSMs originate in control theory and map an input sequence $x(t)$ to an output sequence $y(t)$ through a hidden state $h(t)$:

\(h'(t) = Ah(t) + Bx(t)\) \(y(t) = Ch(t)\)

Here $A \in \mathbb{R}^{N \times N}$ is the state transition matrix and $B, C$ are projection matrices. After discretization (step size $\Delta$, zero-order hold), the recurrent form is:

\[h_t = \bar{A} h_{t-1} + \bar{B} x_t, \quad y_t = C h_t\]

Here $\bar{A} = e^{\Delta A}$ and $\bar{B} = (\Delta A)^{-1}(e^{\Delta A} - I) \cdot \Delta B$.

SSMs combine the advantages of RNNs and CNNs:

Mode Equivalent form Advantages
Training Equivalent to a 1D convolution ($y = x * k$, where $k$ is the convolution kernel) Fully parallelizable
Inference Equivalent to a recurrence (RNN style) $O(1)$ memory and constant time per step

S4: structured state spaces

S4 (Gu et al., 2021) is the first practical deep SSM: by restricting the matrix $A$ to a HiPPO-initialized diagonal-plus-low-rank (DPLR) structure, it makes long-sequence training efficient and feasible, significantly surpassing Transformers on long-sequence benchmarks such as Long Range Arena.

Limitation of S4: the $A, B, C$ matrices are fixed across all time steps (time-invariant), so it cannot dynamically choose what to attend to or ignore based on the input content; it is essentially a static filter.

Mamba (2023)

The core innovation of Mamba (Gu & Dao, 2023) is selectivity: $B, C$ and the step size $\Delta$ are made input-dependent:

\[\Delta, B, C = \text{Linear}(x_t)\]

This lets the model dynamically adjust its state update and forgetting rate according to the content of the current token. Under common parameterizations, $\Delta$ affects the state decay and input injection strength after discretization, so it cannot be simply equated with a fixed “memory switch”.

Hardware-aware parallel scan: A parallel prefix scan is applied directly to the recurrence $h_t = \bar{A} h_{t-1} + \bar{B} x_t$, and the whole computation is done in GPU SRAM to avoid frequent write-backs to HBM. Memory complexity is $O(N)$, and training speed is comparable to a Transformer’s.

Mamba block structure (per layer):

flowchart LR
    X["Input x"] --> L1["Linear projection × 2"]
    L1 --> |"main path"| CONV["1D convolution<br/>(short-range local perception)"]
    CONV --> ACT["SiLU activation"]
    ACT --> SSM["Selective SSM<br/>(Δ,B,C from the input)"]
    SSM --> MUL["⊗ Gating"]
    L1 --> |"gated path"| MUL
    MUL --> L2["Linear projection"]
    L2 --> Y["Output y"]
    style SSM fill:#c4e1ff

Mamba 2(2024)

Mamba 2 (Dao & Gu, 2024) introduces state space duality (SSD), showing that Mamba’s selective SSM is a structured special case of linear attention. This allows optimization and implementation within a unified framework:

  • Matrix-multiplication blocks (tensor cores) replace the scan, making it 2–8× faster than Mamba 1
  • The state size can be expanded to larger dimensions, improving model capacity

Transformer vs. Mamba

Dimension Transformer Mamba
Sequence complexity (compute) $O(N^2)$ $O(N)$
Sequence complexity (memory) $O(N^2)$ (attention matrix) $O(N)$ (activations)
Inference KV cache $O(N)$ (grows with context) $O(1)$ (fixed-size hidden state)
Training parallelism Fully parallel Parallel scan (nearly fully parallel)
Global dependency modeling Explicit attention (strong) Implicit state propagation (optional)
Long-sequence performance Degrades significantly Stays linear
In-context retrieval Very strong Relatively weak (lossy state compression)

Representative work

Model Year Key innovations / applications
S4 2021 Structured SSM, SOTA on Long Range Arena
H3 2022 SSM + attention hybrid for language tasks
Mamba 2023 Selective SSM + hardware-aware algorithm; competes with Transformers in language modeling
Mamba 2 2024 SSD unified framework, 2–8× faster
Vision Mamba / VMamba 2024 Mamba for vision: image classification and segmentation
MambaByte 2024 Byte-level language modeling without a tokenizer
Jamba 2024 Alternating Mamba and Transformer layers (AI21 Labs), 47B parameters
Zamba 2024 Small-scale Mamba-Transformer hybrid with SOTA inference efficiency
RWKV 2023 RNN-like linear attention: Transformer-style training with RNN-style inference
RetNet 2023 Retentive attention proposed by Microsoft: parallel training and recurrent inference

Recent progress: combining recurrent memory and attention

Alternative architectures should distinguish pure SSMs, linear attention, and hybrid models. Finite-dimensional recurrent states help control decoding-cache size, but tasks differ in their requirements for exact retrieval, forgetting, and long-term memory. Hybrid designs retain some full attention while introducing cheaper recurrent layers.

Kimi Linear (2025) is one example: Kimi Delta Attention (KDA) extends Gated DeltaNet with fine-grained gating and is combined with MLA layers. It demonstrates an approach to balancing expressivity and long-context efficiency under a fixed training setup. Cache and throughput gains depend on the mixing ratio, sequence length, and implementation; the entire hybrid model should not be described as having exclusively linear computation. Original paper

11.5 Deep learning and LLMs

Deep learning (especially the Transformer) is the technical foundation of large language models (LLMs). The division of labor between this article and the LLM training survey is as follows:

This article focuses on The LLM training survey focuses on
General deep learning principles and classic architectures End-to-end details of the LLM engineering stack
Basic mechanisms of CNN / RNN / Transformer Scaling laws (Chinchilla, Kaplan)
General methods for optimizers, normalization and regularization Pre-training data engineering and curriculum learning
Introductory concepts of MoE / PEFT / quantization / RLHF 3D parallelism (DP / TP / PP), ZeRO, FSDP
Overview of mainstream benchmarks Long-context training (YaRN, NTK interpolation)
Basic ideas of knowledge distillation KV cache optimization, continuous batching, PagedAttention, speculative decoding

For the engineering details of LLM pre-training, fine-tuning and inference, see:

LLM Training Survey

Covers topics such as scaling laws, distributed training, hybrid parallelism, the KV cache, GQA/MQA implementation internals, long-context extrapolation and inference acceleration.


12. Summary

One main thread runs through the whole article: the three-step machine learning framework (loss → architecture → optimization) × two core goals (optimization vs. generalization). Every training technique falls precisely into one cell of this 3×2 coordinate system. See the method taxonomy table in Section 7 for details.

Core conclusion:

Effective deep learning training requires matching the intervention to the problem, rather than adding more techniques. Inspect training loss first: if it does not decrease, address optimization before adding Dropout. Low training loss with high validation loss points to generalization, where data augmentation and regularization become relevant. This diagnostic principle is a foundation for effective experimentation and recurs throughout the article.


13. Primary sources for this update

These sources support the additions made on 2026-10-28. Method years generally refer to first public release, rather than conference publication.

Feedback