1. Introduction

Deep learning (Deep Learning) is a machine learning method with multi-layer neural network as the core. AlexNet’s breakthrough in the ImageNet Challenge in 2012 accelerated the popularity of deep learning in fields such as computer vision, natural language processing, speech recognition, and scientific computing. Representative systems such as GPT, BERT, Stable Diffusion, and AlphaFold are also built on deep neural networks.

The power of deep learning lies in its ability to automatically learn multi-level abstract feature representations from raw data (pixels, words, signals) without the need for manual feature engineering. However, training a high-performance deep network is not easy - problems such as vanishing gradients, overfitting, and learning rate adjustment have long troubled researchers, which has given rise to a set of systematic training techniques.

To understand deep learning, you need to grasp two levels at the same time: architecture design (how to build the network) and training method (how to effectively optimize the network). Both are indispensable. If you only know how to stack network layers like building blocks without understanding what problem each training technique solves, you will often fall into the dilemma of “adding Dropout will only make it worse” or “replacing Adam will not improve anything”.

This article aims to systematically sort out the core principles and key technologies of deep learning, and provide a searchable map for learning, reproduction, and research. The full text is developed according to the following context:

• §2–3 Basics: machine learning three-step framework, neurons and backpropagation
• §4 Architecture: The main line of evolution of MLP → CNN → RNN/LSTM → Transformer, and supplemented by graph neural network (GNN)
• §5 Training optimization: optimizer, learning rate scheduling, initialization, normalization, residual connection
• §6 Loss and regularization: Selection of various types of Loss, Dropout, data augmentation, AdamW
• §7 Method classification summary: Map all techniques to the “three steps × two goals” coordinate system
• §8 Benchmark: MNIST/CIFAR/ImageNet/GLUE/MMLU/HumanEval/Long Context and other classic reviews
• §9 Efficient training and inference: Engineering-level optimization technologies such as mixed precision, gradient clipping/checkpointing, and Flash Attention
• §10 Model compression and adaptation: MoE, LoRA/PEFT, quantification, knowledge distillation and other deployment-oriented lightweight methods
• §11 Cutting-edge architecture and large models: RLHF, generative model (AE/VAE/GAN), diffusion model, Mamba/SSM, LLM and other major paradigms in recent years

Reading Suggestion: Beginners can read in the order of §2–§6; when training the model, give priority to §5–§7 and §9; when focusing on large models, focus on reading §4.3, §10 and §11. The accuracy, parameter count and year in the article are used to illustrate the historical context and are affected by the model version, data processing and evaluation protocol. When reproducing the experiment, the original paper or official report should prevail.

2. Overview of deep learning basics

2.1 What is deep learning?

Deep learning is a branch of machine learning, using Artificial Neural Network (ANN) as the foundation model. “Deep” usually means that the model contains multiple layers of nonlinear transformations; it does not have a fixed layer threshold that applies to all tasks. Stacking multiple layers enables the network to extract increasingly abstract features layer by layer.

The basic computing unit of a neural network is neuron (Neuron):

Here, $x_i$ is the input, $w_i$ is the weight, $b$ is the bias, and $f$ is the activation function. Multiple neurons are stacked into layers, and multiple layers are connected in series to form a deep network, which ultimately forms a complex mapping function from input to output.

Schematic diagram of

Universal Approximation Theorem

One of the important theoretical basis for the expressive ability of neural network is universal approximation theorem (Cybenko 1989, Hornik 1991):

When conditions such as continuity and non-polynomial activation functions are met, a single hidden layer feedforward network can approximate any continuous function with arbitrary accuracy on a compact set as long as the width is sufficient.

This conclusion answers the fundamental question of “Why can neural networks work?”, but there are also two key limitations that must be clarified:

The advantage of depth : Theoretically (Telgarsky 2016 ) There is a class of functions that deep networks can express with far fewer units than shallow networks. It should be noted that the theorem describes the expression ability and parameter efficiency. It does not guarantee that the optimization process will find these parameters, nor does it directly guarantee the generalization performance.

Three major driving factors for the explosion of deep learning

2.2 Machine learning three-step framework

Any machine learning method can be broken down into three steps. Understanding this framework is the basis for all subsequent training techniques:

1. Step 1 - Define Loss Function (loss function): Measure the difference between the model output and the correct answer, that is, “how to judge good or bad”
2. Step 2 - Determine the function search space (Model Architecture): Select the network structure and delineate the scope of candidate functions, that is, “where to search”
3. Step 3 - Optimization: Find the optimal function that minimizes Loss in the search space, that is, “how to search efficiently”

The ultimate goal of training is to find a function whose Loss is low on the training set (Training Set) and remains low on the validation set (Validation Set). Whether the training error can be reduced mainly reflects Optimization, and the gap between training error and verification error reflects Generalization; the two will affect each other in practice.

2.3 Two core goals

Deep learning training techniques are divided into two categories according to the problems they solve:

Judgment Principle: First look at the training Loss, and then look at the gap between the training and validation curves. When the training loss cannot be reduced, check the learning rate, initialization, normalization and data pipeline first; when the training loss is already low but the validation loss is significantly higher, then consider regularization and data augmentation. Actual problems may involve both types of factors.

2.4 Development Timeline

flowchart LR
    subgraph G1990 ["1990s laying foundation"]
        A["LeNet 1998<br/>CNN laying foundation"] --> B["Backpropagation<br/>Popularize"]
    end
    subgraph G2012 ["2012-2014 break out"]
        C["AlexNet 2012<br/>ImageNet breakthrough"] --> D["Dropout 2014"]
        D --> E["Adam 2014"]
    end
    subgraph G2015 ["2015-2017 deepen"]
        F["ResNet 2015<br/>Skip Connection"] --> G["BN 2015"]
        G --> H["AdamW 2017"]
    end
    subgraph G2018 ["2018-2022 pre-training"]
        I["BERT/GPT 2018<br/>pre-training paradigm"] --> J["Transformer<br/>Dominate"]
        J --> R["RoPE 2021<br/>Rotational position encoding"]
        R --> K["LoRA 2021<br/>Efficient fine-tuning of parameters"]
    end
    subgraph G2022 ["2022 so far large model"]
        L["Chinchilla<br/>2022 Scaling Laws"] --> M["ChatGPT 2022<br/>RLHF Alignment"]
        M --> N["Flash Attn 2/3<br/>2023-2024"]
    end
    B --> C
    E --> F
    H --> I
    K --> L

2.5 Main abbreviation

• ANN: Artificial Neural Network
• MLP: Multi-Layer Perceptron
• CNN: Convolutional Neural Network
• RNN: Recurrent Neural Network
• LSTM: Long Short-Term Memory (long short-term memory network)
• SGD: Stochastic Gradient Descent
• BN: Batch Normalization
• LN: Layer Normalization
• LR: Learning Rate
• PEFT: Parameter-Efficient Fine-Tuning
• SFT: Supervised Fine-Tuning
• RLHF: Reinforcement Learning from Human Feedback (reinforcement learning from human feedback)

3. Neural network basics

3.1 Multi-layer perceptron

Multi-Layer Perceptron (MLP) is the most basic form of neural network. It consists of an input layer, several hidden layers (Hidden Layer) and an output layer. Each layer is fully connected (Fully Connected):

Fully connected means that each neuron in layer $l$ is connected to all neurons in layer $l-1$, and the parameter count is the product of the number of neurons in the two layers. When the input dimension is extremely large (for example, the color image of 1000×1000 has 3 million dimensions after expansion), the number of fully connected parameters will exceed billions, which is not only difficult to train, but also extremely easy to overfitting. Specialized architectures such as CNN were proposed to solve this problem.

3.2 Activation function

Activation functions introduce nonlinearity to neural networks, allowing them to fit complex functions. Without nonlinear activation, a multi-layer network is equivalent to a single-layer linear model.

The success of ReLU (Rectified Linear Unit) lies in the fact that the gradient in the positive area is always 1, which effectively alleviates the vanishing gradient problem of deep networks and enables stable training of networks with dozens or even hundreds of layers.

3.3 Backpropagation algorithm

Backpropagation is the core algorithm for training neural networks. Based on the chain rule, the gradient of Loss for each parameter is transferred from the output layer back to the input layer layer by layer:

flowchart LR
    Loss["Loss<br/>Calculate losses"] -->|"∂L/∂ŷ"| OutLayer["output layer<br/>Calculate gradient"]
    OutLayer -->|"∂L/∂h²"| H2["Hidden layer 2<br/>Calculate gradient"]
    H2 -->|"∂L/∂h¹"| H1["Hidden layer 1<br/>Calculate gradient"]
    H1 -->|"∂L/∂W¹"| Update["Parameter update<br/>gradient descent"]
    style Loss fill:#ff9999
    style Update fill:#99ff99

The Computation Graph automatically records the forward calculation path, and during back propagation the gradient is transferred along the path in the opposite direction. PyTorch’s autograd mechanism is based on this principle. Users only need to define forward calculations, and the framework automatically completes gradient calculations.

A minimal numerical example of backpropagation

Consider the simplest single hidden layer network: $y = w_2 \cdot \sigma(w_1 \cdot x)$, with loss $\mathcal{L} = \frac{1}{2}(y - t)^2$. Set $x=1, w_1=0.5, w_2=0.8, t=0.5$, activate $\sigma$ and take Sigmoid.

forward:

reverse (chain rule stepwise backtracking):

observes: the gradient $\partial \mathcal{L}/\partial w_2 \approx 10^{-3}$ near the output, the gradient $\partial \mathcal{L}/\partial w_1 \approx 10^{-4}$ near the input - even though there are only 2 layers, the gradients are attenuated by $\sim 3$ times. Every additional layer in the deep network requires one more multiplication of $\sigma’(\cdot) \leq 0.25$. This is the source of the vanishing gradient.

Three variations of gradient descent

There are three forms of gradient descent depending on the number of samples used per update:

Modern deep learning almost all uses Mini-batch SGD. The “SGD” spoken in the industry usually refers to the mini-batch version. The choice of batch size directly affects training stability, hardware utilization and final generalization - large batches converge quickly but generalization is slightly worse (Keskar et al., 2017 “On Large-Batch Training”), which is an important parameter adjustment dimension.

4. Classic network architecture

4.1 Convolutional Neural Network

Convolutional Neural Network (CNN) is a standard architecture for processing grid-shaped data (images, time series signals). CNN imposes two key constraints on MLP:

Receptive Field (receptive field): Each neuron only observes a local area of the input (such as the kernel of 3×3), rather than the entire image. Local patterns (edges, textures) in images can be detected using only local perception, without the need for a global view.

Parameter Sharing (parameter sharing): Similar neurons in different positions share the same set of parameters (filter/convolution kernel). The same pattern (such as a horizontal edge) appearing anywhere in the image should be processed by the same detector - this constraint brings translation equivariance (Translation Equivariance); only after pooling or global aggregation, the model will obtain a certain translation invariance.

Core hyperparameters of convolutional layers

output space size formula:

Regular convolution ($d=1$) simplifies to $\text{out} = \lfloor (\text{in} + 2p - k)/s \rfloor + 1$.

Special convolution

Pooling layer (Pooling)

Pooling is done on the feature map Downsampling , reduce the spatial size, expand the receptive field, and provide certain translation invariance. Pooling layer No parameters to learn .

The meaning of GAP: The fully connected layer of VGG-16 occupies the 123M parameters (accounting for the total parameters 89%); this part can be omitted directly after switching to GAP. This is because the parameters of ResNet and EfficientNet are much smaller than those of VGG one of the key reasons.

A typical CNN consists of alternating convolutional layers (extracting features) and pooling layers (downsampling), finally followed by a fully connected layer or GAP output prediction. The following is the structure of LeNet-5 (LeCun, 1998), the foundational work of CNN:

CNN architecture evolution comparison

Key Insight: ResNet-50 (25M parameters) outperforms VGG-16 (138M parameters) with only 18%; EfficientNet-B0 (5.3M) reaches the same accuracy as VGG, and the number of parameters is only 4%. More parameters ≠ higher performance, architectural design is crucial.

ResNet residual block structure

4.2 Recurrent Neural Network and LSTM

Recurrent Neural Network (RNN) is specially designed to process sequence data and passes information between time steps through the hidden state $h_t$:

RNN uses the same set of parameters ** $W_h, W_x$ at each time step, making it naturally adaptable to variable-length sequences. The following figure shows the “unfolding” of RNN in the time dimension:

flowchart LR
    x0["x₀"] --> h0["h₀"]
    x1["x₁"] --> h1["h₁"]
    x2["x₂"] --> h2["h₂"]
    x3["x₃"] --> h3["h₃"]
    h0 -->|"Wₕ"| h1
    h1 -->|"Wₕ"| h2
    h2 -->|"Wₕ"| h3
    h0 --> y0["y₀"]
    h1 --> y1["y₁"]
    h2 --> y2["y₂"]
    h3 --> y3["y₃"]
    style h0 fill:#e1f5ff
    style h1 fill:#e1f5ff
    style h2 fill:#e1f5ff
    style h3 fill:#e1f5ff

However, standard RNN faces serious long-range dependencies problem: when the gradient is backpropagated in the time dimension, the matrix $W_h$ needs to be multiplied $T$ times. When the maximum singular value of $W_h$ $< 1$, gradients decay exponentially with sequence length (vanishing gradients), $> 1$ exponentially explodes (exploding gradients), making it difficult for the model to remember distant contexts.

LSTM (Long Short-Term Memory, Hochreiter & Schmidhuber, 1997) solves this problem by introducing gating mechanism (Gating Mechanism):

LSTM maintains two states: hidden state $h_t$ (short-term memory) and cell state $c_t$ (long-term memory).

Internal data flow of a single LSTM cell:

flowchart LR
    subgraph INPUT ["input"]
        XT["xₜ"]
        HT1["hₜ₋₁"]
        CT1["cₜ₋₁"]
    end
    CONCAT["Splicing [hₜ₋₁, xₜ]"]
    XT --> CONCAT
    HT1 --> CONCAT
    CONCAT --> FGATE["oblivion door fₜ<br/>σ(W·)"]
    CONCAT --> IGATE["input gate iₜ<br/>σ(W·)"]
    CONCAT --> CAND["candidate c̃ₜ<br/>tanh(W·)"]
    CONCAT --> OGATE["output gate oₜ<br/>σ(W·)"]
    CT1 --> MUL1["× fₜ<br/>forget old memories"]
    FGATE --> MUL1
    IGATE --> MUL2["× c̃ₜ<br/>write new memory"]
    CAND --> MUL2
    MUL1 --> ADD["＋"]
    MUL2 --> ADD
    ADD --> CT["cₜ<br/>new cell state"]
    CT --> TANH["tanh"]
    TANH --> MUL3["× oₜ"]
    OGATE --> MUL3
    MUL3 --> HT["hₜ<br/>New hidden state"]
    style CT fill:#ffe1c4
    style HT fill:#c4e1ff
    style CT1 fill:#ffe1c4
    style HT1 fill:#c4e1ff

cell status update formula:

Why can gating alleviate vanishing gradients? Cell state update $c_t = f_t \odot c_{t-1} + \ldots$ is a additive path (compare to RNN’s multiplicative recursion $h_t = f(Wh_{t-1} + \ldots)$). When forgetting gate $f_t \approx 1$, $c_{t-1}$ is passed to $c_t$ with almost no attenuation, and the gradient can remain stable when back propagating along the cell state - similar to ResNet’s skip connection, but occurring in the time dimension rather than the depth dimension.

GRU: a simplified version of LSTM

Gated Recurrent Unit (GRU, Cho et al., 2014) simplifies the 3 gates of LSTM into 2 gates, and merges the cell state and hidden state:

Empirical conclusion: Chung et al., 2014 System experiments show that GRU and LSTM have equivalent performance on most tasks. GRU has fewer parameters and faster training, and is more commonly used in short and medium sequence tasks (conversation, emotion classification); LSTM still has advantages in tasks that require strict long-term memory (machine translation of long sentences, speech recognition).

Bidirectional RNN

The standard RNN can only utilize the information of past (from left to right). But for many tasks (named entity recognition, sentiment analysis, reading comprehension), the context of and future is equally critical - for example, to determine whether “bank” is “bank” or “river bank”, you need to look at the following words.

Bidirectional RNN (Schuster & Paliwal, 1997) runs RNN in two directions at the same time, output splicing:

flowchart LR
    x1["x₁"] --> fh1["→h₁"]
    x2["x₂"] --> fh2["→h₂"]
    x3["x₃"] --> fh3["→h₃"]
    fh1 --> fh2 --> fh3
    x1 --> bh1["←h₁"]
    x2 --> bh2["←h₂"]
    x3 --> bh3["←h₃"]
    bh3 --> bh2 --> bh1
    fh1 --> c1["[→;←] h₁"]
    bh1 --> c1
    fh2 --> c2["[→;←] h₂"]
    bh2 --> c2
    fh3 --> c3["[→;←] h₃"]
    bh3 --> c3
    style c1 fill:#c4e1ff
    style c2 fill:#c4e1ff
    style c3 fill:#c4e1ff

Limitations: Bidirectional RNN must see the entire sequence to calculate, cannot be used for autoregressive generation tasks (no future information when generating the next word). Therefore:

• is suitable for: sequence annotation, classification, NER, and reading comprehension (BiLSTM-CRF was the mainstream solution for NER in the years 2016–2018)
• is not suitable for: Decoder side of language model and machine translation

BERT’s MLM (Masked Language Model) can be seen as Transformer’s generalization of “bidirectional modeling” - ensuring that bidirectional attention does not leak the target through mask.

Teacher Forcing: Key techniques for training sequence generative models

There is a subtle choice when training a Seq2Seq / RNN Decoder - what is the input when decoding step $t$?

Problems with Teacher Forcing——Exposure Bias: The model has never “seen” its own wrong predictions during training. Once a certain step of prediction is biased during inference, it will go further and further wrong (the prototype of error accumulation/hallucination).

Modern solution:

• Scheduled Sampling: Gradually mixed into model prediction in the later stage of training
• Minimum Risk Training / REINFORCE: directly use the overall quality of the generated sequence as the training signal
• Transformer + Causal Masking: Compute all positions in parallel at once (see §4.3). It is still essentially Teacher Forcing, but the efficiency is much higher than RNN stepwise recursion.

Representative work: GRU (2014, a simplified version of LSTM), Seq2Seq (2014), Attention + LSTM (2015), Bidirectional LSTM (proposed by 2005, BiLSTM-CRF 2016 widely used)

4.3 Transformer

Transformer (Vaswani et al., 2017) has completely changed NLP and even the entire deep learning landscape. It completely abandons loops and convolutions, and only directly models the global dependence between any two positions in the sequence through the self-attention mechanism (Self-Attention).

Core Mechanism: Scaled Dot-Product Attention

• Query (Q): represents “what am I looking for” (such as the current word).
• Key (K): stands for “what characteristics do I have” (used to match Q).
• Value (V): represents “the information I want to transmit” (the content after successful matching).
• $\sqrt{d_k}$ Scaling: When the dimension is large, the dot product result may be extremely large, causing softmax to enter a region with extremely small gradients. Scaling keeps gradients stable.

A specific example: the sentence “The cat sat on the mat”

Let’s say the model is processing the verb “sat” and it needs to find “who sat down” (the subject). The weights below are for visual illustration only and are not the output of a real model:

• The dot product of Qofsat and Kofcat is the largest (matching “singular noun subject”), and the attention weight after softmax is $\approx 0.7$
• There is also a certain match (noun) between Qofsat and Kofmat, but the semantic role is wrong and the weight is $\approx 0.15$
• The final output of sat $\approx 0.7 \cdot V_\text{cat} + 0.15 \cdot V_\text{mat} + \ldots$ - mainly absorbs the semantics of cat

This is how Attention works: each token broadcasts a “query”, all tokens respond with “How many do I match”, and finally the aggregated information is weighted by the degree of match. Q/K/V are generated from three different linear projections $W_Q, W_K, W_V$ from the same input, this flexibility allows the model to learn a variety of complex information routing patterns.

Multi-Head Attention

Map the input to $h$ different subspaces to calculate the attention separately, and then splice the results: \(\text{MultiHead}(Q,K,V) = \text{Concat}(head_1, \ldots, head_h) W^O\) Intuition: Different “heads” can focus on different information at the same time (for example, one head focuses on grammatical relationships and the other on semantic relationships).

Schematic diagram of the

Attention variants: MHA → MQA → GQA → MLA

As the context of large models becomes longer, the GPU memory usage of KV Cache (which caches K and V of all historical tokens during inference) becomes a bottleneck. For this reason, the multi-head structure of Attention has evolved into multiple variants, the core of which are reducing the number of K/V heads to compress the KV Cache:

flowchart LR
    subgraph MHA ["MHA: every Q Head independent KV"]
        Q1["Q₁"] --> KV1["K₁,V₁"]
        Q2["Q₂"] --> KV2["K₂,V₂"]
        Q3["Q₃"] --> KV3["K₃,V₃"]
        Q4["Q₄"] --> KV4["K₄,V₄"]
    end
    subgraph GQA ["GQA: Group sharing KV(g=2)"]
        Q1G["Q₁"] --> KVA["K,V group A"]
        Q2G["Q₂"] --> KVA
        Q3G["Q₃"] --> KVB["K,V group B"]
        Q4G["Q₄"] --> KVB
    end
    subgraph MQA ["MQA: all Q Share 1 group KV"]
        Q1M["Q₁"] --> KVS["K,V Share"]
        Q2M["Q₂"] --> KVS
        Q3M["Q₃"] --> KVS
        Q4M["Q₄"] --> KVS
    end

Why GQA is often used: MQA presses K/V to the 1 group, which may cause quality loss; MHA’s KV Cache is larger. GQA is a compromise between quality and GPU memory - e.g. LLaMA-2 70B uses 8 groups, 64 Q heads, and the KV Cache is about MHA’s 1/8; Actual gains still depend on model and task.

MLA’s innovation: DeepSeek uses a low-rank matrix to compress K/V into a $\sim$128-dimensional latent space. Only the latent vector is cached during inference, and the complete K/V is restored when needed. GPU memory has significant advantages in extremely long contexts.

Key components

1. Positional Encoding

Why does need position encoding? The core of Transformer - the self-attention mechanism is permutation equivariant (Permutation Equivariant) when there is no position encoding.: The order of input tokens changes, and the output will also be rearranged in the same way. When calculating $O = \sum \alpha_i V_i$, the model has no additional clues to determine the order, so the sentences “you hit me” and “I hit you” cannot be reliably distinguished, and position information must be injected.

Absolute position encoding: Sinusoidal design The original Transformer constructs a position vector using predefined sine and cosine functions: \(PE_{(pos, 2i)} = \sin\left(\frac{pos}{10000^{2i/d_{model}}}\right)\) \(PE_{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{2i/d_{model}}}\right)\)

Intuition: multi-frequency pointer (Clock Hands) We can think of each pair of $(\sin, \cos)$ as a “pointer” on a two-dimensional plane.

• low dimension ($i$ is small): high frequency, the pointer rotates extremely fast. Similar to the “second hand” of a clock, a few Tokens will complete a circle.
• high dimension ($i$ is large): low frequency, pointer rotation is extremely slow. Similar to the “hour hand”, it may take tens of thousands of Tokens to make one revolution. Transformer is like observing a dashboard composed of dozens of pointers with different rotation speeds, thereby accurately locking the absolute position of the current Token in the sequence.

Mathematical properties: Modeling relative position The subtlety of the author’s choice of sine and cosine functions is that it allows the model to be expressed through linear transformations relative position . According to the angle formula of trigonometric functions, there is an equation that is only related to the relative distance $r$ The relevant transformation matrix $M_r$ , making: \(PE_{pos+r} = M_r \cdot PE_{pos}\) This means that the model can more easily capture the distance information between two words instead of just the absolute coordinates when calculating Attention.

RoPE Position coding: Rotary Position Embedding design

Absolute position encoding adds position information to the Input Embedding, which indirectly affects Attention; ALiBi forcibly subtracts the distance deviation from the Attention Score. RoPE chose another path: directly encodes the position information into the vector itself in a rotational manner before doing the inner product of Q and K.

Core idea: Use rotation instead of addition

For each two dimensions of the vector ($d_0, d_1$), RoPE treats it as a vector on a two-dimensional plane, and then rotates the corresponding angle according to the location of the token $m$:

The rotation matrix is a standard two-dimensional rotation matrix:

For the $d$-dimensional vector, all dimensions are grouped into two groups, a total of $d/2$ groups, each group uses a different base angle $\theta_i$:

This design is in line with Sinusoidal PE - different dimensions correspond to different frequencies, low dimensions rotate quickly and high dimensions rotate slowly, which together form a unique position fingerprint.

RoPE How to naturally encode relative position?

When Q is at position $m$ and K is at position $n$, the inner product of the two is:

The inner product result only relies on the relative distance $m-n$ rather than the absolute coordinates. This is the neat thing about RoPE - there is no need to explicitly model relative distances like in Relative PE, the geometric nature of the rotation guarantees this automatically.

The relative position properties of

Geometric Proof of Translation Invariance

Assume that “cat” is at position 1 and “fish” is at position 3, calculate the Attention value $A$. Now insert 100 unrelated tokens in front, “cat” changes to position 101, and “fish” changes to position 103. The relative distance between the two remains unchanged (still 2), and the rotation angle $(m-n)\theta$ in the inner product remains unchanged, so the Attention value is still equal to $A$.

The geometry is more intuitive: Q rotates $N\theta$, K rotates $N\theta$, both rotate synchronously, and the inner product (the cosine of the included angle) remains unchanged.

Compatibility of with engineering acceleration

RoPE only modified Q and K itself. The calculation process of Attention is exactly the same as the original version, so it is naturally compatible with all Attention acceleration technologies:

• Flash Attention: directly available without modifying the operator;
• KV Cache: Directly cache the rotated $\text{K}^{(m)}$, read it out and use it without injecting the position again.

This is also one of the key engineering reasons why RoPE ultimately won - it is not only effective, but also perfectly compatible with the entire engineering ecosystem.

Common misunderstandings clarified

Many people think that RoPE guarantees “the farther the distance between Q and K, the smaller the Attention” like ALiBi. Actually RoPE does not guarantee this - rotation will produce a periodic oscillation pattern, and Attention fluctuates in a zigzag manner with distance.

This is an advantage: RoPE allows the model to learn “although distant, but still highly relevant” Attention patterns (such as long-range dependence), while ALiBi hard suppresses long-range attention.

Evolution of position encoding:

2. Point-wise feedforward network (Point-wise FFN) After each attention layer, a fully connected block (usually $d_{model} \to 4d_{model} \to d_{model}$) is followed to introduce nonlinear transformation: \(\text{FFN}(x) = \text{max}(0, xW_1 + b_1)W_2 + b_2\)

3. Residual connection and normalization (Add & Norm) Each sub-layer uses residual connections and cooperates with layer normalization (LayerNorm). There are currently two mainstream layouts:

• Post-LN (original version): First calculate the sub-layer and then add the residual to make LN. Good performance but difficult to train at deep level.
• Pre-LN (commonly used in modern large models): Do LN first and then calculate the sub-layer, which is usually easier to train stably; whether Warmup is needed still depends on the model, optimizer and training settings.

The difference between Encoder and Decoder

Transformer uses an encoder-decoder architecture. The core difference between the two is the masking (Masking):

• Encoder: Bidirectional attention, each word can see all words in the sequence.
• Decoder: Use Masked Self-Attention to ensure that only the first $t-1$ words can be seen when generating the $t$ word (to prevent information leakage); it also contains Cross-Attention, used to pay attention to the output of the Encoder.

The specific form of causal mask (Causal Mask) - a lower triangular matrix of $n \times n$ (taking the sequence length 4 as an example):

implements: before softmax, set the attention score of the mask position to $-\infty$ (actually use a large negative number such as -1e9), and after softmax, the weight of these positions becomes 0. This ensures the causality of “the $t$ token can only see the position of $\leq t$”, and also realizes parallelization of Teacher Forcing during training - no longer recursive step by step like RNN, but forward prediction of the next token at all positions at once.

Attention visualization: What did the model really learn?

One of the greatest charms of Attention is Interpretability ——The attention weight of each head in each layer can be directly visualized as a heat map, revealing the “attention mode” of the model. Typical findings:

Tools such as BertViz (Vig, 2019) and attention-flow (Abnar & Zuidema, 2020) can interactively display these patterns. Clark et al., 2019 “What Does BERT Look At?” systematically analyzed BERT’s 144 headers and found that a large number of headers capture clearly interpretable syntax/semantic functions - this is important evidence that Transformer transcends the “black box” label.

RNN vs Transformer comparison

Representative work: BERT (2018, pure Encoder), GPT series (2018-to date, pure Decoder), T5 (2019, Encoder-Decoder), ViT (2020, treat image blocks as sequences).

Vision Transformer (ViT): Transformer enters the visual field

ViT (Vision Transformer, Dosovitskiy et al., 2020) proved for the first time that pure Transformer can directly defeat CNN in processing image tasks with sufficient data, ending the nearly ten-year dominance of CNN in the field of vision.

Core Design: Treat images as “sentences”

flowchart LR
    IMG["Original picture<br/>224×224×3"] --> PATCH["Chunking<br/>16×16 patch"]
    PATCH --> FLAT["flattened to 196 a<br/>768 dimensional vector"]
    FLAT --> EMBED["Patch Embedding<br/>(linear projection)"]
    EMBED --> POS["+ Position Embedding<br/>+ [CLS] token"]
    POS --> TRANS["Transformer<br/>Encoder × L"]
    TRANS --> CLS["take [CLS] output"]
    CLS --> HEAD["MLP Head<br/>Classification"]
    style IMG fill:#e1f5ff
    style HEAD fill:#c4ffc4

Key steps:

1. Patch Split: Cut the $224 \times 224$ image into $14 \times 14 = 196$ patches of $16 \times 16$
2. Patch Embedding: Each patch is flattened into a $16 \times 16 \times 3 = 768$ dimensional vector, and the token is obtained through linear projection
3. [CLS] Token: Add a learnable “[CLS]” token before the sequence, and its final output is used for classification
4. Position Embedding: Add learnable position encoding (ViT uses learnable PE instead of Sinusoidal)
5. Transformer Encoder: Standard Transformer stack, usually 12–32 layers
6. classification head: take [CLS] output and connect to MLP

Inductive Bias and Data Requirements for ViT

Empirical conclusion (ViT original paper):

• Data < 1M (such as ImageNet-1k): ViT is significantly worse than ResNet
• Data ~14M (ImageNet-21k): ViT ties ResNet
• Data ~300M (JFT-300M): ViT significantly outperforms

This verifies a core principle of deep learning: less inductive bias × more data = more flexible architecture and higher upper limit.

The evolution of ViT

ViT not only replaces CNN as the visual backbone, but more importantly allows images and text to share the same Transformer architecture - this is the basis for large multi-modal models such as CLIP, LLaVA, and GPT-4V to unify vision and language in a concise way.

4.4 Graph Neural Network (GNN)

CNN processes grid data and RNN/Transformer processes sequences, but in reality a large amount of data is a graph (Graph) structure - a user-item bipartite graph of social networks, molecular structures, knowledge graphs, and recommendation systems. This type of data has a variable number of nodes, no fixed order, and an uncertain number of neighbors. The regular structure assumptions of CNN/RNN no longer hold true. Graph Neural Network (GNN) is specially designed for this type of non-Euclidean data.

Core mechanism: Message Passing

The unified paradigm of GNN is neighborhood aggregation: each node repeatedly “collects neighbor information → aggregates → updates its own representation”. The update of layer $l$ can be written as:

Here, \(\mathcal{N}(v)\) is the neighbor set of node $v$, $\text{AGG}$ is the aggregation function (sum/mean/maximum/attention weighting) of permutation-invariant, and $\sigma$ is nonlinear. After stacking $L$ layers, each node can perceive the subgraph structure within $L$ hops - this is the same as CNN expanding the receptive field layer by layer.

flowchart LR
    U1["neighbor u₁<br/>express h(u₁)"] --> AGG["aggregation AGG<br/>Sum/mean/attention"]
    U2["neighbor u₂<br/>express h(u₂)"] --> AGG
    U3["neighbor u₃<br/>express h(u₃)"] --> AGG
    V0["node v<br/>old representation h(v)"] --> UPD["update<br/>σ(W·agg + B·h(v))"]
    AGG --> UPD
    UPD --> V1["node v<br/>new representation h(v)′"]
    style V0 fill:#c4e1ff
    style V1 fill:#c4ffc4
    style AGG fill:#ffe1c4

Classic GNN comparison

Task types and core challenges

GNN Three types of tasks: node level (node classification, such as determining the topic of a paper in a citation network), edge level (link prediction, such as recommendation system), graph level (whole graph classification, such as molecular property prediction).

Main challenge - Over-smoothing: When the number of layers is too deep, all nodes express convergence and lose discriminability. Therefore, GNN usually only uses the 2–4 layer, which is much shallower than CNN/Transformer. Residual connections, jump knowledge (JKNet), PairNorm, etc. are common mitigation methods.

The relationship between and Transformer: Transformer can be regarded as GNN defined on a fully connected graph - each token is a node, and self-attention aggregates all other nodes with attention weight. GAT’s neighbor attention has the same origin as Transformer’s self-attention. In recent years, Graph Transformer (such as Graphormer) has migrated position coding and global attention to graphs, achieving SOTA in graph-level tasks such as molecules.

Representative work: GCN (2017), GraphSAGE (2017), GAT (2018), GIN (2019), Graphormer (2021); applications cover AlphaFold (protein residue contact graph), drug discovery, traffic prediction and recommendation systems.

5. Training optimization technology

5.1 Gradient Descent and Optimizer

The updated formula of the standard gradient descent is:

Here, $\eta$ is the learning rate, and $g_t$ is the gradient of Loss to the parameters. In this way, all parameters share the same learning rate. However, in the actual loss surface, the gradients in different directions are very different, and it is difficult to take into account the fixed learning rate. This gave rise to the adaptive optimizer.

Momentum method: rolling down a hill like a ball

Momentum (Polyak 1964, re-developed by Sutskever et al., 2013 in deep learning) simulates the inertia of a small ball rolling down a hillside in physics:

in $\beta \in [0, 1)$ is the momentum coefficient (typical value $0.9$ ), $v_t$ is gradient exponential moving average ——Equivalent to “inertia”.

Physical intuition:

Nesterov Accelerated Gradient (NAG) is an improved version of momentum: it first “takes a step ahead” according to the current momentum and calculates the gradient at the predicted position, making the inertia more sensitive to the upcoming slope - “look-ahead momentum”.

Enabled in PyTorch via torch.optim.SGD(momentum=0.9, nesterov=True).

Optimizer evolution

flowchart TD
    SGD["SGD<br/>stochastic gradient descent<br/>1951"] -->|"Add momentum"| Momentum["Momentum SGD<br/>Suppress oscillation"]
    SGD -->|"Adaptive LR"| Adagrad["Adagrad 2011<br/>Cumulative gradient squared"]
    Adagrad -->|"exponential moving average"| RMSprop["RMSprop 2012<br/>Adapt to unstable states"]
    Momentum -->|"combine"| Adam["Adam 2014<br/>Momentum + RMSprop<br/>Current default"]
    Adam -->|"Correction L2"| AdamW["AdamW 2017<br/>Standard for large models"]
    AdamW -->|"second order information"| Sophia["Sophia 2023<br/>Second order Hessian"]
    AdamW -->|"Orthogonalization"| Muon["Muon 2024<br/>Gradient Orthogonalization"]
    AdamW -->|"Shampoo Equivalent"| SOAP["SOAP 2024<br/>Shampoo+AdamW"]

Comparison of mainstream optimizers

Note: The acceleration ratio of Sophia/Muon/SOAP comes from the experimental results of the original paper under specific settings. The actual performance varies greatly due to model size, data distribution, and hyperparameter tuning. In the production environment, AdamW is still the most empirically stable default choice.

Adam’s core formula (Kingma & Ba, 2014):

Standard hyperparameters: $\beta_1=0.9$, $\beta_2=0.999$, $\epsilon=10^{-8}$. Adam’s $m_t$ is responsible for the direction (can cross the saddle point); $v_t$ is responsible for the size (adaptive adjustment of the learning rate of each parameter). The two complement each other and jointly solve the two core problems of gradient descent.

5.2 Learning rate scheduling

There is a contradiction between a fixed learning rate that is too large (oscillation) and too small (too slow). Standard solution for large language model training: Warmup + Cosine Decay

Other commonly used schedulers

selection principle:

• trains from scratch CNN: StepLR or Cosine Annealing (Warm Restart for long training)
• fine-tuned pretrained model: Warmup + Linear/Cosine (low peak LR, usually $1-5 \times 10^{-5}$)
• LLM pre-training: Warmup + Cosine (peak LR $\sim 3 \times 10^{-4}$, Warmup accounts for 0.5–2%)
• validation set available and uncertain number of epochs: ReduceLROnPlateau
• Rapid Prototyping/Competition: 1Cycle (Smith said it can achieve “super-convergence” effect)

5.3 Parameter initialization

Different initial parameters $\theta_0$ may lead to convergence to different local optimal solutions.

Xavier initializes (Glorot & Bengio, 2010, suitable for symmetric activation such as Sigmoid/Tanh):

This method takes into account the variance stability of forward activation and reverse gradient, and is the default choice for symmetric activation functions.

Kaiming initializes (He et al., 2015, suitable for ReLU activation):

ReLU will set half of the input to zero, and Xavier’s variance assumption no longer holds, so Kaiming amplifies the variance 2 times to compensate. This initialization keeps the activation value variance stable in the early stages of dozens of layers of ReLU network training, preventing it from exploding or disappearing.

Selection Principle: Sigmoid/Tanh commonly used Xavier, ReLU and its variants commonly used Kaiming; the actual initialization also needs to be combined with the residual structure, normalization and framework default implementation.

pre-training as initialization (Pre-training): train on large-scale data first, and then transfer parameters to the target task. Improve Optimization (better starting point) and Generalization (learn common features) at the same time.

5.4 Normalization method

Normalization (Normalization) forces the output of each layer of the network to be within a reasonable range, making the loss surface flatter and the learning rate easier to adjust.

Comparison of normalization methods

Batch Normalization(BN, Ioffe & Szegedy, 2015):

Here, $\gamma, \beta$ is a learnable scaling and translation parameter, which gives the model the ability to “cancel normalization”.

Key differences between BN training and inference

BN behaves differently during training and inference, which is the most common pitfall in engineering practice:

Common bug: In PyTorch, model.eval() will automatically enable running stats when switching to inference mode; if you forget to call it, the current batch statistics will still be used during inference, resulting in randomization of the output when batch size=1.

Limitations of BN and the necessity of LN

BN has three fatal shortcomings, which correspond to the three reasons why Transformer chose LN:

Layer Normalization(LN, Ba et al., 2016):

Layer normalization normalizes across the feature dimensions of each sample, independently of batch size and sequence length. This is why it is standard in Transformers.

normalized dimension compared to intuition (taking the 4D feature map with shape [N, C, H, W] as an example):

RMSNorm (2019): Remove mean normalization and only retain variance scaling:

Typically slightly less computationally expensive than LN, and similar in many language models; adoption depends on architecture and training recipe.

5.5 Residual connection

Deep networks (100+ layers) face the coexistence problem of gradient vanishing and exploding gradients. Skip Connection/Residual Connection (He et al., ResNet, 2015):

That is, the input $x$ (identity mapping) is directly superimposed on the original transformation $F(x)$. Even if the effect of $F(x)$ is weak, the gradient can still flow directly back to the shallow layer through the constant path, which greatly alleviates the vanishing gradients and enables stable training of extremely deep networks such as ResNet-152.

Variants of residual connections

After ResNet, the residual connection developed multiple variants, each weighing “identity degree” and “information density”:

Key insights of Pre-activation ResNet (v2): ReLU in ReLU(F(x)+x) of the original ResNet will truncate part of the gradient; v2 moves BN-ReLU inside $F$, the residual path is completely identity mapping, and the gradient penetrates losslessly. This enables stable training of ResNet-1001 (1001 layer!). Transformer’s Pre-LN layout is essentially a generalization of the same idea.

DenseNet offers a counterintuitive result: each layer concatenates all preceding layers, which seems likely to cause an explosion in parameter count. Feature reuse allows each layer to remain narrow (often with a growth rate of only 32 channels), so the total parameter count can be smaller than that of a ResNet of the same depth. The trade-off is high GPU memory use because many intermediate activations must be retained.

Skip Connection has become a standard component of almost all architectures in modern deep learning (ResNet, Transformer, U-Net, Diffusion UNet). What is improved is Optimization - it does not solve “whether deep networks can express complex functions” (shallow layers can also do so), but “whether deep networks can be trained by SGD”.

6. Loss function and regularization

6.1 Loss function

Loss function (Loss Function) is the first step of the three-step framework of machine learning, which defines “how to measure the quality of the model.” Choosing an appropriate loss function is crucial for model training.

Mean Squared Error (MSE)

Mean Squared Error (MSE) is suitable for regression task, which measures the mean difference between the predicted value and the true value:

• Advantages of **: differentiable everywhere, simple gradient calculation; heavier penalty for larger errors (square amplification effect)
• Disadvantages: Extremely sensitive to outliers; gradients may explode when the prediction error is large

Mean Absolute Error (MAE)

• Advantages: More robust to outliers (linear penalty instead of square)
• Disadvantages: Not differentiable at \(y_i = \hat{y}_i\) (need to use Huber Loss to compromise)

Huber Loss

Use MSE (smooth and differentiable) when the error is small, use MAE (anti-outlier) when the error is large, and $\delta$ controls the switching threshold. Regression branch commonly used for object detection.

Cross-Entropy loss (Cross-Entropy)

Cross-Entropy Loss (Cross-Entropy Loss) is suitable for classification tasks, including image classification and language model (the nature of the next token prediction is multi-classification).

The first step: Convert the logits output by the network into probability distribution through Softmax:

The second step: Calculate the cross entropy with the real label (take the logarithmic probability of the real category, add a negative sign):

Here, $y^*$ is the real category, and \(\hat{p}_i\) is the one-hot label.

Why not use accuracy as Loss? The accuracy is a step function. When the parameters change slightly, the Loss is almost always zero, the gradient cannot be calculated, and Gradient Descent cannot be performed. Cross-Entropy is differentiable everywhere, and the smaller the value, the higher the Accuracy.

numerical stability: In actual implementation, Softmax and Cross-Entropy are combined (LogSoftmax + NLLLoss) to avoid $e^{z_i}$ numerical overflow.

Label Smoothing

Label Smoothing (Szegedy et al., 2016, widely popularized by Inception-v3 with the original Transformer) is a simple but powerful regularization trick for Cross-Entropy.

motivation: one-hot label requires the model to predict the correct category 1.0, the others are 0 - this forces the model to push the logit to infinity to fully fit, resulting in overconfidence (overconfidence), generalization Poor, distillation effect is weak.

approach: soften the hard label \(\hat{p}_i\) to:

Here, $K$ is the number of categories, and $\epsilon$ is the smoothing coefficient (typical value $0.1$). The expected probability of the correct class drops to $0.9$, and the remaining $0.1$ is evenly distributed to other classes.

effect:

• Suppress model overconfidence, calibration (calibration) is better - that is, the actual accuracy when predicting 90% is also close to 90%
• Slightly improved generalization (approximately 0.2–0.5% higher Top-1 accuracy on ImageNet)
• Improve teacher soft label quality for knowledge distillation (soft distribution carries more information than one-hot)

Note: Label Smoothing will lose a small amount of confidence information, which needs to be weighed in tasks that require confidence selection (active learning, rejection).

Binary Cross-Entropy (BCE)

Two classification tasks (using Sigmoid for the output layer):

Focal Loss

Focal Loss (Lin et al., 2017, proposed in RetinaNet) is specially designed for scenes with serious imbalance in the category - for example, there are far more background boxes than foreground boxes in target detection (1: 1000), and there are few positive samples in medical images.

Standard Cross-Entropy still gives a non-negligible loss for easy classification samples (prediction probability $p \to 1$), and a large number of easy samples will “swamp” the gradient of a small number of difficult samples. Focal Loss is multiplied by a modulation factor $(1-p)^\gamma$ in front of CE:

• $\gamma$ (focusing parameter, typical value 2): When $p \to 1$ $(1-p)^\gamma \to 0$, almost no gradient is generated; when $p \to 0$ (difficult sample) the factor approaches 1, gradient is normal
• $\alpha$ (class balancing, typical value 0.25): Static adjustment of positive and negative class weights

Effect: For the first time, RetinaNet allowed a single-stage detector to equal the accuracy of the two-stage Faster R-CNN, and Focal Loss played an important role. Since then, it has been widely used in long-tail classification, anomaly detection, and medical image segmentation.

KL divergence

KL Divergence (Kullback-Leibler Divergence) measures the difference between two probability distributions, often used in knowledge distillation and variational autoencoders (VAE):

Mathematical relationship between cross entropy and KL divergence

Cross-Entropy and KL divergence are not two independent losses; Different perspectives of the same quantity :

where $H(P) = -\sum_i P(i) \log P(i)$ is the entropy of the true distribution $P$.

infers: When $P$ is a one-hot hard tag, $H(P) = 0$, so CE $\equiv$ KL. So in standard classification tasks, “minimizing Cross-Entropy” and “minimizing KL divergence” are exactly equivalent.

Practical differences:

• Hard label (one-hot): using CE, simple calculation
• Soft labels (distillation, Label Smoothing, RLHF reference strategy): Use KL, because when $H(P) \neq 0$, CE contains a non-optimizable constant term, KL more cleanly measures the “distance from the target distribution”

Contrastive/Triplet Loss

Metric Learning (Metric Learning) The goal is to learn an embedding space so that “similar samples” are close and “heterogeneous samples” are far away. Two types of classic losses:

Contrastive Loss (Hadsell et al., 2006): Given a pair of samples $(x_i, x_j)$, the labels $y=1$ represent the same type and $y=0$ the heterogeneous one:

Here, $d$ is the embedding distance (usually Euclidean), and $m$ is the margin hyperparameter.

Triplet Loss (Schroff et al., 2015, FaceNet): Given a triplet (anchor $a$, positive $p$, negative $n$):

The intuition is that “the distance + margin between the positive sample and the anchor point is smaller than the distance between the negative sample and the anchor point”. FaceNet achieves 99.63% accuracy on LFW with its learned face embeddings.

InfoNCE (Contrastive Predictive Coding, Oord et al., 2018): The mainstream form of modern contrastive learning, one positive sample performs softmax on $N-1$ negative samples:

where $\tau$ is the temperature coefficient. SimCLR, MoCo, CLIP are all based on InfoNCE - this type of loss drove the explosion of self-supervised representation learning in 2020–2023.

InfoNCE is essentially Cross-Entropy of $N$ classification - positive samples are the only “correct category”, and negative samples are “other categories”. This once again confirms the universality of CE.

Loss function selection quick check

6.2 Dropout

Dropout (Srivastava et al., 2014): Set $p$ as the dropout rate, set the neuron output to zero with probability $p$ during training, and divide the retained activation by $1-p$ (inverted dropout). Turn off Dropout during inference and use the full network directly, so no additional scaling is required.

Intuition: forcing the network to predict correctly even in the absence of some neurons, preventing excessive co-adaptation between neurons, is equivalent to training a large number of sub-networks with different structures at the same time and obtaining the integration effect.

Timing of use: When there is a clear gap between training and verification errors, Dropout can be used as a regularization option; if the training Loss itself cannot be reduced, the optimization settings should be checked first. Too strong Dropout may further slow down convergence.

6.3 Data augmentation

Data Augmentation Artificially expands the amount of data by applying semantic-preserving transformations to training samples:

Mixup Formula: $\tilde{x} = \lambda x_i + (1-\lambda)x_j$, $\tilde{y} = \lambda y_i + (1-\lambda)y_j$

Note: Data augmentation transformations must maintain label semantics. If the task is to determine the direction of the bird’s head, left and right flipping is not possible; if the task is speaker identification, speaker switching is not possible.

Timing of use: Data augmentation is usually used to improve generalization and distribution robustness, but the enhancement intensity must match the task. If the Training Loss cannot be reduced, you should first confirm that the enhancement does not destroy the label semantics, and then check the optimization settings.

6.4 L2 regularization and AdamW

L2 regularization (Weight Decay) adds the L2 norm penalty of the parameters in Loss to make the optimization bias toward solutions with smaller absolute values of parameters (more “simple” functions):

AdamW (Loshchilov & Hutter, 2017) fixed a common error in adding L2 regularization to Adam: the traditional method merges the regularization gradient with the ordinary gradient and then scales them uniformly by Adam, causing the regularization effect to be “diluted” by the adaptive learning rate. AdamW instead directly performs Weight Decay on the parameters, and then performs Adam update:

θ = θ × (1 - lr × λ)       # Weight Decay Acts directly on parameters (without adaptive term scaling)
θ = θ - lr × Adam_update    # Adam Normal update

AdamW is the current standard optimizer for large language model training, and is usually used with gradient clipping.

6.5 Semi-supervised learning and self-supervised learning

In real scenarios, labeled data is scarce and unlabeled data is cheap. Semi-supervised learning (Semi-supervised Learning) and self-supervised learning (Self-supervised Learning) are the two mainstream paths for utilizing unlabeled data.

Core idea

Relationship to modern large models

The pre-training of large language models is essentially the largest self-supervised learning - “predicting the next token” (Causal LM) or “predicting the masked token” (Masked LM) on massive unlabeled texts, and then adapting to downstream tasks through fine-tuning (SFT, RLHF) with a small amount of labeled data. The success of the Pre-training + Fine-tuning paradigm makes “scarcity of annotated data” no longer the main bottleneck for the implementation of deep learning.

7. Method classification summary

flowchart TD
    A["Neural network training techniques"] --> B["Step 3: Improve Optimization"]
    A --> C["Step 2: Improve network architecture"]
    A --> D["Step 1: Improve Loss with data"]
    B --> B1["Adam / AdamW<br/>adaptive learning rate+momentum"]
    B --> B2["LR Scheduling<br/>Warmup + Cosine Decay"]
    B --> B3["Kaiming Init<br/>Parameter initialization"]
    B --> B4["Pre-training<br/>improve at the same time Opt+Gen"]
    C --> C1["CNN<br/>Receptive Field + Parameter sharing<br/>improve Generalization"]
    C --> C2["Skip Connection<br/>Mitigating vanishing exploding gradientss<br/>improve Optimization"]
    C --> C3["BN / LN / RMSNorm<br/>Major improvements Optimization"]
    D --> D1["Cross-Entropy<br/>make Optimization feasible"]
    D --> D2["Dropout<br/>improve Generalization"]
    D --> D3["Data Augmentation<br/>improve Generalization"]
    D --> D4["AdamW / Weight Decay<br/>improve Generalization"]

Comparison table of objectives of each method:

7.1 Training troubleshooting quick check

When encountering a training abnormality, first determine the problem based on the curve, and then choose to make changes. By changing only a few factors at a time and recording the data, random seeds, and evaluation protocols, the results can be easily compared.

8. Commonly used experimental benchmarks

Benchmarks are tools for measuring progress. This section sorts out classic benchmarks by field - MNIST/ImageNet for CV, GLUE/MMLU for NLP, as well as tasks such as coding, mathematical reasoning, and long context. The models, data, prompts and evaluation scripts of different papers may be different. The scale and historical results in the table are suitable for establishing concepts and should not be directly used as cross-paper rankings.

8.1 Computer Vision Benchmark

MNIST

CIFAR-10 / CIFAR-100

The effects of Dropout, Batch Normalization, ResNet, and Data Augmentation have been extensively verified on this benchmark.

ImageNet(ILSVRC)

ImageNet Top-1 Accuracy evolution (see table in Section 4): AlexNet (56.5%) → VGG (71.5%) → ResNet (76.0%) → EfficientNet (84.4%). These numbers come from classic models and training recipes from different eras, and are mainly used to illustrate the evolution of the architecture.

8.2 Natural Language Processing Benchmark

GLUE / SuperGLUE

BERT (2018) achieved leading results on GLUE when it was released; as the benchmark gradually became saturated, the researchers subsequently launched the more difficult SuperGLUE.

MMLU(Massive Multitask Language Understanding)

MMLU is suitable for observing the knowledge coverage and multi-disciplinary reasoning of the model, but the scores will be affected by prompts, calibration methods and training data contamination, and the evaluation protocol should be fixed when comparing.

Language model perplexity (Perplexity)

Penn Treebank (PTB)/WikiText is the standard benchmark for traditional language models. The evaluation index is perplexity (PPL) - the lower the better, indicating that the model is more certain in predicting the next token. PPL is still suitable for comparing language models under the same corpus, word segmentation and modeling goals, but it cannot represent knowledge, code or reasoning ability alone.

Code Competency Benchmark

To evaluate LLM coding ability, it is required for almost all cutting-edge model releases.

Usage Suggestions: HumanEval/MBPP is suitable for testing function-level code generation, and SWE-bench is closer to real warehouse repair; when reporting results, the number of sampling times, test set version and whether tool calls are allowed should also be stated.

Mathematical Reasoning Benchmark

GSM8K is more focused on basic multi-step arithmetic, while MATH and AIME have higher requirements for competition mathematical reasoning. As model capabilities improve, training data contamination, sampling strategies, and answer verification methods will significantly affect performance, and you cannot just look at a single score.

Long context benchmark

As the context window expands from 2K→32K→128K→1M, simple perplexity can no longer reflect the long context capability of the model, and special benchmarks have emerged:

Usage Suggestions: Needle-in-a-Haystack mainly tests positioning capabilities and cannot replace real long document Q&A. Actual evaluation should also focus on location, document length, question type, retrieval interference, and accuracy of answer citations.

9. Efficient training and inference

9.1 Mixed precision training

Modern GPUs support FP16/BF16 operations. Mixed-precision training completes forward propagation and gradient calculations with low precision (saving GPU memory 50%, accelerating calculations 2-3×), and performs parameter updates with FP32 (ensuring numerical stability).

BF16 (Brain Float 16) has wider index bits (8 bits vs 5 of FP16) bits), with a larger numerical range, and has become the preferred low-precision format for large language model training.

9.2 Gradient Clipping

Occasional exploding gradientss (loss spikes) during large model training can lead to training crashes. Gradient clipping prevents this problem by limiting the upper bound on the L2 norm of the gradient:

The usual threshold is $\tau = 1.0$, which is the standard companion to AdamW.

9.3 Gradient Checkpointing

During training, backpropagation requires all intermediate activation values in the forward process to calculate gradients - under deep networks, the activation value GPU memory can occupy several times the parameters themselves, becoming the main overhead of GPU memory for large model training.

Gradient checkpointing (Chen et al., 2016) retains activations only at selected checkpoint layers. During backpropagation, it recomputes the missing intermediate activations with another forward pass, trading approximately 33% additional computation for activation memory on the order of $O(\sqrt{N})$.

The use of gradient checkpoints in combination with mixed precision, ZeRO, and tensor parallelism is a standard GPU memory optimization method for training large models with hundreds of billions of parameters (gradient_checkpointing=True for HuggingFace Transformers, torch.utils.checkpoint for PyTorch).

9.4 Flash Attention

The bottleneck of standard attention is that the $N \times N$ attention matrix needs to be written to GPU HBM GPU memory. The GPU memory complexity is $O(N^2)$, which is extremely expensive for long sequences.

Flash Attention reduces GPU memory traffic and accelerates computation while retaining exact attention rather than approximating it. Availability depends on hardware, sequence length, and kernel implementation.

10. Model compression and adaptation

10.1 Mixture-of-Experts (MoE)

Mixture of Experts(MoE) The core idea: split a large FFN into $N$ independent “expert”, led by an Router/Gating Network Select top- for each token $k$ Experts perform calculations. Only activating some experts means - The number of parameters is huge, but only a small part of them is used each time. .

flowchart LR
    X["Token<br/>hidden state"] --> R["Router<br/>(small MLP)"]
    R -->|"top-2 scores"| S["softmax Weighted"]
    X --> E1["Expert 1<br/>FFN"]
    X --> E2["Expert 2<br/>FFN"]
    X --> EN["Expert N<br/>FFN"]
    E1 -.->|"selected"| S
    E2 -.->|"selected"| S
    S --> Y["weighted output"]
    style E1 fill:#c4e1ff
    style E2 fill:#c4e1ff
    style EN fill:#f0f0f0

Core indicators

Representative work

Key challenges: Load balancing (preventing all tokens from flooding to a few experts) - usually adding auxiliary losses or capacity constraints to avoid overloading a few experts. The statistical calibers of total parameters and activated parameters are not uniform. When comparing across models, the routing strategy and FFN scale should be viewed at the same time.

10.2 Parameter efficient fine-tuning (PEFT)

As the number of pretrained model parameters expands to tens or hundreds of billions, Full Fine-tuning GPU memory and storage costs tend to be high. Parameter-Efficient Fine-Tuning(PEFT) Only train a very small number of additional parameters (usually < 1% ), which can approximate the full parameter fine-tuning effect.

Comparison of mainstream PEFT methods

LoRA core formula

For pretrained weight $W_0 \in \mathbb{R}^{d \times k}$, LoRA freezes $W_0$ and only trains low-rank decomposition $B \in \mathbb{R}^{d \times r}, A \in \mathbb{R}^{r \times k}$ ($r \ll \min(d,k)$, typical $r=8, 16, 64$):

Forward calculation: $h = W_0 x + BAx$.

Advantages:

• trains: only $B, A$ requires gradients and optimizer state, GPU memory requirements drop to $\sim 1/100$
• storage: One task only needs to save $BA$ (MB level), a set of base molds can load LoRA for hundreds of tasks
• inference: $BA$ can be merged into $W_0$, with zero inference overhead
• switching task: load different LoRA weights to switch scenes in seconds

LoRA/QLoRA are common strong baselines for open source large model fine-tuning, but the best solution still depends on the task, base model, quantization method and available GPU memory.

10.3 Model quantization (Quantization)

quantization compresses the model weights (and activations) from FP16/BF16 to INT8/INT4 or even lower bit width, with a small loss of accuracy in exchange for model size reduced to 1/4~1/8, inference speed improvement 2-4×.

Quantization types

Modern quantization methods

Experience: INT4 can often significantly reduce GPU memory usage, but the quality loss depends on the model, calibration data, quantization granularity and inference kernel; latency, throughput and accuracy should be measured on the target task before deployment.

10.4 Knowledge Distillation

Knowledge Distillation (Hinton et al., 2015) migrates the capabilities of a large “teacher” model to a small “student” model, allowing the small model to achieve performance close to that of the large model during deployment.

Core idea: Soft labels are more informative than hard labels

The one-hot hard label only tells students “This picture is a cat”; while the soft probability distribution output by the teacher model can tell students: “This picture 90% is a cat, 8% is a dog, 1% is a fox, 0% is a car”—the relative relationship between these categories (“A dog is closer to a cat than a fox”) implies the teacher’s knowledge.

Distillation loss:

Here:

• $p_t^{(T)} = \text{softmax}(z_t / T)$ is the output softened by temperature $T$ for teachers (the distribution is smoother at $T > 1$)
• $p_s^{(T)}$ is the softened output corresponding to the student
• $\alpha$ Balancing hard label supervision and soft label distillation

Three Paradigms of Distillation

Representative work

In LLM, distillation can act on logits, hidden representations, preference data, or generate trajectories. It is often used in combination with quantization, pruning, or parameter sharing, but the capability gap between the teacher model and the student model, data distribution, and generation temperature all affect the final effect.

11. Cutting-edge architecture and large models

11.1 Reinforcement Learning from Human Feedback (RLHF)

RLHF (Reinforcement Learning from Human Feedback) is the key to the success of ChatGPT - aligning the pretrained large model from “speaking” to “understanding instructions and in line with human preferences”.

flowchart LR
    PT["pre-training LLM<br/>(GPT-3)"] --> SFT["Step 1: SFT<br/>Supervised fine-tuning"]
    SFT --> RM["Step 2: Training reward model<br/>Humans sort multiple answers"]
    RM --> PPO["Step 3: PPO reinforcement learning<br/>Maximize rewards"]
    PPO --> ALIGNED["Aligned model<br/>(ChatGPT)"]
    style PT fill:#e1f5ff
    style ALIGNED fill:#c4ffc4

Three-stage process

Alternatives to RLHF

DPO has become a common preference optimization method because the training process is simple and there is no need to separately train the reward model. PPO, DPO, rejection sampling, and other methods can also be used in combination; the specific training process of closed-source models is often not fully disclosed.

11.2 Generative model: from AE to GAN

The discriminative model learns \(p(y\mid x)\) (given input prediction label), and the generative model learns the data distribution \(p(x)\) itself, so it can sample new samples “out of nothing”. There are four major families of deep generative models, which continue to evolve along the trade-off of “sampling quality/training stability/latent space controllability”: autoencoder (AE) → variational autoencoder (VAE) → generative adversarial network (GAN) → diffusion model (detailed in the next section).

Autoencoder (AE)

The encoder $f$ compresses the input into a low-dimensional latent vector $z$, the decoder $g$ reconstructs $\hat{x}$, and the training target is the reconstruction error \(\mathcal{L} = \|x - g(f(x))\|^2\). AE is good at dimensionality reduction / denoising / representation learning, but the latent space has no probability constraints and irregular distribution. cannot randomly sample from it to generate new samples - it is essentially a compressor, not a real generative model.

Variational Autoencoder (VAE, Kingma & Welling 2013)

VAE adds probabilistic structure to the latent space: the encoder outputs a distribution \(q_\phi(z\mid x)\) (instead of a single point) and constrains it to be close to the prior \(p(z)=\mathcal{N}(0,I)\). Training to maximize evidence lower bound (ELBO):

The former term is the reconstruction term, and the latter term is the regular term that pulls the latent distribution toward the prior. Through the reparameterization trick \(z = \mu + \sigma \odot \epsilon\) (\(\epsilon \sim \mathcal{N}(0,I)\)), the sampling can be differentiated and the gradient can be returned. After training, it can be generated by directly sampling from the prior \(p(z)\). VAE sampling is stable, the latent space is continuous and interpolable, but the generated image is blurry.

Generative Adversarial Network (GAN, Goodfellow et al. 2014)

GAN uses to counter the game to bypass the explicit likelihood: the generator $G$ maps noise into false samples, and the discriminator $D$ learns to distinguish between true and false, and the two alternately optimize a minimax objective:

Image generated by GAN Sharp and high quality (StyleGAN’s faces are almost lifelike), but training is known for being unstable—— mode collapse , the game is difficult to converge and requires a lot of techniques (WGAN, spectral normalization, etc.) to stabilize it.

List of four major families

flowchart LR
    AE["AE<br/>Compressed reconstruction"] --> VAE["VAE 2013<br/>probability latent space"]
    VAE --> GAN["GAN 2014<br/>confrontational game"]
    GAN --> DIFF["Diffusion 2020<br/>Denoising generation"]
    style AE fill:#e1f5ff
    style DIFF fill:#c4ffc4

The diffusion model combines the training stability of VAE and the high quality of GAN, and has become the mainstream of current generative modeling - the next section will expand.

11.3 Diffusion Models

Diffusion model is the most successful generative paradigm in deep learning in recent years, driving epoch-making work such as Stable Diffusion, DALL-E 3, Midjourney, and Sora. It is another pillar of deep generative modeling alongside Transformer.

Core idea: noise addition and denoising

flowchart LR
    X0["x₀<br/>clear image"] -->|"Forward: progressively adding noise"| XT["xₜ<br/>pure noise"]
    XT -->|"Reverse: learning to denoise"| XM["x̂₀<br/>generate image"]
    style X0 fill:#c4ffc4
    style XT fill:#ffcccc
    style XM fill:#c4e1ff

forward process (Forward / Diffusion): gradually add Gaussian noise to the real data, and after $T$ steps (usually 1000) it becomes an approximate standard normal distribution:

Since the properties are deducible, $x_t$ can be obtained from $x_0$ in one step: \(x_t = \sqrt{\bar{\alpha}_t} \cdot x_0 + \sqrt{1-\bar{\alpha}_t} \cdot \epsilon\), where \(\bar{\alpha}_t = \prod_{s=1}^t (1-\beta_s)\).

Reverse process (Reverse / Denoising): Train a neural network \(\epsilon_\theta(x_t, t)\) to predict the noise added at each step. The training loss is:

reasoning , sampled from the standard normal distribution $x_T$ , then use \(\epsilon_\theta\) Gradually remove noise, $T$ After the step, the generated sample is obtained.

Why are diffusion models more successful than GANs?

Representative work

From UNet to DiT: Transformerization of Diffusion Networks

The early diffusion model uses UNet (encoder-decoder CNN with skip connection) as the denoising network. 2022 appeared DiT (Diffusion Transformer, Peebles & Xie, 2022) years later, using Transformer to replace UNet - Sora, Stable Diffusion 3 all use DiT. This once again confirms the trend of Transformer unifying multiple fields.

11.4 State space model and Mamba

Transformer’s attention mechanism affects sequence length $N$ The computational and GPU memory complexity of $O(N^2)$ , which is extremely expensive when processing long sequences. State Space Model (SSM) Provides an alternative architectural route with linear complexity.

SSM Basics: Continuous-Time Linear Dynamic Systems

SSM originates from cybernetics and maps the input sequence $x(t)$ to the output sequence $y(t)$ through the hidden state $h(t)$:

\(h'(t) = Ah(t) + Bx(t)\) \(y(t) = Ch(t)\)

Here, $A \in \mathbb{R}^{N \times N}$ is the state transition matrix, and $B, C$ is the projection matrix. After discretization (step size $\Delta$, zero-order preservation), the recursive form is obtained:

Among them $\bar{A} = e^{\Delta A}$, $\bar{B} = (\Delta A)^{-1}(e^{\Delta A} - I) \cdot \Delta B$.

SSM combines the advantages of RNN and CNN:

S4: Structured state space sequence model

S4 (Gu et al., 2021) is the first practical deep SSM: restricting the matrix $A$ to HiPPO initialized diagonal plus low rank (DPLR) The structure makes long sequence training efficient and feasible, significantly surpassing Transformer on long sequence benchmarks such as Long Range Arena.

Limitations of S4: The $A, B, C$ matrix is fixed for all time steps (time-invariant) and cannot dynamically choose which information to focus on or ignore based on the input content - essentially a static filter.

Mamba: Selective state space model (2023)

The core innovation of Mamba (Gu & Dao, 2023) is selectivity (Selectivity): convert $B, C$ and step size $\Delta$ Designed for input-dependent:

This allows the model to dynamically adjust the state update and forgetting speed according to the content of the current token; under common parameterization, $\Delta$ will affect the state decay and input injection intensity after discretization, and cannot be simply equated to a fixed “memory switch”.

hardware-aware parallel scan: directly performs a parallel prefix scan on the recursion of $h_t = \bar{A} h_{t-1} + \bar{B} x_t$, and the entire process is completed in the GPU SRAM to avoid frequent writing back to HBM. The GPU memory complexity is $O(N)$, and the training speed is comparable to Transformer.

Mamba block structure (each layer):

flowchart LR
    X["input x"] --> L1["linear projection × 2"]
    L1 --> |"main path"| CONV["1D Convolution<br/>(short range local perception)"]
    CONV --> ACT["SiLU activate"]
    ACT --> SSM["Selectivity SSM<br/>(Δ,B,C from input)"]
    SSM --> MUL["⊗ Gating"]
    L1 --> |"gated path"| MUL
    MUL --> L2["linear projection"]
    L2 --> Y["output y"]
    style SSM fill:#c4e1ff

Mamba 2(2024)

Mamba 2 (Dao & Gu, 2024) introduces State Space Duality (SSD), showing that Mamba’s selective SSM is a structured special case of linear attention. This enables optimization and implementation within a unified framework:

• Use matrix multiplication block (tensor cores) instead of scanning, faster than Mamba 1 2–8×
• Expand the state size to larger dimensions and improve model capacity

Transformer vs Mamba comparison

Representative work

Current situation and prospects: Mamba and SSM series are currently the most popular Transformer alternative architectures, demonstrating competitive performance in fields such as language, vision, and genomics. However, pure SSM is still weaker than Transformer in terms of general reasoning and in-context learning capabilities. The mainstream trend is Mamba + Transformer hybrid architecture (such as Jamba, Zamba) to take into account long sequence efficiency and global reasoning capabilities.

11.5 Deep learning and large language model

Deep learning (especially Transformer) is the technical foundation of large language models (LLM). The division of labor between this article and the LLM training review is as follows:

For specific engineering details of LLM pretraining/finetuning/inference, see:

Overview of LLM training technology

Covers topics such as Scaling Laws, distributed training, hybrid parallelism, KV Cache, GQA/MQA internal implementation, long context extrapolation, inference acceleration, etc.

12. Summary

There is only one main thread running through the whole text: machine learning three-step framework (Loss → Architecture → Optimization) × two core goals (Optimization vs Generalization). Each training technique accurately falls on a certain grid on this 3×2 coordinate system. For details, see the method classification summary table in Section 7.

Core conclusion:

The training technique of deep learning is not that the more the better, but to the right medicine - Look at the training Loss first: if it cannot be reduced, it is an Optimization problem, do not add Dropout; low training Loss and high verification Loss are a Generalization problem, only then consider data augmentation and regularization. This is the basic skill for efficient alchemy, and it is also the judgment principle repeatedly emphasized in this article.