Octopedia The Mathematical Foundations of Modern Transformers
Softmax
\[\text{softmax}(\mathbf{x}) = \frac{1}{\sum_{j=1}^{n} e^{x_j}} \begin{pmatrix} e^{x_1} \\ e^{x_2} \\ \vdots \\ e^{x_n} \end{pmatrix}\]Softmax Vector Form
\[\text{softmax}(x_i) = \frac{e^{x_i}}{\sum_{j=1}^{n} e^{x_j}}\]Attention (Scaled Dot Product)
\[\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V\] \[\text{scores} = \frac{QK^T}{\sqrt{d_k}}\] \[\text{weights} = \text{softmax}(\text{scores}) = \frac{\exp(\text{scores})}{\sum \exp(\text{scores})}\] \[\text{output} = \text{weights} \cdot V\]Multi-Head Attention
\[\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, \ldots, \text{head}_h)W^O\]where each head is:
\[\text{head}_i = \text{Attention}(QW_i^Q, KW_i^K, VW_i^V)\]General Attention (with arbitrary score function)
\[\text{Attention}(q, K, V) = \sum_{i=1}^{n} \alpha_i v_i\]where:
\[\alpha_i = \frac{\exp(f(q, k_i))}{\sum_{j=1}^{n} \exp(f(q, k_j))}\]Layer Normalization
\[\text{LayerNorm}(x) = \gamma \odot \frac{x - \mu}{\sigma} + \beta\]where:
\[\mu = \frac{1}{d} \sum_{i=1}^{d} x_i, \quad \sigma = \sqrt{\frac{1}{d} \sum_{i=1}^{d} (x_i - \mu)^2}\]Residual Connections
\[\text{output} = x + \text{Sublayer}(x)\]Feed-Forward Network
\[\text{FFN}(x) = \max(0, xW_1 + b_1)W_2 + b_2\]or with GELU activation:
\[\text{FFN}(x) = \text{GELU}(xW_1 + b_1)W_2 + b_2\]GELU Activation
\[\text{GELU}(x) = x \cdot \Phi(x) = x \cdot \frac{1}{2}\left[1 + \text{erf}\left(\frac{x}{\sqrt{2}}\right)\right]\]Approximation
\[\text{GELU}(x) \approx 0.5x\left(1 + \tanh\left[\sqrt{\frac{2}{\pi}}\left(x + 0.044715x^3\right)\right]\right)\]Positional Encoding
\[PE_{(pos, 2i)} = \sin\left(\frac{pos}{10000^{2i/d}}\right)\] \[PE_{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{2i/d}}\right)\]Transformer Block
\[z_l = \text{LayerNorm}(x_{l-1} + \text{MultiHead}(x_{l-1}))\] \[x_l = \text{LayerNorm}(z_l + \text{FFN}(z_l))\]Cross-Entropy Loss
\[\mathcal{L} = -\sum_{i=1}^{|V|} y_i \log(\hat{y}_i)\]Key-Query-Value Projections
\[Q = XW^Q, \quad K = XW^K, \quad V = XW^V\]Causal Mask (for decoder)
\[\text{mask}_{i,j} = \begin{cases} 0 & \text{if } i \geq j \\ -\infty & \text{if } i < j \end{cases}\]RMSNorm (alternative to LayerNorm)
\[\text{RMSNorm}(x) = \frac{x}{\text{RMS}(x)} \odot g\]where:
\[\text{RMS}(x) = \sqrt{\frac{1}{d}\sum_{i=1}^{d} x_i^2}\]BERT (Bidirectional Encoder)
BERT Input Representation
\[\text{Input} = \text{TokenEmb} + \text{SegmentEmb} + \text{PositionEmb}\]Masked Language Model Loss
\[\mathcal{L}_{\text{MLM}} = -\sum_{i \in \mathcal{M}} \log P(x_i | \text{context})\]where \(\mathcal{M}\) is the set of masked positions.
Next Sentence Prediction Loss
\[\mathcal{L}_{\text{NSP}} = -\log P(\text{IsNext} | [\text{CLS}])\]Total BERT Pre-training Loss
\[\mathcal{L}_{\text{BERT}} = \mathcal{L}_{\text{MLM}} + \mathcal{L}_{\text{NSP}}\]GPT (Decoder-only)
Autoregressive Language Modeling
\[P(x_1, \ldots, x_T) = \prod_{t=1}^{T} P(x_t | x_1, \ldots, x_{t-1})\]GPT Loss
\[\mathcal{L}_{\text{GPT}} = -\sum_{t=1}^{T} \log P(x_t | x_1, \ldots, x_{t-1})\]Causal Self-Attention
\[\text{Attention}_{\text{causal}}(Q, K, V) = \text{softmax}\left(\frac{QK^T + M}{\sqrt{d_k}}\right)V\]where mask \(M_{i,j} = 0\) if \(i \geq j\), else \(-\infty\).
T5 (Encoder-Decoder)
Text-to-Text Transfer Loss
\[\mathcal{L}_{\text{T5}} = -\sum_{t=1}^{T} \log P(y_t | y_{<t}, x)\]Encoder-Decoder Attention
\[\text{CrossAttention}(Q_{\text{dec}}, K_{\text{enc}}, V_{\text{enc}}) = \text{softmax}\left(\frac{Q_{\text{dec}}K_{\text{enc}}^T}{\sqrt{d_k}}\right)V_{\text{enc}}\]RoBERTa Improvements
Dynamic Masking (no fixed formula, but concept)
- Masking pattern changes during training
Removed NSP Loss
\[\mathcal{L}_{\text{RoBERTa}} = \mathcal{L}_{\text{MLM}}\]ELECTRA
Generator Loss (small MLM model)
\[\mathcal{L}_G = \sum_{i \in \mathcal{M}} -\log P_G(x_i | \mathbf{x}_{\backslash \mathcal{M}})\]Discriminator Loss (replaced token detection)
\[\mathcal{L}_D = \sum_{i=1}^{n} -\mathbb{1}(x_i^{\text{corrupt}} = x_i) \log D(x_i^{\text{corrupt}}, i) - \mathbb{1}(x_i^{\text{corrupt}} \neq x_i) \log(1 - D(x_i^{\text{corrupt}}, i))\]Combined ELECTRA Loss
\[\mathcal{L}_{\text{ELECTRA}} = \mathcal{L}_D + \lambda \mathcal{L}_G\]DeBERTa
Disentangled Attention
\[A_{i,j} = \{Q_i, K_j\} + \{Q_i, \delta_{i,j}^{rel}\} + \{\delta_{i,j}^{rel}, K_j\}\]Enhanced Mask Decoder
\[P(x_i | \mathbf{x}_{\backslash i}) = \text{softmax}(H_i W + b)\]where \(H_i\) includes absolute position information.
BART (Denoising Autoencoder)
Denoising Loss
\[\mathcal{L}_{\text{BART}} = -\log P(\mathbf{x} | \text{corrupt}(\mathbf{x}))\]Various Corruption Functions
- Token masking: \(\text{corrupt}_{\text{mask}}(\mathbf{x})\)
- Token deletion: \(\text{corrupt}_{\text{delete}}(\mathbf{x})\)
- Text infilling: \(\text{corrupt}_{\text{infill}}(\mathbf{x})\)
- Sentence permutation: \(\text{corrupt}_{\text{permute}}(\mathbf{x})\)
ALBERT
Factorized Embedding
\[E = V \times H \rightarrow E = V \times E + E \times H\]where \(E \ll H\).
Cross-layer Parameter Sharing
\[W_l = W_{\text{shared}} \quad \forall l\]Sentence Order Prediction (SOP)
\[\mathcal{L}_{\text{SOP}} = -\log P(\text{InOrder} | [\text{CLS}])\]Common Components
Attention Score Computation
\[e_{i,j} = \frac{(x_i W^Q)(x_j W^K)^T}{\sqrt{d_k}}\]Attention Weight Normalization
\[\alpha_{i,j} = \frac{\exp(e_{i,j})}{\sum_{k=1}^{n} \exp(e_{i,k})}\]Multi-Head Concatenation
\[\text{MultiHead}(Q,K,V) = \text{Concat}(\text{head}_1, \ldots, \text{head}_h)W^O\]Dropout in Attention
\[\text{Attention}_{\text{dropout}} = \text{Dropout}(\text{softmax}(\text{scores}))V\]Learned Positional Embeddings
\[\text{PE}_{\text{learned}}(i) = \mathbf{p}_i \quad \text{where } \mathbf{p}_i \in \mathbb{R}^d\]Relative Positional Encoding
\[e_{i,j} = \frac{x_i W^Q (x_j W^K + r_{i-j})^T}{\sqrt{d_k}}\]Core SBERT Architecture
Sentence Embedding Generation
\[\mathbf{u} = \text{BERT}(\text{sentence})\] \[\mathbf{s} = \text{POOL}(\mathbf{u})\]where \(\mathbf{u} = [\mathbf{u}_1, \mathbf{u}_2, \ldots, \mathbf{u}_n]\) are token embeddings and \(\mathbf{s}\) is the sentence embedding.
Pooling Strategies
CLS-token Pooling
\[\mathbf{s} = \mathbf{u}_{\text{[CLS]}}\]Mean Pooling
\[\mathbf{s} = \frac{1}{n} \sum_{i=1}^{n} \mathbf{u}_i\]Max Pooling
\[\mathbf{s} = \max_{i=1,\ldots,n} \mathbf{u}_i\]Attention-based Pooling
\[\alpha_i = \frac{\exp(\mathbf{u}_i^T \mathbf{w})}{\sum_{j=1}^{n} \exp(\mathbf{u}_j^T \mathbf{w})}\] \[\mathbf{s} = \sum_{i=1}^{n} \alpha_i \mathbf{u}_i\]Training Objectives
Classification Objective (3-way softmax)
\[o = \text{softmax}(W_t [\mathbf{u}; \mathbf{v}; \vert\mathbf{u} - \mathbf{v}\vert])\]where \(\mathbf{u}\) and \(\mathbf{v}\) are sentence embeddings,
\(\vert\mathbf{u} - \mathbf{v}\vert\) is element-wise absolute difference.
Classification Loss
\[\mathcal{L}_{\text{class}} = -\sum_{i} y_i \log(o_i)\]Regression Objective
\[\text{similarity} = \cos(\mathbf{u}, \mathbf{v}) = \frac{\mathbf{u} \cdot \mathbf{v}}{|\mathbf{u}| |\mathbf{v}|}\]Regression Loss (MSE)
\[\mathcal{L}_{\text{reg}} = (\text{similarity} - \text{label})^2\]Siamese Network Architecture
Twin Network Processing
\[\mathbf{u} = f_{\theta}(\text{sentence}_A)\] \[\mathbf{v} = f_{\theta}(\text{sentence}_B)\]where \(f_{\theta}\) is the same BERT model with shared parameters.
Triplet Loss (for some variants)
Triplet Objective
\[\mathcal{L}_{\text{triplet}} = \max(|\mathbf{a} - \mathbf{p}|^2 - |\mathbf{a} - \mathbf{n}|^2 + \epsilon, 0)\]where \(\mathbf{a}\) is anchor, \(\mathbf{p}\) is positive, \(\mathbf{n}\) is negative, and \(\epsilon\) is margin.
Multiple Negatives Ranking Loss
MNR Loss
\[\mathcal{L}_{\text{MNR}} = -\log \frac{\exp(\text{sim}(\mathbf{a}, \mathbf{p}))}{\sum_{i=1}^{N} \exp(\text{sim}(\mathbf{a}, \mathbf{n}_i))}\]Inference/Similarity Computation
Cosine Similarity
\[\text{sim}(\mathbf{u}, \mathbf{v}) = \frac{\mathbf{u} \cdot \mathbf{v}}{|\mathbf{u}| |\mathbf{v}|}\]Euclidean Distance
\[\displaylines{ \text{dist}(\mathbf{u}, \mathbf{v}) = |\mathbf{u} - \mathbf{v}|_2 \\ \\ d(u, v) = \sqrt{\sum_{i=1}^{n} (u_i - v_i)^2} \\ d(\mathbf{u}, \mathbf{v}) = \|\mathbf{u} - \mathbf{v}\|_2 = \sqrt{(u_1 - v_1)^2 + (u_2 - v_2)^2 + \cdots + (u_n - v_n)^2} }\]Manhattan Distance
\[\displaylines{ \text{dist}(\mathbf{u}, \mathbf{v}) = |\mathbf{u} - \mathbf{v}|_1 \\ d_{manhattan}(u, v) = \sum_{i=1}^{n} |u_i - v_i| \\ }\]Fine-tuning Variants
Sentence Pair Classification
\[\text{logits} = W[\mathbf{u}; \mathbf{v}; \mathbf{u} \odot \mathbf{v}; |\mathbf{u} - \mathbf{v}|] + b\]Where
- \(\mathbf{u} \odot \mathbf{v}\) is element-wise multiplication
- \([\cdot; \cdot]\) denotes concatenation
The key innovation of SBERT is generating fixed-size sentence embeddings that can be compared using simple similarity measures, unlike standard BERT which requires expensive cross-attention for sentence pair tasks.