Everything in blog post was first written by hand, and then converted into Markdown blog format by LLM. I used it for my learning and studies.

Small versus large models

1. Attention’s share of floating-point operations drops from 35% to 17% when going from 760 million to 170 billion parameters, whereas the share for multilayer perceptrons goes from 44% to 80%.
2. Behavior emerges when the number of training floating-point operations is around $10^{22}$ or $10^{24}$.

What from CS336 transfers to frontier models?

1. Mechanics. How things work (for example, the Transformer).
2. Mindset. Squeezing the most out of hardware and taking scaling seriously.
3. Intuitions. Which data and modeling decisions yield good accuracy.

The bitter lesson

• Wrong interpretation. Scaling is all that matters, and algorithms do not matter.
• Right interpretation. Algorithms that scale are what matter.

$$\text{Accuracy} = \text{Efficiency} \times \text{Resources}$$

Efficiency is more important at larger scales.

Current language model landscape

• Pre-neural / 2010s. N-gram language models.
• Neural (2010s). Long short-term memory, sequence to sequence, Adam, attention, Transformer, and mixture of experts.
• Early foundation models (late 2010s). ELMo, BERT, and T5.
• Scaling (2020s). GPT-2 and GPT-3, PaLM, Chinchilla, and scaling laws.
• Open models …

The fundamentals are the same (attention, kernels, and optimization). The specifications are different (longer context and inference efficiency).

Basics

• Goal. Train a basic language model.
• Components. Tokenization, model architecture, and training.

Tokenization

A tokenizer converts between raw inputs (bytes) and sequences of integers (tokens).

• Popular approach. Byte-pair encoding (BPE).
• Intuition. Break input into frequently occurring chunks.
• Efficiency. Reduce context length (1,000 to 250 bytes). Adaptive computation.
• The dream. A tokenizer-free model architecture.

Model architecture

Starting point. The original Transformer from 2017.

Refinements

• Activations. Rectified linear unit and Swish gated linear unit.
• Position embeddings. Sinusoidal and rotary position embeddings.
• Normalization. Layer normalization, root mean square normalization, query-key normalization, and pre-normalization versus post-normalization.
• Attention. Full, sparse or local attention, group-query attention (GQA), and multi-head latent attention (MLA).
• Recurrence, state-space models, and linear attention. Mamba and Gated DeltaNet.
• Multilayer perceptrons. Dense and mixture of experts.
• Shape. Hidden dimension, depth, number of heads, and number of experts.