All learning paths

LLMs from scratch

How a language model works inside, one operation at a time: attention, queries, keys and values, embeddings, transfer learning and vector databases.

7 articles · about 19 min of reading

0 of 7 read

Start with step 1

An LLM, piece by piece

  1. Step 1: Episode 0: Why Code LLMs From ScratchUnderstanding how LLMs work without a PhD in mathematics: what this series promises, what you will learn, and the one idea that ties every episode together.2 min
  2. Step 2: Episode 1: Attention Is Just 4 Mathematical OperationsBehind Transformers and the attention mechanism there are only four simple operations: matrix multiplication, transposition, scaling and softmax. Here they are, in Rust.5 min
  3. Step 3: Episode 2: Q, K, V — Why 3 Matrices Instead of One?Query, Key, Value: why real models project each word into three different spaces, and what that separation of roles changes for attention.2 min
  4. Step 4: Episode 3: Embeddings — Absolute at the Start, Contextual at the EndIs an embedding the word on its own or the word in context? Both, at different moments. Plus the three engineering details around attention: multi-head, masking and position encoding.3 min
  5. Step 5: Episode 4: Transfer Learning — Reusing Existing KnowledgeWhy nobody trains a model from zero: where transfer learning comes from, what it saves you, and its three strategies — feature extraction, fine-tuning and LoRA.2 min
  6. Step 6: Episode 5: The Key Concepts Around Transfer LearningPre-training, fine-tuning, catastrophic forgetting, freezing, domain shift and learning rate: the transfer learning terms you hear all the time, explained simply with an analogy for each.2 min
  7. Step 7: Episode 6: Embeddings and Vector DatabasesFrom keywords to meaning: how embeddings and cosine similarity let you search by proximity, why you need a vector database, and how it all comes together as a RAG.3 min