Learning Path We’ll build this in dependency order: Tokens → Embeddings → Neural-network representations → Transformer layers → Causal self-attention → Contextual representations → Next-token probabilities → Generation We’ll verify each major step before moving forward. 1. Tokens An LLM does not directly process words as human-readable concepts. Text is first converted into tokens, which are discrete units produced by a tokenizer. For example, a sentence might be represented approximately as: “Transformers process context.” → [token₁, token₂, token₃, token₄] The exact token boundaries depend on the tokenizer. Why this matters: the model's numerical computation begins from these token representations after token IDs are converted into vectors. 2. Embeddings Token IDs are categorical identifiers. A neural network does not treat an ID such as 1842 as inherently more meaningful than 927 merely because the number is larger. An embedding layer maps each token ID to a learned vector: TOKEN ID → VECTOR REPRESENTATION These vectors provide numerical representations that subsequent neural-network layers can transform. 3. Why Attention Is Needed Consider: “The animal didn't cross the road because it was exhausted.” To interpret “it,” useful information may exist elsewhere in the sequence. In a decoder-only LLM, causal self-attention allows each position to incorporate information from relevant earlier positions while preventing it from attending to future tokens that would not yet be available during next-token prediction. Conceptually: TOKEN REPRESENTATIONS → CAUSAL ATTENTION INTERACTIONS → CONTEXT-AWARE REPRESENTATIONS The important point is that attention does not simply “look up the meaning of a word.” It computes interactions between representations within the available context. 4. Connecting the Mechanism The causal chain is: Text ↓ Tokenization ↓ Token IDs ↓ Embeddings ↓ Transformer transformations ↓ Contextual representations ↓ Output projection → logits ↓ Probability distribution over possible next tokens The model then uses a decoding procedure to select the next token. Understanding Check If a decoder-only LLM could attend to future tokens while making a prediction, what fundamental problem would that create during training? A strong answer should recognize that the model would have access to information that would not be available when actually generating the next token, creating information leakage and invalidating the intended causal prediction setup. Next step: we can examine self-attention mathematically, including queries, keys, values, attention scores, and the causal mask, and connect those calculations to the next-token probability distribution.