How transformers read

A transformer reads left to right, one token at a time, through a stack of identical layers (tens of them). Each layer refines every token's representation using the tokens around it. Three ideas carry almost all the intuition: embeddings, attention, and the forward pass.

1. Embeddings — words as coordinates

Each token ID maps to a vector: a list of a few thousand numbers learned during training. Tokens with similar meaning end up near each other in this space — king − man + woman ≈ queen is the famous (slightly mythologized) demo. The embedding is the model's only starting view of a token; everything else is computed from it.

Order is not free: transformers see sets, not sequences, so a positional signal is added to each embedding marking where it sits. Without it, "dog bites man" and "man bites dog" would look identical.

2. Attention — deciding what matters

For each token, attention asks: which earlier tokens should I listen to right now? It scores every pair with three roles:

• Query — what this token is looking for.
• Key — what each other token offers.
• Value — what actually flows over once matched.

In "the animal didn't cross the street because it was too tired," the head resolving it puts most weight on animal. Nothing magical — just learned match-scores, computed for all pairs at once (which is also why long contexts are quadratically expensive without tricks like the KV cache).

3. The forward pass — refine, refine, predict

One layer = attention (mix information across positions) followed by a feed-forward network (process each position independently, where most of the model's stored "knowledge" lives). Repeat 30–100 times. The final layer outputs a score for every vocabulary entry; normalized, that is the next-token probability distribution:

tokens in  → [embed] → layer₁ → layer₂ → … → layerₙ → scores
"the capital of France is"  →  Paris (92%), Lyon (3%), …

Sampling picks from that distribution (more in the lesson on generation), appends the token, and the whole stack runs again for the next one. That loop — one full forward pass per token — is what you pay for at serving time.