Transformers can't read words — they need float vectors. Token embeddings convert each word-ID into a vector of numbers that carries meaning. Position embeddings add a second vector that tells the model where in the sentence that word sits. Both are lookup tables trained by gradient descent. They get added together to form the actual input the transformer sees.
Think of them as two separate dictionaries the model consults every time it reads a token.
[vocab_size × C][max_seq_len × C]Why two separate tables? Meaning and position are independent signals. "cat" means the same thing whether it's at position 0 or position 20. By keeping them separate, the model can learn each signal independently, then combine them.
The tokenizer and embeddings are two completely different things that happen at different stages.
| Property | Tokenizer | Embeddings |
|---|---|---|
| What it does | text → integer IDs | integer IDs → float vectors |
| Output type | int[] | float[][] |
| Part of neural network? | NO — preprocessing | YES — first layer |
| Trained via backprop? | NO — fixed rules | YES — updated every step |
| Has learnable weights? | NO | YES |
| Can change after training? | NO | NO (frozen at inference) |
Mental model: Tokenizer is a dictionary lookup (static rules). Embedding is a neural network weight matrix (learned numbers). The tokenizer's job ends the moment it produces integer IDs. Embeddings take over from there.
Both embedding tables are just matrices of floating-point numbers. They start random. Every training step, the model makes a prediction, measures how wrong it was, and nudges all weights — including the embedding rows — in the direction that reduces error.
Correct. In both embedding tables, each row is one vector and the number of columns is the length of that vector.
C in Karpathy's nanoGPT · d_model in the original paper · hidden_size in HuggingFace · n_embd in GPT-2 config · dim in Llama — all the same thing.Columns (= C = vector width) are identical on both tables. They MUST be — you can't add vectors of different lengths. What you saw in bbycroft's visualization was a difference in rows, not columns.
Here's what's actually different between the two tables:
In bbycroft's toy model: only 3 unique tokens exist (A, B, C) → token table has 3 rows. The sequence has 6 positions → position table has 6 rows. Both tables are equally wide (same C). The visual difference is height (rows), not width (columns).
When you actually use the tables to look up values for a 6-token input, both produce a [6 × C] result — same shape — and those get added.
Each matrix has a different answer for what sets its row count.
| Matrix | Row count | Example | Set by |
|---|---|---|---|
| Token embed TABLE | vocab_size | 50,257 (GPT-2) | Tokenizer vocabulary — how many unique tokens the tokenizer knows |
| Position embed TABLE | max_seq_len | 1,024 (GPT-2) | Model design choice — max context window (how long an input it can handle) |
| Token lookup RESULT | T (current seq len) | 6 (for 6 input tokens) | The actual input you're running — changes every request |
| Position lookup RESULT | T (current seq len) | 6 (same as above) | Always matches token lookup result — they must be added together |
// The two permanent tables (exist for the lifetime of the model) token_embed_table shape: [vocab_size, C] // e.g. [50257, 768] pos_embed_table shape: [max_seq_len, C] // e.g. [1024, 768] // The runtime lookup results (change every request) tok_lookup shape: [T, C] // e.g. [6, 768] for a 6-token input pos_lookup shape: [T, C] // same shape, ALWAYS input_embed shape: [T, C] // tok_lookup + pos_lookup → into transformer // C is identical in ALL of the above
bbycroft shows you the lookup result — a small slice pulled out of a much larger table. The full tables live permanently in model memory for the entire lifetime of the model.
[T × C] result — only the rows for your current input tokens. If T=6 you see 6 rows. The rest of the table is not shown.[vocab_size × C] — one row for every possible token. GPT-2: 50,257 rows × 768 cols = ~38.6M numbers. Always in memory. You only "activate" the rows matching your input.[max_seq_len × C] — one row per possible position slot. GPT-2: 1,024 rows × 768 cols = ~786K numbers. Always in memory. Row 0 used for first token, row 1 for second, etc.// GPT-2 Small token_table [50_257 × 768] = 38_597_376 params (~38.6M floats) position_table [ 1_024 × 768] = 786_432 params (~786K floats) // Llama-3 8B token_table [128_000 × 4_096] = 524_288_000 params (~524M floats — 6.5% of total model!) position_table → RoPE — no explicit table, zero extra params // What bbycroft shows (for a 6-token input) tok_lookup_result [6 × 768] = 4_608 values ← tiny slice pos_lookup_result [6 × 768] = 4_608 values ← tiny slice
The full table is always present. But during any single training step, only the rows that were actually used get updated. This is the key insight:
Summary: The full table is always in memory and always in the model. bbycroft only renders the T rows you asked for. Training updates only the rows used in each batch via backprop. Over billions of steps the whole table gets shaped into something meaningful.