🧠 Inside the LLM

A visual deep-dive

Inside the LLM

A visual deep-dive into how Large Language Models think, process, and generate — from raw text to output token.

#Transformers#Attention#Embeddings#Training#Decoding
📝Input🔤Tokenizer🔢Embeddings📍Position🔁Transformer ×N📊Projection✅Output
Begin the Journey ↓

Chapter 01

The Big Picture

A bird's-eye view of every step a token travels through.

Chapter 02

Tokenization

Turning raw text into the numerical units a model can process.

What is a token?

Words break into colored pieces

A token is a subword unit — not always a full word.

Thequickbrownfox

Subword split of "unbelievable":

unbelievable
GPT-4 uses ~100,000 unique tokens.

Algorithms

Three families

🔵 BPE — Byte Pair Encoding

GPT series. Merges most frequent character pairs iteratively.

🟢 WordPiece

BERT, DistilBERT. Maximizes language-model likelihood.

🟣 SentencePiece

LLaMA, T5, Mistral. Language-agnostic, works on raw bytes.

Token → ID → Embedding

Lookup table

"dog"→1234→row 1234
"cat"→582→row 582
ModelVocab
BERT30,522
GPT-250,257
GPT-4~100,000

Chapter 03

Embeddings & Positional Encoding

Tokens become vectors — and the model learns where they sit.

Token embeddings

A point in dmodel-dimensional space

"King" → [0.23, -0.85, 0.41, 0.67, 0.12, …]

Rows = tokens · Columns = dimensions.

King − Man + Woman ≈ Queen

BERT: dmodel = 768 · GPT-3: dmodel = 12,288

Positional encoding

Two ways to inject order

Sinusoidal

Original Transformer (2017).

PE(pos,2i)=sin(pos/10000^(2i/dmodel))

RoPE

LLaMA, Mistral, Gemma.

Rotates Q and K vectors by an angle proportional to position.

p0p1p2p3

RoPE generalizes better to longer contexts than fixed sinusoidal encodings.

Chapter 04

The Transformer Block

The repeating unit stacked N times — attention, residuals, normalization, and an MLP.

Input EmbeddingMulti-Head Attention+ xAdd & LayerNormFeed-Forward Network+ xAdd & LayerNormOutput Representation

Multi-Head Attention

Multiple parallel attention heads let the model attend to different relationships at once. See Chapter 5 →

Residual Connection

Output = x + Sublayer(x)

Gradients flow directly through the skip path — this is what makes deep stacks of 96+ layers trainable.

Layer Normalization

LN(x) = γ · (x − μ) / (σ + ε) + β

Stabilizes training by normalizing feature distributions per token.

Feed-Forward Network

FFN(x) = GELU(xW₁ + b₁)W₂ + b₂

Dimensions expand dmodel → 4·dmodel → dmodel.

Chapter 05

Attention Mechanisms

The most important concept in modern AI — explained six ways.

Attention is the mechanism that lets every token decide which other tokens to focus on. It is the core innovation of the Transformer architecture.

Scaled Dot-Product

Foundation

Q · K · V

Q
K
V
  1. ① scores = Q · Kᵀ
  2. ② scores = scores / √dₖ
  3. ③ mask (if needed): scores[future] = −∞
  4. ④ weights = softmax(scores)
  5. ⑤ output = weights · V
Attention(Q,K,V) = softmax(QKᵀ / √dₖ) · V
The
cat
sat
down
The
cat
sat
down

Q asks "what am I looking for?", K says "what do I contain?", V says "what will I give you?". Attention finds which K matches Q best, then returns those V values.

Multi-Head Attention

GPT · BERT · LLaMA · Claude

Many specialized views

Q, K, Vhead 1out1head 2out2head 3out3head 4out4Concat · Wᴼ
headᵢ = Attention(QWᵢQ, KWᵢK, VWᵢV)
MultiHead = Concat(head₁,…,headₕ) · Wᴼ
HeadWhat it learns
1Subject–verb agreement
2Coreference (he/she/it → noun)
3Positional patterns
4Semantic similarity
5–8Syntax, rare patterns…

GPT-3 has 96 heads × 96 layers.

Masked Self-Attention

GPT · LLaMA · Claude

Can only look LEFT — never the future

The
cat
sat
.
The
1
0
0
0
cat
1
1
0
0
sat
1
1
1
0
.
1
1
1
1

Set −∞ at masked positions before softmax → softmax(−∞) = 0 → those tokens get zero attention weight.

MaskedAttn(Q,K,V) = softmax((QKᵀ / √dₖ) + M) · V
where M[i,j] = 0 if j ≤ i, else −∞

Like writing a story — you can only reference what you've already written.

Cross-Attention

T5 · BART · Whisper

Bridges encoder ↔ decoder

Encoder out (K,V)Decoder in (Q)Cross-Attention
CrossAttn(Qdec, Kenc, Venc) = softmax(Qdec · Kencᵀ / √dₖ) · Venc

English "The cat sat" → Encoder → K, V. French Decoder Q → CrossAttn(Q, K, V) → "Le chat s'assit".

Grouped Query Attention

LLaMA 2/3 · Mistral · Gemma

Share K/V across query heads

With 32 query heads you'd need 32 K/V pairs — the KV cache explodes during inference. GQA shares K/V across groups.

MHA
Q1→KV
Q2→KV
Q3→KV
Q4→KV
4 KV
GQA
Q1Q2→KV
Q3Q4→KV
2 KV
MQA
Q1Q2Q3Q4→KV
1 KV

✅ 2–4× less KV cache memory

✅ Faster inference, same quality

✅ Enables longer context windows

LLaMA 3 70B uses 8 KV heads for 64 query heads (8:1).

Flash Attention

GPT-4 · Claude · Gemini · LLaMA 3

Same math — way smarter memory

Standard attention stores the full N×N matrix in slow GPU HBM. For N=8192: 8192² × 4B ≈ 256MB per layer.

Flash Attention tiles Q, K, V into blocks and processes one at a time in fast SRAM — never materializing the full matrix.

StandardFlash
MemoryO(N²)O(N)
SpeedBaseline2–4× faster
Full matrix stored✅ Yes❌ No
Numerically identical—✅ Yes
The math is identical — just computed smarter. No quality tradeoff.

Flash 1 (2022) → Flash 2 (2023) → Flash 3 (2024).

Chapter 06

Feed-Forward Network

The MLP between attention layers — where most of the model's facts live.

Architecture

Expand · activate · contract

x · 768→W₁ · 3072→🔥 GELU→W₂ · 768
FFN(x) = GELU(xW₁ + b₁)W₂ + b₂

Activation curves

GELUReLU

Why it matters

FFN layers store world knowledge

Geva et al. (2021) showed individual FFN neurons activate for specific facts — "capital of France", "author of Hamlet", and so on.

LayerGPT-3 params
Total175B
FFN layers~113B (65%)
Attention~58B

Remove the FFN and the model loses most of its factual knowledge.

SwiGLU(x) = (Swish(xW₁) ⊗ xWV) · W₂

LLaMA replaces GELU with SwiGLU for better empirical results.

Chapter 07

Layer Norm & Residuals

Two tiny tricks that make 96-layer models actually train.

Residual connections

y = x + Sublayer(x)
xSublayer(x)+y

Without residuals, gradients vanish in deep networks. The skip path keeps gradients alive through 96+ layers.

∂L/∂x = ∂L/∂y · (1 + ∂Sublayer/∂x)

Layer normalization

LN(x) = γ · (x − μ) / √(σ² + ε) + β
BeforeAfter

Pre-norm (LN before sublayer) trains more stably than the original post-norm. GPT-3 and LLaMA use pre-norm.

Chapter 08

Decoding Strategies

The LLM outputs a probability distribution over ~100k tokens. How do we pick the next word?

Greedy

Always pick the top token

✅ Fast · ❌ Repetitive · ❌ Boring

Beam Search

Keep top-k candidate sequences

k=1 = Greedy · k=5 typical · k=10+ translation. ✅ Better quality · ❌ Slow

Top-k Sampling

Sample from the top k tokens

P'(t) = P(t) / Σ P(top-k)

✅ Creative · ❌ k is context-sensitive

Top-p (Nucleus)

Smallest set covering p% mass

sample from {t : Σᵢ≤t P(i) ≥ p}

✅ Adapts to context · used by default in most LLMs · p=0.9–0.95

Temperature

Scale logits before softmax

P = softmax(logits / T)

T = 0.1 — spiky

T = 1.0 — natural

T = 2.0 — flat

Rule of thumb: Coding T=0.0–0.3 · Chat T=0.7 · Creative T=1.0–1.2

Chapter 09

Model Variants

Three architectures, three jobs — same Transformer DNA.

🔵 Encoder-Only

BERT · RoBERTa · DistilBERT · ELECTRA

Attention: Bidirectional — every token sees every other

Training: Masked Language Modeling (predict [MASK])

Best for: Classification · NER · QA · Embeddings

Not for generation

🟣 Decoder-Only

GPT-4 · LLaMA · Mistral · Claude · Gemini

Attention: Causal — lower triangular mask

Training: Next-token prediction (autoregressive)

Best for: Chat · Code · Reasoning · Generation

Context: 4k (GPT-3) → 128k (GPT-4) → 1M+ (Gemini 1.5)

🟢 Encoder–Decoder

T5 · BART · mT5 · Whisper · MarianMT

Attention: Encoder bidir + decoder causal + cross-attention

Training: Span corruption / seq-to-seq

Best for: Translation · Summarization · Speech-to-text

Heaviest of the three

Full comparison

FeatureEncoder-OnlyDecoder-OnlyEnc–Dec
Sees contextBoth directionsLeft onlyBoth
Primary taskUnderstandingGenerationSeq2Seq
KV cache neededNoYesYes
Cross-attentionNoNoYes
ExamplesBERTGPT-4T5

Chapter 10

Training Pipeline

From random weights to an aligned assistant.

1

Pre-Training

Common Crawl · Books · Wikipedia · GitHub · ArXiv.

Trillions of tokens, weeks on thousands of GPUs.

L = −Σ log P(tᵢ | t₁,…,tᵢ₋₁)

Output: a language model — but it doesn't follow instructions.

2

Supervised Fine-Tuning (SFT)

~10k–100k expert-written prompt → response pairs.

Output: model learns to follow instructions helpfully.

3

RLHF

  1. ① Collect human preference pairs (A vs B).
  2. ② Train a reward model on preferences.
  3. ③ Use PPO to maximize reward model score.

Output: aligned model, less harmful.

4

Modern Alternatives

DPO — skip the reward model, optimize preferences directly.

L = −log σ(β·log(π(y_w)/π_ref(y_w)) − β·log(π(y_l)/π_ref(y_l)))

RLAIF — AI feedback (Claude / GPT-4) replaces humans.

Constitutional AI — model critiques and revises itself.

Chinchilla Scaling Law

For compute-optimal training: 20 tokens per parameter. LLaMA 2 7B → 140B tokens · 70B model → 1.4T tokens.

compute →lossChinchilla-optimalUndertrainedOvertrained

Chapter 11

Modern Efficiency Tricks

The four engineering ideas behind today's fastest, longest-context models.

⚡ KV Cache

Never recompute the past

Naïve: every step recomputes K, V for all prior tokens. Wasteful.
Cached: store K, V from step 1, reuse at step 2.
mem = 2 · dmodel · num_layers · seq_len · bytes

LLaMA 3 70B @ 8k context ≈ 16 GB just for KV cache.

RoPE

Rotary positional embeddings

RoPE(x, pos) = x · e^(i·θ·pos)
  • ✅ Relative positions preserved
  • ✅ Extrapolates to longer contexts
  • ✅ No learnable parameters

Used by LLaMA · Mistral · Gemma · GPT-NeoX · Falcon.

LoRA

Fine-tune billions on one GPU

Freeze W, add small trainable matrices A and B.

W' = W + α · (A × B)
W (frozen)+AB

7B model → fine-tune only ~4M params instead of 7B.

Mixture of Experts (MoE)

Sparse — activate only 2–4 experts

tokenrouterexpert 1expert 2 ✓expert 3expert 4 ✓Σ out
MoE(x) = Σᵢ Gᵢ(x) · Eᵢ(x)

Mixtral 8×7B: 8 experts, top-2 active, 13B active per token. GPT-4 (rumored): 8 experts, ~220B active of ~1.8T total.

Chapter 12

Glossary

Every term you'll bump into, in plain English.

Token

Architecture

The atomic unit of text a model processes — often a subword piece.

Embedding

Architecture

A learned dense vector representation of a token.

Query (Q)

Attention

Vector representing what a token is looking for.

Key (K)

Attention

Vector representing what a token contains.

Value (V)

Attention

Vector representing the content a token will contribute if attended to.

Attention Score

Attention

Dot product Q·Kᵀ before softmax.

Attention Weight

Attention

Softmax-normalized attention score in [0,1].

Attention Head

Attention

One independent set of Q/K/V projections in a multi-head layer.

Transformer Block

Architecture

A repeating unit: attention + residual + norm + FFN + residual + norm.

Encoder

Architecture

Stack that produces bidirectional representations of input.

Decoder

Architecture

Causal stack that generates tokens one at a time.

Self-Attention

Attention

Attention where Q, K, V come from the same sequence.

Cross-Attention

Attention

Decoder Q attends to encoder K and V.

Masked Attention

Attention

Causal mask preventing tokens from seeing future positions.

Multi-Head Attention

Attention

Several attention heads run in parallel, then concatenated.

Grouped Query Attention

Efficiency

Multiple query heads share fewer K/V heads to shrink KV cache.

Flash Attention

Efficiency

Tiled attention that avoids materializing the full N×N matrix.

Layer Normalization

Architecture

Normalizes features per token to stabilize training.

Residual Connection

Architecture

Skip connection y = x + Sublayer(x) that lets gradients flow.

Feed-Forward Network (FFN)

Architecture

Two-layer MLP applied position-wise after attention.

GELU

Architecture

Smooth activation used in Transformer FFN layers.

SwiGLU

Architecture

Gated activation variant used by LLaMA-family FFNs.

Softmax

Decoding

Function that turns logits into a probability distribution.

Temperature

Decoding

Scalar that sharpens or flattens the next-token distribution.

Top-k Sampling

Decoding

Sample only from the k highest-probability tokens.

Top-p Sampling

Decoding

Sample from the smallest set covering p% of probability mass.

Beam Search

Decoding

Keep the top-k partial sequences at every step.

Greedy Decoding

Decoding

Always pick the single highest-probability token.

Context Window

Architecture

Maximum number of tokens the model can attend to at once.

KV Cache

Efficiency

Stored K/V tensors from previous tokens reused at inference.

Positional Encoding

Architecture

Signal that injects token order into embeddings.

RoPE

Efficiency

Rotary positional encoding that rotates Q/K by angle ∝ position.

ALiBi

Efficiency

Adds a position-dependent bias to attention scores, no embeddings needed.

BPE

Architecture

Byte Pair Encoding — merges frequent character pairs into tokens.

WordPiece

Architecture

Tokenization that maximizes language-model likelihood.

SentencePiece

Architecture

Language-agnostic tokenizer that operates on raw bytes.

Vocabulary

Architecture

The full set of tokens a tokenizer can produce.

Pre-training

Training

Self-supervised next-token prediction on massive corpora.

Fine-tuning

Training

Further training on a smaller, task-specific dataset.

SFT

Training

Supervised fine-tuning on prompt → response pairs.

RLHF

Training

Reinforcement learning from human feedback via a reward model.

DPO

Training

Direct Preference Optimization — optimize preferences without a reward model.

RLAIF

Training

RL from AI feedback instead of human labels.

Reward Model

Training

Model trained to predict human preference between two responses.

LoRA

Efficiency

Low-Rank Adapters fine-tune via small additive matrices.

MoE

Efficiency

Mixture of Experts — sparsely activates a few FFN experts per token.

Router

Efficiency

Gating function in MoE that picks experts per token.

Expert

Efficiency

One of several parallel FFNs in an MoE layer.

Scaling Laws

Training

Empirical power-laws relating loss, data, parameters, compute.

Chinchilla Law

Training

Compute-optimal training uses ~20 tokens per parameter.

dmodel

Architecture

Hidden dimension of token representations.

dff

Architecture

Inner dimension of the FFN — typically 4 × dmodel.

dₖ

Attention

Per-head dimension used inside attention.

Num Heads

Attention

Number of parallel attention heads per layer.

Num Layers

Architecture

Depth of the Transformer stack.

Batch Size

Training

Number of sequences processed in one training step.

Gradient

Training

Vector of partial derivatives of the loss w.r.t. parameters.

Backpropagation

Training

Chain-rule algorithm that computes gradients through the network.

Cross-Entropy Loss

Training

Loss comparing predicted distribution to one-hot true label.

Perplexity

Training

exp(loss) — average per-token uncertainty.