KL Divergence & the Quantization Sweet Spot

Step 1

Your model is a 54 GB brain 🧠

Qwen 3 (27 billion knobs) is amazing — but it's too big to fit on most computers.

Step 2

The fix: shrink the numbers 📦

Every knob in the brain is a number. We round those numbers to fewer digits — that's quantization. Smaller file, slightly blurrier brain.

🪙 Trade-off: smaller file = less memory + faster, but a little bit of the magic gets lost. The chart measures exactly how much is lost.

Step 3

The ruler: KL divergence 📏

How do we measure "how blurry"? We ask both brains the same questions and compare their answers. KL divergence = how different the two answers are.

Step 4

The chart: size vs. quality 📉

Now the picture comes together. Each dot is a different compression setting. Two axes:

↔️ X-axis (sideways)

File size (GB)

left = smaller file

↕️ Y-axis (up & down)

KL divergence

up = worse quality

Step 5

The sweet spot = the elbow 🍯

Look at the bend in the curve. That's everything.

← Before the elbow

The file shrinks a LOT, but quality barely drops. Great deal. Shrink away!

Past the elbow →

The file shrinks a little more, but quality falls off a cliff. Bad deal.

🎯 The sweet spot is the smallest file you can get before the cliff — the last dot that's still low on the KL ruler. For these models that's usually 4-bit (Q4_K_M), around 16 GB: ~75% smaller, but still nearly as smart.

Step 6

Reading the dots on your chart 🔍

The jargon decoded:

IQ1 / IQ2 = super aggressive (1–2 bit) → tiny but dim Q4 / IQ4 = the sweet spot Q5 / Q6 / Q8 = gentle → big but sharp