GGUF — Local AI Model Format

01 —

What Is GGUF

One file. Everything inside. Runs anywhere.

GGUF is a binary container format for AI language models, created by Georgi Gerganov — the author of llama.cpp. It replaced its predecessor GGML in August 2023 and became the standard for local AI within months.

Inside every .gguf file

Model Weights

The billions of numbers that ARE the model's knowledge. This is 98% of the file by size.

Inside every .gguf file

Tokenizer Vocabulary

~32k entries mapping text fragments to the integer IDs the model actually processes.

Inside every .gguf file

Architecture Config

Layer counts, attention head sizes, context window length — no separate config.json needed.

Inside every .gguf file

Metadata

Model name, author, license, quantization type, and any custom fields the creator added.

File naming anatomy

Meta-Llama-3-70B-Instruct-Q4_K_M.gguf

● Model name ● Variant / task ● Quantization level ● Format

Think of it like an .exe for an AI brain — one file, drag and drop, run anywhere. No framework install, no config file hunt, no GPU required.

02 —

Problems It Solves

What local AI looked like before GGUF existed

⚠ Models were split across dozens of files

Before

model-00001-of-00014.bin
model-00002-of-00014.bin
model-00003-of-00014.bin
… (14 files)

After

Meta-Llama-3-70B-Q4.gguf


One file. Done.

⚠ Loading required a full ML framework

Before

pip install torch transformers
accelerate sentencepiece
protobuf bitsandbytes…

After

./llama-cli \
  -m model.gguf \
  -p "Hello"

⚠ Normal computers couldn't run models at all

Before

Required: NVIDIA A100 80GB
Cost: $10,000–$30,000
FP32 Llama-70B: 280GB VRAM

After

16GB laptop RAM runs
a capable 7B model.
Q4 Llama-70B: 40GB RAM.

⚠ No metadata — config files were separate

Before

tokenizer_config.json
config.json
special_tokens_map.json
generation_config.json

After

All embedded in the
.gguf file header.

Self-contained.

The core unlock: GGUF made running AI locally as simple as running any other program — no PhD required, no cloud dependency, no recurring API cost.

03 —

Quantization

How GGUF shrinks a 280GB model into something you can actually run

A model's weights are stored as numbers. At full precision, each is a 32-bit float (4 bytes). Quantization rounds them down to fewer bits — trading a tiny amount of accuracy for a massive size reduction. The math is brutal:

70,000,000,000 params × 32 bits = 280 GB
70,000,000,000 params × 4 bits = 35 GB + overhead → Q4_K_M: ~40 GB

Level	Bits	Size (Llama 3 70B)	Quality	Notes
FP32 (full)	32	280 GB	100%	Research only. $10k+ GPU.
FP16	16	140 GB	99%	Still needs a huge GPU.
Q8_0	8	70 GB	97%	Near-perfect. High RAM req.
Q5_K_M	5	49 GB	94%	Great quality/size balance.
Q4_K_M ★	4	40 GB	91%	Community sweet spot.
Q3_K_M	3	29 GB	85%	Noticeable degradation.
Q2_K	2	20 GB	71%	Emergency / hobbyist only.

Q4_K_M — 4-bit quantized with K-means optimization on important weight clusters. 91% of the original quality at 14% of the original size. A Q4_K_M Llama-3-70B outperforms a full-precision Llama-3-8B on most benchmarks. You're compressing the representation, not the intelligence.

04 —

Alternatives

Every format compared — and when you'd actually reach for something else

Format	Single file	CPU / RAM	Quantized	Metadata	Verdict
PyTorch .bin	✗	✗	✗	✗	❌ Fragmented, GPU-gated
SafeTensors	✓	✗	✗	✗	⚠ Safer loading, still GPU-only
GGML (old)	✓	✓	✓	✗	⚠ GGUF's predecessor, abandoned
ONNX	✗	✓	✗	✗	⚠ Production deploy, not local LLMs
EXL2	✓	✗	✓	✓	⚠ Better quality, GPU required
GGUF	✓	✓	✓	✓	✅ The standard

The real reason GGUF won isn't purely technical — it arrived at the exact moment LLaMA models leaked/released and millions of people wanted to run AI locally. The ecosystem (Ollama, LM Studio, Jan, llama.cpp) formed around it fast, and network effects did the rest.

You have a good GPU → use EXL2. ExLlamaV2 gets better quality at the same file size and runs 2–3× faster on a 3090/4090. GGUF's CPU advantage is irrelevant when you have VRAM.

You're deploying to production → use ONNX or TensorRT. GGUF is optimized for local single-user inference. Production serving at scale needs different tradeoffs: batching, hardware-specific kernels, memory layout.

You're fine-tuning → stay in SafeTensors / PyTorch. GGUF is read-only for inference. Training still happens at full precision. You convert to GGUF after for distribution.

05 —

Why So Large

The disk holds the brain. There's no compression trick that makes that cheap.

A GGUF file IS the model. Not a reference to it — the actual numbers encoding billions of learned associations. It has to be big because intelligence is big.

Where the bytes go in a typical Q4_K_M file:

Model weights 98%

The quantized parameters — the knowledge itself

Tokenizer vocab 1.2%

~32k token embeddings and special tokens

Metadata + headers 0.8%

Architecture config, name, license, quant info

The disk is just storage. When you run the model, it loads into RAM (or VRAM):

Storage / Disk

Holds the file at rest.

Needs: just the file size

Cheap — 2TB SSD is fine

RAM

Loads model to run it. This is the real constraint.

Needs: file size + ~20% overhead

Bottleneck for most people

VRAM (GPU)

Optionally offloads layers for speed.

Optional, but 10–20× faster inference

Speed multiplier

Model	Q4_K_M size	RAM needed	Runs on
Llama 3.2 3B	~2 GB	4 GB+	Any modern laptop
Llama 3.1 8B	~5 GB	8 GB+	Most laptops
Mistral 7B	~4.5 GB	8 GB+	Most laptops
Llama 3 70B	~40 GB	48 GB+	Mac Studio / workstation
Llama 3 405B	~230 GB	256 GB+	Server-class machine

Bottom line: disk is cheap. 40GB on a 2TB SSD is 2% of your drive. Always check RAM first before downloading a model — that's the spec that actually matters.

06 —

Common Questions

The things people actually wonder about after learning GGUF

The quantized file size is just the weights at rest. When the model runs, it needs several additional memory regions:

KV cache (Key-Value cache) — the largest variable cost. Every token in the context window requires cached attention keys and values for all layers. A 70B model at 4096-token context adds ~4–8 GB. At 32k context it can add 30+ GB.
Activation buffers — temporary tensors holding intermediate values during each forward pass (computing one layer at a time). These get reused between tokens but must exist in full.
Runtime graph — the model's computation graph loaded into memory by llama.cpp / Ollama. Metadata, layer pointers, thread synchronization structures.
OS memory mapping overhead — mmap() pages the file in lazily, but the OS page table entries and buffer cache itself consume ~1–3% on top.

The "20%" rule is a rough minimum for short context windows. In practice, if you set --ctx-size 8192, that KV cache alone can double your actual RAM usage over the file size.

Practical rule: always check --ctx-size settings in Ollama / LM Studio. The default context length is where most people get surprised by RAM usage.

The file is compact on disk, but running a model isn't like playing a video file — it's live computation. Two things happen the moment you load it:

Dequantization buffer — Q4 weights are stored at 4 bits, but most CPUs can't do arithmetic in 4-bit. llama.cpp dequantizes small weight blocks to FP32/FP16 just-in-time for each computation, holding a working buffer of the current layer's dequantized weights in RAM.
KV cache pre-allocation — when you start the model, the runtime pre-allocates the full context window's KV cache upfront so it never needs to grow dynamically mid-conversation. This is reserved immediately at startup.

So the memory footprint is: file weights (mmap'd) + dequant buffer + kv cache + activations. The file itself often isn't fully copied into RAM — the OS maps it and loads pages on demand — but everything else is freshly allocated.

GGUF's performance story is: unmatched on CPU, mid-tier on GPU. Here's the landscape:

Format / Runtime	Hardware	Tokens/sec vs GGUF	Why
EXL2 + ExLlamaV2	GPU only	2–3× faster	Custom CUDA kernels, fused ops
GPTQ + AutoGPTQ	GPU only	1.5–2× faster	GPU-native quantized matmul
AWQ	GPU only	1.5–2× faster	Activation-aware, less accuracy loss
vLLM (FP16)	GPU, multi-user	5–10× throughput	Continuous batching, PagedAttention
MLX (Apple)	Apple Silicon only	1.3–1.8× faster	Unified memory, Metal GPU ops
GGUF + llama.cpp	CPU / any GPU	Baseline	Best CPU perf, ok GPU perf

The catch: none of those alternatives run on CPU-only. If you don't have a discrete GPU, GGUF is the only viable option. If you do have a GPU, EXL2 is the performance leader for interactive single-user inference.

vLLM's "5–10× throughput" is measuring something different — it's about serving many users simultaneously via batching, not single-token latency. For one person chatting, it's closer to parity with GGUF on GPU.

At the same nominal bit-width, quantized model files are roughly the same size across formats — they're all storing the same number of parameters. The bit-depth determines the floor.

But there are meaningful differences in practice:

GGUF vs SafeTensors: SafeTensors stores at FP16/BF16 (16 bits) — no quantization. A GGUF Q4 file is ~8× smaller than a SafeTensors version of the same model. This isn't GGUF being clever; it's quantization doing the work.
GGUF vs EXL2 at "4-bit": EXL2 uses non-uniform quantization — it allocates more bits to important weight clusters and fewer to unimportant ones. A "4-bit average" EXL2 can be slightly smaller than GGUF Q4 at better quality, because it's spending bits where they matter.
GGUF vs GPTQ/AWQ at 4-bit: Files are nearly identical in size. The quantization schemes are different, but both land at ~4 bits per parameter.
GGUF overhead: GGUF bundles tokenizer + metadata inline, adding ~100–300 MB over a raw weight file. GPTQ/AWQ models often ship with these as separate files, so the raw weights are fractionally smaller but the total download is similar.

Format	Llama 3 70B approx.	Notes
SafeTensors (FP16)	~140 GB	No quantization
GPTQ 4-bit	~38 GB	Separate tokenizer files
AWQ 4-bit	~38 GB	Separate tokenizer files
GGUF Q4_K_M	~40 GB	Tokenizer bundled in
EXL2 4-bit avg	~36–38 GB	Better quality, GPU only

GGUF is not smaller than GPU-based 4-bit formats. What makes it distinctive is CPU compatibility and self-contained bundling — not compression ratio.