Convert and Quantize Hugging Face Models to GGUF for llama.cpp

What you'll build / learn

You'll take a stock Hugging Face model (Qwen3-0.6B), convert it to GGUF, and quantize it to 4-bit with llama.cpp — turning a 1.5 GB safetensors checkpoint into a ~400 MB file that runs locally on your CPU or GPU. The same three commands work for any supported architecture, so swap in whatever model you actually care about.

Prerequisites

Verified against llama.cpp release b10375 (August 2026) with Python 3.12 on macOS 15 and Ubuntu 24.04.

• Git, CMake (≥ 3.14), and a C++17 compiler — Xcode Command Line Tools on macOS, build-essential + cmake on Debian/Ubuntu.
• Python 3.10+ with venv. The conversion deps pin torch 2.11.0 (CPU wheel) and transformers 4.57.6.
• ~4 GB free disk for this model: 1.5 GB download, 1.2 GB converted file, 0.4 GB quantized file. Budget roughly 4× a model's parameter count in bytes if you bring your own.
• No GPU required. Conversion and quantization are CPU-only operations; a GPU only helps at inference time.
• No Hugging Face account needed for Qwen3-0.6B (Apache-2.0, ungated). Gated models like Llama need hf auth login first.

1. Clone and build llama.cpp

You need the source checkout either way — the conversion script lives in the repo root — so build the binaries from it too:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config Release -j 8

This produces llama-quantize, llama-cli, and friends under build/bin/. On macOS, Metal support is compiled in by default; on NVIDIA boxes add -DGGML_CUDA=ON to the first cmake call if you want GPU inference later. Prebuilt binaries from the releases page also work, but you still need this repo for the Python script.

2. Install the Python conversion dependencies

The converter is a Python script with its own pinned requirements. Keep them in a venv so the pinned torch/transformers versions don't fight your global site-packages:

python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install -r requirements.txt

This pulls a CPU-only PyTorch wheel (via the download.pytorch.org/whl/cpu index the requirements file specifies), so it won't drag in CUDA libraries. It also installs transformers, which ships the hf CLI you'll use next. One caveat from the llama.cpp docs: the pins install transformers 4, and some very new models (Gemma 4, for example) need transformers 5 — pip install -U transformers is safe if conversion complains about an unrecognized model type.

3. Download the model from Hugging Face

Grab the full repo — the converter needs config.json and the tokenizer files, not just the weights:

hf download Qwen/Qwen3-0.6B --local-dir Qwen3-0.6B

You'll end up with model.safetensors (1.5 GB) plus config and tokenizer files in Qwen3-0.6B/. For a gated model, run hf auth login with a token from huggingface.co/settings/tokens before downloading.

4. Convert to GGUF

GGUF is llama.cpp's single-file format: weights, tokenizer, and metadata together, laid out for mmap-friendly loading. Convert at full precision first — you always quantize from a bf16/f16 GGUF, never re-quantize an already-quantized file, because each lossy pass compounds the error:

python3 convert_hf_to_gguf.py Qwen3-0.6B \
  --outfile Qwen3-0.6B-BF16.gguf \
  --outtype bf16

The script logs each tensor as it maps and writes it, then finishes with:

INFO:hf-to-gguf:Model successfully exported to Qwen3-0.6B-BF16.gguf

--outtype auto (the default) also works — it matches whatever precision the source weights use. And if you'd rather skip step 3 entirely, --remote streams tensors straight from the Hub: python3 convert_hf_to_gguf.py Qwen/Qwen3-0.6B --remote --outfile Qwen3-0.6B-BF16.gguf --outtype bf16.

5. Quantize to 4-bit

Now shrink it. Q4_K_M is the community default for a reason: roughly 4.5 bits per weight with the quality-critical tensors kept at higher precision, which costs little accuracy on most models:

./build/bin/llama-quantize Qwen3-0.6B-BF16.gguf Qwen3-0.6B-Q4_K_M.gguf Q4_K_M

It runs in under a minute for a model this size, printing per-tensor lines as it converts. Other useful targets: Q8_0 (near-lossless, ~2× smaller than bf16), Q5_K_M (middle ground), Q3_K_M and below (only when you're desperate for RAM). Run ./build/bin/llama-quantize --help for the full list with per-type size/quality estimates.

Verify it works

Check the sizes first:

ls -lh Qwen3-0.6B-*.gguf

-rw-r--r--  1 you  staff   1.2G Aug 12 10:41 Qwen3-0.6B-BF16.gguf
-rw-r--r--  1 you  staff   397M Aug 12 10:44 Qwen3-0.6B-Q4_K_M.gguf

Then actually run the quantized model. -st (single turn) answers one prompt and exits instead of dropping into interactive chat:

./build/bin/llama-cli -m Qwen3-0.6B-Q4_K_M.gguf \
  -p "Explain what GGUF is in one sentence." -st -n 256

After the loader output you should see a coherent answer (Qwen3 emits a <think>…</think> reasoning block first — that's normal), something like:

GGUF is a binary file format used to store quantized large language
models for efficient local inference with llama.cpp.

If the model loads, generates fluent text, and exits cleanly, your conversion and quantization are good.

Troubleshooting

ERROR:hf-to-gguf:Model <name>ForCausalLM is not supported — the architecture isn't in your checkout's converter, usually because the model is newer than your clone. git pull, re-run pip install -r requirements.txt, and rebuild. If it still fails, the architecture genuinely isn't supported yet — search the llama.cpp issues/PRs for it.

ValueError: The checkpoint you are trying to load has model type '<x>' but Transformers does not recognize this architecture — your transformers is too old for the model. Run pip install -U transformers inside the venv (the llama.cpp docs explicitly bless this over the pinned version).

ModuleNotFoundError: No module named 'gguf' (or 'torch') — you're running the script outside the venv, or skipped step 2. Run source .venv/bin/activate and retry; check which python3 points into .venv.

GatedRepoError: 403 Client Error … Access to model <x> is restricted — the model requires accepting a license on its Hugging Face page. Accept it in the browser while logged in, then hf auth login with a read token and re-download.

Next steps

Serve your quantized model over an OpenAI-compatible API with ./build/bin/llama-server -m Qwen3-0.6B-Q4_K_M.gguf, then point any OpenAI client at localhost:8080. To squeeze quality out of aggressive quants (Q3 and below), generate an importance matrix with llama-imatrix on a calibration text file and pass it to llama-quantize --imatrix. For sharing, hf upload pushes your GGUF to a Hub repo so others can pull it with llama cli -hf you/your-model-GGUF. And if you want quants without any local setup, the GGUF-my-repo space runs this exact pipeline in your browser.