Run Mixtral 8x7B Locally with llama.cpp and Benchmark MoE vs. Dense

What you'll build / learn

You'll compile llama.cpp from source, run a 4-bit quantized Mixtral 8x7B locally, and use llama-bench to prove the mixture-of-experts (MoE) trade-off with your own numbers: generation speed tracks the ~12.9B active parameters per token, while memory cost tracks all 46.7B.

Prerequisites

Verified against llama.cpp build b10261 (August 4, 2026) and the current Hugging Face hf CLI. Commands are for macOS/Linux; Windows users should work inside WSL2.

• Disk: ~40 GB free (both models).
• Memory: the Q4_K_M Mixtral weighs 26.49 GiB. You need one of: 32 GB+ system RAM (CPU-only), a GPU with 32 GB+ VRAM for full offload, or a 24 GB GPU plus ~16 GB RAM using the MoE CPU-offload flag covered in step 5. On Apple Silicon, 48 GB unified memory is comfortable; on 32/36 GB machines use the Q3_K_M file (21.0 GiB) instead.
• Toolchain: git, CMake ≥ 3.14, and a C++17 compiler (Xcode Command Line Tools on macOS, build-essential on Debian/Ubuntu).
• NVIDIA only: CUDA Toolkit 12.x installed, with nvcc on your PATH.

llama.cpp ships new tagged builds daily and flags do change — everything below is checked against b10261.

1. Build llama.cpp

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

On macOS, Metal GPU support is enabled by default, so a plain build is all you need:

cmake -B build
cmake --build build --config Release -j 8

On Linux with an NVIDIA GPU, turn on the CUDA backend:

cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j 8

Binaries land in build/bin/. Confirm the two you'll use exist:

ls build/bin/llama-cli build/bin/llama-bench

2. Download the models

Install the Hugging Face CLI and pull single files — never clone a GGUF repo, it holds every quantization level:

curl -LsSf https://hf.co/cli/install.sh | bash

hf download mradermacher/Mixtral-8x7B-Instruct-v0.1-GGUF \
  Mixtral-8x7B-Instruct-v0.1.Q4_K_M.gguf --local-dir models

hf download Qwen/Qwen3-14B-GGUF \
  Qwen3-14B-Q4_K_M.gguf --local-dir models

That's 28.4 GB for Mixtral and 9.0 GB for Qwen3-14B, our dense baseline. Qwen3-14B is the fair fight: its 14.8B parameters sit right next to Mixtral's 12.9B active parameters, and both files use the same Q4_K_M quantization, so any throughput gap comes from architecture, not quant format.

Skip the famous TheBloke Mixtral quants from December 2023 — they predate llama.cpp's 2024 switch to merged expert tensors and the GGUF chat-template metadata. The mradermacher conversion above uses the modern format.

3. Smoke-test Mixtral

./build/bin/llama-cli -m models/Mixtral-8x7B-Instruct-v0.1.Q4_K_M.gguf \
  -c 8192 -ngl 99

-ngl 99 offloads every layer to the GPU (recent builds also accept -ngl auto); -c 8192 caps the context so the KV cache doesn't inflate memory use. Because the GGUF carries a chat template, llama-cli drops you straight into interactive conversation mode. Ask it something, confirm you get coherent output, then exit with Ctrl+C — it prints token-throughput timings on the way out. Watch the load logs: a line per layer ending in offloaded 33/33 layers to GPU (32 blocks plus the output layer) means the whole model is resident on the GPU.

For a scripted one-shot run instead of chat, use -no-cnv with Mixtral's [INST] template:

./build/bin/llama-cli -m models/Mixtral-8x7B-Instruct-v0.1.Q4_K_M.gguf \
  -c 4096 -ngl 99 -no-cnv -n 128 \
  -p "[INST] Explain mixture-of-experts routing in two sentences. [/INST]"

4. Benchmark MoE vs. dense

llama-bench measures two phases: pp (prompt processing, compute-bound) and tg (token generation, memory-bandwidth-bound). Generation is where MoE shines, because each token only reads the router plus 2 of the 8 experts per layer.

./build/bin/llama-bench -m models/Mixtral-8x7B-Instruct-v0.1.Q4_K_M.gguf \
  -m models/Qwen3-14B-Q4_K_M.gguf -p 512 -n 128 -ngl 99

Each test runs 5 repetitions by default and reports mean ± stddev; add -r 3 if you're impatient.

5. Fit it on a smaller GPU with --n-cpu-moe

If Mixtral doesn't fit your VRAM, don't drop whole layers with a lower -ngl. Use --n-cpu-moe N, which keeps the MoE expert weights of the first N layers in system RAM while attention, norms, and the router stay on the GPU. Experts are ~96% of Mixtral's weights but only a quarter of them fire per token, so this costs far less speed than evicting entire layers:

./build/bin/llama-cli -m models/Mixtral-8x7B-Instruct-v0.1.Q4_K_M.gguf \
  -c 8192 -ngl 99 --n-cpu-moe 12

On a 24 GB card, start around --n-cpu-moe 12 (Mixtral has 32 layers) and lower it until you hit an out-of-memory error, then back off. llama-bench takes the same flag as -ncmoe and accepts a comma-separated sweep, so you can find the knee of the curve in one command:

./build/bin/llama-bench -m models/Mixtral-8x7B-Instruct-v0.1.Q4_K_M.gguf \
  -ngl 99 -ncmoe 12,16,24,32 -p 512 -n 128

Verify it works

Success looks like a Markdown table from step 4 in this shape (these reference numbers are from a 64 GB M3 Max — your absolute values will differ by hardware):

| model                | size      | params  | backend    | ngl | test   | t/s            |
| -------------------- | --------: | ------: | ---------- | --: | ------ | -------------: |
| llama 8x7B Q4_K_M    | 26.49 GiB | 46.70 B | Metal,BLAS |  99 | pp 512 | 235.11 ± 1.82  |
| llama 8x7B Q4_K_M    | 26.49 GiB | 46.70 B | Metal,BLAS |  99 | tg 128 | 32.40 ± 0.21   |
| qwen3 14B Q4_K_M     |  8.38 GiB | 14.77 B | Metal,BLAS |  99 | pp 512 | 341.27 ± 2.05  |
| qwen3 14B Q4_K_M     |  8.38 GiB | 14.77 B | Metal,BLAS |  99 | tg 128 | 35.87 ± 0.33   |

The result that matters is the ratio, not the absolute numbers: Mixtral's tg 128 should land within roughly 25% of the dense 14B's, despite carrying 3.2× the parameters. That's expert routing working — per token it reads about the same number of bytes as a 13B dense model. For contrast, a dense 46.7B model at Q4_K_M would stream all 26.49 GiB through memory for every token and generate roughly 3× slower. The MoE bill arrives in the size column instead: 26.49 GiB resident versus 8.38 GiB.

Troubleshooting

ggml_backend_cuda_buffer_type_alloc_buffer: allocating 26550.00 MiB on device 0: cudaMalloc failed: out of memory — the full Q4_K_M doesn't fit your card. Add --n-cpu-moe 12 (raise N until it loads) rather than cutting -ngl, or use the Q3_K_M file.

CMake Error at ggml/src/ggml-cuda/CMakeLists.txt (message): CUDA Toolkit not found — CMake can't see nvcc. Install CUDA Toolkit 12.x, then reconfigure with the compiler pinned: CUDACXX=/usr/local/cuda/bin/nvcc cmake -B build -DGGML_CUDA=ON.

Mac loads the model but generation crawls, with a log warning that allocated size exceeds recommendedMaxWorkingSetSize — macOS caps GPU-wired memory at roughly 70% of unified RAM, so a 26.49 GiB model thrashes on 32/36 GB machines. Raise the cap (resets on reboot) with sudo sysctl iogpu.wired_limit_mb=30720, or switch to Q3_K_M.

hf: command not found — the installer put the binary in ~/.local/bin, which isn't on your PATH. Run export PATH="$HOME/.local/bin:$PATH" (add it to your shell rc), or pip install -U huggingface_hub to get the same CLI via pip.

Next steps

Swap llama-cli for llama-server and you get an OpenAI-compatible API on localhost:8080 with the same flags, including --n-cpu-moe. Run llama.cpp's llama-perplexity tool over both models to see the quality side of the trade — speed parity means little if the 47B MoE doesn't beat the dense 14B on your workload. Then point the same benchmark harness at newer MoE models like Qwen3-30B-A3B or OpenAI's gpt-oss-20b, where sparser routing (3B–4B active) makes the effect even more dramatic. When you outgrow static Q4_K_M files, look into imatrix-calibrated quants, which measurably cut quantization loss at the same size.