Topic

#Quantization

12 articles on Quantization — news, releases, guides and analysis from the SourceFeed engine.

Tutorial 1d ago 0

Convert and Quantize Hugging Face Models to GGUF for llama.cpp

Turn any Hugging Face checkpoint into a 4-bit GGUF that runs fast and small on your own hardware.

Mariana Souza

Latent Reasoning Escapes the Lab, Bolted Onto DeepSeek-V4

A 36M-parameter head makes V4-Flash think silently in latent space — and the serving hacks matter more than the benchmarks.

Article · 4d ago2

Run Mixtral 8x7B Locally with llama.cpp and Benchmark MoE vs. Dense

Compile llama.cpp, run quantized Mixtral 8x7B, and measure the mixture-of-experts speed-versus-memory trade-off yourself.

Tutorial · 1w ago0

An 8B Fine-Tune Now Fits in 4 GB of VRAM

Soup streams frozen decoder layers through the GPU one at a time, and proves the arithmetic is bit-exact.

Article · 1w ago1

Your SSD Is the New VRAM for Local LLMs

Swiftlet streams sparse Qwen experts off disk, fitting an 80B model into 4.3 GB of Mac RAM.

Article · 1w ago1

$Cloudflare's Quantization Math Is Right. The Disclosure Isn't$

Cloudflare's Quantization Math Is Right. The Disclosure Isn't

FP8 caches and INT4 weights are becoming table stakes; what providers tell you at purchase time is the real fight.

Article · 1w ago0

Quantize the Decode, Not the Prefill

Cloudflare's production numbers for Kimi K2.6 and GLM 5.2 show where trillion-parameter inference actually gets cheaper.

Article · 1w ago5

An LLM on an $8 Chip Is Mostly a Memory Trick

The ESP32-S3 demo matters less for what it writes than for how it splits hot and cold parameters.

Article · 2w ago0

Bonsai 27B Puts Real Agents on Phones

A 3.9 GB 1-bit 27B model clears the iPhone memory gate that blocked every prior build of this size.

Article · 4w ago1

Quantize and Run Llama 3.2 on Apple Silicon with llama.cpp

Build llama.cpp with Metal, convert Llama 3.2 3B to Q4_K_M GGUF, and benchmark real prompt-processing and generation throughput on your specific chip.

Tutorial · 1mo ago0

Demystifying Integer Quantization for Neural Network Inference

A low-level look at the mathematics and hardware mechanics that shrink massive models without destroying accuracy.

Article · 1mo ago0

Xiaomi's MiMo-V2.5-Pro-UltraSpeed Pushes a 1T Model Past 1000 Tokens/Sec on Commodity GPUs

Through FP4 quantization, block-level speculative decoding, and the TileRT system stack, Xiaomi claims trillion-parameter decode speeds normally reserved for custom silicon — on a single 8-GPU node.

News · 2mos ago5