#On Device Ai

Your SSD Is the New VRAM

Swiftlet streams MoE experts from SSD, fitting an 80B Qwen in 4.3GB of RAM — and a 35B on an iPhone.

Article · 1w ago1

A Solo Dev Squeezed Usable TTS Into 9.36M Parameters

Inflect-Micro-v2 isn't the breakthrough its headline implies, but it resets the floor for offline speech on cheap CPUs.

Article · 2w ago2

Small Models That Know When to Phone Home

Cactus Hybrid ships Gemma 4 with a built-in confidence probe that decides when to hand off to the cloud.

Article · 2w ago2

Bonsai 27B Puts Real Agents on Phones

A 3.9 GB 1-bit 27B model clears the iPhone memory gate that blocked every prior build of this size.

Article · 4w ago1

Quantize and Run Llama 3.2 on Apple Silicon with llama.cpp

Build llama.cpp with Metal, convert Llama 3.2 3B to Q4_K_M GGUF, and benchmark real prompt-processing and generation throughput on your specific chip.

Tutorial · 1mo ago0

Apple Core AI and the Local LLM Tax

Apple's new Core AI framework optimizes generative models for Apple Silicon, but developers must navigate compilation and specialization hurdles.

Article · 1mo ago1

Claude Slots Into Apple's Foundation Models Framework

A new Swift package lets the same on-device session API call Claude in the cloud, swapping models by one argument.

News · 1mo ago3