Over the past month, a wave of new local language models has arrived that run on everyday hardware instead of data-center rigs. The common thread is aggressive 4‑bit quantization and distillation from much larger teachers, techniques that shrink memory footprints enough to fit on an 8 GB laptop GPU or even a phone.
Models That Fit in a Backpack
Liquid AI’s LFM3 1.7B model, released October 27, runs in a 4‑bit quantized form using roughly 1 GB of RAM and supports 16 languages. A single Ollama command — ollama run lfm3 — gets it going on a laptop.
On November 13, the Haru‑4B voice‑dialogue model appeared with an official Mac app. Its makers report about 65 milliseconds of latency on a MacBook, showing that multimodal audio models are starting to work locally, though memory requirements were not disclosed.
For developers with an 8 GB RTX 3060, Qwen3.8‑9B‑Distill has become a go‑to choice. It is a fine‑tuned version of Qwen3.5‑9B trained on outputs from the far larger Qwen3.8‑2.4T A95B model. The distillation lifts MMLU scores by roughly 20 percentage points while costing only 1.5 percentage points on GSM8K math accuracy.
How Quantization and Distillation Cut the Bill
Four‑bit quantization stores each weight in half the space of the standard 8‑bit format, letting models like LFM3 1.7B squeeze into 1 GB of RAM. Distillation goes a step further: a massive teacher model generates training data for a compact student, transferring reasoning patterns without the parameter count. Qwen3.8‑9B‑Distill demonstrates the trade‑off — stronger general knowledge, slightly weaker arithmetic.
Memory‑mapping, the same trick llama.cpp uses on Android, lets the runtime load only the model shards needed for the current token. PocketPal AI on a Galaxy S21 (8 GB RAM) runs Gemma 2 2B this way, keeping the app alive without loading the whole file at once.
Benchmarks and Real‑World Gotchas
OpenClaw’s agent benchmark on an RTX 3090 paired a 9 B model with an embedding model and measured 11.4 GB of GPU memory consumed just by the two models sitting idle. A single one‑line question ballooned to an 18,415‑token prompt because the agent stuffs system prompts, tool definitions and workspace files into every request. The first reply took 1 minute 10 seconds — 59 seconds of that was model loading — while subsequent turns settled around 2 to 5 seconds thanks to Ollama’s prompt cache.
Long system prompts and agent scaffolding remain the hidden memory tax on local inference. Developers planning to run agents locally need to budget for that overhead, not just the base model size.
Tooling Catches Up
Ollama moved from v0.40.8 to v0.40.16 in the same month. Version 0.40.11 added Mac text‑to‑speech support, and 0.40.16 cut memory use for “Qwen3.8 Flash Next” on Apple silicon by about 27 GiB. New model support now lands within days of release. LM Studio shipped a bug‑fix for M5 Macs in version 0.4.27.
On Android, apps like PocketPal AI, ChatterUI and SmolChat bridge the llama.cpp runtime through a thin native layer, making on‑device inference practical on phones a few years old.
What Developers Can Expect Next
The hardware ladder is clarifying. A 16 GB machine can now host coding assistants such as gpt‑oss‑20b under an Apache 2.0 license, while Gemma 4 and Moonshot’s Kimi K3.1 Pro still target workstation‑class GPUs. Preferred Networks’ PLaMo 3 Translate, released November 5, offers 53‑language translation in four sizes up to 31B, with the largest 4‑bit variant needing 18.66 GB — proof that specialized distilled models are filling niches once reserved for cloud APIs.
As 4‑bit quantization and distillation become standard, the next frontier is multimodal models like Haru‑4B that speak while listening, and agent frameworks that tame the prompt‑size explosion measured by OpenClaw.
Sources
- A summary of local LLM news from the past month|ゆっくりSi
- Large language model
- 4 small but powerful local LLMs worth trying out on your computer this weekend
- Best Local LLM for Coding: $0 Weights vs $2 API [2026]
- I just replaced Duolingo with a local LLM that took two minutes to set up on my Android phone
- Running an AI Agent on a Local LLM: What I Measured on One GPU
Photo: Christian Wiediger / Unsplash (Unsplash License)





