Testing local models in daily use
Local models are the favorite topic of every conversation about privacy and cost. The appeal is obvious: no per-token price, no data leaving the building, no API that can go down. I spent a month finding out how much of that actually survives contact with daily work – on an Apple Silicon Mac running Ollama, with real tasks instead of benchmarks.
The starting point
The deciding factor is not the model, it's the memory. A model has to fit entirely into RAM as weights, or it becomes unusably slow. On a machine with 16 GB, you hit the wall at roughly 8 to 9 billion parameters once the OS and a browser are also running. With 32 GB, 14B models come into reach; with 64 GB, a 32B model – but then most of it is spoken for.
That's the first honest finding: local models are a memory question, not a performance question. The chip is almost always fast enough. The RAM is not.
What works well
There's a set of tasks where local models genuinely keep up – exactly the ones that don't need brilliant language quality, but reliability:
- Extraction. Pulling names, dates, and prices out of unstructured text. An 8B model does it cleanly and quickly, at zero running cost.
- Classification. "Which category does this request belong to?" Local models almost never fail at these – and they're precisely what shows up most often in automations.
- Embeddings. Vectorization is its own model class anyway and should run locally. It's expensive at volume through an API and identical in quality locally.
- Short summaries. Condensing an email, a paragraph, or a ticket into three sentences. Good enough that you can't tell the difference from the cloud.
The common thread: everything where the answer is unambiguous. Where there is one correct result and no room for style.
Where it shows
And then there's the other side. As soon as the task demands that the phrasing itself be the product, the difference becomes visible – and quickly:
- Long, nested reasoning. An 8B model loses the thread on multi-step inferences. It sounds confident but is subtly off – and that error is harder to spot than an obvious one.
- Instructions over longer texts. The longer the context, the more often an instruction from the start is simply forgotten. The large cloud models hold up noticeably longer.
- Nuanced phrasing. Tone, audience, finesse. A local model writes correctly, but flat. For a blog post or a client email, that's not enough for me.
The pattern is always the same: local models don't fail with errors, they fail with mediocre answers that look correct. That's the more dangerous variant.
The numbers I measured
Speed is rarely the problem. An 8B model delivers around 40 to 50 tokens per second on a current Apple Silicon chip – perfectly fine for interactive use. A 14B model lands at 20 to 25. Only the large models, which no longer fit in memory anyway, get slow.
What does get annoying is the first start. A model has to be loaded into memory before it answers. Depending on size, that's 10 to 30 seconds. For one call per hour it's irrelevant; for a feeling of responsiveness in daily use, it isn't.
How I split it up now
After a month, my boundary is clear, and it matches what I've been preaching for a while – except now I've measured it myself:
- Local: embeddings, extraction, classification, short summaries. The grunt work.
- Cloud: everything where language quality or long reasoning matters. The blog, consulting, multi-step inference.
That boundary isn't ideology, it's a question of cost and quality. Running an 8B model locally costs nothing, and it's fully sufficient for half of my tasks. For the other half it would be the wrong trade-off – I'd save on tokens and pay with worse answers.
The real gain of the month wasn't a specific model, but a clearer picture of what I actually need. Local models aren't a replacement for the cloud. They're a tool for the part of the work where the cloud was oversized to begin with.