The New Local AI Flex Is Memory, Not Model Size
Local AI has moved past the old question of whether a laptop can run a useful model at all. It can. The better question is what trade was made to make the model fit.
That trade is usually hiding in the details: quantization, context length, cache compression, CPU offload, speculative decoding, sparse activation, or a backend that is doing something clever with Apple silicon. None of those are bad. Some are excellent. But they are not the same thing as saying a full large model simply “runs in 2 GB RAM” in the way a normal reader would understand it.
Apple silicon makes these demos especially tempting because the memory story is genuinely different. The CPU and GPU are not fighting over separate pools in the same way a desktop PC with a discrete graphics card does. Apple’s Metal documentation exposes unified-memory capability for devices, and Apple’s MLX project is explicitly built for machine learning on Apple silicon. That matters. It does not repeal arithmetic. Apple documents unified memory at the Metal device level, and MLX is described by Apple as an array framework for Apple silicon.
So yes, local AI on Macs is getting better. Also yes, a screenshot and a punchy headline are not a capacity plan.
Model Parameters Are Not The Memory Requirement
The first trap is treating the parameter count as the whole memory bill. It is only the most visible line item.
A model with 26 billion parameters does not always activate every parameter for every token if it uses a mixture-of-experts design. A 4-bit quantized file is much smaller than a bfloat16 checkpoint. A long context can eat memory through the KV cache even when the weights fit comfortably. A backend can offload some work to CPU, stream pieces, compress cache data, or use custom kernels. The result can be real and still be narrower than the marketing sentence makes it sound.
Google’s Gemma documentation is a useful example because it publishes model variants and memory-oriented deployment details instead of relying on vibes. The Gemma 4 model overview lists multiple sizes, including a 26B A4B mixture-of-experts variant, and shows that precision and deployment format change the memory picture materially. Google’s Gemma model documentation is the kind of source you want next to any local inference claim.
That does not mean every third-party benchmark using Gemma is wrong. It means the words need units. “Runs” can mean one-token proof of life. It can mean slow CPU fallback. It can mean a tiny context. It can mean useful interactive speed. Those are different products wearing the same jacket.
The Six Receipts Every Local AI Demo Should Provide
If a tool claims a large model fits into surprisingly little memory, the right response is not cynicism. It is a checklist.
- Exact model identifier: The base model name is not enough. You need the specific checkpoint, revision, and whether it is instruction-tuned, distilled, merged, or modified.
- File format and quantization: GGUF, MLX, safetensors, 2-bit, 3-bit, 4-bit, 8-bit, mixed precision, and custom formats all change the answer.
- Context length used in the demo: A model that fits at 2K context may not behave the same way at 32K, 128K, or 256K. Long context is where many tidy claims go to quietly die.
- Peak resident memory: Startup memory is less useful than peak memory during prefill and generation. If the demo only reports file size, keep your wallet in your pocket.
- Backend and acceleration path: MLX, llama.cpp with Metal, Core ML, custom Metal kernels, CPU fallback, and hybrid execution are not interchangeable.
- Throughput and prompt shape: Tokens per second should include prompt processing and generation. A short chat prompt is not the same as summarizing a 70-page document.
This is the same discipline teams already need when evaluating AI answers. If a result will affect code, operations, or money, ask what was measured and how. Notavello’s checklist for checking AI-generated technical answers uses the same principle: separate the claim from the proof before reusing it.
Why Apple Silicon Makes The Claim Plausible Enough To Test
The reason these demos keep happening on M-series Macs is not just fan culture. Apple silicon is a good local inference target because unified memory gives the GPU access to a large shared pool, and because frameworks such as MLX and Metal-aware inference engines can avoid some of the awkwardness of moving data between CPU RAM and dedicated VRAM.
That architecture is useful for developers. A Mac with enough unified memory can be a practical local AI workstation: private drafts, offline coding help, local document search, embeddings, quick evals, and experiments that do not need a cloud bill. It is also pleasant in the most boring way: one laptop, one power cable, no driver séance.
But unified memory also creates a measurement trap. People often say “RAM” when they mean file size, available memory, resident memory, GPU-allocated memory, memory pressure, or compressed memory. macOS can make a workload look survivable right until it starts swapping, throttling, or crawling. A demo that technically completes is not necessarily a demo you would want inside a daily workflow.
The practical rule is simple: if the model is supposed to be interactive, measure it while doing the actual job. Ask it to summarize your real document length. Ask it to edit a real code file. Ask it to run for 20 minutes, not 20 seconds. Local AI that only works under stage lighting is still stage lighting.
What To Test Before You Build Around A Tiny-RAM Claim
Before adopting a local model setup, create a small acceptance test. This does not need to be academic. It needs to match your work.
- Cold start: Time how long the model takes to load after a reboot.
- Short prompt speed: Send a normal chat-sized request and record tokens per second.
- Long prompt prefill: Paste or load the kind of source material you actually use.
- Peak memory: Watch memory pressure, process memory, and swap during the heaviest request.
- Thermal behavior: Run repeated prompts long enough for the machine to heat up.
- Quality at the chosen quantization: Ask domain-specific questions where you know the answer.
- Failure mode: Deliberately exceed the context or memory budget and see whether the tool errors clearly or produces nonsense.
The last one matters more than people admit. A local AI tool that fails loudly is usable. A local AI tool that silently drops context is a bug generator with a chat box.
For teams, save the test prompt, model hash, launch command, and results in a plain text note inside the project. Future you will not remember whether the good result came from the 4-bit build, the 8-bit build, the smaller context, the old backend, or the day the laptop was not also running fifty browser tabs. Future you is unreliable. Write it down.
The Honest Promise Of Local AI
The point of local AI is not to beat every frontier cloud model. That is the wrong contest. The point is to make certain work private, cheap at the margin, scriptable, offline-capable, and fast enough without sending every prompt across the internet.
That is a strong promise. It does not need miracle memory claims to be useful. A smaller model that reliably handles local search, draft cleanup, code explanation, test generation, log triage, or structured extraction can be more valuable than a huge model squeezed through a straw.
So treat bold local inference demos as leads, not conclusions. If someone shows a large model running in a shockingly small memory footprint, be interested. Then ask for the receipts. The real breakthrough is not the headline number. The real breakthrough is a setup you can reproduce on Thursday afternoon, while your actual work is open, without the machine turning into a hand warmer with opinions.