Most people assume AI only lives in the cloud: you send your text to a company’s servers, they run a giant model, and they send an answer back (and bill you per token). But you can also run capable models directly on your own computer or phone. No cloud, no per-message cost, nothing leaving your device. Here is how that works and when it is worth doing.
Open-weight models #
The models you can run yourself are called open-weight models. The “weights” are the giant set of numbers that make up a trained model, the thing it learned during training. When a lab releases those weights openly, anyone can download them and run the model on their own hardware, instead of it living locked on a company’s servers.
This is the foundation of local AI. A number of strong open-weight model families exist, and several are good enough that, for everyday tasks, you would struggle to tell them apart from the big cloud services. Once you have the weights, running the model is yours to do, for free, as much as you want.
The catch: these models are big #
There is a reason everyone does not already run AI locally: capable models are large, often tens of gigabytes, and running them at full size demands serious memory and a strong graphics card. Load a full-size model onto a normal laptop and it either will not fit or crawls.
This is where the second key idea comes in.
Quantization: shrinking the model to fit #
Quantization is the technique that makes local AI practical on normal hardware. The model’s weights are stored as numbers at some precision. Quantization reduces that precision, storing each number more compactly, which dramatically shrinks the model’s size and memory needs and speeds it up.
The tradeoff is a small loss of quality. Squeezing the numbers down loses a little fidelity, so a quantized model is very slightly less sharp than the full version. But the trade is lopsided in your favor: modest quantization can cut a model’s size by more than half while the drop in everyday quality is often barely noticeable. It is what lets a model that “should” need a data-center card run acceptably on a good laptop, or even a phone.
Why bother running locally #
Given that the cloud is easy, why go through this? Several real reasons, and they line up neatly with living cheaply and privately:
- It is free per use. After the one-time download, every message costs nothing. No per-token billing, no subscription, no meter running. For heavy use, this adds up fast.
- It is private. Your text never leaves your device. For anything sensitive (personal notes, private documents, confidential work) that is a real advantage over shipping it to someone else’s server.
- It works offline. No internet, no problem. A local model runs on a plane, in a dead zone, or during an outage.
- No rate limits or lock-in. Nobody can throttle you, change the terms, or deprecate the model out from under you. You have the weights.
This is exactly the thinking behind the offline AI chat app I built: a real language model running entirely on your phone, private and free to use. It is also the frugal angle in using cheaper models instead of premium subscriptions.
The honest tradeoffs #
Local AI is not a free lunch in every dimension:
- You need decent hardware. A capable local model wants a good chunk of memory and ideally a strong graphics card. Modest machines can run smaller models, but there is a floor.
- Frontier quality still lives in the cloud. The very best models are enormous and run on hardware you do not have at home. A good local model handles the vast majority of everyday tasks well, but for the hardest reasoning and coding, the top cloud models still edge ahead.
- There is some setup. It is easier every year, with friendly apps that download and run models for you, but it is still a step beyond opening a website.
The bottom line #
For a large share of what people use AI for (writing, summarizing, answering questions, coding help) a quantized open-weight model on your own machine is more than good enough, and it is free, private, and offline. Reserve the paid cloud models for the moments you genuinely need frontier capability. It is the same principle as the rest of this site: match the tool to the task, and stop paying for premium when the free version does the job.