Run a Private AI Model on Your Own Computer — Properly
Run a Private AI Model on Your Own Computer — Properly
Running a language model on your own machine has stopped being a weekend science project. The installers are one click, the model files are a single command, and the weights are free. What has not changed is the part that decides whether the result feels usable or unbearable: how much memory you have, and how much of it you spend on context.
Most people who try local AI and give up do not fail at installation. They install correctly, load a model that is slightly too big for their graphics card, watch it crawl at three words per second, and conclude that local AI is not ready. It is ready. Their model was running on the CPU and nobody told them.
This guide does one thing properly: gets a private model running on your machine, at a speed you will actually tolerate, with a verification step so you know it worked. Roughly 45 minutes, most of it download time.
Before you start: prerequisites
Check these before you download anything. Two of them are hard requirements and will fail loudly; the third is the one that quietly ruins the experience.
Operating system. LM Studio requires macOS 14.0 or newer on Apple Silicon — Intel Macs are not supported at all. On Windows it runs on x64 and on Arm (Snapdragon X Elite) systems, and on x64 it requires a CPU with the AVX2 instruction set. On Linux it ships as an AppImage and asks for Ubuntu 20.04 or newer, with the caveat in its own documentation that releases newer than 22 are not well tested.
GPU support. Ollama supports Nvidia cards with compute capability 5.0 or higher and driver version 531 or newer; cards in the 5.0–6.2 compute range need driver 570 or newer. On the AMD side it requires the ROCm v7 driver stack on both Linux and Windows. Apple GPUs are handled through Metal. If your card sits outside those lists, Ollama also ships a Vulkan backend, enabled by default, which covers most Intel and other GPUs on Windows and Linux.
Memory — the one that matters. LM Studio recommends at least 16 GB of system RAM and at least 4 GB of dedicated VRAM. That is a floor, not a target. The section below turns it into a number you can actually shop against.
Cost: zero. Both tools are free, and the model weights discussed here are openly licensed — OpenAI released its gpt-oss models under Apache 2.0.
Step 1 — Work out what fits before you download
A model's memory footprint is not mysterious. It is parameter count multiplied by bytes per parameter, plus the key-value cache, plus about a gigabyte or two of runtime overhead. A four-bit quantised weight occupies roughly 0.56 bytes; an eight-bit weight occupies one byte; full sixteen-bit precision occupies two.
We ran that arithmetic across the model sizes people actually download. Everything below is our own calculation, shown so you can check it: parameters × bytes-per-weight, converted to gibibytes, then a practical floor that adds roughly 2 GB for the cache and runtime at a modest context window.
| Model size | Weights at 4-bit | Weights at 8-bit | Weights at 16-bit | Practical memory floor |
|---|---|---|---|---|
| 4B | 2.1 GiB | 3.7 GiB | 7.5 GiB | ~4 GB |
| 8B | 4.2 GiB | 7.5 GiB | 14.9 GiB | ~8 GB |
| 14B | 7.3 GiB | 13.0 GiB | 26.1 GiB | ~12 GB |
| 21B (gpt-oss-20b) | 10.4 GiB | 19.6 GiB | 39.1 GiB | ~16 GB |
| 30B | 15.7 GiB | 27.9 GiB | 55.9 GiB | ~20 GB |
| 70B | 36.7 GiB | 65.2 GiB | 130.4 GiB | ~40 GB |
The 21B row is the useful sanity check. OpenAI states that gpt-oss-20b runs within 16 GB of memory, because its mixture-of-experts weights ship natively in a 4.25-bit MXFP4 format. Our independent arithmetic lands at 10.4 GiB of weights and a ~16 GB practical floor — the same answer from a different direction. Hugging Face's own write-up of the release gives the same 16 GB figure, so the number is corroborated twice.
The practical reading: 8 GB of VRAM comfortably runs an 8B model, 12 GB runs a 14B, and 16 GB of VRAM or unified memory opens up gpt-oss-20b. A 70B model at four-bit precision does not fit on any consumer graphics card currently sold, which is why it belongs on a Mac with large unified memory or not at all.
Step 2 — Pick your tool, then install it
Choose one
LM Studio if you want a graphical application: browse models, click download, chat. It handles the model catalogue, the chat interface, and document chat without touching a terminal.
Ollama if you want something other software can talk to. It runs a local HTTP server on 127.0.0.1 port 11434, which means editors, note apps and scripts on your machine can use the model as a backend. Everything in the rest of this guide uses Ollama, because that is where the configuration levers are.
Install it, then pull a model sized from the table above:
ollama pull gpt-oss:20b ollama run gpt-oss:20b
On macOS and Windows, Ollama updates itself — the menubar or taskbar item offers "Restart to update". On Linux you re-run the install script to upgrade. Models land in ~/.ollama/models on macOS, /usr/share/ollama/.ollama/models on Linux, and C:\Users\%username%\.ollama\models on Windows. If your system drive is small, set OLLAMA_MODELS to a different directory before you download 20 GB into the wrong volume.
Step 3 — Set the context window deliberately
This is the step almost everyone skips, and it is the one that determines whether local AI feels like a tool or a toy.
Context length is how many tokens the model can hold in memory at once. Ollama's FAQ states a default context window of 4,096 tokens. Its context-length documentation describes something more nuanced — defaults scaled to available VRAM: 4K context below 24 GiB of VRAM, 32K between 24 and 48 GiB, and 256K at 48 GiB and above. The two pages are framed differently, which is worth knowing before you assume you know what your install is doing. Check rather than guess.
The trade-off is direct: every doubling of context roughly doubles the key-value cache, and that cache comes out of the same VRAM your weights are sitting in. Ollama's documentation is explicit that required RAM scales with parallel requests multiplied by context length. Push context up carelessly and you evict your own model onto the CPU.
For agents, coding tools and anything that reads web pages, Ollama recommends at least 64,000 tokens. Set it when starting the server:
OLLAMA_CONTEXT_LENGTH=64000 ollama serve
Inside a running chat you can change it per session with /set parameter num_ctx 8192, and through the API with the num_ctx option. If memory is tight, enabling quantised key-value cache buys headroom: setting OLLAMA_KV_CACHE_TYPE=q8_0 uses roughly half the memory of the default 16-bit cache, and Ollama's own documentation describes the precision loss as usually unnoticeable.
Step 4 — Verify it actually worked
Do not judge by feel. Run one command while the model is loaded:
$ ollama ps
NAME ID SIZE PROCESSOR CONTEXT UNTIL
gpt-oss:20b c6eb396dbd59 12 GB 100% GPU 65536 4 minutes from now
The PROCESSOR column is the whole test. 100% GPU means the model loaded entirely onto the graphics card — this is what you want. 100% CPU means it is running in system memory and will be slow. A split reading such as 48%/52% CPU/GPU means part of the model spilled, and that partial spill costs far more speed than the percentage suggests.
If you see anything other than 100% GPU, work down this list in order: drop to a smaller model or a smaller quantisation; reduce the context length; switch the KV cache to q8_0; close whatever else is holding VRAM, browsers included. Reinstalling fixes none of this and wastes an afternoon.
Other things that go wrong. On Linux, Ollama sometimes loses track of an Nvidia GPU after a suspend/resume cycle and silently falls back to the CPU; reloading the UVM driver with sudo rmmod nvidia_uvm && sudo modprobe nvidia_uvm is the documented workaround. On unsupported AMD cards, the HSA_OVERRIDE_GFX_VERSION variable lets you point at a close LLVM target. On mixed integrated/discrete laptops where the Vulkan integrated GPU misbehaves, set GGML_VK_VISIBLE_DEVICES to the discrete GPU's index. And if downloads crawl under WSL2 on Windows 10, the documented culprit is Large Send Offload on the vEthernet (WSL) adapter.
Step 5 — Make it genuinely private
"Local" and "private" are not automatically the same thing. Modern Ollama builds ship cloud models and web search alongside local inference, and the company's own FAQ distinguishes between the two: prompts stay on your machine when you run locally, while cloud-hosted models are processed on their servers. If your reason for doing any of this is that the data must not leave the building, turn the cloud path off explicitly rather than assuming you are not using it.
// ~/.ollama/server.json
{
"disable_ollama_cloud": true
}
Restart, and the logs should read Ollama cloud disabled: true. The environment variable OLLAMA_NO_CLOUD=1 does the same job. And leave the bind address alone unless you have a reason: Ollama listens on 127.0.0.1 by default, and changing OLLAMA_HOST to 0.0.0.0 exposes an unauthenticated model server to your whole network.
When not to bother
Skip this if your machine has 8 GB of RAM and no discrete GPU — you will get a 4B model producing text slowly, which teaches you the mechanics but will not replace anything. Skip it if your work depends on the largest frontier models; a 20B model running on your desk is genuinely useful for drafting, summarising, extraction and code completion, and genuinely not competitive on hard reasoning. And skip it if you are on an Intel Mac, where LM Studio does not run at all and your GPU options are thin.
The honest case for doing it is narrower than the enthusiasm suggests, and stronger where it applies: documents you are contractually not allowed to upload, work on aeroplanes and bad hotel Wi-Fi, high-volume repetitive tasks where per-token billing adds up, and the plain fact that a model on your disk cannot be deprecated out from under you.
The interesting shift in local AI is not that the models got smarter — it is that the memory maths quietly moved a whole tier of hardware into range. When a 20B-parameter model ships natively at 4.25 bits per weight rather than being squeezed down after the fact, the constraint stops being "do you own a workstation" and becomes "does your laptop have 16 GB". That is an enormous population of machines. Every guide we read while researching this piece treated quantisation as a compression trick applied to a finished model; the more consequential version is quantisation as a design decision made before training ends, which is what makes the difference between a model that technically fits and one that fits with room for a usable context window. The documentation gap follows from the same place. Vendors publish install commands and hardware compatibility lists, but almost nobody publishes the one number that predicts your experience — the ratio of your VRAM to your model plus cache. That number is why two people running identical software on similar-looking laptops report wildly different verdicts on whether local AI is any good.
What we still don't know: whether natively-quantised releases become the norm or stay an OpenAI-and-a-few-others practice, and how much quality the format actually costs relative to a post-hoc 4-bit conversion — no vendor publishes a like-for-like comparison, and we have not run one.
Sources
- Ollama — Hardware support (docs/gpu.mdx) (primary) — Nvidia compute capability and driver versions, AMD ROCm v7 requirement, Vulkan backend, GPU selection variables, suspend/resume workaround.
- Ollama — Context length (docs/context-length.mdx) (primary) — VRAM-tiered context defaults, 64,000-token recommendation for agents and coding, ollama ps PROCESSOR verification.
- Ollama — FAQ (primary) — 4,096-token default, model storage paths, KV cache quantisation types, cloud disabling, bind address, WSL2 networking.
- LM Studio — System Requirements (primary) — macOS 14.0 and Apple Silicon requirement, AVX2, 16 GB RAM and 4 GB VRAM recommendations, Linux AppImage caveats.
- OpenAI — Introducing gpt-oss (primary) — Apache 2.0 licensing, MXFP4 native quantisation, 16 GB memory figure for gpt-oss-20b.
- Hugging Face — Welcome GPT OSS — independent corroboration of the 21B parameter count, MXFP4 format and 16 GB figure.
Specifications verified September 7, 2026 and subject to change. GadgetGlow Bytes does not test hardware and did not benchmark these tools; all figures are from vendor documentation or from our own arithmetic, which is shown in full above. GadgetGlow Bytes does not receive products from manufacturers for coverage.
.webp)
Comments
Post a Comment