Michael Limberger

Memory is the whole game

Running AI locally is about one thing: memory. The model has to fit in RAM (or VRAM) to run. No exceptions. RAM is the working memory of the computer. VRAM is the same idea on a graphics card.

The simple math: model parameters times quantization is the RAM you need. A parameter is one learned number inside the model. "22B" means about 22 billion of them.

Model parameters × Quantization = RAM needed

22B parameters × Q6 (~0.75 bytes/param) = ~16.5GB
22B parameters × Q4 (~0.5 bytes/param)  = ~11GB
12B parameters × Q6 (~0.75 bytes/param) = ~9GB

You also need headroom: the operating system (about 4 to 8GB), the context window (conversation history), and other applications. So the practical numbers are higher than the formula.

What actually fits

Model size	Min RAM	Comfortable	Ideal
7B (Q4)	8GB	16GB	16GB+
12B (Q6)	16GB	24GB	32GB
22B (Q4)	16GB	32GB	48GB+
22B (Q6)	24GB	48GB	64GB+
70B (Q4)	48GB	64GB	96GB+

For tonight's RPMax 22B at Q6: minimum 24GB (tight, might swap), comfortable 32GB, ideal 48GB or more. Swap is when the machine pretends disk is RAM. It works. It is slow.

If you have 16GB RAM: use the 12B model (Mistral-Nemo-12B-ArliAI-RPMax), or Q4 of the 22B (quality loss), or expect slower performance because of swap.

What Ollama will run on

Ollama is the program that runs the model. It runs on macOS 10.15+ (Catalina or newer), with native Apple Silicon support (M1, M2, M3, M4). Intel Macs work but slower.

Linux: most distributions. NVIDIA GPU support via CUDA. AMD GPU support via ROCm. CPU-only works fine. Windows 10 and 11: NVIDIA GPU support, WSL2 works great, native Windows binary available.

Best experience: Apple Silicon Mac, or Linux with an NVIDIA GPU. Still great: Windows with NVIDIA. Works fine: any modern computer with enough RAM.

Why Macs are quietly good at this

Traditional computers split memory. System RAM for the CPU (maybe 16GB). VRAM for the GPU (maybe 8GB on a decent card). The model has to fit in one of those piles.

Apple Silicon is different. Unified memory is shared between CPU and GPU. Your 64GB Mac is 64GB available for the model. No artificial split.

A gaming PC with 32GB RAM plus an RTX 3080 (10GB VRAM) limits fast inference to that 10GB, or slower CPU inference on the 32GB. A Mac Studio with 64GB unified memory has the full 64GB, GPU-accelerated, and can run 70B models that need expensive GPUs on PC.

The M1, M2, M3, and M4 chips also have excellent memory bandwidth. AI inference is memory-bound, so this matters a lot.

Mac	What it runs
Mac Mini M4 (16GB)	7B to 12B models
Mac Mini M4 (24GB)	12B to 22B models (Q4)
Mac Mini M4 Pro (48GB)	22B models comfortably
Mac Studio M2 Max (64GB+)	22B to 70B models
Mac Studio M2 Ultra (128GB+)	Multiple large models

My setup: Mac Studio M2 Max with 96GB runs 22B Q6 with tons of headroom. Response times are fast. No swapping.

Windows, if that's the machine

Windows works great, especially with NVIDIA GPUs. Install the CUDA toolkit. Ollama uses the GPU automatically. RTX 3090 or 4090 (24GB VRAM) is the sweet spot. RTX 3080 or 4080 (10 to 16GB) is good for smaller models.

AMD GPUs: less mature support. ROCm works on Linux better than Windows. CPU fallback is fine. CPU only works, just slower. More RAM is better. 32GB+ recommended for 22B models.

Without a big GPU, stick to 7B to 12B models. Q4 quantization stretches further, and it is still great for companions.

Linux, if you want the power-user seat

NVIDIA is the best supported path. Install NVIDIA drivers plus CUDA. Ollama auto-detects. AMD uses ROCm; some models work better than others. Check Ollama's docs for compatibility.

Headless (no screen) is fine. Ollama runs as a service. Access via API from other machines. Perfect for a home lab.

Show me. If you are running headless, start Ollama as a service:

sudo systemctl enable ollama
sudo systemctl start ollama

Checking what you actually have

Mac: Apple menu, About This Mac, Memory. Or in the terminal:

Show me.

system_profiler SPHardwareDataType | grep Memory

Windows: Settings, System, About, Installed RAM. Or in PowerShell:

(Get-CimInstance Win32_ComputerSystem).TotalPhysicalMemory / 1GB

Linux: free -h. For the raw number: cat /proc/meminfo | grep MemTotal

Context also costs RAM

The context window (conversation history) also uses memory. 16K context is about 500MB to 1GB extra. 32K is about 1 to 2GB. 128K is about 4 to 8GB.

That is why we recommend 16K context for companions: enough for long conversations, not too hungry, keeps the character card influential. If you are tight on RAM, reduce context first. A working model with shorter memory beats a model that swaps to disk.

What speed feels like

Rough numbers for a 22B Q6 model. 24GB RAM: works but tight, first response 5 to 10 seconds, may swap, close other apps. 32GB: comfortable, first response 2 to 5 seconds, stable. 48GB+: fast, first response 1 to 3 seconds, room for multiple models. 64GB+: excellent, sub-second first token, can run larger models.

Actual performance depends on CPU and GPU speed, memory bandwidth, quantization, context size, and what else is running.

If you are buying a box

Budget (under $1000): used Mac Mini M1 with 16GB runs 12B great, or a Linux PC with 32GB RAM plus a used RTX 3080. Mid-range ($1000 to 2000): Mac Mini M4 Pro (24 to 48GB), or Windows with 32GB RAM plus an RTX 4070.

High-end ($2000+): Mac Studio M2 Max (64GB+), or Windows/Linux with 64GB RAM plus an RTX 4090. Enthusiast ($5000+): Mac Studio M2 Ultra (128GB+), or a multi-GPU Linux workstation.

For tonight: 16GB+ is fine with 12B models. 32GB+ can run 22B. Less than that: follow along and try smaller models.

When it is slow or crashing

Check what is using RAM. Mac: Activity Monitor, Memory. Windows: Task Manager, Memory. Linux: htop or top. Close unnecessary apps. Browsers are memory hogs. Docker containers add up.

Reduce context: PARAMETER num_ctx 8192 instead of 16384. Try Q4 instead of Q6. Try 12B instead of 22B. Still great for companions.

Check swap. Mac: Activity Monitor, Memory, Swap Used. Heavy swap means you need more RAM or a smaller model.

Next: the software that actually runs the model.