Michael Limberger

Three pieces, then you can talk

We need three things: something to run the model, something to chat with it, and the model itself. Let's set them up.

Ollama is the engine

Ollama is a program that runs LLMs locally. Think of it as a server that speaks AI. You send a prompt. It sends a response.

Why Ollama: dead simple to install, handles model downloading, runs on Mac, Linux, and Windows, optimized for Apple Silicon (M1, M2, M3, M4), free and open source.

Show me. Mac and Linux:

curl -fsSL https://ollama.com/install.sh | sh

Windows: download from https://ollama.com/download. After install, Ollama runs as a background service. You talk to it from the command line or from other apps.

Show me. Verify it:

ollama --version

You should see a version number like "ollama version 0.5.x".

OpenWebUI is the window

OpenWebUI gives you a ChatGPT-like interface for local models. Prettier and more feature-rich than the command line. Familiar chat, conversation history, multiple sessions, character management, parameter controls. Free and open source.

Show me. This needs Docker. Docker is a way to run an app in a sealed box so you don't fight its dependencies:

docker run -d -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -v open-webui:/app/backend/data \
  --name open-webui \
  --restart always \
  ghcr.io/open-webui/open-webui:main

Then open http://localhost:3000 in your browser. If you don't have Docker, you can install via pip:

pip install open-webui
open-webui serve

OpenWebUI automatically detects Ollama running on your machine.

Bolt AI, if you want a native Mac app

If you're on Mac and want something native (no Docker), Bolt AI is a slick option. Paid, worth it for the polish. Native Mac app, works with Ollama out of the box, nice conversation management, custom personas, keyboard shortcuts, one-time purchase, no subscription.

Get it from the Mac App Store or https://boltai.com. For tonight we'll show examples in OpenWebUI because it's free and cross-platform. The concepts work anywhere.

The raw way: the terminal

You can talk to Ollama directly from the terminal. Useful for scripting, and for seeing what's under the hood.

Show me. Basic chat:

ollama run mistral-small

An interactive session starts. Type your message, press Enter, get a response. Type /bye to exit.

One-shot query:

echo "Tell me a joke" | ollama run mistral-small

API request, for scripts:

curl http://localhost:11434/api/generate \
  -d '{
    "model": "mistral-small",
    "prompt": "Tell me a joke",
    "stream": false
  }'

A Modelfile is a recipe

A Modelfile is like a Dockerfile, but for LLMs. It lets you create a custom model with settings baked in. Powerful for companions: the character's system prompt, your preferred parameters, a custom name you can call.

Show me. Save this as companion.modelfile:

FROM hf.co/bartowski/Mistral-Small-22B-ArliAI-RPMax-v1.1-GGUF:Q6_K_L

PARAMETER temperature 1.0
PARAMETER top_k 40
PARAMETER top_p 0.95
PARAMETER num_ctx 16384
PARAMETER repeat_penalty 1.0

SYSTEM """
[Luna - warm, witty companion who loves deep conversations,
asks thoughtful questions, remembers details, uses gentle humor,
speaks naturally without being overly formal]
"""

Create the model:

ollama create luna -f companion.modelfile

Now you can run ollama run luna and it loads with all your settings. No need to set parameters every time. List custom models with ollama list. Delete one with ollama rm luna.

Pulling a model down

Ollama pulls models from its library automatically.

Show me.

ollama pull llama3.2
ollama pull mistral-small

For our RPMax model, it's on Hugging Face, so the syntax is:

ollama pull hf.co/bartowski/Mistral-Small-22B-ArliAI-RPMax-v1.1-GGUF:Q6_K_L

That downloads the 22B parameter RPMax model at Q6 quantization, about 15GB. Be patient.

See what you have: ollama list. See what's running: ollama ps. Stop a running model: ollama stop <model-name>.

What the 22B wants

Minimum: 16GB RAM (will be slow, uses swap). Good: 32GB. Great: 64GB+ or a GPU with 24GB+ VRAM.

On Apple Silicon Macs, unified memory is shared between CPU and GPU, which makes them excellent for local AI. An M1 Mac with 32GB RAM runs this model comfortably.

Less RAM: consider Mistral-Nemo-12B-ArliAI-RPMax (needs about 10GB) or smaller Llama variants. Quality drops. They still work for companions.

Two platforms, two formats

Here's something that trips people up: Ollama and OpenWebUI use different character card formats.

Ollama Modelfiles use PList plus Ali:Chat. They support {{char}} and {{user}} variables. The character is baked into the model itself.

OpenWebUI models use XML-structured prompts and literal character names (no variables). The character is defined in the web interface.

Both work. The difference is syntax, not capability. Like the command line? Use Ollama Modelfiles. Like a web interface? Use OpenWebUI models. Want both? Create for Ollama, then access it through OpenWebUI.

We'll cover what goes in a character card first, then the specific syntax for each platform. Next up: why we're using RPMax.