Michael Limberger

The chat window can stay the same. The smart part happens upstream: search, pins, checks, and a model that never needs to see the raw clinic library unfiltered.

Retrieval means we find the useful pages, put them in the prompt, and generate from that. The layer you own is the one that does that.

Same interface

Nothing the user touches changed. They still open OpenWebUI. They still type into a chat box. They still pick a model from a dropdown. There was no retraining for anyone. No new tool to learn, and no migration.

The familiarity is deliberate, and it is why the system actually gets used. The trap many engineers fall into is building a better solution that requires people to learn something new. The new solution dies on the launchpad because the cost of switching is higher than the cost of the old broken thing. The proxy is engineered to be invisible. Same dropdown. Same chat box. New abilities underneath.

Everything smart now happens upstream

The intelligence moved. It is not in the model's memory (that was fine-tuning, part 3, failed). It is not locked inside the chat app (that was built-in RAG, part 8, quietly broken). It is in the proxy, one hop upstream of the app, where you can see it, control it, script it, and reuse it.

That single move, from inside-the-app to upstream-of-the-app, is the entire contribution of this night.

The arc

The whole night collapses into this table.

Stage	Where	What happened
Fine-tuning (parts 3, 4)	Inside the model's weights	Taught style, not facts. Wrong tool.
RAG, the concept (part 5)	At question time	Open-book test. Read, do not memorize.
Built-in RAG (parts 6, 7, 8)	Inside the chat app	Works, but locked in and quietly broken.
RAG Proxy (parts 9, 10)	Between the app and the model	RAG as infrastructure. One config to wire.
The stack (part 11)	870 lines of Perl	Lean beats heavy. Restraint is the design.
The extensions (part 12)	Same core, many front ends	Modularity is the payoff.

All of it collapses into the closing line: same interface, smart stuff upstream.

The opening promise, kept

The night started with a promise: you would watch a local AI become an expert on a language that did not exist last week. No training, and no cloud.

That demo works because of every architectural decision in this talk. The model never trained on the language because training is the wrong tool (part 3). The retrieval finds the right passages because RAG is the right tool (part 5). The retrieval happens in a place you control because the proxy is upstream of the app (part 9). Nothing leaves the building because the whole stack is local (parts 2, 11).

Every "wow" moment in the demo is a structural property of the architecture, not a trick. That is what makes it reproducible. Anyone in this room could build it.

870 lines of Perl and a Mac

This is not a slogan. Anyone with patience and a laptop could build this. The components are open-source. The architecture is documented. The literature backs the technique.

What you do not need: a research team, a cloud account, a vendor relationship, a budget line item, permission to send your data somewhere. What you do need: a laptop with enough RAM to run a small model, an afternoon to install the four components, documents you actually want to ask questions of.

That accessibility is the political point underneath the technical one. AI does not have to be rented. It can be local, private, and yours. That has always been true. The tools for it just had to catch up.

The sentence behind the sentence

If someone asks you to sum up the whole night in the hallway afterward, this is it:

RAG is not a feature you turn on. It is a layer you own.

The closing slide says that in seven friendlier words. But that is the idea.

Fine-tuning tried to put knowledge in the model. Wrong place. Built-in RAG tried to put it in the app. Wrong place. The proxy puts it in a layer you control. Right place. The version that actually works is the one that lives in a layer you own.

The path home

If you have documents you would like an AI to be expert on, and you cannot or will not send them to the cloud, the path is: install the stack (Ollama plus OpenWebUI, four commands, part 7). Try built-in RAG first. It might be enough for your use case. If you hit the four failure modes (embed truncation, context window, lock-in, fixed pipeline), move the RAG out into a proxy. The proxy can be 870 lines of Perl or 1500 lines of Python or whatever language fits your team. The architecture is what travels.

You did not need a tutorial for any of that, technically. You needed someone to tell you the path was real and the shape of it was small. That was this night.

Where can you get this? The scripts (Brain.pm, ragproxy.pl, ragproxy-ingest.pl) are the whole system. The companion site at ragproxy.dentistry.utoronto.ca has the references and links.

What would I do differently next time? Honest answer: I spent five months on fine-tuning before reading the research. The lesson: check whether the problem is already solved before solving it. The papers were there the whole time.

What is next for the project? Hybrid search is the natural next step, combining keyword search with vector search for better recall on rare terms. More brains. More front ends. The architecture is built to extend (part 12's point).

Can you use this for your own organization? Yes, and that is the design goal. The architecture is small enough to copy, the components are open-source, and the licensing on the AI models we use (Apache 2.0 for Ministral 3) does not restrict commercial use.

The one piece of advice for someone starting this journey: start with the simplest thing that works. OpenWebUI's built-in RAG is fine for most people. Only move to a proxy when the four failure modes actually bite you. Premature architecture is its own failure mode.

Build the layer

RAG is not a feature you turn on. It is a layer you own.