Michael Limberger
Need me? Email mike@limberger.ca
AI
What I learned
The fine-tuning journey first
I went on a fine-tuning journey first. Fine-tuning means extra training on a model that already exists. Like adding bookmarks in a book you already own. You are not reprinting the book. You are marking "talk like this."
It taught style, not the answer. It sounded like our shop. It did not recall the procedure. Retrieval means we find the useful pages, put them in the prompt, and generate from that. The facts went in the prompt, this question, from our files.
Pins again
The corpus is a folder of markdown files. A pin is one of those files. A cheat sheet tells the proxy: this topic, always paste this file. Search still runs on the rest.
Reranking would re-score search hits with a second model. We parked it. Pins already put the must-see file in.
A harness after the model talks
A harness is software that helps guide the LLM (large language model). My version runs after the model talks. The prompt is a request. The harness is a check. The model cannot sweet-talk the checker.
Think of a compiler: you do not trust the author because
they sound confident. You run perl -c on the
program. If Perl says the syntax is broken, it is not done.
How we check clinic answers
We keep a list of real policy numbers from our library. If the model cites Policy H.99 and H.99 is not on that list, we stop. Fluent is not the same as true.
If the retrieved hours say Monday to Friday and the model
adds Saturday, that Saturday did not come from our pages.
The harness can catch that. Same idea for TOAD or Perl: if
the answer includes a program, we run the real language
check (toad -c or perl -c) on
that program.
Weak model, strong cage
The LLM is a guesser, even when it is huge. We do not need it to be a lawyer. We need retrieve, pin, then software that the model cannot charm. Small or large, we own the checker.