Michael Limberger

How we built it

Approach: RAG Proxy on Faculty hardware, normal chat apps, policy chunks, pins, web UI.

A local model runs on hardware we own and writes the answers. Open WebUI is the chat window in the browser.

Production: Lightning on a Spark

Production runs NVIDIA Nemotron-3.5-Lightning on a DGX Spark (NVIDIA GB10, 128 GB unified memory). It is the latest NVIDIA model that fits this box cleanly, and it shipped in August 2026.

MoE is mixture of experts: many specialist sub-networks, only a few fire per token. About 30 billion parameters on disk, about 3 billion active per token. Hybrid: Mamba-2 plus MoE plus attention. Built to run on one Spark.

NVFP4 is NVIDIA 4-bit floating point: Spark-native 4-bit. More squeezed than 8-bit. The engine is vLLM, a GPU server that keeps the model loaded, streams tokens as they are written, and speaks the same chat protocol the browser already uses.

Card: Nemotron-3.5-Lightning on Hugging Face.

Homelab: Qwen on a Studio

Homelab is Qwen 3.6 Heretic on a Mac Studio M2. Qwen is from Alibaba. Reasoning model, and also MoE, same idea as Lightning. 6-bit compression: more squeezed than 8-bit, smaller on disk.

Base: Qwen 3.6 35B-A3B · uncensored cut: Heretic.

Open-book on purpose

Retrieval means we find the useful pages, put them in the prompt, and generate from that. The model does not memorize well. Search the library. Write from what you found.

A proxy is a program in the middle of two other programs. Same idea as a reverse proxy in nginx. The chat window talks to our proxy. The proxy talks to the model. Protocol level. That middle hop searches our docs, glues the pages together, and forwards the packed question to Lightning in production, or Qwen in the homelab.

Try it: github.com/wildcard-wizard/rag-proxy.

You
          |
          v
        +------------------+
        | Open WebUI       |  chat in the browser
        +--------+---------+
                 |
                 v
        +------------------+
        | RAG Proxy        |  middle hop (nginx idea)
        |  1. cheat sheet  |  which pin, this topic
        |  2. search       |  other markdown in the folder
        |  3. pack prompt  |  pin + hits + your question
        +--------+---------+
                 |
                 v
        +------------------+
        | vLLM             |  local model, already loaded
        |  Lightning / Qwen|
        +------------------+

A folder of markdown, plus pins

The corpus is just a folder of markdown files. That is the library. A pin is one of those files, kept short. You can open it in an editor like any other doc. Nothing is hidden inside the model. Search still runs on the rest of the folder. The pin is extra: we always paste that whole file in when the question is about that topic.

How the proxy knows which pin: we gave it a short list of rules. Not another library. A cheat sheet the proxy reads before it talks to the model. Example rule: if the question is about a needlestick, always paste needlestick-pin.md. That list is a settings file for the proxy. You can think of it as a table: topic → file.

We did not add reranking

Search is a first guess. It returns the closest pages, in a rough order. Reranking is a second pass: a smaller model re-scores those pages against this exact question, then keeps the top few. Extra machinery. Extra GPU. And it cannot save you if the right page never made the first list.

A pin skips that luck. Needlestick question → always paste needlestick-pin.md. The must-see page is in the prompt even if search ranked it fifth. Search still runs. We parked the second ranker. Pins are doing the job for the pages that must not be missed.

corpus/                         the library (just markdown)
          Policy-H03.md                 normal doc. Search may find this
          clinic-hours.md
          needlestick-pin.md            also markdown. This is the pin
        
        Cheat sheet the proxy reads:
          needlestick question  →  always paste needlestick-pin.md
          hours question        →  always paste clinic-hours.md

You ask about a needlestick. The proxy looks at that cheat sheet, pastes needlestick-pin.md onto the prompt. Search may still add Policy H.03. Two markdown files. One extra rule that says "this topic, this file."

# needlestick-pin.md

Use Policy H.03 only.
Do not invent a policy number.
If H.03 is not in the pages we found, say you do not have it.