Teaching an AI to Say "I Don't Know"
Here's a question that sounds simple until you actually try to answer it: if you hand an AI your health insurance documents and ask it a question, will it give you the right answer? And, just as important, will it admit when it doesn't know instead of confidently making something up?
That second question is the one that actually kept me up at night, so I built a project to test it.
Version 1: keep it simple
I started small. Two real health plan documents (Summary of Benefits and Coverage forms, the standardized format every US health plan has to produce), a batch of questions I wrote and verified by hand against the source text, and a straightforward setup: paste both documents into the prompt, ask the question, grade the answer. I used promptfoo to run the eval and score everything automatically.
This is called "long-context stuffing," and it's the easiest way to build something like this. At two documents, it worked great. The model had everything it needed sitting right in front of it.
Version 2: making it real
A static eval is a nice portfolio piece, but I wanted something a visitor could actually poke at. So I built a small Cloudflare Worker that holds my API keys server-side and lets the live dashboard make real model calls, rate-limited so nobody could run up my bill. Now anyone could ask the model a question themselves and watch it answer live instead of just reading my pre-baked results.
This is also where I learned my first real lesson about running live AI demos on a budget: rate limits exist for a reason, and building around them (not just hoping you never hit them) is part of the job.
Version 3: the point where copy-paste stops working
Two documents in a prompt is easy. Six documents is a different problem. Long-context stuffing doesn't scale, it gets slow, it gets expensive, and more importantly, it isn't how real retrieval-augmented generation (RAG) systems actually work in production. So I scrapped the approach and rebuilt it properly.
Here's what that actually meant:
Chunking. Each source document gets split into small, independently retrievable pieces, one per benefit category, one per coverage example, that kind of thing. About 40 chunks per document, roughly 260 total across all six plans.
Embedding. Every chunk gets converted into a vector with Voyage AI's embedding model, so "meaning" becomes something you can measure with math instead of just string-matching.
Retrieval. When a question comes in, it gets embedded the same way, and I pull back the five chunks whose vectors are most similar to the question's vector. No vector database needed, at this scale a plain linear scan runs in well under a millisecond.
Generation. The model gets only those five retrieved chunks, not the whole library, and has to answer strictly from what it was given.
I also expanded the document set from 2 to 6 real plans (a PPO, an HDHP, two HMOs, an EPO, and a second PPO, sourced from CMS, Auburn University, the State of Illinois, Northwestern University, Cochise County AZ, and CalPERS), wrote 88 new test questions, and, this is the part I'm most proud of, started grading every answer twice. Did retrieval pull back the right document? And separately, was the final answer correct? Those are two different failure modes with two different fixes, and most evals only check the second one.
The part where everything broke
I'd be lying if I said this went smoothly. Building the retrieval pipeline was the fun part. Actually running it was where I hit a wall, twice.
The embedding step calls Voyage's API for every chunk, and my Voyage account, being new and cardless, was capped at 3 requests per minute. My build script fired off nine batches almost instantly, got rate-limited after the third one, and died without ever writing the output file. Then the eval step failed too, for the exact same reason, cascading from a file that never got created in the first place. Eighty-eight test questions, eighty-eight errors, zero real results.
The fix wasn't to add a credit card and buy my way out of it (tempting, but not the point of a project about doing things properly). It was to make the scripts respect the limit: pace every request to stay under three per minute, and retry with backoff if I still got rate-limited anyway. Slower, but it works without needing to hand Voyage a payment method at all. The whole index build now takes a few minutes instead of failing in ten seconds, and I'd rather have "slow but honest" than "fast but broken."
What the real numbers actually say
Once everything ran clean, here's what came back: 85.2% answer accuracy and 90.9% retrieval accuracy across all 88 questions.
The number I care about most isn't either of those, though. It's this: the model never once stated a wrong fact with confidence. Every single failure was the model correctly saying "not stated in the document" instead of guessing, even when I threw questions at it specifically designed to tempt a wrong guess, like asking about a plan that doesn't exist, or borrowing a detail from the wrong plan's paperwork. Zero hallucinations across all of that.
The most interesting weak spot turned out to be cross-plan comparisons, questions like "which plan has the lower deductible, A or D?" Retrieval found the right chunks from both plans 100% of the time. But the model only got the actual comparison right 60% of the time. That's not a retrieval problem, that's a reasoning problem: the model sometimes had both numbers sitting right in front of it and still didn't put them together correctly. Splitting the grading into two separate metrics is exactly what surfaced that distinction. A single pass/fail score would have just called all of those "wrong" without telling me why.
What's next
I'm treating this as the second entry in a small portfolio of LLM evaluation projects, and honestly the biggest shift for me wasn't the tech stack, it was moving from "give the model everything and hope it stays grounded" to actually measuring each piece of the pipeline separately. Retrieval and reasoning fail differently, and if you only check the final answer, you'll never know which one to fix.
If you want to poke at it yourself, the live dashboard has an "ask the benefits library" panel where you can ask your own question and watch the real pipeline run, retrieval, chunks, and all. The full write-up and source are on GitHub.
Check it out at: https://benefits.stephpawlowski.com/
Full write-up on Github at https://github.com/stephpawlowski/benefits-qa-eval