The Plan That Doesn't Exist
If you hand an AI your documents and ask it a question, will it give you the right answer? That's worth checking. But the question I actually cared about was the second one: will it admit when it doesn't know, or will it just make something up and say it with confidence?
Two documents was the easy version
I started small. Two real health plan documents (the standardized Summary of Benefits and Coverage form every US health plan has to produce), a batch of questions I wrote and checked against the source text myself, and a simple setup: paste both documents into the prompt, ask the question, grade the answer. I used promptfoo to run it and score everything automatically.
This approach has a name, long-context stuffing, and it's the easiest way to build something like this. At two documents it worked fine. The model had everything it needed sitting right in front of it.
Making it something you could actually try
A static eval is a nice thing to point to, but I wanted a visitor to be able to poke at it themselves. So I built a small Cloudflare Worker to hold my API keys server-side and let a live dashboard make real model calls, rate-limited so nobody could run up my bill. Now anyone could ask a question and watch the model answer live instead of reading my pre-baked results.
I also learned something here that had nothing to do with AI: rate limits exist for a reason, and building around them ahead of time is part of the job, not something to bolt on after your bill surprises you.
Six documents broke it
Two documents in a prompt is easy. Six is a different problem. Long-context stuffing gets slow and expensive fast, and it isn't how real retrieval systems work in production anyway. So I scrapped it.
The real version splits each document into small chunks, one per benefit category or coverage example, turns each chunk into a vector so "meaning" becomes something you can measure, and then only hands the model the five chunks most relevant to the question instead of the whole library. The model has to answer strictly from what it's given.
I expanded the document set to six real plans (a PPO, an HDHP, two HMOs, an EPO, and a second PPO, sourced from CMS, Auburn University, the State of Illinois, Northwestern University, Cochise County AZ, and CalPERS), wrote 88 new test questions, and started grading every answer twice: did retrieval pull the right document, and separately, was the final answer actually correct. Those are two different failure modes with two different fixes, and most evals only check the second one.
Everything broke, twice
Building the pipeline was the fun part. Running it is where I hit a wall.
The embedding step calls an API for every chunk, and my account, being new and cardless, was capped at three requests a minute. My script fired off nine batches almost instantly, got rate-limited after the third, and died without writing anything. Then the next step failed too, for the same reason, cascading from a file that never got created. Eighty-eight questions, eighty-eight errors, zero real results.
I didn't add a credit card to make the problem disappear. I made the script respect the limit instead: pace every request, retry with backoff if it still got throttled. Slower, but it works without asking anyone for payment info. The whole build takes a few minutes now instead of failing in ten seconds.
What the numbers actually say
Once it ran clean: 85.2% answer accuracy, 90.9% retrieval accuracy, across 88 questions.
The number I actually care about is that the model never once stated a wrong fact with confidence. Every failure was the model correctly saying "not stated in the document," even on questions built specifically to tempt a wrong guess. One question asked whether Plan H, the university's dental-only plan, requires a referral to see a specialist. Plan H doesn't exist anywhere in the library. The model said so, and explained exactly why it couldn't answer. Another asked about a specific drug copay waiver under the wrong plan. The model caught that too, and named the correct plan the detail actually belonged to.
The weak spot was comparisons. A question like "which plan has the lower deductible, A or D" found the right chunks from both plans every time, but only got the actual comparison right 60% of the time. That's not a retrieval failure, it's the model having both numbers in front of it and still not doing the arithmetic correctly. Splitting the grading in two is what surfaced that, a single pass/fail score would have just called those wrong without saying why.
The live dashboard has an "ask the benefits library" panel where you can ask your own question and watch retrieval and generation run for real.
Check it out at: https://benefits.stephpawlowski.com/
Full write-up on Github at https://github.com/stephpawlowski/benefits-qa-eval