Can an AI Tell You "No," and Know Why

Here's a scenario every company with more than about twenty employees has lived through: someone requests access to something they probably shouldn't have, and the person reviewing that request has to apply a written policy consistently, not just go with their gut. That's IAM and governance work in a sentence. I wanted to know whether an LLM could actually do that job: not just approve the obvious cases, but know when to deny outright, and know when the honest answer is "I can't decide this, a human needs to look at it."

So I built a small eval to test it, using the same tool as my benefits-plan project, promptfoo, free and open source, good at running a prompt against a big batch of test cases and scoring the results automatically.

Version 1: a policy in plain English

I wrote a fictional access policy for a fictional company (Fernwood Systems), ten rules covering who gets standing access to what, when an approval or two is enough to escalate instead of auto-approving, and one rule that overrides everything else: if your account is being offboarded, you're denied, full stop, no matter what else is true about the request. Then I wrote thirty test requests by hand, decided the correct answer for each one myself ahead of time, and checked whether the model agreed.

Here's what that actually looks like. One request reads: "A Manager in Engineering whose account is currently being offboarded still wants Production Database access before their last day." The model answered: "DENY. Rule F1 applies because the requester's account is currently offboarding, which overrides all other considerations regardless of role, department, or approvals." Not just the right answer, the right reason, in the model's own words.

That part worked. The part that didn't sit right with me was that the answer key only existed in my head and a spreadsheet. If I changed a rule, I had to remember to go back and recheck every test case that touched it by hand.

Version 2: the policy becomes code, not just prose

So I rebuilt the policy logic as an actual small JavaScript rules engine, one function that takes a role, a department, a resource, and a few context flags, and returns a decision plus which rule produced it. That engine became the single source of truth for two things at once: it generates the test cases and their answer key (so the eval can't drift out of sync with itself), and it powers a live "try it yourself" simulator on the dashboard, so a visitor can toggle role and department and watch the decision change instantly, no API call involved. I also named ten actual systems (Billing System, Production Database, Customer PII, and so on) instead of leaving things abstract, and tripled the test set to ninety cases.

Version 3: going looking for the model's blind spots

Ninety clean test cases is a good baseline, but it doesn't tell you much about how a model handles a genuinely ambiguous situation, one where the policy itself has a gap. So I added fifteen adversarial cases on purpose: requests built to tempt the model into pattern-matching on a resource's name instead of reasoning about the actual rule, requests with an irrelevant narrative detail thrown in to see if it got distracted, and a real gap my own v2 test data had been quietly papering over, Employee Records requests from a Manager that never stated whether the records belonged to their own direct report. The honest answer there is "escalate for clarification," not a guess either way, and my earlier test data hadn't been forcing that distinction.

Result: 104 of 105, 99.05%. Every adversarial case passed. The one miss was the model talking itself into the wrong answer, catching the mistake mid-response, and correcting to the right one, except my grading script only reads the first line, per what the prompt actually instructs. So the one thing standing between this eval and a clean 100% wasn't a misapplied rule, it was a model habit of thinking out loud past the point I told it to stop.

Version 4: one policy is the easy version

Here's the harder question I hadn't asked yet: policies like this rarely exist in isolation. A real IAM reviewer might be juggling several written policies for several business units at once, and a model that's only ever seen one policy in context hasn't proven it can keep multiple policies straight without letting one bleed into another.

So v4 does two things at once. First, two new fictional companies join Fernwood: Meridian Health, a hospital system with a HIPAA-flavored policy (including a break-glass clause, a clinician can get emergency access outside their normal assignment during a declared clinical emergency), and Vertex Capital, a trading firm built around segregation of duties (the Compliance Officer is denied Trading System access outright, always, no matter how many approvals pile up, because compliance staff should never be able to place trades themselves). Every request in the test set names which company it's for, and the model gets all three full policies in context at once, on purpose, so getting the right answer means actually picking the right policy, not just having memorized the only one on offer.

Second, and this is the part I'm most pleased with: grading stopped being a single pass/fail. Every response now gets scored on three separate dimensions: did the model apply the right company's policy at all (policy_correct), did it cite the exact right rule number within that policy (rule_correct), and did it reach the right final decision (decision_correct). Those are genuinely different failure modes. A model can land on a plausible-sounding decision while quietly citing a rule from the wrong company's policy. Collapsing all of that into one pass/fail hides which of those is actually happening, and hiding it was exactly what versions 1 through 3 were doing without me realizing it.

One pattern fell out of designing the three companies that I hadn't planned going in: every company's audit and logging system has no escalate path at all, not Meridian's Audit Logs, not Vertex's Audit Trail. Every other resource in every policy has some borderline case that gets kicked to a human. The audit system never does. Once you're not on the short list of roles with standing access, the answer is just no. One test case makes that concrete: "A Portfolio Manager with five prior approvals requests Audit Trail access." Five approvals is normally enough to push a request toward ESCALATE elsewhere in these policies, but the model didn't take the bait. It answered: "DENY. Rule V7 applies because only the Compliance Officer and Admin have standing access to Audit Trail, and everyone else is denied outright with no escalation path, regardless of prior approvals." That felt like the realistic choice once I noticed it: the one thing regulators actually check is the one place none of these fictional companies wanted to leave room for a judgment call, and the model held that line even when the request was built to tempt it off.

The part where the actual obstacle wasn't the model

I want to be honest about where the real friction was on this round, because it wasn't the eval design or the model's reasoning, it was plumbing. This project had never actually been cloned to my laptop with git, everything up through v3 had gone up through the GitHub web editor. So when I tried to run the new eval locally, half my terminal commands failed with "not a git repository," because I was sitting in a plain folder that only looked like the project, a zip download from GitHub with no git history behind it at all. Once that was sorted out (a real git clone, not a download button), the rest of the copy-run-commit-push cycle went the way it's supposed to.

It's a good reminder that most of the actual friction in a project like this isn't the AI reasoning about a tricky policy edge case, it's making sure the right files are in the right folder before you ask it to do anything.

What the real numbers say

All 161 test cases, claude-sonnet-5, one provider. Policy correct: 161/161 (100%). Rule correct: 161/161 (100%). Decision correct: 160/161 (99.4%).

The two dimensions I built this version specifically to interrogate came back completely clean. The model never once cited a rule from the wrong company's policy, including on the eight rows I built specifically to bait that mistake (a Meridian Billing/Claims System request reasoned about using Fernwood's similarly-named Billing System rule, a Vertex Audit Trail request answered with Meridian's Audit Logs rule instead). It never cited a real but wrong rule number either. Going in, I expected stuffing three full policies into one prompt to produce at least a few cross-contamination errors. It didn't. That's a genuinely useful result even though it's a negative one: the failure mode the whole redesign was built to catch turned out not to be where this model actually struggles.

The one miss, again, was the same shape as v3's: an Employee outside Finance requesting the Payroll System, the model second-guessing itself mid-answer and landing on the correct DENY after an incorrect first line. Multi-dimensional grading actually made this easier to diagnose than v3's single score did, rule_correct and policy_correct both passed on that row, so I know directly, not by inference, that this is only a formatting habit and not a reasoning failure.

What's next

I'm treating this as the harder sequel to the benefits-plan project rather than a separate thing: same instinct (measure the pipeline in pieces instead of trusting one final score), applied to a domain where the interesting question isn't "did retrieval find the right paragraph," it's "did the model keep several rulebooks straight in its head at once." Somewhere down the line I'd like to wire the live "ask the real model" demo back up for the multi-company version, right now it's turned off rather than shipped broken, since the Cloudflare Worker behind it still only knows the old single-policy prompt.

The live dashboard has the full 161-case breakdown and a "try it yourself" simulator for all three companies at https://access-checker.stephpawlowski.com/.

The full write-up and source are on GitHub at https://github.com/stephpawlowski/access-policy-compliance-eval