The Rule To Escalate

Every company with more than about twenty employees has lived through this: someone requests access to something they probably shouldn't have, and whoever reviews it has to apply a written policy consistently, not just go with their gut. That's IAM and governance work in a sentence. I wanted to know whether an LLM could actually do that job. Not just approve the obvious cases, but know when to deny outright, and know when the answer is "a human needs to look at this."

A policy in plain English

I wrote a fictional access policy for a fictional company, ten rules covering standing access, when an approval or two should escalate instead of auto-approving, and one rule that overrides everything else: if your account is being offboarded, you're denied, full stop. Then I wrote thirty test requests by hand, decided the right answer for each one myself, and checked whether the model agreed.

One request: a manager whose account is currently being offboarded still wants database access before their last day. The model denied it and named the exact rule, offboarding overrides everything else regardless of role or approvals. Right answer, right reason, in its own words.

What didn't sit right was that the answer key only existed in my head and a spreadsheet. Change a rule, and I had to remember to recheck every test case that touched it, by hand.

Turning the policy into code

So I rebuilt the policy as an actual small rules engine: one function that takes a role, a department, a resource, a few context flags, and returns a decision plus which rule produced it. That engine became the source of truth for two things at once. It generates the test cases and the answer key together, so they can't drift out of sync. And it powers a live simulator on the dashboard, toggle a role and department and watch the decision change instantly, no model call involved.

I also named ten real systems instead of leaving things abstract, and tripled the test set to ninety cases.

Looking for where it would actually break

Ninety clean cases is a good baseline, but it doesn't tell you how a model handles a genuinely ambiguous situation, one where the policy itself has a gap. So I added adversarial cases on purpose: requests built to bait pattern-matching on a resource's name instead of the actual rule, and a real gap my own test data had been quietly covering up, a manager requesting employee records without ever stating whether they belonged to their own direct report. The right answer there is "ask," not a guess either way, and my earlier tests hadn't been forcing that distinction. The model got 104 of 105 right, and the one miss was the model correcting itself mid-answer in a way my grading script wasn't set up to catch, not a wrong rule.

Then I went further and stopped testing one company. A real reviewer juggles several policies at once, so I added two more fictional companies with genuinely different rules: a hospital system with a break-glass clause for clinical emergencies, and a trading firm where the compliance officer is denied trading access outright, always, because compliance staff should never be able to place trades themselves. Every request named which company it was for, and the model got all three full policies at once, so getting it right meant picking the correct policy, not just having memorized the only one on offer.

I also stopped grading with one pass/fail score. Every response now gets checked on three separate things: did it use the right company's policy at all, did it cite the right rule number, and did it land on the right decision. Those are genuinely different failure modes. A model can reach a plausible decision while quietly citing a rule from the wrong company.

The best thing I found came out of just designing the three policies: every company's audit log has zero escalation path. Not one exception, anywhere. Every other resource has some borderline case that gets kicked to a human. Auditing never does. One test case makes it concrete: a portfolio manager with five prior approvals, normally more than enough to push a request toward escalation elsewhere, requests audit trail access. The model denied it anyway and named the rule: standing access only, no escalation, regardless of history. That's the one place none of these fictional companies left room for a judgment call, and the model held the line even when the request was built to tempt it off.

The real obstacle wasn't the model

This project had never actually been cloned to my laptop, everything through the third round had gone up through GitHub's web editor. So when I tried to run the new eval locally, half my terminal commands failed because I was sitting in a folder that only looked like the project, a zip download with no git history behind it. Once I actually cloned it properly, the rest went the way it's supposed to. Most of the real friction on a project like this isn't the model reasoning about a tricky edge case. It's making sure the right files are in the right folder before you ask it to do anything.

What the numbers say

All 161 test cases, one model, three companies. Policy correct 161/161. Rule correct 161/161. Decision correct 160/161.

I expected stuffing three full policies into one prompt to cause at least a few cross-contamination errors, a Meridian billing request reasoning about Fernwood's similarly-named system, a Vertex audit request citing Meridian's rule instead. It didn't happen once, including on the rows built specifically to bait that mistake. That's a useful result even though it's a negative one, the failure mode the whole redesign was built to catch turned out not to be where this model actually struggles.

The one miss was the same shape as before: a correct decision reached after the model second-guessed itself mid-answer, caught by a grading script that only reads the first line. Splitting the grading three ways made it easy to tell that apart from an actual reasoning failure. It wasn't one.

The live dashboard has the full breakdown and a try-it-yourself simulator for all three companies.

Check it out at: https://access-checker.stephpawlowski.com/

Full write-up on Github at https://github.com/stephpawlowski/access-policy-compliance-eval