Four Failures, One Bug
For background, check out: Building AI Tooling That Knows What It's For
When I built my accessibility triage tool, I hit a bug I couldn't resolve no matter how much time I threw at it. I had to decide: ignore it, or write up what was actually going on.
The bug shows up
Quick recap: the accessibility triage tool scans a URL and gives back violations, grouped issues, and how to fix them. Most sites work fine. Of course I hit one that didn't, and it failed spectacularly.
This is a good time to remind you that I am a product manager, not an engineer.
The first few times, the scan timed out. The Worker I'd set up waited for the page's network activity to go idle before scanning, and this particular site never stops making requests in the background. That condition was never going to arrive. Fine. I loosened it and kept going.
The bug won’t stop bugging me
Then it failed a second way. The browser session closed mid-scan. I added a retry, thinking it might be a one-time timing thing. That retry failed a third way, because it tried to reuse a session that had already died. Turns out you can't pick a conversation back up once the other person's already left the room.
I fixed that too, every retry gets a fresh session now. I thought that would solve it.
It did not.
Failure four had nothing to do with my code: I'd hit Cloudflare's account-level limit on how many browser sessions I could spin up. A different limit than the one I'd built into the tool myself. All that retrying and recalibrating had burned through it, and the scan still wouldn't work for this one site.
At this point I had two theories. Maybe the site's bot protection was killing my automated browser. Or maybe the page itself, with all its video and animation, was running the browser out of memory before the scan could finish. Both seemed plausible. I had no way to tell them apart.
I just wanted an answer
I wanted a root cause. I ran another test to figure out what the scan could actually handle: a different, similarly complex site, likely without the same bot-management tooling. It scanned like a dream, first try. That ruled out "any heavy page breaks this." It didn't rule out either of my two remaining theories. Maybe it's the bot protection. Maybe that one page really is too much for my tool.
Saying what I don't know
I didn't get to pick a theory and write an error message around it. But I found one piece of evidence I did have access to, product manager coding on a tight budget: when a browser's renderer process crashes, it fires a specific signal, different from a session closing for some other reason. I set up a listener for that signal. If it fires, the tool tells a visitor the page is too resource-heavy to scan. If a session closes without that signal, the tool says the site appears to be blocking automated browsers instead.
This site appears to block automated browser scanning, which some sites do deliberately (bot/anti-automation protection). Try a different URL.
"Appears to" is doing real work in that sentence. I'm not claiming proof of bot protection, a crash that doesn't happen doesn't prove one thing, it just tells me something else closed the connection. But that's still more than a generic error would've told anyone.
Why I kept poking
The hardest part of trusting AI isn't whether it can get a right answer. It's whether it knows where it's hit the edge of what it actually knows. Anyone selling an "AI-powered" tool is making these same tradeoffs somewhere. Not everything resolves cleanly in the real world. Does the tool say it will anyway? Does it paper over the gap with something that sounds more finished than it is?
I'd like to think I'd have shipped this version even without hitting all four failures. But I'm glad I did hit them, because by the time I committed this version, saying here's what I know and here's what I don't wasn't a hard call to make.
Check it out at: https://a11y-triage.stephpawlowski.com
Full write-up on Github at https://github.com/stephpawlowski/accessibility-triage-tool