When I Couldn’t Resolve a Bug in my Project
For background info, check out this post: Building AI Tooling that Matters + Fun with Accessibility
When I built my accessibility triage tool, I hit a bug that I couldn’t resolve despite throwing more and more time at it. I had to make a decision - do I ignore, or do I write up what is actually going on?
The bug pops up
Quick recap: the accessibility triage scans a URL and then gives information on violations, batched issues, and how to resolve them. Most sites work.
But I, of course, hit one that didn’t, and it failed spectacularly.
This is a good time to remind you that I am a product manager, not an engineer.
The first few times the site scan failed, it timed out. The Worker I set up would wait for the page’s network activity to go idle before scanning, and this particular site - the thorn in my side - never stops making requests in the background. So.. that condition never arrived.
That’s fine. I loosened the condition it was waiting on, and kept trying.
The bug won’t stop bugging me
Then it failed in a different way. The browser session closed out during the scan mid-request. I added a retry because I thought it may be a one-time timing thing. That retry then failed a THIRD way because it was trying to reuse a session that had already died.
It turns out you can’t pick a conversation up, when someone has already left the room.
A software engineer likely would have known to avoid this situation from the get-go.
So I fixed that too, and every retry will get a fresh session. I thought that would solve it.
It did not.
ANOTHER failure - we’re at four now - came on the scene, and this one had nothing to do with my code: I hit my Cloudflare’s account-level limit on how many browser sessions I could spin up. This was a different limit than the one I’d built into the tool.
All that retrying and recalibrating had burned through my limit.
And the accessibility scan still wouldn’t work for this one site.
At this point I had a few theories. Could it be the site’s bot protection kicking in and killing my automated browser? Could it be that the page itself had so much going on - video, animation, media - that the browser was running out of memory before the scan completes?
Both theories seem plausible. And I hadn’t defined a way to tell them apart.
I just wanted an answer
I wanted to solve it, and get this site to work when scanned. I wanted to find a root cause and resolve it.
I ran another test to try to settle what the scan could handle. I found a different, JS-intense site that is similarly complex to render but (likely) doesn’t have the same bot-management tooling running on it. And that site scanned like a dream, first try. That rules out the theory that ‘any heavy page would ruin the scan’s completion.’ This was useful. I felt a little better. It didn’t rule out either theory though - maybe it’s the bot protection. Maybe that site page really IS too complex for my tool.
Shipping my honest version of the tool
Instead of picking a theory and writing an error message, I finally found a piece of evidence that was available to me (product manager coding on a tight budget).
When a browser’s renderer process crashes, it fires a specific signal that’s different from a session that closes for some other reason. I set up a listener for that signal. If it fires, the scanner tool tells a visitor that the page they’re trying to scan is too resource-heavy. If a session closes without that signal firing, the tool says that the site ‘appears’ to be blocking automated browsers, instead.
This site appears to block automated browser scanning, which some sites do deliberately (bot/anti-automation protection). Try a different URL.
Through my language, I hope to indicate to the user that this appears to be happening, and some sites do this deliberately. But I’m not stating it as definitive. It’s not proof for the bot-protection case. The absence of a crash doesn’t always mean you were blocked. But it does mean that something else closed that connection.
Why I kept poking
I think one of the hardest parts of trusting AI is not ‘can it get a right answer’ but ‘does it know where it’s hitting its own limit of information.’ If someone is selling you an ‘AI-powered’ tool, like a scanning tool, they have to make these same choices and tradeoffs. Not everything will cleanly resolve in the real world. Does the pricey tool say it will? Does it mask the issue with something that sounds more complete than it is?
I want to say that I’d have shipped this version even if I hadn’t hit all these issues. But I am really glad that I did find these problems, because by the time I committed the version of the tool, saying “here’s what I know, and here’s what I don’t know” wasn’t a hard call to make.
Check it out at: https://a11y-triage.stephpawlowski.com
Full write-up on Github at https://github.com/stephpawlowski/accessibility-triage-tool