The Part of My AI Tool That Shouldn’t Have Been AI

Back in June I wrote about running Verilay on Verilay and getting a different grade almost every time. Same app, same code, different answer. For a tool whose entire job is telling you whether something is safe, that's not a great look, and I said so.

I've now fixed a big part of that. The fix was to take the AI out.

Not all of it. Verilay is still built on Claude and always will be. But there was one job I had given the AI that it was never the right tool for, and once I saw it I couldn't unsee it.

What it was doing

Verilay reads your code and tells you what's in it. One of the things it checks is whether you've left a password or an API key sitting in a file — a real and very common problem in apps built with Lovable or Replit, because the AI that built your app will cheerfully paste a key straight into the code if you're not watching.

The way I had built that check was, roughly:

Here are 25 of your files. Have a look through them for anything that looks like an API key.

And Claude would look, and usually find them, and explain them beautifully.

It took me an embarrassingly long time to notice the three things wrong with that.

It was only looking at 25 files. Verilay picks the 25 files most likely to matter and sends those. If your app has 300 files — which is normal — then the other 275 were never looked at. A key sitting in file number 26 was invisible. Not "probably found", not "less likely to be found". Invisible.

It gave a different answer each time. Not wildly different, but enough. Ask an AI to skim for something and you're asking for a judgement, and judgements vary. That was the wobbling grade from my June post, showing up in the one place where wobble is least acceptable.

It cost money every single time. Every analysis Verilay runs costs me a few cents in AI credits. Some of those cents were being spent asking a very sophisticated reasoning model to do something a wristwatch could do.

The thing I finally noticed

Finding an API key is not a judgement call.

An Amazon Web Services key always starts with the letters AKIA followed by exactly sixteen more characters. Not usually. Always. A Stripe live key always starts sk_live_. A private key file always begins with a line saying -----BEGIN PRIVATE KEY-----.

These aren't things you weigh up. They're things you look up. It's the difference between "is this a good password?" — a judgement — and "does this postcode exist?" — a lookup.

I was paying for judgement on a lookup.

So I wrote plain code to do it instead. A few hundred lines, no AI involved. It knows the shapes of around twenty different kinds of key, it reads every file in your project, it costs nothing, and it gives the same answer every single time because it isn't deciding anything. It's matching.

What changed

That last row is the one I care about most. Before, the tool said something looked risky. Now it says exactly which file, exactly which line, and the first few characters of what it found. You can go and look. You don't have to take anyone's word for it, mine or the AI's.

The example that made it worth doing

Here's the one that convinced me.

If your app uses Supabase — and if you built with Lovable, it almost certainly does — you have two keys. They look nearly identical. Both are long strings of gibberish starting eyJ.

One of them is the anon key. It's meant to be in your app, visible to everyone. That's by design and it's fine.

The other is the service role key. It ignores every privacy rule you have set up. Anyone holding it can read, change or delete every row in your database — every customer, every address, every phone number — regardless of what permissions you configured.

Putting the second one where the first one belongs is, I think, the single most damaging mistake a non-developer can make with an AI-built app. And you would never spot it by eye, because the two look the same.

The plain code can tell them apart. It unpacks the key and reads what's inside, which takes about a thousandth of a second. It checks every file, so it doesn't matter where the key is hiding.

The old way — asking an AI to skim 25 files — might have caught it. Might not have. Might have caught it on Tuesday and missed it on Wednesday.

That's not good enough for something that serious.

So what is the AI still doing?

Everything that actually needs thinking about.

Because here's what the plain code produces:

SUPA001  critical  src/config.ts:2  eyJhbG...

Which is useless to you. It's useless to almost everyone, honestly. It's a fact with no meaning attached.

The AI turns it into this:

Your Supabase master key is in this file. It looks almost identical to the safe public key, but it ignores every privacy rule you've set up — anyone who finds it can read, change or delete every row in your database, including other people's personal details.

Remove it from this file, then go to Supabase → Project Settings → API and roll the service_role key.

That's the part I can't write rules for. Understanding what your app does, working out which of your problems matters most, and explaining it to someone who has never written a line of code — that's judgement, and it's exactly what the AI is extraordinary at.

The mistake wasn't using AI. The mistake was using it for the wrong half.

The bit you can actually use

If you're building anything with AI — an app, a tool, an automation — it's worth going through it and asking one question of each part:

Is this a judgement, or a lookup?

Judgement is anything where a thoughtful person could reasonably disagree. What does this app do? Which of these problems is most urgent? How do I explain this to someone who's frightened? Give those to the AI. It's better at them than most people.

Lookups are anything with a right answer that doesn't change. Does this string match a known pattern? Is this version number higher than that one? Is this date in the past? Write those as ordinary rules — or ask the AI to write the rules for you once, and then run the rules forever.

You'll spend less, wait less, and get the same answer twice in a row.

There's a version of this you may already be doing without noticing. If you've ever asked ChatGPT to add up a column of numbers and got a slightly wrong total, that's the same mistake in miniature. Arithmetic is a lookup. Don't ask for judgement on it.

One more thing, since I'm being honest

While I was in there, I found three real security holes in Verilay itself. In the security tool. Not theoretical ones — the sort that would have been worth writing up if I'd found them in someone else's app.

They're fixed now, and I'll write about them separately, because they deserve more room than a footnote and because one of them is genuinely interesting: the dangerous part never went anywhere near the box where users type things, which is exactly why I hadn't spotted it.

Verilay's code has always been public, at github.com/ekbm/verilay. I've said before that a tool asking you to trust it should let you check. That cuts both ways. It means anyone can find what I missed — and someone finding it is a much better outcome than nobody looking.