I ran my own free tool on my own app and got an F. Fixing the one real vulnerability didn’t explain why the score jumped to a perfect A — the real answer was a file that had quietly stopped telling the truth.
I ran Verilay — the tool I built — on my own app, BuildBook. It came back with a grade of F. 23 critical findings, 18 warnings, 41 “dependencies” flagged as risky.
If you’ve built something with Lovable, Bolt, or Replit, you’ve got dependencies too, whether you’ve ever seen that word or not. Think of them as ingredients your app borrows instead of making from scratch — a calendar widget, a way to generate PDFs, a login system. Nobody hand-builds all of that themselves. You (or rather, your AI builder) reach for hundreds of these pre-made pieces, bundle them together, and that’s most of what your app actually is.
Which means when one of those borrowed ingredients has a known security problem, it’s now a problem in your app too, even though you never wrote a line of it.
That’s what a “dependency vulnerability” scan checks. And I want to walk you through what happened when I ran that check on my own app — not because the vulnerability itself was dramatic, but because of something much quieter I found underneath it, that I think is worth understanding if you’re building with AI tools too.
Verilay’s free scan checks every one of your app’s ingredients against a public list of known security problems. It came back and said: 41 out of 963 ingredients have a documented issue, 23 of them serious.
On purpose, the free scan doesn’t tell you which 41 — that detail is reserved for the paid version. Instead, it hands you a ready-to-paste message and tells you to give it to your own AI builder, so it can go find the specifics. So that’s what I did: copied the message Verilay gave me, pasted it into Lovable, and asked it to investigate.
Here’s the first thing worth knowing: not every “vulnerability” is something a stranger could actually use against you.
Lovable’s investigation split the 41 flagged ingredients into two very different piles. Most of them — the majority — were tools that only run on my computer while I’m building the app. Things like the tool that packages everything together before it goes live. A security problem in a tool like that is a real thing, technically, but nobody visiting BuildBook could ever touch it. It’s like a security flaw in the factory machine that boxes up a product, versus a flaw in the product itself — only one of those can hurt a customer.
Buried in that pile was one real one: a login-related library called react-router, sitting on an old version with a documented bug called an “open redirect.” In plain terms: a link that looks like it’s taking you somewhere trusted, but can be secretly rigged to bounce you somewhere else instead — the exact kind of thing someone could exploit right after a person logs in, since BuildBook uses that library to decide where to send you after signing in.
That one, Lovable fixed: one small version bump, no other changes needed. Small, targeted, done.
I ran Verilay again to confirm the fix. Same result. Still 41 flagged, still an F.
Turned out the fix had been made inside Lovable, but hadn’t actually been pushed out to the real, live version of the code yet — a boring, easy-to-miss gap between “I told it to fix this” and “the fix is actually live.” Once I double-checked and made sure it had gone out properly, I ran the scan a third time.
This time: zero critical, zero warnings. Grade A. Every single ingredient checked out clean.
Not “most of them.” All of them. And that’s the part that didn’t make sense to me.
I’d only asked Lovable to fix the one real issue — the login redirect bug. That shouldn’t have had anything to do with the other 40 unrelated build-tool flags. So I asked Lovable directly: tell me exactly what changed, don’t summarize, I want the real answer.
It turned out two completely separate things had happened, back to back, that I’d assumed were one:
The actual fix was exactly as small as it looked — just that one login library, bumped to a newer, safer version. Nothing else touched.
Everything else got cleaned up by accident, because of something unrelated: the file that’s supposed to be an exact, accurate list of every ingredient the app actually uses — its “lockfile” — had quietly fallen out of date. It was missing dozens of ingredients the app was genuinely using (things for generating PDFs, handling 3D graphics, running tests) that it simply didn’t know about anymore. When a routine “refresh this list” step ran, it rebuilt that file from scratch — and rebuilding it happened to update all those old, flagged build-tools to their current, safe versions at the same time.
Two unrelated events, landing back to back, that looked from the outside like one satisfying fix.
This is the part I actually want you to remember.
That lockfile has exactly one job: be an honest, accurate record of what your app is really running. Nothing more. And it had been quietly wrong — missing real ingredients that were genuinely in use — for who knows how long, without a single symptom. Nothing broke. Nothing looked off. The app worked completely normally the entire time it was wrong.
The only reason anyone found out is that a security check happened to go looking at exactly the right moment, and someone (well — Lovable) refused to just say “looks fine” without checking.
That’s the pattern I keep running into, over and over, building a tool that reads AI-built apps for a living: the scariest problems aren’t the ones that break something visibly. They’re the ones that just quietly stop being true, and keep on working anyway, right up until the day someone actually checks.
A few honest things worth taking away, if you’re building with Lovable, Bolt, Replit, or anything similar:
A vulnerability count is only as honest as the record it’s reading from. If that record has drifted — like mine had — the number you’re looking at could be wrong in either direction: hiding a real risk that isn’t showing up, or scaring you over something nobody can actually reach.
Not every flagged issue is equally dangerous. A scanner can’t always tell the difference between “a stranger could exploit this tomorrow” and “this only matters to someone who already has access to your build tools.” That’s exactly why the free scan hands you a prompt instead of a verdict — go get your AI builder to actually look, rather than panicking at a raw number.
When something fixes itself more than you expected, ask why. I almost just wrote down “we fixed one bug and it cleared 41 issues” and moved on — which would have been a nice story and a wrong one. The real answer took one more question to find, and it turned out to be the more useful thing to know.
The good news, this time, was real. It just wasn’t the good news I thought I was getting — and the extra five minutes it took to find out the real reason taught me more than the fix itself did.