Why Your Security Scanner and Your AI Builder Disagree (and the GitHub Sync Trap)

Finding the problem was the easy part.

It started with monitoring

I don’t just scan my own apps once. Verilay checks them for me every day: the code on GitHub and the live site. If something changes, a grade drops, or a new problem appears, it emails me. That’s monitoring, and it’s the part of Verilay I rely on most.

This week, two of those emails arrived on the same morning. One app is built with Lovable, the other with Replit. Both grades had dropped to an F.

Nothing in the apps had changed overnight. Verilay had got better at reading the lists of packages your app depends on, so it could suddenly see problems that had been sitting there all along. Between the two apps: almost 90 packages with known security problems.

Then the deep scan

The daily check is quick on purpose. It reads enough to notice a change, not your whole codebase. So for the full picture, I ran a deep scan: Verilay reads every file, checks every layer of the app, and writes one full report with a ready-to-paste fix prompt for each problem.

Now I knew what was wrong.

That sounds like the hard part. It wasn’t.

The hard part was everything after.

Then the fix

For each problem, Verilay gives you a prompt to paste into your AI builder. So I did what anyone would do. I copied it, pasted one into Lovable and the other into Replit.

Here’s what I expected: “Done, all fixed.”

Here’s what I got instead: questions.

The back-and-forth

Replit came back with a different number. Verilay said 38 packages in that app had problems. Replit ran its own check and found only one. For a moment I wondered which of them to believe, and I’m the person who built the scanner.

Then I remembered. A few days earlier, Replit had emailed me its own security alert, and I’d fixed most of those packages right there in Replit. But those fixes had never been synced to GitHub, and GitHub is what Verilay reads. Verilay had scanned the old code, correctly. Replit was looking at the new code. Both were right.

That left the one real problem: a package that checks incoming data could be fed a huge list and run out of memory.

The proper fix was risky. Fixing that one package properly meant a big upgrade that could break forms, the database setup and error messages. Replit asked whether to go ahead. I didn’t know that either. In the end we chose a smaller, safer move: block oversized lists before they ever reach the package, and leave the big upgrade for a calmer day.

The safe fix nearly broke the app. Before publishing, I asked Replit one more thing: will these new limits get in the way of normal use? They would have. “Select all” on more than 100 documents would have failed. So would a long blog post. Nobody would have noticed until a real user hit it and saw “Failed to save”. We raised the limits and added clear messages.

Lovable asked me to make a call. On the other app, it updated the safe packages one at a time, checking the app still built after each one. Then it stopped and asked whether to carry on with the major upgrades. Most of those were tools that only run while the app is being built. Visitors never download them. But I only knew that because I asked Lovable to sort everything into “visitors get this” and “build tools only” first. After that sort, two problems were left that actually mattered, and both got fixed.

The builder found things the scan didn’t. While it was in there, Lovable found two more weak spots in the backend that my scan hadn’t flagged, and tightened them up as well.

And at the very end, the same trap again. The new fix was published, but it hadn’t been pushed to GitHub either. If I’d rescanned straight away, Verilay would have said “not fixed”, and I’d have started all over again. Lesson learned twice in one day: whatever your builder fixes, make sure it reaches GitHub before you scan.

A tip if anything checks your GitHub

This goes for Verilay’s monitoring and for any other tool that watches your repo: it can only see what’s on GitHub. If your builder’s changes don’t reach GitHub, the tool keeps checking old code. It will warn you about problems you’ve already fixed, and it will miss anything new.

So turn on syncing with GitHub in your builder, and check it’s actually working:

• Connect your project to GitHub in your builder’s settings, and switch on automatic syncing if your builder offers it.
• If it doesn’t sync on its own, push your changes yourself after every fix. In Replit, I had to do that from its Git panel.
• Check now and then: open your repo on GitHub and look at the date of the latest change. If it’s older than your last edit in the builder, the sync isn’t happening.

What I learned

Over two sessions I went back and forth with the builders more than a dozen times. Every step needed a judgement call:

• Which of these numbers do I believe?
• Is it safe to say yes to this question?
• Will this fix break something my users rely on?
• Is it actually fixed, or did the builder just say so?

A developer answers those in seconds. If you built your app with AI because you don’t write code, each one is a wall. And that’s where most people stop. Not at the scan. At the second reply from their builder.

I also learned something less comfortable: my own scanner made the first step harder than it needed to be. On the Lovable app, the first thing the builder did was shrink Verilay’s list, because Verilay counted build tools as if your visitors downloaded them. That’s on me, and it’s the next thing I’m fixing. A scan should tell you what actually reaches your users first.

Coming soon: fixing it end to end, with Verilay

Verilay will always give you the free prompt. If you’re comfortable working with your builder, that’s all you need.

For everyone else, I’m building fix sessions: a guided way to see a problem through to the end, as a paid option alongside the free prompt.

• Verilay checks your GitHub code is up to date, then sorts the problems into what your visitors get, what’s exposed on your backend, and what’s only a build tool.
• You send the prompt to your builder, then paste its reply back into Verilay.
• Verilay tells you what the builder changed, what it skipped, and what else it touched. When the builder asks you a question, Verilay tells you how to answer.
• Before you publish, Verilay asks your builder to check the fix won’t break normal use.
• Then Verilay rescans your code and gives you a fix record: Fixed, Protected (blocked for now, proper fix later), or Left on purpose.

“Fixed” will mean one thing only: Verilay rescanned your code and the problem is gone. Not that your builder said so, and not that your whole app is now secure.

So the full loop looks like this: monitoring notices something changed, a deep scan shows you exactly what, and a fix session sees it through to “Fixed”.

If you’d like to be one of the first to try fix sessions, email me at [email protected]. If you’d like Verilay to watch your apps the way it watches mine, tell me on the monitoring page. Until then, the free scan and its prompts are at verilay.dev.