Risk

Is AI-generated code safe to ship?

Most of your product's code was probably written by a machine. What that changes about your risk, the questions to ask your engineering team, and the controls that actually matter.

3 min readBy Mayur Kale

Key takeaways

  • AI-generated code is not inherently unsafe. Unreviewed code is, and generation has made it far cheaper to produce code than to check it.
  • The measurable risk with hard numbers attached is dependencies: models import packages that do not exist at rates around one in twenty, and attackers register those names.
  • Ask your team one question: has review and testing capacity grown at the same rate as code output? If not, your defect rate is rising whether or not you can see it yet.
  • Watch approval speed against change size. Bigger changes approved faster is the clearest early signal that review has become a formality.
  • You do not need to read code to govern this. Four questions and two metrics are enough.

If you are funding software built in 2026, a large share of it was written by a machine. That is not a scandal and it is not something to stop, it is how software is now made, and the alternative is paying considerably more for the same result.

But it does change your risk profile, in ways that are specific enough to manage. You do not need to read code to do it.

The actual problem is arithmetic

Writing code got dramatically cheaper. Reading and understanding code did not.

That is the whole thing. Every consequence below follows from it. When production is cheap and verification is not, the natural drift is toward shipping more code that fewer people have genuinely read, not because anyone decided to, but because that is where the path of least resistance leads.

DORA's 2025 research found this playing out at industry scale: AI adoption correlating with higher delivery throughput and higher instability, with more change failures and more rework. Their reading is that AI exposes existing bottlenecks in testing and review rather than creating new problems.

Four risks that reach the business

Dependencies nobody checked. This is the one with hard numbers. A USENIX Security study generated 2.23 million code samples across sixteen models and found them importing packages that do not exist at rates from 5.2% on commercial models to 21.7% on open-source ones. A 2026 replication on frontier models measured 4.62%–6.10%, better, not solved.

The invented names are predictable, which means attackers can register them in advance and wait. The attack has a name, slopsquatting, and it is not hypothetical.

The fix is mechanical and cheap: lockfiles committed, a registry allowlist or proxy, and a build that fails if a package appeared out of nowhere. Ask whether you have it.

Security flaws that look like ordinary code. Missing permission checks, unsafe handling of user input, secrets in the wrong place. Generated code is fluent and well-formatted, which means it triggers less scrutiny than messy code doing the same wrong thing.

Silent failures. Generated code has a habit of catching errors, logging them and continuing. The system stops failing loudly, so you learn about problems from customers rather than from monitoring.

Code nobody can explain. The expensive one. It costs nothing until something breaks at 3am and the person on call is reading it for the first time, under pressure, with no idea why it was built that way.

The perception problem

One research finding is worth knowing because it undermines the most common way this gets evaluated.

METR ran a randomised controlled trial: sixteen experienced developers, 246 real tasks from their own backlogs, randomised as to whether AI tools were allowed. Developers were 19% slower with AI, and estimated afterwards that they had been 20% faster.

That population was unusual, maintainers working in code they knew intimately, roughly the worst case for these tools. The result should not be read as "AI makes everyone slower". But the perception gap is the part that generalises, and it means one thing for you:

What to ask, and what to watch

Four questions for your engineering lead. None require a technical answer you cannot evaluate:

  1. Has review capacity grown as fast as code output? If output tripled and the same two people review everything, that is the finding.
  2. Are we shipping bigger changes and approving them faster than a year ago? This is the signature of review becoming a formality.
  3. Do we verify dependencies? Lockfiles, allowlist, a check that packages exist and are maintained.
  4. Could someone other than the author debug the payment path at 3am? If the honest answer is no for any critical system, that is your priority.

Two metrics to watch, both of which you probably already have:

  • Change failure rate, the share of releases that cause a problem. If it is rising, generation has outpaced verification.
  • Review time against change size, larger changes approved faster is the clearest early warning available.

A policy worth having

Short, and about standards rather than tools:

  • The review bar is identical regardless of who or what wrote the code
  • Changes have a size limit, held even when generation makes bigger ones easy
  • A human writes or approves the test that defines "done" before implementation
  • Dependencies are verified before they enter the build
  • A named person is accountable for every merged change

Policies that try to restrict which tools engineers use do not work. They get ignored, and they teach people not to mention what they are doing, which removes your visibility at exactly the moment you need it.

The reasonable position

Use the tools. They genuinely lower the cost of building software, and refusing them puts you at a disadvantage against competitors who do not.

Then fund the other half. If code output rises and review capacity does not, you have not become more productive, you have moved work from before the release to after it, where it is more expensive and your customers find it first.

Frequently asked questions

Is it safe to ship code written by AI?

It is as safe as your review process makes it. The code itself is often perfectly good; the risk comes from volume. Generation has become dramatically cheaper while reviewing and understanding code has not, so the natural drift is toward shipping more code that fewer people have genuinely read. That drift is the risk, not the tool.

What are the actual risks of AI-generated code in a product?

Four that matter commercially. Dependencies nobody vetted, including packages that do not exist and have been registered by attackers. Security flaws in ordinary-looking code, particularly around permissions and user input. Error handling that fails silently, so problems reach customers before they reach you. And code nobody on your team can explain, which becomes expensive the first time something breaks at 3am.

What should I ask my engineering team about this?

Has our review capacity grown as fast as our code output? Are we shipping larger changes and approving them faster than a year ago? Do we have a check that every dependency we import actually exists and is maintained? And could someone other than the author debug the payment path at 3am? Those four questions surface almost everything that matters without requiring you to read any code.

Should we have a policy about AI use in engineering?

A short one is worth having, less to restrict use than to set expectations. The useful content: the review standard is identical regardless of who wrote the code, changes have a size limit, dependencies must be verified, and someone is accountable for every merged change. Policies that try to restrict which tools engineers use tend to be ignored and teach people not to mention it.

Does using AI mean we need fewer engineers?

Not in the way it is usually pitched. The work shifts rather than disappearing, less time writing, more time specifying, reviewing and verifying. DORA's 2025 research found AI adoption correlating with both higher delivery throughput and higher instability, which is what happens when the generating half speeds up and the checking half does not. Teams that cut headcount on the assumption of pure speedup tend to discover the second half later.

References & further reading

  1. [1]
  2. [2]
  3. [3]
  4. [4]
  5. [5]

Turn the trend into shipped software

I contract as a senior fullstack and data engineer, building the production systems your customers and your reporting depend on.

Explore services↗