AI Now Writing Code That Humans Can’t Even Understand

As programmers make more and more use of generative AI in the workplace, their responsibilities are quickly transitioning from creating new products from scratch to babysitting a large language model that cobbles together code.

It’s gotten to the point where some engineers are no longer able to make sense of the code being spat out by an AI, as chip optimization software company Infinity founder Jeremy Nixon told Business Insider.

That doesn’t bode well, considering researchers are finding that the widespread use of AI can lead to cognitive decline and skill atrophy, including among computer scientists. Developers are afraid that software engineers are drifting further and further away from the act of actually writing code and learning new skills required to architect larger projects as their employers continue to push for more AI automation.

Even employees at frontier AI labs are aghast at peers who are struggling to make sense of the outputs of AI models. Earlier this year, independent research firm SemiAnalysis technical staff member Jordan Nanos revealed in an interview that OpenAI engineers had no idea what they were looking at while scrolling through the code of their own graphics processing unit kernel, the program at the core of the company’s efforts to better make use of available hardware to train and run AI models.

“The point is, the guys who were scrolling through this code with us, it was kind of clear that they know a whole bunch about hardware, they know a whole bunch about the concepts in the system, and they just have no idea what this MLA kernel that they’re showing us for DeepSeek actually does,” Nanos told Fireside Alpha‘s Mansour Karam.

“You can go line by line and it’s like nope, nope, nope,” he added. “The AI understands it, the AI tests it, and you see the results. It produces correct kernels that perform really well.”

At the time, Nanos argued that the “actual code is not necessarily something that a human has to reason about deeply,” because the “AI knows how to manipulate the data movement and the processing elements on the hardware that you’ve given it.”

One danger is that errors or hallucinations will fall through the cracks as human oversight fades. Case in point, Amazon was forced to implement a 90-day “code safety reset” earlier this year following a series of outages that disrupted customer orders. While the company pushed back on claims that AI was to blame, internal meeting notes revealed that managers were concerned about the “high blast radius” of “gen-AI assisted changes.”

A recent analysis by software company Undo found that the transition to AI agents is opening the floodgates. Some 35 percent of AI-generated code “has not been fully comprehended before engineering teams push it to production,” the firm said.

Meanwhile, 29 percent of software engineer respondents said that they’d witnessed a loss of productivity due to “engineers needing to ‘unpick’ AI-generated code” multiple times per month, while a whopping 94 percent said they saw such an incident at least once in the past six months.

The sheer amount of time engineers are now spending each day going through AI-generated code raises a nagging question: is the use of the tech more trouble than it’s worth?

To people like Nanos, it’s not necessarily a matter of incompetence. It’s “simply an illustration of a paradigm shift in software development,” as AI researcher Sergey Cleftsow argued in a recent Medium post.

“In other words, AI has advanced to the point where there’s no need to worry about many things anymore,” he wrote. “Developers simply need to define the architecture and correctness criteria; the neural network will handle the rest.”

But whether such optimism will hold out as software developing becomes ever more unrecognizable remains an open question.

Besides, researchers are equally concerned that a new generation of engineers are growing up in an environment that’s actively undermining the value of understanding the code itself. And that doesn’t bode well for an increasingly digital world order that’s putting AI front and center.

More on AI code: Software Engineers Say They’re Losing the Ability to Code Now That AI Does It for Them