Government lacks ability to verify AI labs’ claims, experts say

One of the key findings in IST’s report is that the government lacks the ability to verify frontier AI labs’ claims about what their products can and will do.

“Respondents pointed to chronic underinvestment in test and evaluation, the absence of independent evaluation processes, and no reliable way to benchmark frontier capabilities,” the report said.

That underinvestment means that government leaders cannot verify how AI systems will behave before integrating them into military technology and other critical platforms. “Without its own benchmarks, the [government] is left to take capability claims on faith rather than measurement.”

Experts were also deeply skeptical that current AI systems could be trusted to act independently. “Many see hallucination as intrinsic to AI models, making systems highly problematic to rely on without significant, detailed human checking,” IST observed. “As one put it, AI can make a good analysis better and a bad one worse.”

More than eight in 10 respondents said AI would “very likely” improve all of the cyber defense capabilities they were asked about — from vulnerability discovery to malware analysis to incident response — but 57% also said AI was helping attackers more than defenders in the near term.

The biggest cybersecurity risk associated with AI is its ability to find and exploit vulnerabilities, according to the survey, whose respondents ranked that risk first by a wide margin.

And while legal and regulatory frameworks ranked second on respondents’ list of the biggest hurdles to combating AI threats, respondents were split over the right way to regulate AI. Roughly one-quarter of experts advocated for cross-sector federal requirements, 31% supported sector-specific requirements and 36% favored a hybrid approach.

Experts encouraged policymakers not to focus too much on AI’s unique capabilities when evaluating AI-related risks, saying that basic cybersecurity failures were likely to be the biggest cause of issues such as “loss-of-control scenarios, adversarial manipulation, and the theft of model weights and weapons designs.”

“The bottleneck is deploying cyber defense at scale, not the technology itself,” IST said, adding that weak cybersecurity “will enable many of these other risks.”