Hospitals Can’t Outsource AI Accountability

Watch more: TechReg With Alaap Shah of Epstein Becker Green

Healthcare organizations spent the first phase of the artificial intelligence (AI) boom asking whether the technology worked. The next phase will be defined by a harder question: Who is responsible when it does not?

AI is moving deeper into clinical decisions, patient communications, claims administration and health-data exchange as the regulatory environment remains fragmented. Federal agencies are reconsidering parts of their oversight, states are pursuing separate approaches and existing healthcare, privacy and consumer-protection laws are being applied to systems they were never designed to govern.

That does not create a regulatory vacuum. It creates a more complicated form of exposure.

“Self-governance matters because defensibility matters. We have already seen that risk is manifesting with respect to the use of AI technology in the healthcare sector,” Alaap Shah, member of the firm at Epstein Becker Green, told Competition Policy International (CPI), a PYMNTS company, in an interview.

As a result, AI-native governance capabilities are becoming both legal protection and commercial infrastructure across healthcare.

Existing Law Still Applies to Healthcare AI

One of the most dangerous assumptions in enterprise AI is that organizations can wait for lawmakers to produce a clear rulebook. Healthcare companies do not have that luxury. Laws governing privacy, discrimination, consumer protection, contracts and professional duties already apply to AI-enabled activity, even when they do not mention artificial intelligence directly.

“There are existing bodies of law that, while not passed or promulgated for the reason of AI, are still applicable to AI solutions,” Shah said, noting that the question for healthcare firms is not simply whether an AI-specific law applies but whether the system creates risks covered by older legal obligations.

A patient-facing model that produces different recommendations across demographic groups may generate discrimination exposure. A clinical tool that influences a physician’s decision may become relevant in a malpractice case. AI is not replacing healthcare’s existing liability structure. It is entering it.

The Food and Drug Administration (FDA) continues to regulate certain software as a medical device, but its approach leaves room for interpretation. Shah called the boundary “still a little murky.”

“There are still going to be situations where a company may push right up against that line and FDA may come back and say, ‘Actually, this is something we’d like to regulate,’” he said, noting that state medical boards add another layer.

A developer may describe a product as decision support. A medical board may see a system behaving like an unlicensed clinical operator.

“I think there’s going to be fragmentation for quite a while,” Shah said.

Healthcare’s AI Governance Must Become a Defensible Record

There’s another kink in the AI machine for healthcare. Healthcare organizations often do not control the models they deploy. Vendors may own the system logs, training process and performance data, but when a tool fails inside a hospital, the hospital may still be expected to explain what happened. That makes contracts central to AI governance.

Hospitals need more than standard promises about security and compliance. They need logging requirements, preservation obligations, audit rights and clear rules governing access to the evidence a model produces.

“To the extent that any events could be logged in the AI processing, that is something that needs to be happening so we can understand how that AI operated and why the input led to the output,” Shah said.

A clinician cannot defend a decision influenced by a system whose operation cannot be reconstructed. Nor can a hospital investigate an adverse event if the vendor controls the relevant records. The black box is becoming a contractual problem.

AI also challenges one of healthcare’s most familiar privacy safeguards: de-identification.

Removing direct identifiers can reduce risk, but AI systems can combine datasets, infer attributes and reconnect information that appeared anonymous in isolation.

“It’s a real possibility that AI algorithms could re-identify individuals if sufficient data gets put in, even if de-identified in the first instance,” Shah said, adding that bias presents a similar problem.

AI can influence treatment, prior authorization, insurance coverage and patient communications. When those systems produce different outcomes for protected groups, organizations may face litigation, regulatory scrutiny or public backlash regardless of federal enforcement priorities.

That makes internal governance more important. Organizations need to inventory their AI systems, classify them by risk, assign accountable owners, train employees, document controls and allocate responsibility across vendors. The goal is not to prove that failure was impossible. It is to prove that the organization was not careless.

Watch the full PYMNTS TV interview with Alaap Shah to hear more about:

Alaap Shah is a member of the firm at Epstein Becker Green