Health systems building AI agents must balance trust and token budgets

In Part 1 of this two-part series, we described how, despite the myriad challenges and necessary requirements for health systems to build agentic AI tools that work, Advocate Health and ECU Health were finding success with the Epic EHR-integrated Agent Factory platform.

In using the Agent Factory platform to create and deploy autonomous artificial intelligence agents, both health systems have tackled several challenges that get to the heart of no-code AI.

Andy Crowder, Advocate's chief data and AI officer, and Jacob Parrish, ECU's vice president of clinical operations, spoke candidly about their efforts to address long-term trust in AI and the affordability of deploying AI agents at their organizations.

Applying rigid architectural rules for agents

Advocate's agents were co-created during intense "immersion events" every six weeks, which involved frontline doctors, nurses and pharmacists who teamed up directly with internal and Epic engineers to build the workflows, Crowder said.

That collaboration helped ensure that the health system's AI Studio adheres to a strict "human in the loop" rule for anything created and deployed in its EHR.

"I'm not allowing the AI to take an action or a reaction without oversight by humans and clinicians that are in it all day long," said Crowder. "I don't allow it to order, I don't allow it to diagnose, I don't allow it to do any of those things without approval by the clinician who's administering whatever the function of the process is."

While there are a series of real-time monitoring tools in place to secure data and prevent agents from exposing it, "real-time exception logging" by clinicians supports the quality of their outputs.

"It said to do X, I didn't do it, and why," he explained. "And our teams look at those in real time, just like any other type of system abnormality; and we will disable them if we find that they are not performing appropriately."

Crowder said Advocate is also planning AI literacy training for all of its 162,000 employees.

But from a system-level perspective, addressing AI errors is "going to be really important as we move forward," and Crowder is thinking creatively about how to resource that critical IT function for the long-term viability of the health system's AI agency.

While Epic may have tools to monitor and observe AI, "we're going to be using solutions from multiple vendors in this space," he said, noting that a new proof of concept currently in process aims to "even up our skills in this space and our competencies."

"Even though we've got a human in the loop, we know that there are going to be some processes that maybe the human is just where it escalates to," said Crowder. "And we can offload some of that work.

"Some people will call that monitoring, some people call that agent orchestration, but that's an architectural capability that we're designing right now."

Parrish noted that ECU Health had already worked with AI outside of Epic and had a framework in place to oversee the quality of AI outputs.

"When [Agent Factory] came along, there was really no change to governance because we already had it. And then being partnered with Epic, it made it easier, on this particular project, because the data is in Epic and it's already protected through all the normal things," he said.

"We didn't have to worry about standing up anything else."

Token budgeting and focusing intentions

Managing large language model (LLM) spending requires analyzing fractional token usage, and it's a challenge for health systems confronting this type of budgeting for the first time, Parrish said.

It's possible to overspend on a pilot without knowing it, he said.

"We had to work through some of the token utilization, which is something new to all of us," Parrish explained. "We know what a token is, but this is a fraction of a token. And what is that going to mean from a budget perspective?

"Am I getting ready to spend $1,000,000 and I have no idea? Or am I getting ready to spend $100?"

While Agent Factory is LLM model-agnostic, customers pay token costs that Epic incurs. Healthcare organizations must manage a token budget and decide which autonomous agents deliver an ROI.

The opportunities of LLMs, and the necessary token budgeting required to take advantage of them, present a paradigm shift in clinical engineering. IT team structures must change, according to Jeremy Harper, chief information officer at Owl Health Works and board member of Indiana HIMSS.

To save time and money, AI project lifecycles must be tied to reality, he said. His framework for screening AI projects can help determine when AI is a powerful accelerator and when it's not.

That framework asks, "Where can we use large language models to facilitate and speed up a process, and where is it just mildly hopeless?" he told Healthcare IT News earlier this year.

Once ECU understood how token budgeting works, there were change management challenges waiting at the next hurdle.

Employees, when encountering the new agents, will ask what else the technology can do, Parrish said.

"It's human nature to want to go to the next thing, which is exciting, and that's where we've come up with a couple of our other ideas," he said.

However, Parrish said ECU has remained "very intentional from a leadership perspective" with the function of its first agents created as a result of this year's Epic build-a-thon.

"We truly want the feedback so we can make [a specific agent] better, because again, the end state is to make the workflow easier," he said.

ECU's teams using the agents are doing a really good job of saying, 'This one was good,' or 'No, this one was bad,'" Parrish said. "Every summary, we're able to get real-time feedback."

With user feedback consistent and metrics fed into dashboards, the tech team can make adjustments quickly, he added.

Andrea Fox is senior editor of Healthcare IT News.Email: [email protected]Healthcare IT News is a HIMSS Media publication.