From Pilot to Proof: How Health Systems Can Show Their AI Is Working

Many health systems struggle to prove AI ROI, cutting tools too early or funding them on promise alone. A fair test starts with a sound workflow, a clear baseline, and set decision points to scale, improve, or retire each tool.

Health systems are particularly incentivized to chase artificial intelligence’s (AI) promised efficiencies amid ongoing industry pressures including staff shortages, physician burnout, shrinking budgets from Medicaid cuts. In 2025, 27 percent of health systems paid for commercial AI licenses, triple the rate across the US economy.

With that expenditure comes the complicated question of how to justify it. An overly hasty or narrow approach to return on investment (ROI), for instance, often leads organizations to discontinue a program before addressing implementation gaps, which does not just incur the loss of the technology’s current benefits but blocks the path to workflows that will define the next era of healthcare. On the other hand, sustaining AI programs based on the hope that they’ll someday generate return can be just as costly.

To stay competitive, health systems need to decide whether AI is delivering value. First, they must ensure that they’re not automating a broken process. Then systems must define the outcome they want, design the workflow that produces it, and establish the data, ownership, and measurement to make improvement sustainable. Here’s how to start.

Ambient Scribes and Radiology: Where Healthcare AI Value Actually Shows Up

Two early success stories reveal what AI in healthcare looks like in practice—and why decisions about how and when to assess its value are so important.

Ambient AI documentation

Arguably the first generative AI product to show value and impact at scale, ambient scribes listen to doctor–patient conversations, generally via an app on the physician’s phone, and convert the conversation into a note for the physician to sign off on before it’s entered into the electronic health record (EHR). Ambient scribes immediately saturated the market—more than sixty companies offered ambient solutions by early 2025—and many health systems adopted them expecting that they would lead to a greater number of patient visits per day.

That increase in visits was modest. According to a 2026 study across five major health systems, adoption was associated with a 1.7 percent rise in visit volume and a conservatively estimated $167 per month in visit revenue per clinician, modeled at Medicare rates. As fees for these tools range from $200 to $600 per doctor per month, leadership might conclude that the program is not worth the cost.

But that framing misses where value did appear. Ambient scribes may also support more complete documentation and coding, but health systems should validate any resulting reimbursement or quality-reporting benefits locally. A 2025 study found that just one month with an ambient AI scribe was associated with a drop of roughly 13 percentage points in self-reported burnout among those working in ambulatory clinics. A health system focused only on visit volume would overlook these benefits to its own detriment.

Radiology

Radiology has become both a blueprint for how to apply AI in clinical work and a cautionary tale against guessing how these technologies will develop. Built on predictive AI, radiology accounts for more than three-quarters of the AI devices the Food and Drug Administration has authorized, and this oversight has made health systems more comfortable adopting them. These devices’ early promise notoriously led scientist Geoff Hinton to claim that people should stop training to become radiologists.

In the decade since Hinton’s statement, the radiologist staff at the Mayo Clinic has grown 55 percent, even as the Clinic now uses more than 250 AI models. For example, it uses an AI tool to measure the volume of kidneys, a task physicians formerly performed by hand.

Thus, AI can augment the work of radiologists rather than replace them. After deploying AI imaging tools across more than fifty hospitals, Mercy Health cut scan-to-result time from nearly twenty hours to just over one for the slowest-handled cases flagged for acute findings; and flagged more than 50,000 incidental findings in five months. More than 5,600 patients received faster follow-up as a result.

A health system that measured radiology AI by savings on radiologist salaries would have found virtually no value.

How to Evaluate Each Type of AI

Not every problem needs AI. The better first move often is to optimize existing EHR and enterprise resource planning (ERP) functionality or use simpler automation.

Where AI is the right answer, it typically comes from one of three sources. The source doesn’t change the core discipline of evaluation—every tool needs a baseline, owner, and defined measure of value—but it shapes how that evaluation works in practice depending on workflow, risk, integration, and cost.

• Platform-embedded AI. Enterprise systems often are already running in hospital systems and generally introduce AI updates (e.g., an EHR tool could begin drafting replies to patient portal messages automatically). To measure gains, roll out new features in stages so departments using them can be compared with those that aren’t.
• Point solutions. These are often standalone products from startups and specialty vendors built for a single job, like the ambient scribes discussed above. They are easy to judge too quickly. Proper implementation may require adjusting a tool to your workflow (a prior authorization tool refined for another system may handle large national payers well but stall on regional plans) or adjusting your workflow to the tool (asking physicians to document more consistently so an automated coding tool can read their notes).
• Homegrown AI. A health system may build its own function, as Mayo Clinic has done with many of its imaging models. But these tools require a potentially large investment: engineers, data architects, cybersecurity, and staff focused on human-centered design. Without a vendor, the organization also carries the full burden of maintenance and support alongside the assessment it owns for any tool.

Best Practices for Proving AI ROI

Leaders across the industry are investing in AI, automation, and new operating models. AI’s value must be assessed across financial, operational, and clinical outcomes. Here are five foundational best practices health systems can use to capture benefits:

1. Inventory AI already in use, its owners, costs, and measured outcomes. For new deployments, establish a baseline and credible comparison before rollout. For existing tools, reconstruct a pre-deployment baseline or use an appropriate comparison group. Without this step, organizations weigh without zeroing their scale.
2. Match the evaluation horizon to the use case, and set decision gates. Some organizations pull AI programs within twelve to eighteen months because they weren’t operationally ready to take full advantage of the technology. That can become a lose-lose: for example, if physician documentation is too inconsistent for an AI system to code from it, a portion will be sent back for manual review, reducing efficiency gains.

Closing that gap starts with a readiness assessment and a defined remediation plan. Even within a multiyear program, early deployments should have near-term milestones (often within ninety days) for adoption, workflow performance, and safety. Financial and clinical outcomes may require longer evaluation periods. Set checkpoints for adoption, workflow performance, safety, and benefits; and at each checkpoint decide whether to scale, remediate, replace, or retire the tool.

3. Measure realized value, not just activity. Time saved is not automatically money saved. A credible evaluation:

• • tracks actual use, cycle times, exceptions, rework, and human review
  • shows how freed capacity becomes reduced expense, avoided hiring, added contribution margin, or improved collections
  • separately measures safety, access, patient outcomes, quality, and clinician experience
  • counts the full cost, including implementation, integration, training, oversight, maintenance, and licensing

Incremental revenue is not incremental margin.

4. Prioritize high-value, feasible, well-governed workflows. Most AI use cases in health systems today are administrative—tools for revenue cycle, patient access, and documentation, like ambient scribes—rather than clinical. These can be good starting points, but the category alone doesn’t make a workflow ready or low risk. Early wins in well-chosen workflows build capabilities and confidence to expand responsibly into more complex use cases.
5. Build an operating model with clear ownership. Create an AI center of excellence that sits between technology and operations, enabling departments through common standards, reusable capabilities, and oversight across each tool’s lifecycle. Understanding, designing, and assessing these systems takes time and is not a muscle organizations naturally have. Operational leaders should own workflow change and outcomes; finance should validate benefit assumptions and track whether they’re realized; technology teams should own integration and support with clinical, compliance, privacy, and security oversight matched to risk. Validation, safety oversight, and benefit accountability stay with the health system, even for purchased tools—vendors can provide support, if needed.

Develop a Fair Test

Health systems should neither abandon AI prematurely nor sustain it on promise alone. Earning buy-in from physicians will require giving them the support and training to use these tools effectively. Leadership confidence grows when operational and clinical improvements are demonstrated and financial benefits are validated, not simply projected.

A fair test starts with a well-designed workflow, the simplest effective technology, an agreed baseline, and accountable owners. It measures financial, operational, and clinical value over an appropriate horizon with explicit decisions to scale, improve, or retire each tool.