
Healthcare
Healthcare AI Readiness Guide
A medical device approval tells you a tool met requirements for its intended use. It does not tell you the tool works on your patients, in your workflow, at your settings. Only local evidence does.
By Kesho Partners
9 minute read
In most industries an AI error costs money or trust. In healthcare it can cost a life, and the patient may never know a machine was involved. Clinicians are adopting AI quickly, vendors are switching it on inside the systems hospitals already run, and regulators are moving from one-off approval to lifecycle oversight. The organizations that do this well treat every AI tool that touches diagnosis, triage, treatment or the clinical record as part of clinical care: owned by clinicians, tested locally and monitored for harm.
Adoption has outrun governance
The US FDA had authorized more than 1,600 AI-enabled medical devices by September 2026. The American Medical Association found 81% of US physicians using AI in their practice in 2026, up from 38% in 2023. Most of that use started bottom-up, for summarizing research and drafting documentation.
Clinicians want a say. In the same survey, 85% wanted to be consulted or directly involved in decisions about adopting AI, and 88% were concerned about losing skills. Governance that is run by IT or procurement, and handed to wards, misses both points.
Why a vendor's numbers are not your numbers
University of Michigan researchers tested a sepsis prediction model built into a major electronic health record. The vendor reported accuracy (AUC) of 0.76 to 0.83. On Michigan's patients it measured 0.63, missing two-thirds of sepsis cases while alerting on 18% of all admissions. The vendor disputed the method, saying each hospital must tune its alert threshold.
Both points carry the same lesson. Performance depends on local patients and local settings, and only local testing shows it.
Accuracy claims are also being enforced. In 2024 the Texas Attorney General settled with a company whose generative AI summarized patient records in Texas hospitals, after alleging that its advertised hallucination rate was likely inaccurate. The company denied wrongdoing. The state's message to buyers was that hospitals must judge whether AI products are appropriate and train their staff.
Bias can hide in a sensible-looking target
A widely used algorithm for identifying patients who needed extra care predicted future health costs, not illness. Because less had been spent on Black patients with the same needs, it ranked them as healthier. Correcting it would have raised the share of Black patients flagged for extra care from 17.7% to 46.5%.
Nobody designed that bias. It came from the choice of what to predict. That is why overall accuracy is not enough: a tool must be checked for every patient group the organization serves.
Sources: [6]
Where regulation is heading
Clinical AI is governed by medical device law, data protection and health information law, and professional rules. All three are tightening around the full life of a tool, not just its approval.
| Jurisdiction | Development | What it means for a provider |
|---|---|---|
| International | WHO guidance on large multi-modal models, 2024 | Over 40 recommendations, including for providers. |
| European Union | AI in medical devices needing a notified body is high-risk from August 2028 | Device approval and AI Act compliance are assessed together. |
| United Kingdom | National Commission's 44 recommendations, all accepted by government in October 2026 | Post-market surveillance becomes central. |
| United States | Non-discrimination rule for patient care decision support tools, in effect since May 2025 | Providers must identify and mitigate biased tools. |
| Kenya | Pharmacy and Poisons Board guideline on medical device software, 2026 | Four-tier registration with lifecycle monitoring. |
| South Africa | SAHPRA communication on AI devices, 2025; revised professional guidelines on AI | Device evidence and clinicians' professional duties both apply. |
In plain terms: check device status before you buy, and plan for monitoring after you deploy.
Five questions every health board should ask
- Show me every AI tool in use, with a clinical owner and a risk rating. Include AI features inside the electronic health record.
- Which of these are regulated medical devices, and do we hold the approvals?
- For our most important clinical model, what did we find when we tested it on our own patients?
- Does it perform equally well across age, sex, ethnicity, language and our sites?
- How would we know if an AI tool contributed to a patient being harmed, and who would we tell?
Where to start
Start with the inventory, including device status. Name a clinical owner for each tool. Validate high-risk tools on local patients, by subgroup, before relying on them, and add an AI category to incident reporting so problems surface. Require clinician sign-off for AI-drafted notes and letters, and give staff an approved tool so patient data stays out of public chatbots.
Our guide sets out the full assessment for hospitals, health systems, clinic networks and digital health providers, with the evidence to request, red flags, 42 questions and a 100-day plan.
The Healthcare Executive's Guide to AI Readiness and Governance
Related reading
Sources
- US FDA. Artificial intelligence-enabled medical devices.
- American Medical Association. More than 80% of physicians use AI professionally, March 2026.
- Healthcare Innovation. AMA survey: AI usage among doctors doubles since 2023, March 2026.
- Healthcare IT News. Research suggests sepsis model lacking in predictive power, on JAMA Internal Medicine, 2021.
- Fierce Healthcare. Texas attorney general, Pieces Technologies settle allegations, September 2024.
- Obermeyer et al.. Dissecting racial bias in an algorithm used to manage the health of populations, Science, 2019.
- WHO. AI ethics and governance guidance for large multi-modal models, January 2024.
- UK Government. Government backs recommendations of NHS doctors-led AI Commission, October 2026.
- Holland & Hart. New ACA 1557 non-discrimination rules.
- The Star. New rules set for health apps and medical software, April 2026.
This article is general information, not medical, legal or regulatory advice.
Related service
Healthcare
Independent AI readiness and governance assessments for healthcare organizations, built around patient safety.
Explore the service