What safeguards are necessary to ensure the safe and effective deployment of AI in healthcare?

Most answers emphasize the need for strong governance, human oversight, and ongoing validation, while some highlight transparency, informed consent, and vendor accountability as additional safeguards.

16 answers

Answers

  • Chris Carmody

    Chief Information Officer · Upmc

    Governance and training

    Chris Carmody says the right governance and processes must come first to identify and focus on the top problems, and that training users to understand the why behind the technology is just as important.

    The answer starts at 7:04.

    Becker’s Healthcare Podcast · Listen on Apple Podcasts
  • Munjal Shah

    Senior Product Manager, Wearables · Google

    Output testing

    Munjal Shah says the key safeguard is output testing—testing what comes out of the model rather than trusting what went in—and advises looking for a history of it, comparing it to how the FDA tests drugs.

    The answer starts at 11:54.

    Becker’s Healthcare Podcast · Listen on Apple Podcasts
  • Hal Wolf

    President and Chief Executive Officer · Himss

    Governance, training, processes

    Hal Wolf says deploying AI safely requires governance, training of the people, and processes in place, along with measuring impact on access and quality.

    The answer starts at 6:50.

    The Big Unlock · Listen on Apple Podcasts
  • Aaron Miri, DHA, FCHIME, CHCIO

    EVP, Chief Digital & Information Officer · Baptist Health

    Informed consent

    Aaron Miri says a tolerance level for AI error is acceptable as long as patients choose with informed consent and responsibility, eyes wide open, rather than being unsafe.

    The answer starts at 52:11.

    "Second Opinion" with Christina Farr, Ash Zenooz MD & Luba Greenwood JD · Listen on Apple Podcasts
  • Katherine Adams, MS

    Director, Clinic Operations, Primary Care · Commonspirit Health

    Ongoing Safety Oversight

    Katherine Adams says she needs tech partners' ongoing partnership after go-live to keep AI tools learning, avoid hallucinations and stay fine-tuned, and wants their commitment to patient safety post-implementation.

    The answer starts at 5:20.

    Becker’s Healthcare Podcast · Listen on Apple Podcasts
  • Colin LeClair

    Chief Executive Officer · Connections Health Solutions

    Vendor vetting and contracts

    Colin LeClair says his organization vets vendors for proven track records and viability, and builds safeguards into contracts so it can absorb a technology if the company folds.

    The answer starts at 13:46.

    Becker’s Healthcare Podcast · Listen on Apple Podcasts
  • Amit Malik

    Founder & CEO · Amaha

    Human clinical accountability

    Amit Malik says the human and clinical accountability angle is what is missing and should always be missing from an LLM or AI chatbot, and that AI should not be used for diagnosis, emergency risk, medication changes, or as a sole source of treatment.

    The answer starts at 19:14.

    Say Again?! With Amrita Tripathi · Listen on Apple Podcasts
  • David Niewolny

    Sr. Dir, Global Head of Business Development - Healthcare / Medical · NVIDIA

    Layered Governance

    David Niewolny says AI governance in healthcare works in layers: bounded tools that are safe, secure and validated, controlling data access, and a clear human decision point.

    The answer starts at 35:22.

    Faces of Digital Health · Listen on Apple Podcasts
  • David Niewolny

    Sr. Dir, Global Head of Business Development - Healthcare / Medical · NVIDIA

    Bounded Tools

    David Niewolny says AI governance in healthcare works in layers: bounded tools that are safe, secure and validated, controlling data access, and a clear human decision point.

    The answer starts at 35:22.

    Faces of Digital Health · Listen on Apple Podcasts
  • David Niewolny

    Sr. Dir, Global Head of Business Development - Healthcare / Medical · NVIDIA

    Data Control

    David Niewolny says AI governance in healthcare works in layers: bounded tools that are safe, secure and validated, controlling data access, and a clear human decision point.

    The answer starts at 35:22.

    Faces of Digital Health · Listen on Apple Podcasts
  • David Niewolny

    Sr. Dir, Global Head of Business Development - Healthcare / Medical · NVIDIA

    Human Oversight

    David Niewolny says AI governance in healthcare works in layers: bounded tools that are safe, secure and validated, controlling data access, and a clear human decision point.

    The answer starts at 35:22.

    Faces of Digital Health · Listen on Apple Podcasts
  • Rhett Alden

    CTO, Elsevier Health Markets · Elsevier

    Transparency and validation

    Rhett Alden says transparency is critical for clinical adoption—both in how a response is arrived at and where its information comes from—and points to his team's robust validation framework as another requirement.

    The answer starts at 3:24.

    The AI in Business Podcast · Listen on Apple Podcasts
  • Anirban Maitra

    Director, Perlmutter Cancer Center · Nyu Langone Health

    Qualified physician oversight

    Anirban Maitra says the most important safeguard is using AI safely, which means having qualified physicians analyze and check the results rather than letting AI diagnose without oversight, since the physician's judgment remains the captain of the ship.

    The answer starts at 10:15.

    Becker’s Healthcare Podcast · Listen on Apple Podcasts
  • Hannah Galvin, MD, FAAP, FAMIA, CHCIO

    Chief Health Information Officer · Cambridge Health Alliance

    Robust governance

    Hannah Galvin says her organization was an early adopter of mature AI governance, which they have evolved over time to keep up with industry trends and the need to handle many more tools, and to move useful technology into practice safely.

    The answer starts at 7:37.

    Becker’s Healthcare Podcast · Listen on Apple Podcasts
  • Zeke Emanuel

    Vice Provost for Global Initiatives · University Of Pennsylvania

    Biannual review

    Zeke Emanuel says AI in healthcare must be watched in hundreds of actual patient cases and reassessed on a regular basis, proposing biannual review every two years, with what it is authorized to do adjusted accordingly.

    "Second Opinion" with Christina Farr, Ash Zenooz MD & Luba Greenwood JD · "Second Opinion" with Christina Farr, Ash Zenooz MD & Luba Greenwood JD · Listen at 32:56
  • Kimberly Powell

    Vice President / General Manager, Healthcare at NVIDIA · Nvidia

    Healthcare professionals in the loop

    Kimberly Powell says it is so important to have healthcare professionals in the loop, meaning both that clinicians remain in the loop when using the tools and that companies build deep clinical understanding into their systems.

    Second Opinion with Christina Farr · Watch on YouTube

More HealthTech questions

View all →