How should organizations evaluate AI systems for their intended real-world use?

Most answers emphasize hands-on, context-specific evaluation of AI systems, though some focus on structured risk assessment or systematic tracking of usage and feedback.

5 answers

Answers

  • Jim Rowan

    Principal (Equity Partner) · Deloitte

    Run their own evals

    Jim Rowan says individuals especially need to have their own evals to test their own work against AI tools, describing his own series of prompts he runs as a check on the answers he gets back.

    The answer starts at 37:53.

    DataFramed · Listen on Apple Podcasts
  • Jerry Tworek

    Chief Executive Officer · Core Automation

    Test on real tasks

    Jerry Tworek says the way to know if a model works well is to try it yourself, ideally on the work you are actually doing, and be happy if it does what you wanted.

    The answer starts at 37:57.

    Venture with Grace · Listen on Apple Podcasts
  • Vishal Srivastava, PhD

    Corporate Vice President, Lead Data Scientist · New York Life Insurance Company

    Based on risk

    Vishal Srivastava says evaluations should be risk-based rather than frequency-based, breaking the system into components like technical failure, deployment controls and business context, then applying system-level stress testing before making a deployment decision.

    AI Engineer · Watch on YouTube
  • Ameya Bhatawdekar

    VP, Field CTO · Braintrust

    With a robust eval discipline

    Ameya Bhatawdekar says organizations should build a robust eval discipline with the right tools, automation and systems, which he calls paramount for managing generational model changes and keeping systems working well.

    AI Engineer · Watch on YouTube
  • Jeff Westcott

    Director, Practice Technology, Innovation & AI · Akin Gump Strauss Hauer & Feld LLP

    Track usage, time saved, user feedback

    Jeff Westcott says his organization measures AI adoption through usage numbers like how often and how long people use it, alongside systematically captured anecdotal feedback from lawyers, rather than extrapolating time saved into dollar figures.

    ILTA Voices · ILTA Voices · Listen at 31:33