How should organizations evaluate AI systems for their intended real-world use?
Most answers emphasize hands-on, context-specific evaluation of AI systems, though some focus on structured risk assessment or systematic tracking of usage and feedback.
5 answers
Answers

Jim Rowan
Principal (Equity Partner) · Deloitte
Run their own evals
Jim Rowan says individuals especially need to have their own evals to test their own work against AI tools, describing his own series of prompts he runs as a check on the answers he gets back.
The answer starts at 37:53.
DataFramed · Listen on Apple Podcasts

Jerry Tworek
Chief Executive Officer · Core Automation
Test on real tasks
Jerry Tworek says the way to know if a model works well is to try it yourself, ideally on the work you are actually doing, and be happy if it does what you wanted.
The answer starts at 37:57.
Venture with Grace · Listen on Apple Podcasts
Vishal Srivastava, PhD
Corporate Vice President, Lead Data Scientist · New York Life Insurance Company
Based on risk
Vishal Srivastava says evaluations should be risk-based rather than frequency-based, breaking the system into components like technical failure, deployment controls and business context, then applying system-level stress testing before making a deployment decision.
AI Engineer · Watch on YouTube

Ameya Bhatawdekar
VP, Field CTO · Braintrust
With a robust eval discipline
Ameya Bhatawdekar says organizations should build a robust eval discipline with the right tools, automation and systems, which he calls paramount for managing generational model changes and keeping systems working well.
AI Engineer · Watch on YouTube

Jeff Westcott
Director, Practice Technology, Innovation & AI · Akin Gump Strauss Hauer & Feld LLP
Track usage, time saved, user feedback
Jeff Westcott says his organization measures AI adoption through usage numbers like how often and how long people use it, alongside systematically captured anecdotal feedback from lawyers, rather than extrapolating time saved into dollar figures.
ILTA Voices · ILTA Voices · Listen at 31:33