How should AI product evaluation metrics be defined?

The answers agree that evaluation metrics should connect to broader goals and user experience, but differ on whether to prioritize business alignment, product principles, or specific combinations of usage and satisfaction data.

5 answers

Answers

  • Dave Killeen

    Field Chief Product Officer · Pendo.Io

    KPI Driver Tree Alignment

    Dave Killeen says execs need a strong KPI driver tree aligned around north-star metrics for revenue, cost and risk, with product metrics influencing them and a steady drumbeat on progress, because otherwise people push AI as the solution.

    Atlassian · Watch on YouTube

  • Jonathan Evens

    Product Lead · Google Deepmind

    Product Principles Foundation

    Jonathan Evens says evaluations should be built on product principles, arguing that before evaluations you must decide what experience you're trying to create, with evaluations as the first translation of this.

    Mind the Product · Watch on YouTube

  • Srinivasan Raghavan

    Chief Product Officer · Freshworks

    Usage and Satisfaction

    Srinivasan Raghavan says the answer is a combination of usage metrics and satisfaction metrics, gathered via in-product surveys of about 5–10% of users monthly, because together they give signals that make it more effective.

    DataCamp · Watch on YouTube

  • Mina Doroudi

    Principal Product Manager · Microsoft

    Clear Meaning and Learning

    Mina Doroudi says principal PMs should define what an eval score means and present it as a learning process, since evals are still a learning process for the whole industry.

    Exponent · Watch on YouTube

  • Todd Olson

    CEO / Founder · Pendo.Io

    Scores and User Experience

    Todd Olson says evaluation metrics should pair eval scores with measuring user experience, since evals alone don't make products better and an eval can score well while the user has a different experience.

    DataCamp · Watch on YouTube

More Product Management questions

View all →