What is one specific practice your team uses to create a tight feedback loop for improving AI models, and why?

Most teams use structured systems or direct user feedback to rapidly inform model improvements, though their specific methods—ranging from rubrics to live data—vary widely.

12 answers

Answers

  • Tulsee Doshi

    Sr. Director & Product Lead, GenAI & Gemini Models · Google

    Rubrics from User Feedback

    Tulsee Doshi says her team looks at customer and user feedback to define what good looks like for Gemini, then translates that into rubrics to evaluate the model and turn into data for reward modeling.

    Lenny's Podcast · Watch on YouTube
  • Laura Modiano

    Founder Experience · Openai

    Structured Feedback System

    Laura Modiano says her team captures feedback in a structured system of record, organizing anything written or from calls so it can be routed to the right teams and keep everyone moving in the same direction.

    The European VC · Watch on YouTube
  • Jayesh Govindarajan

    Executive Vice President, Salesforce AI · Salesforce

    Post to Slack Channel

    Jayesh Govindarajan says every forward deployed engineer in his org posts to one Slack channel, tagging each post as a paper cut or feature request, so PMs can match them against the roadmap and learn directly from the field to improve the product.

    Product Growth · Product Growth · Read
  • Emily Man

    Partner · Primary Venture Partners

    Break Down Jobs to KPIs

    Emily Man describes loops as breaking the jobs to be done into discrete component parts aligned to business metrics and KPIs, so the system you build keeps getting better over time.

    Run the Numbers with CJ Gustafson · Watch on YouTube
  • Sam Stephenson

    Founder · Granola

    Ship Early Internally

    Sam Stephenson says his team ships things internally as quickly as they can to small groups of friends and baby users very early, though they hold a high bar for whether it actually works for people before putting it out.

    Peter Yang · Watch on YouTube
  • Karri Saarinen

    CEO, Cofounder · Linear

    Winner Loop for Bug Investigation

    Karri Saarinen says his team uses the "winner loop" to investigate bugs, connecting to data, Sentry and other tools to find the bug's source and write a fix, which saves engineers time so they only verify results.

    Lenny's Podcast · Watch on YouTube
  • Andrew Somervell

    Founder (Product + Technology) · Hamster

    HTML Briefs for Feedback

    Andrew Somervell says his team builds an HTML brief to present information densely so a colleague can quickly react and give feedback, though he notes the workflow of pushing it through GitHub or Slack adds friction.

    Supra Insider · Watch on YouTube
  • Jonathan Evens

    Product Lead · Google Deepmind

    Collect User Interaction Signals

    Jonathan Evens says his team collects user interaction signals such as ratings, clicks, thumbs up and full listens, calling these signals extremely valuable for the reinforcement learning from human feedback loop.

    Mind the Product · Watch on YouTube
  • Hassan El Mghari

    Director of Developer Experience · Together Ai

    Automated Agent Evals

    Hassan El Mghari says his team writes tasks representing real usage with each PM and runs agent evals that spin up a sandbox, launch Claude Code, and grade the result; the transcript doesn't show him giving a reason for this practice beyond describing the loop.

    Product Growth · Product Growth · Read
  • Blaine Billingsley

    Member of Design Staff · Openai

    Run Many Evaluations

    Blaine Billingsley says a huge part of his job, and everyone's, is running evaluations—running the thing many times and judging whether the results are good, heuristically or by rubric—which he calls a new and strange part of the work.

    Mind the Product · Watch on YouTube
  • Sonya Huang

    Partner · Sequoia Capital

    Live Customer Data Feedback

    Sonya Huang describes setting up a system so live customer data creates a feedback loop where the model improves with every customer interaction, as the final step in the journey she outlines.

    Sequoia Capital · Watch on YouTube
  • Shaun Modi

    CEO · Capitol

    One‑Week Release Cycles

    Shaun Modi says his team is on one-week release cycles.

    FinStrat Management, Inc. · Watch on YouTube

More Generative AI questions

View all →