What matters most when evaluating autonomous AI agents?

Noah Shinn

Research Scientist · Sierra

Understandability

Noah Shinn says the focus should be understandability, not capability, and proposes measuring how much a user grasps about what's happening and how well they can predict a request's result.

Emrecan Dogan

Head of Product · Glean

The journey

Emrecan Dogan says the journey matters as much as the destination; he argues that while the ideal destination is no one needing to build agents, reaching it today still requires human coaching and context setting.

The answer starts at 32:13.

Rama Akkiraju

VP, Enterprise AI · Nvidia

Observability

Rama Akkiraju says observability is key, pointing to the need to understand agent behavior before an agent can be certified.

Tara Seshan

Member of Product Staff · Openai

Data access

Tara Seshan says data access—along with cloud infrastructure and reliability—is a huge part of making agents effective, since an agent without access to systems and data won’t be effective.

Angela Jiang

Head of Product, Claude Platform · Anthropic

Error recovery

Angela Jiang says what's needed is infrastructure that lets an agent continue running and recover from its errors, along with ways to work securely and compliantly.

Moe Katib

Founder & CEO · One (formerly Pica)

Trust

Moe Katib says trust is what matters, citing refined access controls — read-only, draft-only, per-action permissions across platforms — as important for the trust he says people need before adopting the tech.