What is one specific practice or architectural choice your organization uses to optimize allocation of compute resources for AI workloads, and why?

The answers highlight diverse strategies for optimizing AI compute allocation, ranging from infrastructure design and scheduling to pricing mechanisms and resource management, with little overlap among approaches.

13 answers

Answers

  • Matt Garman

    Chief Executive Officer · Amazon Web Services (Aws)

    Reserve Capacity Blocks

    Matt Garman says AWS deliberately avoids selling all future capacity to large labs, reserving blocks for enterprises needing small GPU counts, startups that don't exist yet, and its Bedrock service.

    TMT Breakout · TMT Breakout · Read
  • Sam Altman

    Chief Executive Officer · OpenAI

    No token leaderboards

    Sam Altman says OpenAI avoids rewarding high compute usage internally, noting there are no token leaderboards and topping one would not help a researcher's career, instead giving researchers wide latitude unless usage looks abusive.

    TMT Breakout · TMT Breakout · Read
  • Vinoo Ganesh

    CEO & Co-Founder · Keru.ai

    Arithmetic at compute layer

    Vinoo Ganesh says his organization pushes arithmetic to the compute layer rather than the model layer, listing it among the guarantees his team makes.

    AI Engineer · Watch on YouTube
  • Rob Ferguson

    VP Technology and Strategy · Fireworks Ai

    Serverless infrastructure

    Rob Ferguson says his organization provides a serverless infrastructure, as OpenAI and Anthropic do, so that GPUs stay busy, since idle GPU time drives up costs that get passed on to others.

    Mind the Product · Watch on YouTube
  • Eugene Cheah

    CEO / Co-Founder · Featherless Ai

    Nighttime background agents

    Eugene Cheah says a team he cites began scheduling background agents to run at night while the team was asleep, which lets them use their hardware respectively.

    AI Engineer · Watch on YouTube
  • Uttara Sivaram

    Head of Global Policy · Writer

    Batchable workloads

    The passage does not show Uttara Sivaram giving this answer; the quoted remarks about batchable workloads are spoken by Varun, not Sivaram.

    Boston Consulting Group · Watch on YouTube
  • Brett Harrison

    Founder and CEO · Architect

    Mini-auction per prompt

    Brett Harrison says every prompt that comes in triggers a mini auction where the lowest price wins, according to intelligence and routing rules set by the consumer, as a way to get the cheapest possible inference at all times.

    The answer starts at 33:50.

    The Information's TITV · Listen on Apple Podcasts
  • Austin Lyons

    Senior Analyst · Creative Strategies, Inc.

    Prefill-decode disaggregation

    Austin Lyons says his organization starts with prefill-decode disaggregation because prefill and decode have different memory capacity and bandwidth needs, so they could run each on separate hardware SKUs.

    Chipstrat · Chipstrat · Read
  • Chase Lochmiller

    Co-Founder and Chief Executive Officer · Crusoe

    Cohesive RDMA fabric

    Chase Lochmiller says the Abilene campus is designed so its GPUs run on one cohesive RDMA fabric, built from the ground up to operate a single workload across the whole campus.

    The answer starts at 21:15.

    The Twenty Minute VC (20VC): Venture Capital | Startup Funding | The Pitch · Listen on Apple Podcasts
  • Chen Goldberg

    Executive Vice President of Product & Engineering · Coreweave

    Rack manager dashboard

    Chen Goldberg says her organization uses Reki, a rack manager, which ties signals from all systems into a dashboard so they can see how infrastructure is performing and how it affects running jobs and take actions.

    The answer starts at 10:14.

    The Neuron: AI Explained · Listen on Apple Podcasts
  • Sitanshu Gupta

    Director - Inference · Coreweave

    Schedule-based scaling API

    Sitanshu Gupta says they provide an API capability letting customers schedule when to scale capacity up and down, so dedicated inference capacity used for real-time workloads during the day can be opened up for batch processing at night.

    AI Engineer · Watch on YouTube
  • Christian Bhatia Ondaatje

    Co-Founder & CEO · Aranya Inc.

    Federated multi-cluster architecture

    Christian Bhatia Ondaatje says his organization built a federated multi-cluster architecture from the start, with a management cluster and sub-clusters, because scaling is their business, and he says the early decision has paid off.

    The answer starts at 23:29.

    Code Story: Insights from Startup Tech Leaders · Listen on Apple Podcasts
  • Harry Stebbings

    Founder · 20vc

    Use spot instances

    Harry Stebbings says he rents the remainder of his compute needs as spot instances from hyperscalers, after buying part at the best deal and part through long-term contracts, as part of how he allocates capacity.

    20VC with Harry Stebbings · Watch on YouTube

More Generative AI questions

View all →