What is one specific practice or architectural choice your organization uses to optimize allocation of compute resources for AI workloads, and why?
The answers highlight diverse strategies for optimizing AI compute allocation, ranging from infrastructure design and scheduling to pricing mechanisms and resource management, with little overlap among approaches.
13 answers
Answers


Matt Garman
Chief Executive Officer · Amazon Web Services (Aws)
Reserve Capacity Blocks
Matt Garman says AWS deliberately avoids selling all future capacity to large labs, reserving blocks for enterprises needing small GPU counts, startups that don't exist yet, and its Bedrock service.
TMT Breakout · TMT Breakout · Read

Sam Altman
Chief Executive Officer · OpenAI
No token leaderboards
Sam Altman says OpenAI avoids rewarding high compute usage internally, noting there are no token leaderboards and topping one would not help a researcher's career, instead giving researchers wide latitude unless usage looks abusive.
TMT Breakout · TMT Breakout · Read

Vinoo Ganesh
CEO & Co-Founder · Keru.ai
Arithmetic at compute layer
Vinoo Ganesh says his organization pushes arithmetic to the compute layer rather than the model layer, listing it among the guarantees his team makes.
AI Engineer · Watch on YouTube
Rob Ferguson
VP Technology and Strategy · Fireworks Ai
Serverless infrastructure
Rob Ferguson says his organization provides a serverless infrastructure, as OpenAI and Anthropic do, so that GPUs stay busy, since idle GPU time drives up costs that get passed on to others.
Mind the Product · Watch on YouTube

Eugene Cheah
CEO / Co-Founder · Featherless Ai
Nighttime background agents
Eugene Cheah says a team he cites began scheduling background agents to run at night while the team was asleep, which lets them use their hardware respectively.
AI Engineer · Watch on YouTube
Uttara Sivaram
Head of Global Policy · Writer
Batchable workloads
The passage does not show Uttara Sivaram giving this answer; the quoted remarks about batchable workloads are spoken by Varun, not Sivaram.
Boston Consulting Group · Watch on YouTube

Brett Harrison
Founder and CEO · Architect
Mini-auction per prompt
Brett Harrison says every prompt that comes in triggers a mini auction where the lowest price wins, according to intelligence and routing rules set by the consumer, as a way to get the cheapest possible inference at all times.
The answer starts at 33:50.
The Information's TITV · Listen on Apple Podcasts

Austin Lyons
Senior Analyst · Creative Strategies, Inc.
Prefill-decode disaggregation
Austin Lyons says his organization starts with prefill-decode disaggregation because prefill and decode have different memory capacity and bandwidth needs, so they could run each on separate hardware SKUs.
Chipstrat · Chipstrat · Read

Chase Lochmiller
Co-Founder and Chief Executive Officer · Crusoe
Cohesive RDMA fabric
Chase Lochmiller says the Abilene campus is designed so its GPUs run on one cohesive RDMA fabric, built from the ground up to operate a single workload across the whole campus.
The answer starts at 21:15.
The Twenty Minute VC (20VC): Venture Capital | Startup Funding | The Pitch · Listen on Apple Podcasts

Chen Goldberg
Executive Vice President of Product & Engineering · Coreweave
Rack manager dashboard
Chen Goldberg says her organization uses Reki, a rack manager, which ties signals from all systems into a dashboard so they can see how infrastructure is performing and how it affects running jobs and take actions.
The answer starts at 10:14.
The Neuron: AI Explained · Listen on Apple Podcasts

Sitanshu Gupta
Director - Inference · Coreweave
Schedule-based scaling API
Sitanshu Gupta says they provide an API capability letting customers schedule when to scale capacity up and down, so dedicated inference capacity used for real-time workloads during the day can be opened up for batch processing at night.
AI Engineer · Watch on YouTube

Christian Bhatia Ondaatje
Co-Founder & CEO · Aranya Inc.
Federated multi-cluster architecture
Christian Bhatia Ondaatje says his organization built a federated multi-cluster architecture from the start, with a management cluster and sub-clusters, because scaling is their business, and he says the early decision has paid off.
The answer starts at 23:29.
Code Story: Insights from Startup Tech Leaders · Listen on Apple Podcasts

Harry Stebbings
Founder · 20vc
Use spot instances
Harry Stebbings says he rents the remainder of his compute needs as spot instances from hyperscalers, after buying part at the best deal and part through long-term contracts, as part of how he allocates capacity.
20VC with Harry Stebbings · Watch on YouTube
More Generative AI questions
View all →- What design choice improves AI agent privacy?5 answers
- What architectural choices enable reliable agentic AI systems?8 answers
- What responsibilities should AI creators have for their models?12 answers
- How does your team create feedback loops for AI models?12 answers
- How do you balance AI customization and scalability?8 answers



















