What is one specific practice or choice that has been most important for scaling AI data center or compute infrastructure, and why?

The answers split between optimizing hardware, software, cooling, and operational strategies, with some emphasizing technical specialization and others focusing on efficiency, planning, or infrastructure reuse.

15 answers

Answers

  • Chris Lattner

    EVP of Advanced AI Software and Platforms · Qualcomm

    Unified software stack

    Chris Lattner says the key practice is a unified software stack that spans different chips, because with no one-size-fits-all chip, software should let people avoid rewriting the same code for each chip.

    The TWIML AI Podcast with Sam Charrington · Watch on YouTube
  • Sam Altman

    Chief Executive Officer · OpenAI

    Clear Goal & Plan

    Sam Altman says a clear goal and a clear plan to get there, including how decisions will be made along the way, is very important, citing OpenAI's decision to bet on scaling deep learning as an example of that clarity.

    The answer starts at 10:29.

    Stanford Online: University Courses & Lectures · Listen on Apple Podcasts
  • Mark Papermaster

    CTO and EVP · AMD

    Full stack optimization

    Mark Papermaster says full-stack optimization is required, arguing that you cannot achieve the latency and reflex response needed in physical AI without it.

    Imagination in Action · Watch on YouTube
  • Naveen Rao

    [un]CEO · Unconventional Ai

    Smaller Data Types

    Naveen Rao says most efficiency gains in recent chip generations have come from smaller data types, going from 32-bit to 16-bit to 8-bit to 4-bit, because you move and compute with fewer bits.

    20VC with Harry Stebbings · Watch on YouTube
  • Rob Ferguson

    VP Technology and Strategy · Fireworks Ai

    Keep GPUs Busy

    Rob Ferguson says the key is to keep GPUs busy, because idle time on GPUs drives up costs for everybody.

    Mind the Product · Watch on YouTube
  • Eugene Cheah

    CEO / Co-Founder · Featherless Ai

    Dedicated Base Instances

    Eugene Cheah says the biggest cost savings in the market come from deploying a dedicated instance when you know your base workload, since full use of it yields savings, though he notes it depends on traffic patterns.

    AI Engineer · Watch on YouTube
  • Ilya Sutskever

    Cofounder and CEO

    Faster Processors

    Ilya Sutskever says you always want faster processors and more of them, and that faster processors are wanted "any day," though he frames the cost as justified only if what comes out of it outweighs the cost.

    Eye on AI · Watch on YouTube
  • Shayle Kann

    Managing Partner, Frontier · Energy Impact Partners

    Rear-Door Heat Exchangers

    Shayle Kann says his team used rear-door heat exchangers to bring cooling closer to the heat source, which saves fan energy by only moving heat near the compute, though he notes they stop working well beyond about 80kW.

    The answer starts at 10:26.

    Catalyst with Shayle Kann · Listen on Apple Podcasts
  • Amin Vahdat

    Chief Technologist, AI Infrastructure and SVP · Google

    Specialize Hardware

    Amin Vahdat says the more you specialize hardware to a particular workload, the faster and more power-efficient it is, though less flexible, so the workload must be durable enough to justify it.

    Sequoia Capital · Watch on YouTube
  • Gilad Shainer

    Senior Vice President Networking · Nvidia

    Scale-Out GPU Networking

    Gilad Shainer says the industry invested in scale-out networking so all GPUs stay fully synchronized, avoiding jitter where one GPU's data-delivery hiccup makes the others wait.

    SiliconANGLE theCUBE · Watch on YouTube
  • Andrew Feldman

    Founder and CEO · Cerebras

    Data Center Availability

    Andrew Feldman says the whole industry is currently constrained by data center availability, which he says is holding everyone back because data centers move like real estate while the industry tries to move at the speed of software.

    Imagination in Action · Watch on YouTube
  • Austin Lyons

    Senior Analyst · Creative Strategies, Inc.

    Fine-Tune Models

    Austin Lyons says fine-tuning models is the practice, citing an insurance company example where humans correct agentic claims processing and that labeled data is kept internally to fine-tune a model so it gets things right in the future.

    Chipstrat · Chipstrat · Read
  • Rodrigo Liang

    Co-Founder & CEO · Sambanova

    Reuse Infrastructure

    Rodrigo Liang says reusing existing infrastructure where energy and space are already allocated lets state-of-the-art AI run at a much lower cost, making AI available to everybody.

    SiliconANGLE theCUBE · Watch on YouTube
  • Samuel Rodriguez

    Senior Product Manager - Industrial Solutions · Chatsworth Products

    Adopt Liquid Cooling

    Samuel Rodriguez says the biggest driver in thermal management will be the adoption of liquid cooling, because more devices will require it or perform better with it, forcing it into the data center.

    The Data Center Frontier Show · The Data Center Frontier Show · Listen at 8:39
  • Matt Vincent

    Editor-in-Chief/Head of Content (Event) · Data Center Frontier

    Liquid cooling

    Matt Vincent says the solution right now for higher-density chips is liquid cooling, though he notes there is currently room for both liquid and air cooling.

    The Data Center Frontier Show · The Data Center Frontier Show · Listen at 7:12

More Cloud Computing questions

View all →