Disaggregated AI inference delivers 2.5x speed boosts

Large language model inference distinctly splits into two phases—prefill and decode—that impose fundamentally different demands on hardware. Prefill is a compute-bound workload characterized by extremely high arithmetic intensity, reaching up to 4096 FLOPs per byte at 4096 tokens in FP16 precision, which fully saturates GPU tensor cores by processing all input tokens in parallel. This phase benefits from large batch sizes and matrix-matrix multiplications, enabling thousands of GPU cores to be utilized efficiently for prompt processing.

In stark contrast, the decode phase is memory-bound with low arithmetic intensity—typically between 1 to 4 FLOPs per byte—because it generates tokens sequentially, relying heavily on streaming large key-value caches and model weights through high bandwidth memory (HBM). Each token depends on the previous one, preventing parallelization and causing GPUs to spend much of their time waiting on memory transfers rather than computation, which explains why output tokens are often priced three to four times higher than input tokens in API models.

This fundamental dichotomy between compute-intensive prefill and memory-bound decode phases creates a resource management challenge that has shaped the evolution of inference engineering. Running both workloads on the same GPU leads to interference and latency jitter, as a long prefill stalls decode streams sharing the device. The industry’s structural response has been disaggregated inference architectures that separate compute pools optimized for each phase’s bottleneck—compute-dense GPUs for prefill and bandwidth-rich hardware for decode—enabling independent scaling, batching, and hardware selection.

To address decode’s sequential bottleneck, speculative decoding has emerged as a key optimization, employing a smaller model to predict multiple upcoming tokens ahead of the main model’s verification pass. This approach effectively increases throughput by amortizing the cost of token generation, allowing several tokens to be produced for the price of one, and exemplifies how understanding the distinct hardware regimes of prefill and decode informs sophisticated inference optimizations encapsulated in production-ready frameworks like vLLM and kvcached.

By early 2026, the architectural shift to disaggregated inference systems fundamentally transformed large language model (LLM) deployment by separating the prefill and decode phases onto distinct GPU pools. This division addresses the starkly different computational demands of each phase—prefill being compute-heavy and FLOPs-bound with high arithmetic intensity, while decode is bandwidth-heavy with low arithmetic intensity—allowing each to be optimized independently. Such isolation eliminates resource contention and interference, enabling aggressive parallelism for prefill's prompt processing and efficient batching for decode, which together enhance scalability and reduce latency.

Despite these benefits, the disaggregated approach introduces significant challenges around KV cache transfers, as serving a single 512-token request on a 66-billion parameter model can generate over a gigabyte of KV cache data. This demands ultra-high network bandwidth—often exceeding 90 Gbps—to prevent communication overhead from becoming a bottleneck. To mitigate this, topology-aware orchestration strategies such as colocating prefill and decode instances within the same physical node leverage ultra-fast intra-node interconnects like NVIDIA NVLink, which offers up to 600 GB/s bandwidth, thereby maintaining latency guarantees critical for interactive applications.

The disaggregation trend has also spurred hardware specialization and ecosystem shifts, exemplified by Cerebras positioning its CS-4 chip as a decode accelerator optimized for memory bandwidth rather than compute, reflecting the distinct needs of each inference phase. However, this specialization introduces resource allocation challenges, as fixed prefill-to-decode hardware ratios set at procurement may not adapt well over a system’s multi-year lifespan, complicating workload balancing. Furthermore, while GPU-based software attempts can approach high-bandwidth limits, SRAM-only architectures like Cerebras deliver superior bandwidth, highlighting a tradeoff in heterogeneous disaggregated setups.

Google’s ongoing TPU externalization efforts demonstrate how disaggregation enables innovative optimizations such as speculative decoding, which leverages a small drafter model to predict multiple tokens ahead and amortizes expensive weight reads across them without quality loss. This approach effectively addresses the decode phase’s bandwidth-bound nature, where streaming the entire model’s weights from HBM dominates cost and leaves matrix units idle. Although these advances mark a promising start, substantial software stack optimizations remain necessary to fully unlock the performance potential of disaggregated inference architectures.

By early 2026, the AI hardware landscape for disaggregated inference has been shaped by specialized ASICs and heterogeneous chip combinations tailored to distinct workload phases. For instance, the Talis ASIC demonstrated blazing speeds by burning the Llama 3.18B model onto silicon, achieving 16,000 tokens per second, while Nvidia’s strategy leverages specialized compute separation for prefill and decode tasks. AWS’s integration of Trainium chips with Cerebras further exemplifies this trend, optimizing large-scale inference through hardware specialization that aligns with the fundamentally different bottlenecks of prefill and decode workloads—compute-bound tensor operations versus bandwidth-bound streaming of KV cache and weights.

System-level orchestration frameworks have become indispensable to harness the full potential of disaggregated hardware architectures by mitigating resource contention and latency jitter. Scheduling prefill and decode workloads on the same GPU leads to interference and stalls, but disaggregating compute pools and transferring KV cache between them enables independent optimization of batching and hardware selection. This orchestration complexity has elevated frameworks like Ray and companies such as Anyscale to critical roles in modern serving stacks, ensuring smooth workload routing and efficient KV cache management across heterogeneous accelerators.

Heterogeneous disaggregated setups, such as those using Cerebras CS-4 as a decode chip alongside AMD and AWS Trainium for prefill, highlight a fundamental tradeoff: the fixed ratio of prefill to decode resources at hardware purchase limits adaptability as workload profiles evolve. While SRAM-only architectures like Cerebras excel in bandwidth, outperforming GPU-based approaches, GPUs and TPUs offer dynamic resource allocation that better accommodates shifting inference demands. This tension underscores the challenge of balancing peak performance with operational flexibility in disaggregated AI inference systems.

Innovations in interconnects and software protocols, such as AMD’s MORI-IO connector, have demonstrated that disaggregated serving can dramatically improve inference throughput by physically separating prefill and decode onto dedicated GPUs within a single node. This approach eliminates inter-token latency spikes and achieves up to 2.5x higher goodput, leveraging RDMA-based KV cache transfers with concurrent write modes to minimize proxy serialization overhead and reduce time-to-first-token. Such advances provide practical guidance: disaggregation is advantageous when high concurrency and long prompts cause prefill interference or when inter-token latency exceeds service-level objectives, while traditional serving may suffice for low-latency or low-load scenarios.

The fundamental challenge in optimizing LLM inference lies in the starkly different hardware demands of the prefill and decode phases. Prefill is compute-bound, leveraging large matrix multiplications with high arithmetic intensity to saturate GPU cores, whereas decode is memory-bound, generating tokens sequentially with low arithmetic intensity that stresses memory bandwidth. This dichotomy explains why input tokens are cheaper to process than output tokens in API pricing and drives the need for sophisticated resource management strategies.

Continuous batching has emerged as a pivotal software optimization that dynamically manages batch membership at every decoding iteration, replacing finished requests with waiting ones to maintain GPU saturation and reduce latency. Unlike static or dynamic batching, which suffer from inefficiencies due to fixed batch sizes or waiting for the slowest request, continuous batching—implemented by leading frameworks like vLLM, SGLang, TensorRT-LLM, and TGI—ensures high throughput even with variable-length outputs, making it a de facto standard in production LLM serving.

Speculative decoding represents a cutting-edge technique that accelerates the decode phase by employing a smaller model to predict multiple tokens ahead, allowing the larger model to verify these guesses in a single pass and effectively produce several tokens for the cost of one. While this approach yields non-trivial speedups, as seen in advanced methods like DFlash, dSpark, and SpecSpecta, it demands complex training akin to GANs, highlighting the delicate balance between performance gains and engineering complexity.

NVIDIA's open-source Dynamo toolkit exemplifies the evolution of inference software by focusing on efficient data routing and cluster-wide coordination rather than out-of-the-box optimization. Its Shadow Engine Recovery feature slashes LLM downtime from nearly five minutes to just 7.3 seconds by maintaining fully initialized standby engines and leveraging GPU Memory Service to keep model weights in high-bandwidth memory. Integrated with major AI backends and Kubernetes, Dynamo not only enhances fault tolerance and throughput but also paves the way for future innovations like key-value cache sharing, underscoring its role as a critical infrastructure component for scalable, resilient LLM deployments.

Advanced optimization techniques such as chunked prefill and the Encode-Prefill-Decode (EPD) disaggregation further refine inference efficiency by breaking large prompts into manageable token chunks and separating vision encoding from LLM prefill and decode stages. NVIDIA’s Dynamo framework leverages EPD to achieve up to 5x faster time to first token and 7x faster end-to-end response times, especially for multimodal workloads, by routing requests to specialized encoder and PD workers. This modular design, supported by flexible hardware configurations and the NVIDIA Inference Transfer Library, enables dynamic scheduling and scaling, balancing compute and memory demands while minimizing latency spikes.

Amazon SageMaker HyperPod's Disaggregated Prefill and Decode (DPD) architecture exemplifies how separating prefill and decode phases onto distinct GPU pools connected via Elastic Fabric Adapter (EFA) with RDMA can dramatically reduce token generation stalls and latency spikes. By enabling independent tuning of time to first token and inter-token latency, DPD handles mixed long and short prompt workloads seamlessly, optimizing throughput and tail latency without manual routing. This is achieved through a sophisticated multi-layer RDMA stack leveraging vLLM's LMCache, which overlaps compute and KV cache transfers to maximize GPU utilization while minimizing overhead.

vLLM's deployment of disaggregated serving on AMD's 8-GPU MI300X node using the MORI-IO connector further validates the performance gains of separating prefill and decode workloads, achieving a 2.5x increase in goodput by eliminating GPU interference that plagues collocated serving. The RDMA-based connector's write mode allows concurrent data transfers that reduce time-to-first-token overhead and stabilize inter-token latency under high concurrency and long prompts. However, this improvement is workload- and hardware-specific, with disaggregation recommended primarily when tail latency exceeds service-level objectives or prefill interference becomes untenable.

NVIDIA Dynamo's Shadow Engine Recovery showcases a complementary approach to enhancing inference resilience and throughput by maintaining fully initialized standby engines on the same GPUs, enabling failover in just 7.3 seconds—nearly 39 times faster than traditional cold restarts. This technique leverages persistent memory via NVIDIA's GPU Memory Service to keep model weights resident in high-bandwidth memory, allowing instant remapping without reloading or recompilation. Integrated with major AI backends and Kubernetes, Shadow Engine Recovery positions NVIDIA Dynamo as a critical fault-tolerant solution for large-scale AI deployments.

Kubernetes-native frameworks like LLMD and Red Hat's optimized GLM 5.2 clusters demonstrate how disaggregated prefill and decode pods, combined with intelligent KV cache-aware routing, can drastically reduce latency and boost throughput. LLMD's independent scaling of prefill and decode pods addresses phase interference inherent in aggregated serving, while Red Hat's deployment on H200 GPUs achieved 4x faster time-to-first-token and 60% more requests processed by modularly scaling prefill and decode workers and exploiting BF16 KV cache optimizations. Moreover, routing requests to pods with cached key-value states yields a 10x cost advantage over uncached token generation, underscoring routing as a primary lever before scaling GPU resources. These benefits, however, are most pronounced at medium concurrency and rely on high-speed fabrics like RDMA or RoCE for efficient cache transfers.

NVIDIA’s Encode-Prefill-Decode (EPD) disaggregation framework, exemplified by the Dynamo system, marks a significant leap in multimodal AI inference by decoupling vision encoding from language model prefill and decode stages. This separation not only enables independent scaling and batching—resulting in up to 5x faster time to first token and 7x faster end-to-end response times for image-heavy prompts—but also leverages hardware heterogeneity to optimize resource allocation. By assigning lighter vision encoding tasks to cost-effective GPUs like the RTX 6000D and reserving high-end GB200 GPUs for compute-intensive LLM workloads, the disaggregated architecture outperforms traditional homogeneous setups, enhancing throughput and latency especially in mixed workload production environments.

The Dynamo framework’s intelligent orchestration further refines multimodal inference by routing requests based on their modality, directing multimodal inputs to dedicated encoder workers while allowing text-only prompts to bypass this stage and proceed directly to prefill and decode workers. This nuanced task separation not only reduces bottlenecks inherent in vision encoding—which can dominate inference time—but also improves batching efficiency and memory utilization. As a result, production systems benefit from scalable, flexible handling of diverse workloads, underscoring the evolving trend of hardware-software co-design that prioritizes modularity and specialization in AI inference infrastructure.