LLMs hit ludicrous speed: diffusion decoding and dMoE supercharge AI inference, shattering old benchmarks

By early 2026, speculative decoding had emerged as a groundbreaking algorithmic approach to accelerate large language model inference without sacrificing output quality, effectively doubling efficiency. This method leverages a faster draft model to generate candidate tokens, which are then verified in parallel by a larger, more accurate model. A pivotal milestone was achieved when Google integrated speculative decoding natively on its Cloud TPUs powering Google Search, enabling immediate latency and efficiency improvements that quickly established the technique as an industry standard with numerous citations and derivative innovations.

Building on this foundation, Speculative Speculative Decoding (SSD) was introduced as an evolution of traditional speculative decoding, further enhancing inference speed by employing a small proxy model—such as a tiny llama—to draft tokens that a larger model like big llama then verifies. This parallel run-ahead-and-verify strategy not only demonstrated clear speed advantages over existing open-source inference engines but also underscored that these gains stemmed from algorithmic innovation rather than system-level tweaks. As one presenter emphasized, 'it's not the systems, it's the algorithm,' highlighting SSD’s role in pushing the boundaries of efficient AI inference.

The motivation behind these early innovations in speculative decoding transcended mere cost savings or convenience; instead, pioneers framed inference speed as a direct proxy for AI capability. As articulated in a 2026 talk, 'the speed at which you can do inference, the tokens per second, is exactly the peak intelligence that you can deliver.' This paradigm shift positioned faster inference not just as a performance metric but as a fundamental lever to unlock higher peak intelligence and more complex reasoning in large language models.

By early 2026, block diffusion models like DFlash revolutionized speculative decoding by generating entire blocks of tokens in a single forward pass, dramatically outpacing traditional autoregressive drafters such as EAGLE-3. This approach not only bypassed the sequential bottleneck inherent in previous methods but also maintained high acceptance rates by conditioning on hidden features from multiple layers of the target model, ensuring quality token predictions. As reported, DFlash achieved up to an 8.5× speedup in token generation without sacrificing output quality, with practical gains of 2× to 4× for interactive workloads, marking a significant leap in inference efficiency.

Building on DFlash's foundation, DDTree introduced a sophisticated draft tree mechanism that constructs multiple token continuations using per-position distributions, then efficiently verifies these candidates in a single target model forward pass via an ancestor-only attention mask. This best-first heap algorithm not only improved acceptance lengths but also enhanced overall decoding efficiency, positioning DDTree as a leading speculative decoding approach that elegantly balances speed and accuracy.

While DFlash relies on block diffusion to generate token drafts in parallel, EAGLE3—used in models like Gemma 4—takes a different route by employing a lightweight draft head that predicts tokens through fused internal features of the target model rather than diffusion-based methods. Although EAGLE3's public speedup metrics remain undisclosed, its accepted draft lengths closely approach the speculative token counts, indicating meaningful acceleration potential through a novel integration of low-, mid-, and high-level target model signals.

DFlash's integration into popular inference frameworks such as vLLM, SGLang, and Transformers, along with the availability of draft models on HuggingFace for major LLMs like Qwen3, Qwen3.5, and Llama 3.1, underscores its growing adoption and practical impact. This ecosystem support facilitates broader deployment of block diffusion speculative decoding, accelerating the transition from research innovation to real-world application across diverse language models.

By mid-2026, the dMoE framework emerged as a pivotal innovation addressing the inherent incompatibility between diffusion-based large language models (dLLMs) and traditional Mixture-of-Experts (MoE) architectures. Unlike token-level expert selection typical in MoE, diffusion LLMs operate with block parallel decoding, creating a bottleneck in memory and inference efficiency. dMoE ingeniously bridges this gap by aggregating token-level expert distributions into a consolidated block-level distribution, which drastically reduces the number of uniquely activated experts from an average of 69.5 to just 14.6. This consolidation not only slashes memory usage by up to nearly 80% but also accelerates end-to-end inference latency by 14% to 66%, all while maintaining over 99% of the original model's performance, marking a significant stride in scalable, efficient LLM deployment.

By mid-2026, Xiaomi's MiMo-V2.5-Pro-UltraSpeed set a new benchmark in ultra-fast LLM inference by achieving over 1,000 tokens per second on a trillion-parameter model using standard 8-GPU commodity hardware, a feat previously unattainable without custom chips. This breakthrough leveraged innovative techniques like FP4 quantization on expert layers and DFlash speculative decoding, enabling near-zero quality loss while accelerating token generation to speeds roughly 15 to 20 times faster than leading models such as ChatGPT, Claude Opus, and even Google's Gemini Flash. Xiaomi's approach not only shattered speed records but also democratized access by running on widely available hardware, signaling a paradigm shift in deploying high-performance LLMs without specialized infrastructure.

Shortly after Xiaomi's announcement, Google DeepMind unveiled DiffusionGemma, an open-weight diffusion-based language model that similarly broke free from traditional left-to-right token generation by producing entire 256-token blocks in parallel. Running on a single NVIDIA H100 GPU, DiffusionGemma achieved over 1,000 tokens per second and demonstrated up to six times faster inference than standard autoregressive models, with the added ability to self-correct and iteratively refine outputs through bidirectional attention. Although its output quality trails the standard Gemma 4 model, DiffusionGemma’s architecture and Apache 2.0 licensing foster broad adoption across consumer GPUs and cloud platforms, supported by integrations with Hugging Face, vLLM, and Nvidia’s optimized ecosystem.

Both Xiaomi’s MiMo and Google’s DiffusionGemma exemplify a transformative industry trend toward compute-bound, parallelized LLM inference that prioritizes ultra-fast generation on commodity hardware, enabling new real-world applications. Xiaomi’s limited API trial, offering roughly ten times the generation speed at triple the cost, targets enterprise developers eager to harness this speed advantage, while Google highlights DiffusionGemma’s suitability for interactive coding, real-time editing, and complex problem-solving tasks like Sudoku, where its self-correcting diffusion process accelerates convergence. However, Google cautions that DiffusionGemma’s efficiency gains diminish in high-throughput cloud serving scenarios, underscoring the nuanced trade-offs in deploying diffusion-based LLMs across diverse workloads.

By early 2026, the open-source ecosystem for parallel and diffusion-based LLM inference witnessed remarkable growth with z-lab's DFlash models for Qwen3.5 and Thoughtworks' and RedHatAI's EAGLE3 draft heads for Gemma 4 enabling substantial speedups in speculative decoding. These releases, accompanied by detailed model cards and example vLLM commands, fostered active community engagement and experimentation, lowering barriers for developers to integrate and evaluate these novel inference paradigms.

The rapid adoption of diffusion-based inference was further accelerated by the public availability of DFlash on GitHub and its integration into popular frameworks like vLLM, SGLang, and Transformers, with draft models spanning Qwen3, Llama 3.1, and more hosted on Hugging Face. This open tooling ecosystem empowered researchers and practitioners to achieve up to 8.5× speedups without quality loss, underscoring the community’s appetite for transparent, high-performance solutions.

June 2026 marked a watershed moment as Google DeepMind’s open-source DiffusionGemma, a 26B parameter Mixture-of-Experts model licensed under Apache 2.0, shattered speed records by generating over 1,000 tokens per second on a single NVIDIA H100 GPU. The community buzzed around this paradigm shift from autoregressive token-by-token generation to simultaneous block denoising, with viral commentary highlighting the radical departure from decades of left-to-right sequence modeling and sparking widespread ecosystem integration including Hugging Face and NVIDIA support.

Alongside DiffusionGemma, the open-source landscape saw a surge of community-driven initiatives such as Cohere’s first open coding model, Xiaomi’s trillion-parameter ultra-fast MoE, and OpenEnv’s transition to a community-owned project with major contributors like Hugging Face and NVIDIA. However, licensing debates, exemplified by discussions around MiniMax M2.7, revealed ongoing tensions and active discourse within the community regarding model usage and distribution rights, reflecting the complex dynamics of open-source AI innovation.

By mid-2026, a comprehensive tutorial emerged that demystifies building and training a diffusion-based large language model from scratch, specifically a 13-million-parameter LLM. This resource leverages the TinyStories dataset—a synthetic corpus of short stories with vocabulary tailored for young children, generated by GPT-3.5 and GPT-4—enabling efficient training even on modest computational setups. The tutorial’s focus on practical accessibility is underscored by its use of publicly available datasets and free GPU resources, making advanced diffusion model training approachable for a broad audience.

The tutorial’s technical foundation is built on widely adopted open-source tools such as PyTorch, Hugging Face datasets, and transformers libraries, with a clever adaptation of the GPT-2 BPE tokenizer extended by a special <MASK> token to facilitate diffusion-based text generation. This token is appended as a new ID at the vocabulary’s end, illustrating an elegant yet straightforward modification that enables the diffusion process within the model’s architecture. Such detailed guidance bridges the gap between theoretical diffusion concepts and hands-on implementation, empowering practitioners to experiment with diffusion LLMs using familiar frameworks.