Memory, power, and thermal co-design, plus constrained telemetry for AI systems
The short version
Hardware engineering is shifting from isolated component optimization to co-design across memory, power, thermal, and telemetry constraints.
This week’s developments
- CXL memory is entering production-adjacent AI systems, so hardware designers must co-optimize memory expansion, bandwidth, power, and thermals instead of treating them separately.
- Telemetry is becoming a constrained design problem, pushing engineers to build local filtering and adaptive sampling into devices, not rely on always-on logging.
Hardware Design Moves to Memory, Power, and Thermal Co-Design
This week, Astera Labs’ Leo CXL Smart Memory Controllers entered private preview on Microsoft Azure M-series VMs to test CXL memory expansion for production RAG and KV-cache workloads, a clear sign that CXL is moving into production-adjacent AI infrastructure. Intel and Astera Labs’ 2024 inference benchmarks showed CXL memory expanders delivering roughly 85–95% of local DDR5 throughput, while Liqid reported production composable CXL deployments reaching 10–100 TB per server with up to 60% higher TPS and 40% lower P95 latency.
At the same time, AI rack densities have climbed from about 5–10 kW in legacy environments to 80–150 kW today, with next-generation concepts discussed around 600 kW. That is pushing operators from air cooling toward rear-door heat exchangers and direct-to-chip liquid cooling. The design center is shifting from standalone server optimization to integrated memory, power, and thermal co-design: HBM as the bandwidth tier, DDR under pressure as the capacity tier, and CXL as the pooled middle layer.
For hardware engineers, the practical change is clear: your value increasingly comes from workload-aware memory tiering, rack-level power and thermal modeling, and cross-team validation with software and datacenter operators, not from optimizing a board in isolation.
How should we reorganize teams for memory, power, and thermal co-design?
If you're an individual contributor
If you still define your value by board-level optimization alone, you are getting squeezed out; the engineers who become fluent in memory tiering, power budgets, and thermal constraints will be the ones trusted on AI infrastructure programs.
Build credibility in workload-aware design, CXL/DDR/HBM tradeoffs, and rack-level validation now, because the next promotion path favors people who can talk to software and datacenter teams without losing the hardware rigor.
If you manage a team
Your team’s output is no longer judged by isolated hardware wins — the people who can connect memory, power, and cooling decisions to real workload performance will look like the strongest engineers on the floor.
Rebalance coaching toward cross-functional design reviews, thermal/power modeling, and validation with infrastructure partners, or your team will keep producing technically solid work that misses the new system-level decision points.
If you lead the organization
Your org model is at risk if it still treats memory, power, and thermal design as separate specialties; AI infrastructure is forcing a shift toward integrated co-design teams that can ship at rack scale, not just component scale.
Invest in talent and operating models that combine hardware, firmware, and datacenter expertise around AI workload performance, because the next competitive gap will come from who can validate and deploy these systems fastest.
- Your Company Isn’t Ready for AI — The Next Big Idea Club Book of the Day Newsletter, May 21, 2026
Framework for redesigning decision-making, workflows, and learning to compete in the AI era.
- Industrializing Enterprise AI: Building The Push-Button AI Factory For The Agentic Era — Forbes, July 1, 2026
How to build governed, scalable AI infrastructure and operating models for production deployment across hybrid environments.
- Product School CEO | The AI Operating Model for Product Teams — Product School, May 27, 2026
Framework for aligning teams, tools, and incentives to turn AI strategy into measurable operational results.
Telemetry Shifts from Always-On Logging to Constrained Diagnostic Design
Cisco’s adaptive sampling and dynamic scheduling cut telemetry volume by up to about 70% while preserving critical monitoring, and edge-analytics research reduced telemetry energy and bandwidth by up to about 40% through local filtering, aggregation, and deduplication before transmission. Nordic Semiconductor and Soracom reinforced the same direction: process data locally, send only essential events, and avoid streaming raw high-frequency data when smaller payloads, batching, and compression can still support trends, alerts, and diagnostics.
For hardware engineering teams, telemetry is no longer a background logging function; it is becoming a constrained, lifecycle-managed subsystem that must be designed around explicit diagnostic outcomes. The engineering problem is shifting from capturing everything to maximizing diagnostic value per sample, per transmission, and per cloud event, with direct consequences for firmware behavior, duty cycles, power budgets, and embedded processing choices.
For practitioners, the work now includes deciding what not to collect, building edge-side reduction logic, and proving that reduced payloads still support root-cause analysis. Engineers who can balance observability, power, bandwidth, and serviceability will be more valuable across hardware, firmware, and reliability teams.
How should we redesign telemetry to preserve diagnosis while cutting costs?
If you're an individual contributor
If you still think telemetry work means logging more data, your value is eroding fast; the engineers who become indispensable are the ones who can prove a smaller, smarter signal still supports diagnosis, power, and product reliability.
Build edge-side filtering, batching, compression, and sampling logic into your toolkit now, and get comfortable defending what you chose not to collect because your credibility will increasingly come from diagnostic judgment, not data volume.
If you manage a team
Your team’s default behavior of collecting everything is becoming a liability; the people who can design constrained telemetry systems will outpace teams that treat observability as an unlimited backend problem.
Shift coaching toward diagnostic design, power-aware firmware decisions, and root-cause validation with reduced payloads, because your team needs to learn how to preserve serviceability while cutting bandwidth, energy, and cloud noise.
If you lead the organization
Telemetry is no longer a cheap afterthought in the stack — it is now a lifecycle-managed product decision that affects silicon, firmware, cloud cost, and field support, so orgs that keep funding always-on logging will look inefficient and outdated.
Rebalance investment toward edge analytics, constrained observability standards, and cross-functional telemetry architecture, and hire for engineers who can trade off diagnostics against power and bandwidth instead of assuming more data is always better.