The new building block of AI infrastructure co-optimizes compute, memory, packaging, and interconnect into a single optimized engine - by Jerry Chang, Sr. Director, Head of Marketing, Americas and Europe
In previous blogs, we’ve explored how AI has fundamentally changed the trajectory of computing. While Moore's law continues to deliver remarkable advances in semiconductor technology, AI compute demand is growing even faster. The industry has reached a point where simply building faster chips is no longer enough. The road to scalable AI requires a new approach.
Today, the real challenge in AI is system-level co-optimization. Think of it as a team sport: you can't just focus on the processor while ignoring how memory, advanced packaging, power delivery, and cooling work together. To get the best performance per watt and keep the total cost of ownership in check, every single layer of the hardware stack must be designed in tandem. It’s no longer about maximizing individual components, but making sure the entire platform runs as a highly efficient unit.
Where Systems Become Silicon
In my previous blog, we explored why the rack has become the fundamental building block of AI infrastructure. But racks don't perform computation on their own. They’re built from multiple AI compute systems—sometimes called AI blades—each centered on an AI compute module.
This is where systems become silicon.
For decades, engineers have built computer systems by connecting increasingly capable processors, memory devices, networking chips, and I/O components on a circuit board. Today, that relationship is being turned inside out. Rather than treating these as separate devices and integrating them later, the AI infrastructure is built around highly integrated compute modules that combine CPUs, XPUs (GPUs, NPUs, and other accelerators), high-bandwidth memory (HBM), storage, high-speed I/O, advanced packaging, and sophisticated interconnect technologies into a single optimized engine.
The result is far more than a collection of components. Compute, memory, packaging, and interconnect are designed together from the outset, allowing the entire module to function as a unified system. The package is no longer simply something that protects the silicon—it has become an integral part of the architecture itself.
This represents a profound evolution in semiconductor design. For decades, progress came primarily from building faster processors using increasingly advanced process technologies. Those innovations remain critical, but today’s greatest gains come from co-optimizing every aspect of the compute module so that its constituent parts work cohesively.
As the new foundation of AI infrastructure, the compute module has become the primary unit of technology innovation; decisions made at this level ripple upward into system architecture, rack design, and overall economics, ultimately deciding whether AI can be deployed efficiently and at scale.
Scale-up Interconnect and Scale-Out Interconnect
As AI models continue to grow in size and complexity, moving data efficiently has become almost as important as processing it. Every bit transferred consumes energy. Every additional nanosecond of latency reduces overall system efficiency. In large AI deployments, these seemingly tiny effects can add up to enormous differences in operating costs and achievable performance. This is why AI compute modules employ two complementary forms of connectivity: scale-up and scale-out.
Scale-up interconnect connects the processors, accelerators, and memory within a compute module. Its job is to move extraordinary amounts of data over extremely short distances while consuming as little energy as possible. At this level, designers worry about every picojoule because trillions upon trillions of bits are transferred every second.
Scale-out interconnect performs a very different role. It links compute modules together into systems, systems into racks, and racks into entire data centers. Here, designers must balance bandwidth, latency, reach, reliability, and energy efficiency while supporting deployments containing thousands—or even hundreds of thousands—of AI accelerators.
Because neither the compute-module nor rack-level fabric can be optimized in isolation, achieving best-in-class AI performance requires co-optimizing both simultaneously—a technical synergy that, at hyperscale, directly translates into reductions in power, cost, and complexity as infrastructure is increasingly judged by how efficiently it converts energy and capital into useful work.
Packaging: Where Silicon Comes Together
Perhaps the biggest transformation in AI system design involves something that once received relatively little attention: packaging. For decades, packaging served primarily to protect the silicon and provide electrical connections to the outside world. Today's advanced AI systems demand much more of it.
Many of today’s compute modules are built around chiplet-based architectures, enabling compute, memory, and I/O functions to be implemented as specialized silicon components optimized individually before integration into a single package.
Packaging is where silicon comes together. Advanced 2.5D and 3.5D integration technologies allow these diverse components to sit side by side—or even above one another—while communicating with bandwidths that would have been unimaginable only a few years ago.
In effect, the package has become part of the computer's architecture. Performance is no longer determined solely by transistor density or process technology. It increasingly depends on how effectively diverse technologies can be integrated into a coherent system. As heterogeneous integration continues to expand, packaging becomes the medium through which compute, memory, and communications are transformed into a unified engine.
The Compute Module Becomes the Product
For decades, general-purpose computing has served the industry remarkably well. AI, however, rewards specialization. Different workloads place very different demands on compute, memory capacity, memory bandwidth, latency, precision, and data movement. The optimal architecture for training trillion-parameter models may differ significantly from the best architecture for high-volume inference or emerging agentic AI applications. Consequently, AI compute modules are becoming increasingly workload-aware.
Memory hierarchies can be tailored to specific applications. Specialized numerical formats improve efficiency. Compute resources can be balanced differently depending on whether throughput, latency, or energy efficiency is the primary objective. Rather than build a single architecture intended to satisfy every possible workload, designers can increasingly optimize compute modules for the tasks customers perform.
This represents another fundamental shift in the AI infrastructure evolution. Previously, the semiconductor industry measured progress by the capabilities of individual chips. Now, success increasingly depends on how effectively compute, memory, interconnect, packaging, and power are connected in a unified system. The AI compute module embodies that philosophy, transforming what was once a collection of individual components into a single, optimized engine.
As AI deployments continue to scale, competitive advantage will increasingly come from how effectively the entire infrastructure stack is co-optimized—from compute and memory to packaging, interconnect, and system architecture. The AI compute module is where many of those optimizations converge.
In the AI era, systems don't just contain silicon—they become silicon.
FAQs
What is an AI compute module?
An AI compute module is a tightly integrated subsystem that combines compute, memory, storage, I/O, and interconnect in a single optimized engine. These modules are the fundamental building blocks for modern AI systems and racks.
What is the difference between scale-up and scale-out interconnect?
Scale-up interconnect moves the data within a compute module, as it connects processors, accelerators, and memory with extremely high bandwidth and low latency. Scale-out interconnect connects modules, systems, and racks, enabling AI workloads to scale across large deployments.
Why is advanced packaging important for AI?
Advanced packaging technologies, such as 2.5D and 3.5D integration, allow compute dies, memory, and I/O components to be combined into highly efficient multi-die systems. This helps deliver the bandwidth, density, and power efficiency required by modern AI workloads.
