Ep. 030 - Long Live the Short King: Why 4-hi HBM Wins (Memory)
Speakers
Myron Xie
— Substack author, SemiAnalysis
Summary
The discussion explores why AI accelerator manufacturers, notably NVIDIA, are shifting their upcoming designs to feature less High Bandwidth Memory (HBM) capacity per chip. Driven by severe memory supply constraints and the realization that inference performance is bottlenecked by bandwidth rather than capacity, the industry is moving toward lower-stack HBM configurations like 4-hi and 8-hi.
- NVIDIA's Rubin Ultra is expected to ship with 192GB of HBM, significantly less than the originally previewed 1TB.
- Moving to 4-hi or 8-hi HBM stacks improves manufacturing yields and maximizes the number of memory cubes produced per wafer.
- AI model parameter sizes are not scaling as aggressively as previously expected, reducing the need for massive memory capacity per chip.
- The memory supply shortage is expected to persist through the end of the decade due to physical constraints like clean room availability and limited ASML EUV tools.
- As HBM constraints ease via lower-stack designs, supply chain bottlenecks may shift toward TSMC's leading-edge logic wafers and advanced substrates.
Ideas
NVIDIA is reducing HBM capacity per accelerator.
NVIDIA is redesigning its upcoming Rubin Ultra accelerators to feature significantly less HBM capacity (192GB via 8-hi stacks instead of 1TB via 16-hi) in order to ration scarce HBM supply, maximize total accelerator shipments, and optimize for bandwidth over unnecessary capacity.
AI bottlenecks will shift to logic wafers.
As AI accelerator designs shift to lower-stack HBM to improve yields and cube availability, the primary supply chain bottleneck will shift away from memory and toward TSMC's leading-edge logic wafers and advanced packaging substrates.
Memory supply shortages will persist until 2030.
Memory suppliers will continue to benefit from a severe structural supply shortage in both HBM and commodity DRAM that will not ease this decade, driven by physical constraints like clean room availability and long lead times for ASML EUV tools.
ASML tool limits constrain memory capacity expansion.
ASML's limited annual production capacity for EUV lithography tools, constrained by its own highly specialized supply chain for components like smooth mirrors, serves as a hard physical bottleneck preventing rapid expansion of global memory fabrication capacity.
