Why Positron AI is Choosing LPDDR over HBM for Next-Gen LLM | Researcher Conversations at GTC

Summary

Thomas Sohmers discusses Positron AI's hardware strategy for AI inference, focusing on reducing token costs while maintaining speed. He explains their architecture's high memory bandwidth utilization and choice of LPDDR memory over HBM to avoid supply constraints. The Titan server aims to support massive models with high context lengths using commodity technologies like organic substrates.

• Positron AI is transitioning from FPGA to custom silicon with the Osmo chip.
• Architecture optimizes matrix-vector performance for AI inference efficiency.
• Uses LPDDR memory instead of HBM for higher capacity and supply chain advantages.
• Avoids advanced packaging, using organic substrates for cost and scalability.
• Claims 1:1 matrix-matrix to matrix-vector ratio, unlike NVIDIA's worsening ratio.
• Titan server targets 16-trillion parameter models with million-token contexts.
• Emphasizes commodity supply chains to compete with large players.
• Partnership with Oracle and rapid development highlighted.