Kimbo Chen
· SemiAnalysis
· August 03, 2026 at 19:42
· ⏱ 28 min read
| Read on Substack ↗
Summary
Kimi K3's frontier performance comes from combining KDA compressed linear attention, attention-over-depth residuals, and Stable LatentMoE, not from a single breakthrough; the article explains these techniques and their serving implications. For markets, this validates Nvidia B300 as a workable single-node platform for frontier open-weight inference but shows KV cache/HBM capacity is the operational bottleneck, which should keep focus on memory capacity and disaggregated serving.
•Kimi K3 uses a 3:1 ratio of KDA linear attention to MLA full attention; KDA replaces RoPE as the position-aware operator.
•FlashKDA's KDA recurrence is O(T·D²) prefill and O(D²) decode per head, with kernel-level memory access ~O(TC + TD + D²), making decode constant in sequence length.
•Stable LatentMoE halves the expert input dimension to 3584 vs K2's 7168, allowing active experts to double from 8 to 16 without raising MoE communication volume.
•Quantile Balancing sets next-batch router biases from the (1-k/n) quantile of sorted margins, eliminating aux-loss and hyperparameter tuning.
•Attention Residuals over blocks deliver 1.25× compute efficiency vs standard residual connections and lower validation loss; cross-stage caching keeps pipeline parallelism overhead at ~4%.
•On Nvidia B300, Kimi K3 fits one node and 'serves well'; B200 needs pipeline parallelism and DSpark speculative decoding is incompatible with PP.
•Kimi K3 is offered on OpenRouter at a $3/M input, $15/M output floor; agentic trace replay shows median 142k input tokens and 444 output tokens per turn across 65 turns.
•B300 HBM only holds ~3.25M tokens of KV cache after weights; above concurrency 8, cache hit rates collapse to <10% vs a theoretical 95%, tanking throughput.
Article's InferenceX benchmark says 'For B300, the model fits on 1 node and serves well,' while B200 requires pipeline parallelism and 'DSpark also didn't work with PP' — validating Nvidia's latest GP
Article's InferenceX benchmark says 'For B300, the model fits on 1 node and serves well,' while B200 requires pipeline parallelism and 'DSpark also didn't work with PP' — validating Nvidia's latest GPU as the reference platform for serving frontier open-weight models, though HBM capacity limits remain.
Risk: B300 HBM holds only ~3.25M tokens of KV cache; at concurrency >8 cache hit rates fall below 10%, so high-concurrency serving is still constrained.
The article states 'Both Nvidia and AMD had Day 0 recipes on vLLM, boasting DRAM offload and DSpark speculative decoding' — a software-readiness validation that narrows AMD's ecosystem gap for frontie
The article states 'Both Nvidia and AMD had Day 0 recipes on vLLM, boasting DRAM offload and DSpark speculative decoding' — a software-readiness validation that narrows AMD's ecosystem gap for frontier open-weight model inference.
Risk: No throughput benchmark is provided for AMD, so Day 0 compatibility does not prove competitive serving performance.
This newsletter, published August 03, 2026,
features Kimbo Chen
discussing NVDA, AMD.
2 trade ideas extracted by AI with direction and confidence scoring.