AgentX - InferenceXv3: Does CUDA Moat Hold up in Agentic Inferencing?

Cam Quilici · SemiAnalysis · August 24, 2026 at 00:19 · ⏱ 103 min read  | Read on Substack ↗
Summary
Agentic, multi-turn inference has replaced fixed-sequence workloads as the defining production inference pattern—OpenAI enterprise agentic spend overtook ChatGPT spend in April 2026—and the open-source AgentX benchmark shows that real-world inference leadership depends heavily on software, caching, routing, and ecosystem maturity, not just raw chip performance. NVIDIA remains broadly ahead on realistic agentic workloads despite strong MI355X hardware, while AMD's software stack is repeatedly identified as the main competitive drag.
  • •AgentX v1.0 open-sourced a 393-session subset of 8,000+ collected Claude Code/Codex sessions representing roughly $3M in API spend; the full trace corpus contains 3.4 million requests and 610 billion tokens.
  • •OpenAI's Enterprise agentic spending overtook ChatGPT spending in April 2026, and long-context multi-turn agentic workloads now dominate production inferencing.
  • •On DeepSeek V4 Pro, B300 vLLM and B200 SGLang beat AMD MI355X on performance per dollar, while MI355X ATOM beats B200 vLLM on e2e latency—but ATOM is not widely used in production.
  • •AMD's vLLM CPU-KV-offload path was inefficient because hipMemcpyBatchAsync was missing until ROCm 7.14, forcing serialized memcpy, whereas NVIDIA B300 achieved a 91% HBM cache hit rate at 384 concurrent agentic traces.
  • •On the 2.8T-parameter Kimi K3, MI355X ATOM beats even GB300 NVL72 vLLM on performance per dollar across part of the 40-60s e2e latency curve, but Hopper and upstream vLLM were initially immature/unusable for the model on realistic workloads.
  • •On MiniMax M3 432B, B300 TRT-LLM TP2 is the best configuration, while AMD software is described as 'horrible' at high context length; DP-attention is non-optimal because KV cache locality becomes a routing constraint.
  • •On Qwen3.5 397B, NVIDIA SGLang has over 20x better performance at 90 tok/s/user than AMD SGLang, and on GLM 5.3 at 150 tok/s/user NVIDIA has up to 5x better cost efficiency—even free AMD chips would be more expensive per token.
  • •AgentX triggered 70+ upstream optimization PRs across vLLM, SGLang, TensorRT-LLM, ATOM, AITER, Dynamo, LMCache, and Mooncake, including boundary-aware incremental tokenization cutting Qwen3.5 turn processing from 185.1ms to 11.3ms.
Read time 103 min
Length 103,236 chars
Category finance
Ideas
Cam Quilici Substack author, SemiAnalysis
AgentX results repeatedly validate NVIDIA's leadership on realistic agentic inference: B300/B200 vLLM, GB300/GB200 Dynamo TRTLLM, and B300 TRT-LLM TP2 on MiniMax-M3 all lead performance-per-dollar cur
AgentX results repeatedly validate NVIDIA's leadership on realistic agentic inference: B300/B200 vLLM, GB300/GB200 Dynamo TRTLLM, and B300 TRT-LLM TP2 on MiniMax-M3 all lead performance-per-dollar curves, and the article explicitly calls DCP/PCP part of the 'CUDA moat.' Even where AMD's MI355X hardware is competitive, NVIDIA's software stack delivers 20x and 5x advantages on Qwen3.5 and GLM 5.3 respectively. Risk: AMD's ATOM and upstream open-source optimizations are narrowing the software gap, and agentic workload patterns could shift relative TCO faster than expected.
NVDA
Cam Quilici Substack author, SemiAnalysis
The article praises MI355X hardware but repeatedly criticizes AMD software maturity for long-context multi-turn workloads: missing hipMemcpyBatchAsync until ROCm 7.14, unsupported DCP/PCP in the vLLM
The article praises MI355X hardware but repeatedly criticizes AMD software maturity for long-context multi-turn workloads: missing hipMemcpyBatchAsync until ROCm 7.14, unsupported DCP/PCP in the vLLM support matrix, ATOM not production-ready, and 'AMD software performance is horrible' on MiniMax M3. This means strong hardware specs are not translating into wins on realistic agentic benchmarks. Risk: AMD is actively upstreaming ATOM optimizations into vLLM/SGLang and collaborating with SemiAnalysis; several AgentX follow-ups are expected to improve AMD results within weeks.
AMD
Cam Quilici Substack author, SemiAnalysis
The article emphasizes that agentic inference is a distributed systems problem, requiring KV transfer across nodes, PD-disaggregation, and cluster-scale routing. That directly increases the importance
The article emphasizes that agentic inference is a distributed systems problem, requiring KV transfer across nodes, PD-disaggregation, and cluster-scale routing. That directly increases the importance of high-bandwidth, low-latency datacenter networking and Ethernet fabrics, a segment where Arista is a primary beneficiary even though the article does not name it. Risk: The article focuses on software and GPUs, so the networking implication is inferential rather than a quantified performance claim.
ANET
More from SemiAnalysis
Are Open Models Catching Up?
Aug 21, 16:40
Cerebras's Next Generation CS-4: Fast Just Got Faster
Aug 19, 01:32
$12B of US ratepayers' money wasted on a modeling mistake and PJM wants to do it again
Aug 16, 22:27
Ultra-High Interactivity on NVIDIA GPUs? - TileRT InferenceX
Aug 10, 04:51
SpaceX 10GW in 2027 – Why It’s Real, Will Drive $300B ARR for SpaceX, and Why Microsoft Will Be the Largest Offtaker
Aug 07, 20:08

This newsletter, published August 24, 2026, features Cam Quilici discussing NVDA, AMD, ANET. 3 trade ideas extracted by AI with direction and confidence scoring.

Speakers: Cam Quilici  · Tickers: NVDA, AMD, ANET