Ultra-High Interactivity on NVIDIA GPUs? - TileRT InferenceX
Bryan Shan
· SemiAnalysis
· August 10, 2026 at 04:51
· ⏱ 24 min read
| Read on Substack ↗
Summary
The article argues that TileRT software enables standard NVIDIA GPUs to achieve ultra-high interactivity (low latency) inference, a domain previously dominated by specialized ASICs like Cerebras and Groq. By statically compiling decode graphs into persistent kernels, TileRT allows cloud providers to dynamically provision high-speed tiers from existing GPU fleets, significantly threatening the total addressable market of purpose-built inference chips that lack hardware fungibility.
•Premium AI 'fast modes' require ultra-low latency, prompting frontier labs to evaluate specialized inference systems like Cerebras and NVIDIA Groq LPUs over standard GPUs.
•Traditional GPU programming models suffer from kernel setup and teardown overhead, which dominates latency as token generation approaches the sub-millisecond range.
•TileRT solves this by statically compiling the entire decode graph into a single persistent kernel on NVIDIA GPUs, abolishing the kernel as the unit of execution.
•On the InferenceX GLM5 FP8 744B benchmark, TileRT on a single B200 decode server reached up to 500 tokens/s/user, approximately 3x faster than traditional engines on GB300 NVL72.
•TileRT sacrifices aggregate throughput for per-user speed; at 8K/1K, GB300 FP4+MTP delivers 240 total tokens/s/GPU while TileRT delivers 160.4 total tokens/s/GPU.
•Using a disaggregated architecture, throughput-optimized engines like vLLM handle prefill while TileRT handles latency-critical decode, allowing both to share the same GPU pool.
•TileRT's end-to-end decode tail latency was 3.01 seconds, compared to 6.54 seconds for the best NVFP4 + MTP competitor and 18.18 seconds for AMD's MI355X.
•The software-driven approach of TileRT allows providers to reallocate existing GPU capacity dynamically, posing a structural threat to the fixed-hardware models of ASICs.
TileRT software unlocks ultra-low latency capabilities on standard NVIDIA GPUs (like the B200), defending NVIDIA's moat against specialized inference chips by offering customers fungibility between hi
TileRT software unlocks ultra-low latency capabilities on standard NVIDIA GPUs (like the B200), defending NVIDIA's moat against specialized inference chips by offering customers fungibility between high-throughput prefill and low-latency decode workloads.
Risk: TileRT's static ahead-of-time compilation requires significant engineering effort per new architecture, limiting its model catalog and slowing deployment compared to generalized engines.
The article highlights a massive performance gap in ultra-low latency scenarios for AMD's current stack, noting the MI355X decode tail was 18.18 seconds compared to TileRT's 3.01 seconds on NVIDIA.
The article highlights a massive performance gap in ultra-low latency scenarios for AMD's current stack, noting the MI355X decode tail was 18.18 seconds compared to TileRT's 3.01 seconds on NVIDIA.
Risk: AMD has committed to submitting MI455X UALoE72 numbers to InferenceX this year, which could show significant architectural or software improvements.
This newsletter, published August 10, 2026,
features Bryan Shan
discussing NVDA, AMD.
2 trade ideas extracted by AI with direction and confidence scoring.